Mutation testing with PIT

How can we know that our unit tests are good?

Code coverage can't say that we have proper unit tests - it's just a side effect of using TDD.
It says what code has been executed in the unit test - we can write a test which just executes a method and without any assertions we'll have 100% code coverage.

Let's take a look at the following class:

It's a simple class that tells us if a given number is higher or equal to zero.

Let's take a look at the unit test:


When we execute cobertura on this test, we will see 100% coverage, both line and branch, even though it forgets about the case where the number is 0.

Mutation testing is an idea that tries to measure the quality of unit tests. They are run against modified version of the source code (by applying various mutators - hence the name). If the application code is changed, it should produce different result, therefore the unit test should fail. If it doesn't fail, it may indicate problem in the test suite.

PIT (http://pitest.org/) is a bytecode mutation testing system that automatically applies mutators to the application code.

If we run it against above mentioned class, it will produce a nice HTML report saying that 'changed conditional boundary' has survived - which means that we didn't test all boundary conditions.

We can integrate execution of PIT into our build (there are plugins for maven, ant and gradle) or we can execute it within our IDE (plugins for Eclipse and IntelliJ - that one I've created recently).

The project is actively developed, so we can expect more features to come.