There is no doubt that unit tests are necessary to keep good quality of code. People used to check line coverage to determine if they have good tests, but… Have you ever wondered how to test your tests and check if they really test something?
In this post I’ll show you why line coverage is a bad metric and how to use mutation testing in Java.
Mutation is a small change in bytecode. When tests will fail then mutation is killed, otherwise mutation has survived.
There are many types of mutation, for example:
removing method calls to void methods,
removing all conditionals statements:
will be mutated to:
changing return values of method calls:
change true to false, false to true
int byte short
change 0 to 1, otherwise return 0
change x to x+1
There are few libraries which can be used to mutation testing. I chose pitest library.
Example of usage pitest in Java
I wrote a small program to calculate ticket price for passengers, which will be used to demonstrate how to use mutation testing in order to create good tests.
Program has two classes, Passenger class which represents traveler and TicketPriceCalculator which contains logic to calculate ticket price.
There are three basic scenarios to cover in tests: ticket for adult, child and family.
Ok, let’s run pitest to see how good our tests are.
To run pitest we use maven command:
It will output an html report to target/pit-reports/YYYYMMDDHHMI. In this report we can see that we have 100% line coverage, but only 75% mutation coverage. It means that our tests are not as good as they should be.
When you click on class name you’ll see a detail report.
We have one type of mutation which survived, it’s conditional boundary change. We already know, that > operator is mutated to >=. Mutation survived, because when we’ll change > to >= our tests will pass.
We have to add tests for edge cases.
Ok, we covered our edge cases, however if you execute pitest plugin once again you’ll see that we still have two mutations which survived.
Take a look into this code:
Our edge case test scenario passes one adult and one child to this method. Due to AND operator we can change our code to:
and all test are still green, because second condition adultCounter > 1 change the result of this statement to false. The same behaviour we might be observed when we’ll change adultCounter > 1 to adultCounter >= 1.
To cover this cases we should replace our shouldNotCalculatePriceForFamily with two new tests:
After this we finally reached 100% mutation coverage.
To sum up, after this reading you should learn how to use mutation testing to see if your tests are good. You also know that line coverage doesn’t mean that we have covered all cases. Hope you enjoyed and you’ll start to use mutation testing. All code which was used is available on the repository.