The Zalando Testing Team creates tools that improve code quality for development teams. One way to improve code quality is to write tests – and we do write a lot of tests! But are these tests good at catching bugs? If code is refactored and the test suite is still green, can we be confident that we didn’t break anything? This post introduces a new tool that verifies whether or not tests are actually able to find errors in the code.

Mutation Testing

We typically use code coverage to gain insight into how well software is tested. This is a rather bad metric: Code can have a high coverage percentage even without any assertions in tests! Research also suggests that there isn’t a strong correlation between coverage and test suite effectiveness.

One promising approach to measure test quality is mutation testing. The idea is simple: Faults are injected into the source code and tests are run against faulty versions. If the tests start to fail, the tests are good at catching errors. If everything is still green, the tests are doing a poor job.

A faulty version of the software is called a mutant. A mutant can, for example, contain a negated conditional:

if (a == b) {
  // some logic...
}

Which will be changed to:

if (a != b) {
  // some logic...
}

If a test fails for a mutant, the mutant is said to be killed. After mutation tests have been run, a mutation score is calculated:

mutation score = number of mutants killed / total number of mutants

A higher ratio indicates a more effective test suite.

PIT is a popular Java mutation testing tool. It provides Maven integration, command line tools, and Ant support out of the box. Running PIT unfortunately breaks a developer’s work flow and I sincerely hope to see mutation testing as part of a developer’s work in future. An IDE seems like the natural place for a mutation test runner! I’d like to then introduce Zester, an IntelliJ IDEA plugin that makes running mutation tests a pleasant experience for developers.

Zester

Zester was developed in order to improve the quality of tests inside Zalando. It is an IntelliJ IDEA plugin that uses PIT under the hood. It focuses on ease of use: Running mutation tests should be as easy as running JUnit or TestNG tests inside an IDE.

After installing Zester, you can right click a Java file or a package in the Project navigator. The context menu offers you an option to run mutation tests. After starting the tests, a run configuration is saved in similar fashion to JUnit or TestNG configurations. If you need to provide more detailed configuration, you can always open the complete configuration from the “Edit configurations…” menu.

When mutation tests are executing, the console is updated with current progress.

Once mutation tests are finished, a link to the mutation test report is displayed in the console.

From the report we can see that the tests are still passing even with the negated condition at line 14. There is no test to protect you from mistakenly modifying the condition. In larger projects, mistakes like this happen and your tests should be able to detect these errors!

Conclusion

Today’s developers have a wide range of tools available that help them to test code. Mutation testing guarantees that the tests catch every possible edge case and therefore should be part of a developer's toolkit. With Zester and PIT, developers can write tests and refactor with confidence!