Mutation testing is automatically modifying your code to introduce defects, and making sure the unit tests fail. This can mutate source code directly, or mutate the byte code if you’re using a JVM language. A simple diagram, via Thomas Hamilton:
Here’s an example of a code mutation:
try: return int(guid.replace('-', '')) except ValueError as ve: - raise ValueError("Unable to convert uuid that is not all numeric to hrid.", ve) + raise KeyError("Unable to convert uuid that is not all numeric to hrid.", ve)
In this example, the type of the exception raised was changed. In case the test that calls this block was expecting too broad of an exception type to be raised (ah, the infamous
except Exception), the test will continue to pass. So the test would need to be modified to assert that the specific exception that we’re looking for was raised.
Of course, this is totally random and the source code could have been modified in any number of ways. In case the string in the error message was the only thing modified, we probably don’t care that the test suite continues to pass. This is something to keep in mind when using mutation testing - not all mutations are the same.
Why use mutation testing?
In short, mutation tests make sure your testing suite is high quality. They help identify sections of code that are covered, but not thoroughly tested - things like logger statements, side effects of functions, etc. They also help identify weak tests.
Here are some potential issues to keep in mind when using mutation testing:
- Since the full test suite is run for each mutation, execution can be extremely slow for large projects with lots of tests.
- Resolving all errors forces you to write a test that asserts against every single line of code, which may not be worth the effort depending on the system.
- The mutation tests are not necessarily deterministic due to injecting some randomness, so I don’t recommend using them as part of your CI/CD pipeline.
- Last time I used them, we had a separate build that triggered after the CI/CD run to run the mutation testing evaluation and report back if there were any issues.
Mutation Testing Libraries
Here’s a list of libraries that support mutation testing:
- https://pypi.org/project/MutPy/ - no support for excluding lines from being mutated
- https://pypi.org/project/mutmut/ - allows you to exclude lines with a comment pragma
- https://github.com/sixty-north/cosmic-ray - nice UI, most modern option
- PIT Test: https://pitest.org/
- There are a few other libraries available, but PIT Test and Descartes seem to be the only ones actively maintained
- Stryker-net: https://github.com/stryker-mutator/stryker-net
- Example project: https://github.com/bbenetskyy/mutation-tests-workshop
- There are a few others, but they don’t seem very mature