Mutation Testing

Mutation testing is automatically modifying your code to introduce defects, and making sure the unit tests fail. This can mutate source code directly, or mutate the byte code if you’re using a JVM language. A simple diagram, via Thomas Hamilton:

Mutation Testing Diagram

Here’s an example of a code mutation:

     try:
         return int(guid.replace('-', ''))
     except ValueError as ve:
-        raise ValueError("Unable to convert uuid that is not all numeric to hrid.", ve)
+        raise KeyError("Unable to convert uuid that is not all numeric to hrid.", ve)

In this example, the type of the exception raised was changed. In case the test that calls this block was expecting too broad of an exception type to be raised (ah, the infamous except Exception), the test will continue to pass. So the test would need to be modified to assert that the specific exception that we’re looking for was raised.

Of course, this is totally random and the source code could have been modified in any number of ways. In case the string in the error message was the only thing modified, we probably don’t care that the test suite continues to pass. This is something to keep in mind when using mutation testing - not all mutations are the same.

Why use mutation testing?

In short, mutation tests make sure your testing suite is high quality. They help identify sections of code that are covered, but not thoroughly tested - things like logger statements, side effects of functions, etc. They also help identify weak tests.

Here are some potential issues to keep in mind when using mutation testing:

Since the full test suite is run for each mutation, execution can be extremely slow for large projects with lots of tests.
Resolving all errors forces you to write a test that asserts against every single line of code, which may not be worth the effort depending on the system.
The mutation tests are not necessarily deterministic due to injecting some randomness, so I don’t recommend using them as part of your CI/CD pipeline.
- Last time I used them, we had a separate build that triggered after the CI/CD run to run the mutation testing evaluation and report back if there were any issues.