Some time ago, I heard Dan Clarke from the Unhandled Exception podcast mention Mutation testing - the latest episode on this can be found here. I thought this definitely warranted some investigation.
If you skip to the bottom of this post, you’ll see some links to the official docs for Stryker, and to a video that details exactly how to use it.
What is Mutation Testing
The hypothesis here is that, if you’ve written a test, you can test that test by changing an element of the code under test - if the change breaks your test then your test is valid, if it does not, then your test is not.
I’m not completely sure I accept this theory, but I can see its uses. In this post, I’m experimenting with a Calculator class.
Installation
For the purpose of this, I’ll assume that you have some code to test. If you don’t, then you can download the code that I used my my tests here.
You’ll need a terminal window - you can use Windows Terminal, or any other terminal window of your choice; I’ve recently started using the Developer Power Shell (it’s kind of the Visual Studio equivalent of the VS Code Terminal):
The first thing you’ll need to do (unless you’re using other .Net Tools in your project) is to install the manifest:
dotnet new tool-manifest
To install Stryker, use the following command:
dotnet tool install dotnet-stryker
Tests and Usage
The tool cannot work without tests - remember that the purpose of it is to tell you if the tests are useful, not if the tests are there (although you do get some coverage stats from it). Here’s my code:
public static class Calculator
{
public static decimal Add(decimal x, decimal y) =>
x + y;
public static decimal Subtract(decimal x, decimal y) =>
x - y;
}
And here’s the tests that I have:
[Fact]
public void Calculator\_Add\_ReturnsCorrect()
{
// Arrange
// Act
decimal result = CalculatorApp.Calculator.Add(3, 6);
// Assert
Assert.Equal(9, result);
}
As you can see, we’re looking at, at most, 50% test coverage. Let’s run the mutation tool and see what happens:
If you open the URL, you’ll get a coverage report, including any mutants that survived (we’ll come back to the later):
What this is telling us is that we don’t have particularly good test coverage, but what we do have has not survived mutation.
Let’s fill out the test coverage to 100%:
[Fact]
public void Calculator\_Subtract\_ReturnsCorrect()
{
// Arrange
// Act
decimal result = CalculatorApp.Calculator.Subtract(1, 0);
// Assert
Assert.Equal(1, result);
}
Admittedly, this took some gaming of the system, but when you run this, it survives:
Why did that test survive, by the first one didn’t? And what does ‘survived’ mean? Well, you can actually get it to tell you what it does during the mutation by selecting the file in question, and clicking “Expand All”:
What this tells you is that it replaced the code in the Subtract method with 1 (i.e. just return 1), and with x + y, (rather than x - y). The mutation would be ‘killed’ if, upon this change, at least one test failed. All I had to do was to find a test that would survive both scenarios (hence 1 - 0.
Summary
Stryker looks like a really cool and useful tool, but it definitely has its limitations. It identifies test coverage, and any test coverage that isn’t definitive (where there is no assert statement, or where the assert statement is ambiguous; for example, asserting that an exception is not thrown).
I’ve still to run it on a reasonable sized code-base; which I fully intend to do, but I’m not sure that I’d necessarily build this into a CI/CD pipeline (unless you genuinely fear that your developers are gaming the code-coverage stats).
I also have reservations as to whether a code base with 100% test coverage, and 0 surviving mutants is a healthy one, or one in a straight-jacket. Having said that, I definitely think this is a useful tool - it gives you information about your code base that you didn’t have before.