Some time ago, I saw a talk at DDD North about git bisect (it may well have been this one). I blogged about it here. I can honestly say that it’s one of, if not the, most useful thing I’ve ever learnt in 10 minutes!
However, the problem with it is that you, essentially, have to tell it what’s good and what’s bad. In this post, I’ll be detailing how you can write automated tests to determine this, and then link them in.
Using existing tests to determine where something broke
In this example, I’ll be using this repository (feel free to do the same). The code in the repository is broken, but it hasn’t always been, and there are some tests within the repository that clearly weren’t run before check-in, and are now broke (I know this, because I purposely broke the code - although this does happen in real life, and often with good intentions - or at least not bad).
We’re using xUnit here; but I’m confident that any test framework would do the same. The trick is the dotnet test command; from the docs on that command:
If all tests are successful, the test runner returns 0 as an exit code; otherwise if any test fails, it returns 1.
As with the previous post, we need to start with a good and bad commit; for the purpose of this post, we’ll assume the current commit is bad, and the first ever commit was good.
git log
Will give a list of commits:
We need the first, which, for this repo, is:
3cbd757dd4e92d8ab2424c6a1e46a73bef23e056
Now we need to go through the process of setting up git bisect; the process is: you tell git that you wish to start a bisect:
git bisect start
Next, you tell git which commit is bad. In our case, that’s the current one:
git bisect bad
Finally, you tell it which was the last known good one - in our case, the first:
git bisect good 3cbd757dd4e92d8ab2424c6a1e46a73bef23e056
Now that we’re in a bisect, you could just tell git each time which is good and which bad (see the previous post on how you might do that), but here you can simply tell it to run the test:
git bisect run dotnet test GitBisectDemo/
This will then iterate through the commits and come back with the breaking commit:
That’s great, but in most cases you didn’t actually have a breaking test - something has stopped working, and you don’t know why or when. In these cases, you can write a new breaking test, and then give that to git bisect for it to tell you the last time that test passed.
Create a new test to determine where something broke
Firstly, the new test must not be checked in to source control, as this works by checking out code from previous releases. Then create your new test; for example:
namespace GitBisectDemo.Tests
{
public class CalculationTests2
{
[Fact]
public void DoCalculation\_ReturnsCorrectValue()
{
// Arrange
var calculationEngine = new CalculationEngine();
// Act
float result = calculationEngine.DoCalculation(2, 3);
// Assert
Assert.True(result > 4);
}
}
}
This is a new class, and it’s not checked into source control.
Executing a specific test from the command line
We now want to execute just one test, and you can do that using dotnet test like so:
dotnet test GitBisectDemo/ --filter "FullyQualifiedName=GitBisectDemo.Tests.CalculationTests2.DoCalculation\_ReturnsCorrectValue"
You need to give it the full namespace and class name; we can now incorporate that into our git bisect:
git bisect start
git bisect bad
git bisect good 3cbd757dd4e92d8ab2424c6a1e46a73bef23e056
These are the same as before.
Note: if, at any time, you wish to cancel the bisect, it’s git bisect reset
Now, we feed the filtered test run into git bisect:
git bisect run dotnet test GitBisectDemo/ --filter "FullyQualifiedName=GitBisectDemo.Tests.CalculationTests2.DoCalculation\_ReturnsCorrectValue"
And we get a result when the new test would have broken.
That’s two cases covered. The final case is the situation whereby the thing that has broken cannot be determined by an automated test; say, for example, that an API call isn’t working correctly, or a particular process has slowed down. In this situation, we can have git bisect call out to an external executable.
Custom Console App
The first step here is to return a value (exit code) from the console app. In fact, this is deceptively simple:
static int Main(string[] args)
{
var calculationEngine = new CalculationEngine();
float result = calculationEngine.DoCalculation(3, 1);
return (result == 4) ? 0 : -1;
}
Notice that all we’ve done here is change the Main signature to return an int. This console app could now be calling an external API, running a performance test, or anything that has a verifiable result.
Publish the console app
Because we’re calling this from another location, we’ll need to publish this test as a self-contained console app:
dotnet publish -r win-x64
Run the test
Again, the same set-up:
git bisect start
git bisect bad
git bisect good 3cbd757dd4e92d8ab2424c6a1e46a73bef23e056
Finally, we call the console app to run the test:
git bisect run GitBisectDemo/GitBisectDemo.ConsoleTest/bin/Debug/netcoreapp3.1/win-x64/GitBisectDemo.ConsoleTest.exe
References
https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-test