git-bisect: how to find a needle in a haystack
Imagine you've noticed a bug in your project. It may not be something that has failed any test, but it is a bug nonetheless. You've just noticed it, but how long has it been there? How many changes might have been committed since the bug was introduced? Debugging this is going to be a nightmare!
In walks git bisect
It's a wicked tool for exactly the above situation. It performs a binary search for the bad commit and can very efficiently get through hundreds of commits. It can be done manually or automatically.
The example I used here was a problem with how we'd configured the mutation testing tool Pitest. It resulted in an error being printed Passed files have no features!, but Pitest would eventually pass. Weird right?
How to use it automatically
Define your success command. This must be a command that exits
0for success andnot 0for failure. In this case, mine was:sh -c './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'Translating to English:
sh -c '<command>'- execute this shell command and exit to the current shell./gradlew pitest 2>&1- run pitest and print both stderr and stdout to the terminal| grep -q "<search_string>"- pipe the output of pitest to grep and search for the string without printing to the terminal&& exit 1 || exit 0- exit 1 if grep finds a match else exit 0
Find your first good commit by choosing one WAY in the past and checking your success command exits 0 - note its revision number. Do not try to find a recent one - that's what the tool is for!
Start the bisect process with:
git bisect startMark the current commit (master) as bad with:
git bisect badTell bisect which commit is known to be good with:
git bisect good <revision_number>Tell bisect to get to work with:
git bisect run <success_command>e.g.
git bisect run sh -c './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'
If you have defined your success command correctly, it will checkout different revisions, test them with your command, and then know whether to look for the bad commit before or after that point. It will do this until it finds the baddie!
Real World Demo
Here is the entire process I used to find the "Passed files have no features" issue:
$ git bisect start
Already on 'master'
Your branch is up to date with 'origin/master'.
status: waiting for both good and bad commits
$ git bisect bad
status: waiting for good commit(s), bad commit known
$ git bisect good b39b356
Bisecting: 72 revisions left to test after this (roughly 6 steps)
[c343bab24e630bee0912abe923b5e09049204603] fix(US4764533): Fix running the service locally
$ git bisect run sh -c './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'
running 'sh' '-c' './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'
Bisecting: 36 revisions left to test after this (roughly 5 steps)
[4bc5b1444577128ce008a382f4c777548d7d10e0] fix(US3976007): Autogenerate WireMock stubs
running 'sh' '-c' './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'
Bisecting: 17 revisions left to test after this (roughly 4 steps)
[04b239b50334f63fabdcfdeb6cddae657d848e97] Update plugin com.diffplug.spotless to v6.12.1
running 'sh' '-c' './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'
Bisecting: 8 revisions left to test after this (roughly 3 steps)
[1904b509d8b753bcf4613dc35eb62409732be1bd] US4407184: Revert changes to cucumber reports for unit tests due to PiTest
running 'sh' '-c' './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'
Bisecting: 4 revisions left to test after this (roughly 2 steps)
[c894a4d0bd994d33fdeac771f1c742f61bdc4fc8] Merged after successful ./gradlew clean build
running 'sh' '-c' './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'
Bisecting: 2 revisions left to test after this (roughly 1 step)
[0cba5d12cd8300a26b8f518138a6063d62fedf7d] US4407184: Fix LSD database url for staging
running 'sh' '-c' './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'
Bisecting: 0 revisions left to test after this (roughly 1 step)
[2308a73e20f3683f4cc65491a76d880a6d80ba99] US4407184: Publish cucumber reports for unit tests
running 'sh' '-c' './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[49c2b1be8d838eb4bd75793e84ece69f6c0b0cb6] US4407184: Update internal dependencies
running 'sh' '-c' './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'
2308a73e20f3683f4cc65491a76d880a6d80ba99 is the first bad commit
commit 2308a73e20f3683f4cc65491a76d880a6d80ba99
Author: John Doe <john@doe.com>
Date: Mon Jan 23 12:22:30 2023 +0000
US4407184: Publish cucumber reports for unit tests
Jenkinsfile | 2 +-
application/src/test/resources/junit-platform.properties | 3 +++
2 files changed, 4 insertions(+), 1 deletion(-)
bisect found first bad commit
As you can see, git-bisect checks out a commit in the middle of the commit-range, tests it against the success command and then knows to narrow its commit-range to before or after that commit. It does this over and over until it knows exactly which commit the bug started.
Due to the way binary searches work, searching for a bad commit within 256 commits will only perform double the revision tests as within 8 commits. So don't spend time searching for a recent good commit, the tool will do this much quicker than you will!
Note you can also do this manually if your definition of success is too complicated, by marking the first good and bad commits, then manually testing the revision that bisect gives you, then going back and telling it git bisect good or git bisect bad.
Conclusion
git-bisect is an incredibly powerful tool that, while you won't use all the time, is a useful one to keep at the back of the toolbox for when you need it.
