In walks git bisect

It's a wicked tool for exactly the above situation. It performs a binary search for the bad commit and can very efficiently get through hundreds of commits. It can be done manually or automatically.

The example I used here was a problem with how we'd configured the mutation testing tool Pitest. It resulted in an error being printed Passed files have no features!, but Pitest would eventually pass. Weird right?

How to use it automatically

1. Define your success command. This must be a command that exits 0 for success and not 0 for failure. In this case, mine was:

    sh -c './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'
   

   • Translating to English:

     • sh -c '<command>' - execute this shell command and exit to the current shell

     • ./gradlew pitest 2>&1 - run pitest and print both stderr and stdout to the terminal

     • | grep -q "<search_string>" - pipe the output of pitest to grep and search for the string without printing to the terminal

     • && exit 1 || exit 0 - exit 1 if grep finds a match else exit 0

2. Find your first good commit by choosing one WAY in the past and checking your success command exits 0 - note its revision number. Do not try to find a recent one - that's what the tool is for!

3. Start the bisect process with:

    git bisect start
   

4. Mark the current commit (master) as bad with:

    git bisect bad
   

5. Tell bisect which commit is known to be good with:

    git bisect good <revision_number>
   

6. Tell bisect to get to work with:

    git bisect run <success_command>
   

   e.g.

    git bisect run sh -c './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'
   

If you have defined your success command correctly, it will checkout different revisions, test them with your command, and then know whether to look for the bad commit before or after that point. It will do this until it finds the baddie!

Real World Demo

Here is the entire process I used to find the "Passed files have no features" issue:

$ git bisect start
Already on 'master'
Your branch is up to date with 'origin/master'.
status: waiting for both good and bad commits

$ git bisect bad
status: waiting for good commit(s), bad commit known

$ git bisect good b39b356
Bisecting: 72 revisions left to test after this (roughly 6 steps)
[c343bab24e630bee0912abe923b5e09049204603] fix(US4764533): Fix running the service locally

$ git bisect run sh -c './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'
running  'sh' '-c' './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'
Bisecting: 36 revisions left to test after this (roughly 5 steps)
[4bc5b1444577128ce008a382f4c777548d7d10e0] fix(US3976007): Autogenerate WireMock stubs
running  'sh' '-c' './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'
Bisecting: 17 revisions left to test after this (roughly 4 steps)
[04b239b50334f63fabdcfdeb6cddae657d848e97] Update plugin com.diffplug.spotless to v6.12.1
running  'sh' '-c' './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'
Bisecting: 8 revisions left to test after this (roughly 3 steps)
[1904b509d8b753bcf4613dc35eb62409732be1bd] US4407184: Revert changes to cucumber reports for unit tests due to PiTest
running  'sh' '-c' './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'
Bisecting: 4 revisions left to test after this (roughly 2 steps)
[c894a4d0bd994d33fdeac771f1c742f61bdc4fc8] Merged after successful ./gradlew clean build
running  'sh' '-c' './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'
Bisecting: 2 revisions left to test after this (roughly 1 step)
[0cba5d12cd8300a26b8f518138a6063d62fedf7d] US4407184: Fix LSD database url for staging
running  'sh' '-c' './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'
Bisecting: 0 revisions left to test after this (roughly 1 step)
[2308a73e20f3683f4cc65491a76d880a6d80ba99] US4407184: Publish cucumber reports for unit tests
running  'sh' '-c' './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[49c2b1be8d838eb4bd75793e84ece69f6c0b0cb6] US4407184: Update internal dependencies
running  'sh' '-c' './gradlew pitest 2>&1 | grep -q "Passed files have no features" && exit 1 || exit 0'
2308a73e20f3683f4cc65491a76d880a6d80ba99 is the first bad commit
commit 2308a73e20f3683f4cc65491a76d880a6d80ba99
Author: John Doe <john@doe.com>
Date:   Mon Jan 23 12:22:30 2023 +0000

    US4407184: Publish cucumber reports for unit tests

 Jenkinsfile                                              | 2 +-
 application/src/test/resources/junit-platform.properties | 3 +++
 2 files changed, 4 insertions(+), 1 deletion(-)
bisect found first bad commit

As you can see, git-bisect checks out a commit in the middle of the commit-range, tests it against the success command and then knows to narrow its commit-range to before or after that commit. It does this over and over until it knows exactly which commit the bug started.

Due to the way binary searches work, searching for a bad commit within 256 commits will only perform double the revision tests as within 8 commits. So don't spend time searching for a recent good commit, the tool will do this much quicker than you will!

Note you can also do this manually if your definition of success is too complicated, by marking the first good and bad commits, then manually testing the revision that bisect gives you, then going back and telling it git bisect good or git bisect bad.

Conclusion

git-bisect is an incredibly powerful tool that, while you won't use all the time, is a useful one to keep at the back of the toolbox for when you need it.

For demonstration purposes let's imagine I'm a middleman between a customer wanting a cake and a baker. The cucumber scenario looked a bit like this:

Scenario: Tell the baker that the customer wants some cake
    Given the customer has a cake request
    When the request is submitted
    Then a cake requested event is published to the baker

At some point in my test, I wanted to make sure that certain fields in the outgoing cake requested event matched the ones received in the incoming cake request. Because I was intercepting the outgoing event using a Kafka listener, it was in JsonNode format.

So imagine this incoming CakeRequest:

data class CakeRequest(
    val requestId: String,
    val flavour: String,
    val colour: String,
    val wantsSprinkles: Boolean,
)

And an outgoing JsonNode that looks like this:

{
    "flavour":"chocolate",
    "colour":"red",
    "wantsSprinkles":"true"
}

What is the best way to make sure they have the same values (minus the requestId)?

Simple private method

I started with this simple assertion:

assertTrue { hasSameEssentialValues(cakeRequest, resultingEvent) }

private fun hasSameEssentialValues(cakeRequest: CakeRequest, resultingEvent: JsonNode) =
        cakeRequest.flavour == resultingEvent["flavour"].asText()
                && cakeRequest.colour == resultingEvent["colour"].asText()
                && cakeRequest.wantsSprinkles == resultingEvent["wantsSprinkles"].asBoolean()

While this worked fine, there were a few downsides:

• The assertion didn't read fantastically

• It polluted my test file with ugly private methods

• Most importantly, the truth about how to match fields between a CakeRequest and a JsonNode was being mixed in with everything else.

In steps a Hamcrest matcher!

By the power of Hamcrest, I created an assertion that looks like this:

assertThat(resultingEvent, hasSameEssentialValuesAs(cakeRequest))

And a matcher in a separate file that looked like this:

fun hasSameEssentialValuesAs(cakeRequest: CakeRequest) = CakeRequestMatcher(cakeRequest)

class CakeRequestMatcher(private val cakeRequest: CakeRequest) : BaseMatcher<JsonNode>() {

    override fun describeTo(description: Description) {
        description.appendText("The important fields must match")
    }

    override fun matches(resultingEvent: Any?): Boolean {
        if (resultingEvent !is JsonNode)
            return false

        return cakeRequest.flavour == resultingEvent["flavour"].asText()
                && cakeRequest.colour == resultingEvent["colour"].asText()
                && cakeRequest.wantsSprinkles == resultingEvent["wantsSprinkles"].asBoolean()
    }
}

This makes the assertion read brilliantly and follows the single responsibility principle much better.

But there are some downsides:

• The mandatory implementation describeTo() - not useful in my situation

• matches() takes the Any? type as its input, requiring manual type-checking

• The whole thing is complex - it will be harder to read and maintain

Final Takeaways: Reflections and Next Steps

While this was a good exercise where I learned a lot about matchers (and ended up writing this article), in my specific use case I ultimately decided to check the fields in a simple package-level function in its own file. This kept the responsibility in the correct place and made the whole thing less complex.

It means when someone else (or me in 6 months) reads this test, they'll take less time to understand it, and less time to modify it. Which surely is the ultimate goal when writing code, right?

Update:

5 minutes after I published this article I sent the link to my esteemed work colleague who I worked with on this problem, and he replied straight away with a solution that blows mine out of the water. It solves the problem of readability and allows you to abstract the comparison logic away from both the test and the CakeRequest object:

assertTrue { cakeRequest.hasSameEssentialValuesAs(resultingEvent) }

Kotlin has extension functions, duh!!

Where it started

Before we get to the real origins, let me take you through how it originated in my head. I've subscribed to a few definitions over time!

I first heard the term technical debt when I was a junior developer, interacting with teams who were maintaining lots of legacy code. Due to this, there was a growing initiative from middle managers for their teams to reduce their technical debt. This brings us to...

Definition 1: Legacy Technology

Technical debt is synonymous with legacy tech. If you have lots of legacy tech that is expensive to maintain and hard to change, then you must reduce that technical debt.

This definition was easy to learn since it was a pure definition of a tangible thing rather than a concept. And it was all I ever knew until I was told otherwise!

Pros of this definition

• Reducing technical debt sounds much more appealing than dealing with legacy technology so it might speed up decisions to deal with problems

Cons of this definition

• It is reactive. It offers no insight into how problems are caused in the first place

What triggered me to rethink my definition

• I got hot on the fact that everything we write as developers is someday going to have to be modified or deleted and in assuming that technical debt is a thing that we must get rid of, I concluded very confidently that...

Definition 2: Everything is Technical Debt

Every line of code will one day die, and it will cost effort to kill it. So in assuming everything we write is technical debt, we can be mindful of the cost of creating and deploying.

While this is quite reductive, let's explore it:

Pros of this definition

• If we adopted this mindset, then we might be more sparing in the code we write. Think of the "overproduction" or "extra processing" lean wastes.

• It will reduce your ego! No matter how unbelievably beautiful the code you write today is, one day it will be as decrepit as something written in the context of the 1990s might be today. As you write your code, you might now consider how easy it would be to delete it.

Cons of this definition

• It might cause people to be nihilistic. What's the point in trying to do a good job if it's eventually going to be deleted anyway?

• The opportunity cost of not leaning into the metaphor

What triggered me to rethink my definition

• I realised it had absolutely nothing to do with the idea of debt. Why was this term first coined? Someone must have had a metaphor in mind

Definition 3: Getting back to the metaphor

Financial Debt

What is financial debt? It is borrowing money now, with the requirement to pay it back later, often with interest.

It can be bad; unconsciously racking up credit card debt so you can get a fresh wardrobe ready for the summer, with no plan to pay it back might get you into a lot of trouble.

But taking out a business loan to buy some equipment so you can start your new manufacturing business might pay itself back multiple times over.

The difference between these two cases? Only one of them uses the debt to invest in something that will make more money in the future, so you can pay back the debt, and end up with more money than if you hadn't taken out the debt in the first place.

The same applies to the metaphor of technical debt.

Technical Debt

If you unconsciously go faster than your constraints should allow you when developing a new application, then something will have to give. The code might be substandard and hard to maintain. And if you've done this unconsciously, then you might never repay the technical debt by bringing the code up to standard.

On the other hand, if you take shortcuts in your coding to deliver something in a short time frame, you may get buy-in from the business resulting in them investing more in your idea.

BUT! You must be conscious that one day, that debt will have to be repaid, or else interest will accrue. Every day that goes by, more workarounds will have to be made, and something that should take 1 hour to do, ends up taking 3, all due to the substandard code that was written in the first place and never dealt with.

With every decision to move faster than your constraints allow, be it financial or ability, you must make a plan to pay that back as well as the interest that goes along with it.

Definition 3 is the one I've settled on organically and feel comfortable with in my head. It has the benefit of being quite close to the real idea too.

The actual origins of Technical Debt

Ward Cunningham coined the term around 1992 but his most in-depth exploration of the concept is in a video he released almost 20 years after. Rather than for me to try and fail in paraphrasing his ideas, I'll point you to where you can watch or read about his ideas directly.

Conclusion

In my experience, Definition 1: Legacy Technology is by far the most widespread definition people subscribe to, but it misses out on the main point.

Definition 2: Everything is Technical Debt might have some ideas to extrude, but it shouldn't be associated with the term technical debt. It waters down the real idea unnecessarily.

I'm settled on Definition 3 for now, and given it was the original one, something revolutionary would have to come along to dethrone it.

Reading the actual idea behind technical debt by watching Ward's video straight away would have got me to my answer quicker.

But you know what? I learned a lot from my discovery along the way and each of the definitions contributed to my thoughts on developing software.

This is some bold content

this_block = 'some code'

That was a horizontal line ^

This is a list:

• list item 1

• list item 2

This is a blockquote

This is a link to Google

This is an image:

This is a mermaid diagram:

flowchart LR
    START -- Journey --> FINISH

...well that didn't work, did it?

This is an emoji :joy:

ah man...

reached the limitation!