Tuesday, September 20, 2011

The Build is Broken, Now What?

Your team is hard at work, testing and coding and planning, and suddenly the build breaks. Now what can you do? The broken build might not be your fault at all, and besides you have work of your own to do. You could go ahead and practice business-as-usual, but this is probably a bad idea.

Here's what I say:

  • Never pull from a broken build.
    Assume that version control is poison. If you are not the one fixing the build (and shouldn't you be?) then you should leave it be. The last thing you want to do is import brokenness into your local build environment. Import from a "green" build, and you will know that any brokenness is your problem.
  • Test locally before considering pushing code to the CI build, even though it takes a bit more time. There is really no good time to NOT know that you've broken something. A little "due diligence" goes a long way.
  • Never push to the CI server if your local build is broken.
    If your build is broken, it inconveniences only you. If someone is working on the build and you push a second mistake, they won't know if they fixed it or broke it worse. If the build is not broken, why bother pushing errors to it? 
  • If you're not fixing the build, never push a working local change to a broken build.
    This is a kindness akin to not walking in the dirt your mother is sweeping up. If you push a change while the build is broken, the person fixing the build will have yet another update/build/test cycle on his hands before he can push his fix to the server. Don't punish the volunteer by pushing your changes. Wait, pull, merge, retest.

We have installed annunciators for teams. Sometimes they are flashing lights, sometimes a glowing orb, sometimes speakers playing a particular noise, sometimes it's just a person standing up and yelling that the build is down. Usually there is some electronic communication (jabber message or email) as well.

Why do we all need to know that the build is broken, and how long it is broken? So that we can alter our behavior. Someone needs to jump in and fix the build quickly, and everyone else needs to let them or help them. By leaving the build broken, continuing to build and push code as always, or pushing additional errors into the build, we are just getting in the way of the team.

Build annunciators should let you know when you need to get out of the way.  They don't have to be piercing and shrill or overly bright, but they need to keep the team aware that the build is not "green."

It's not just a good idea... well, actually it is.


  1. And remember that it is perfectly valid, and sometimes necessary, to just revert the change (or set of changes) that broke the build.

    The primary objective is to return to green; the secondary objective is to apply the most-recent changes. Languishing for hours or days in red while fiddling with your latest changes is not the best approach.

  2. How about time-boxing the fix-time for build breaks?

    It might sound a bit harsh but I've seen it working perfectly well. If the person who broke the build could not fix it in, let's say 15 minutes, he needs to revert the change back.

    With teams which are working in real CI environments, build breaks might be really annoying. This policy at least helps to keep the pipeline going.


  3. Another of my teams has fully converted to the guidelines here. They consider it an antisocial act to push a non-fix to a broken build, and unprofessional to leave a build broken for any longer than ABSOLUTELY necessary.

    If you commit code and don't wait to be sure it passes the CI test builds, you are persona non grata for a few days.

    It isn't the intolerance that makes me grin, it's seeing a constant commitment to keeping the system in a runnable state. I think people are getting addicted.