June 16, 2015

Agile Software Development Best Practices: Continuous Integration

3Pillar Global’s SVP of Engineering Jeff Nielsen recently recorded a video on the benefits of continuous integration for software development teams. Continuous integration is a system that delivers fast feedback, allowing teams to know how well new code interacts with existing code, as well as recognize immediately when a build breaks.

Watch the video embedded below to hear Jeff’s complete thoughts on continuous integration and to discover his three principles that ensure continuous integration will thrive for any team. You can also read a full transcript of the video below.


For those of you who like to drive your cars a little bit fast, how fast do you think you could safely drive the Mars Rover—that little vehicle that we’ve put up on Mars that’s controlled remotely here from Earth? Before you answer that question, remember that it takes light anywhere from 4 to 24 minutes to travel between Earth and Mars (depending on the relative positions of the planets). So on an average case, you’re talking 10 minutes to get a radio signal from your steering wheel on earth up to the vehicle on Mars. Imagine yourself turning the steering wheel a couple of degrees, 10 minutes up, 10 minutes back, OK now I see that I’m looking at something different. I am going to put my foot on the gas pedal, just a little bit, go forward a few inches, 10 minutes up, 10 minutes back. Oh, I’m about to hit a rock, turn this way – 10 minutes up, 10 minutes back.

You can see that even if you are a very skilled driver, it would take you a long time to get that Mars Rover between point A and point B working in that fashion. The difference, of course, is the speed of the feedback. When you are driving a regular car on earth, you can see immediately what you’re looking at when you turn. Or when you’re putting your foot on the gas pedal you can feel what’s going on there. The speed of the feedback affects how quickly you can drive, and the speed of the feedback affects how quickly you can work in general.

This same principle applies to software development. It’s not uncommon for a developer or a team to have to wait hours or days or weeks sometimes to get feedback on how the code they just added or changed in the system affected things. And that is a very slow way to work.

That’s where Continuous Integration comes in.  Continuous Integration is about setting up fast feedback loops. We talked about fast feedback loops on your individual workstations when we talked about automated unit testing. Continuous integration is about fast feedback that lets you know right away (or very quickly) about how the code you just wrote or changed interacts with everybody else’s code – the problem of integration. And as a practice, it is fairly simple. Two basic pieces go into Continuous Integration. One is that every developer on the team checks in his or her code multiple times per day—frequent small check-ins rather than waiting until a whole piece is done. And the second is that every time that anyone checks in, that is verified by an automated build which includes testing. That is usually handled by some sort of continuous integration server.

So if you’ve got that kind of feedback, I can write some code, I can check it in and within minutes, I can know if I accidentally broke something that someone else was working on or something that I didn’t understand or I failed to check in a file . . .  It can be a very powerful practice if you keep in mind a couple of important principles.

First is that your continuous integration build has to run fast. There is a threshold beyond which people will not wait around for the build to finish.  And I’ve found that that threshold is about 10 minutes. As soon as your CI build creeps up beyond 10 minutes, people will stop checking in if it’s the end of the day, they will stop checking if they are about to go to lunch, and they will probably even go on to something else after they’ve checked in. They are not going to sit around and wait for the results for more than about 10 minutes. So that is a magic number – if you can keep it to five even better—but the build has to run quickly for it to be valuable as a feedback mechanism.

Secondly, the CI build has to be reliable. As soon as you get a couple of false negatives where the build breaks but then you dig into it and it turns out that it’s the build system itself that’s not working properly or you’ve got a bunch of bad tests that are telling me something is broken when it really isn’t, then people start to ignore the Continuous Integration feedback.

The third piece is that you have to make it difficult to ignore the feedback for everyone on the team. Having the CI server send out emails to everyone on the team is just not sufficient. It’s too easy to ignore email or filter it into an mailbox that you’re going to read later. I’ve found things like an audible siren or some other sound that plays in the team area when a build breaks or even flashing lights that are red when a build breaks – those are the kinds of things that are much harder for people to ignore. And I promise you that if you put that in place, people will fix a broken build right away because the sound or the lights or something else will drive them crazy. And that’s what you want. You want a broken build to be a big deal. You want people to address it immediately. That’s the power of Continuous Integration.

So if you would rather drive a Ferrari than the Mars Rover—at least in the speed at which you are able to get stuff done with your software development approach—you need an effective Continuous Integration strategy.