Build Patterns to Boost your Continuous Integration

Software Development Magazine - Project Management, Programming, Software Testing

Scrum Expert - Articles, tools, videos, news and other resources on Agile, Scrum and Kanban

Build Patterns to Boost your Continuous Integration

Julian Simpson, The Build Doctor Limited, http://www.build-doctor.com

Software developers have long had theories, methodologies, and patterns. Practitioners of building and releasing code, on the other hand, can appear to preside over a seat-of-the-pants affair. This difference comes partly from the evolution of many software companies: They typically start small, and a couple of developers put out the first few builds from local machines. But (hopefully) soon enough, the focus shifts from development to pushing product. Eventually, however, disaster strikes, and you realize that until then no one had either the inclination or time to think about practices regarding the release of code.

Well, I have had the inclination and time to think about this. If developers can condense the accumulated knowledge of decades into design patterns, why should build managers not similarly take advantage of experience and mine patterns in build and release? Today, I will share a few. You might find one or more of them handy.

Build Pattern: Façade

Let’s begin with one for when your build scripts start showing their age, reminding you of the "before" pictures for wrinkle remedies. First, do not bother attempting to rewrite the script from scratch. You might just be trading a new set of problems for the old. If you feel that build refactoring is not going to get you where you want to go, try wrapping the build in a different tool and swap out chunks where it makes sense. One of my blog’s readers, Jason, had this to say about Gradle:

"When Gradle consumes an Ant build, it treats the tasks as actual Gradle tasks, so you could override the ant tasks as needed and simplify things until you’re completely ready to replace the old Ant build with a Gradle build."

You can use a similar approach with Maven. Use the Maven Ant plug-in to embed Ant code, and convert over to Maven at your leisure.

Green-Light Build

Continuous Integration, for all its advantages, sometimes feels more like a parking lot than a highway. For when developers have to battle for limited CI capacity, time is wasted while they wait for the feedback they need to confirm proper code integration. Their builds fail to get the prompt service that CI was supposed to deliver, when they must struggle against their colleagues’ host of checkins for new functionality. This can be particularly frustrating when working on critical and time-sensitive tasks such as bug fixes. Then there are the functional tests that jam up the available build agents as the short builds queue up.

What to do? Dedicate some CI capacity to the shorter builds. Your CI system will determine how you implement this. My solution for this issue on CruiseControl (which happened to be the first solution I attempted), was to make a separate server. I have since implemented the same solution on Team City, although in this case I took into account the build agents reserved for fast builds, and added an environment variable to them. If the variable exists in the agent, and the build takes more than 15 minutes, the agent will not use the variable.

While it might appear that I am not optimizing the use of the CI server by doing this, I think that appearance deceives. Salaries are more costly than CI servers, so it makes more sense to optimize the system for people. A little slack does more help than harm.

The Captive Build Tool

Ever struggle to get your build script to run, let alone build? Then think about making one master build tool, that serves as a single installation for building your project or program. But think carefully before you make it - there are tradeoffs.

Perhaps the most important feature of this tool should be its plain vanilla quality. If you can do this, it should be able to deal with any project, and will then surely find its way to the hearts of the developers, who will regard it as a dependable commodity. Early on, the name Ant Farm caught my fancy for this pattern, but then I realized that I would also have to make a NAnt farm and a Phing Pharm. Make it generic enough to avoid:

The temptation to add new features, which would force everyone to upgrade the tool.
Tasks that insist on being on the boot classpath.
It becoming attractive to key dependencies.

Do not skimp when considering tasks or libraries for the tool: just be sure that they are not project specific. This is a utility to make it easy to onboard developers, not a cage to lock them in. You do not want to have the building and testing of your application dependent on the use of the right version of build tool.

Finally, check your build tool into your version control system, and use a location relative to your project(s). That way you can have a go.bat or go.sh: a one-line wrapper script to call the project for the correct build tool. The simpler the script, the better.

Once you have your captive build tool, new developers can be cutting (and building) code before they go to lunch, boosting not only productivity, but newbie self-esteem as well. This pattern brings more love to your team because it frees each developer from the task of downloading all the libraries.

Amnesiac CI Build

Back in 2004, I set up my first CruiseControl build, updating the working copy from the Ant build by using our Version Control System (Perforce). That was how I got the latest version of the code onto the project’s working area. Since I needed to know things about the VCS to do the checkout, and had to tell the build where to find the VCS, I tagged a successful build from Ant, too. What a pain that was.

Bootstrappers went a long way toward alleviating this pain, as they eliminated the duplicated efforts of telling not only both the build and the CI server the same facts about the VCS, but the CI server where to look for changes as well.

So bootstrappers were great as far as they went, but they do not address the vexing issue of tagging. What I used to do was obviate the main build’s need to know things such as VCS info by tagging with a publisher. Enter the new CI tools, that automatically do the tagging whenever a project completes successfully. These tools include Team City, CruiseControl.NET, and Hudson.

Your build need not know a thing about where it really resides, or what its history is, when you push checkouts and tagging to your CI server. Voilà, the Amnesiac Build.

Software Practices Advancement Conference Workshop: Patterns without Developers

Last year, I led a workshop at the SPA Conference [1], held at the British Computer Society’s buildings in London, to gather patterns in the software development process originating from neither the Gang of Four [2] nor the Patterns of Enterprise Application Architecture[3]. Participants did good work, although the process was not very conducive to the creation of highly detailed patterns. A list of pattern names, definitions, and other criteria follows.

Embedded Test Team. This pattern embeds testers with development team, obviating the handovers that slow things down when you have separate testing and development teams. It is an effective pattern, even when team sizes swell to the point where silos and specialization become necessary. Putting the specialists where they are needed the most makes sense regardless of how big a project is.

Blue-green deployments. Using this pattern results in practically no downtime on release, keeping production live. It works by deploying to an unused environment and then switching traffic to hit that unused environment. The name came from inspired by Continuous Delivery[4]. Patterns like Encapsulate Table with View, and NoSQL databases ensure that you can deploy two application versions at once without either one of them throwing database-related errors. It was particularly successful in a project where it was imperative to access the database using stored procedures. Although developers’ productivity suffered because of the stored procedures, when we released a new version there were only seconds of downtime.

Cookie Cutter Servers. In this pattern, you deploy servers as images or automatically built machines, rather than manual or evolved installations. Since it automatically maintains consistency between production and testing servers, you might never again have to maintain consistency on different environments by hand. Tools like Puppet [5] and Chef [6] come in very handy.

Simplicator (Freeman / Pryce). This one calls on you to define your own API, and implement it by a stub or an adapter for testing. It makes testing more deterministic by decoupling service consumers from providers.Published in Growing Object-Oriented Software, Guided By Tests [7].

A/B Deployment. A/B testing allows you to find out which versions of your website are most appealing to users (and hopefully which allow you to sell more product). Google famously testing dozens of shades of blue to find out which performed best, at the cost of a designer’s inner calm. The trick here is continuous deployment to a limited subset of machines, allowing you to gauge improvement of new version over the old before switching all users to the new. You need multiple versions to operate in parallel using interface versioning for this pattern. Timothy Fitz covered this pattern at IMVU[8], but I provisionally took the name from Split Testing.

Virtualization. More of a technique, really: use virtual machines to simulate systems in the operational environment, creating a scale model of the production environment for functional testing. You may find it most useful for end-to-end testing, where you need an operational test environment for each external system to which the software interfaces. In general, it works well for operational procedures, where you want to test for the responses to various events. Using it can be particularly handy at businesses such as banks, where you can build a virtual machine in a fraction of the time it typically takes to get a server approved and delivered to the point where you can use it. It also speeds up your ability to manage change and develop against realistic hardware. If you run Linux on server and desktop, and can deliver virtualized nodes without the tedium of licensing, this becomes a very compelling approach.

I was recently told by a friend that it takes six weeks to get a new virtual machine approved at a large bank. The lead time for real hardware is about twice that. I still stand by this advice, as you may get your server before Christmas.

Time Slicing. The idea here is simple enough: bring down test environments when they are not testing. It uses test hardware more efficiently than would be the case if that hardware were reserved exclusively for testing. Since testing requirements are sporadic, this pattern levels the cost of resources over time. It relies on virtualization and cookie-cutter servers, which also pays off by preventing project delays caused by, for instance, a test environment being booked for months.

Stubs. Using this pattern involves testing a system against stubs and other systems in the environment before testing it in a full pseudo-production environment. Stubbing out external services has multiple advantages. For instance, if your code fails to talk to the stubbed services, you know have more work to do. Using this pattern makes that work less of a hassle than it would have been after deployment.

Atomic deployment. Think of this pattern as transactional deployment: either everything gets deployed, or nothing does. Admittedly, it can be deceptively hard to get going (Can you really roll back that database change? What is the risk of doing that?), but I think you will find it worth the effort. Artifacts of a failed deployment can cause things to break in ways that are difficult to diagnose.

Configuration Repository. This pattern eliminates embedded service identifiers, addresses, and sizings, abstracting parameters out to a configuration service so that it is easier to manage. This way, you can deploy identical software to every environment without adapting. People have been talking about this pattern for a long time, but my erstwhile colleagues Chris Read [9] and Tom Sulston [10] seemed to have successfully used it with Escape[11]. Using it with a NoSQL database would seem useful, as all configuration seems to be represented in name-value pairs.

Four More, for good measure

Aslak Hellesøy created Immediate Test Failure Notification, which prevents having to wait for a long CI build, only to find out a test failed for a trivial reason. [12]

Jon Tirsen coined Fast and the Full Builds. As you would imagine, it speeds up full builds [13]

Sam Newman authored Checkin Gate, which already has been nicknamed Checkin Dance and Movable Checkin Gate [14].

Parumu has described several, notable among which is Binary Deliverable [15], an impressive deployment tool that makes up in predictability what it lacks in subtlety.

References

Related Articles

Continuous Integration: the Cornerstone of a Great Shop

Related Resources

Open Source Software Configuration Management Tools Directory

Click here to view the complete list of archived articles

This article was originally published in the Spring 2011 issue of Methods & Tools

Methods & Tools
is supported by

Testmatick.com

Software Testing
Magazine

The Scrum Expert