Methods & Tools Software Development Magazine

Software Development Magazine - Project Management, Programming, Software Testing

Scrum Expert - Articles, tools, videos, news and other resources on Agile, Scrum and Kanban

Click here to view the complete list of archived articles

This article was originally published in the Spring 2006 issue of Methods & Tools

Continuous Integration: The Cornerstone of a Great Shop - Part 2

Jared Richardson, http://www.jaredrichardson.net

Test Automation

Another practice that a CI system encourages is test automation. Writing and running tests is a huge milestone for many shops and isone of the hallmarks of a great shop. I think test automation is the core of whya CI system adds such benefit. People who recognize the benefit of automatingcommon tasks tend to be more strategic thinkers. They automate everythingpossible, including building and testing, and it frees them up for moreinteresting work. (Of course, this doesn’t eliminate manual testing, but that’s another topic.)

What is an automated test?

  • Binary
  • Automatic
  • Repeatable

Binary

A test with a binary result passes or fails unambiguously. There’s no question about whether the test succeeded. Sometimes a test willreturn a result that requires a judgment call from a tester. The odds are good that you don’t need this.

Work hard to make your tests clean and binary. Write them so they evaluate the result and tell you if it passed or failed.

Automatic

If the test isn't automatic then someone has to set up an environment, start the test, click a button, or look at the results. When thishappens, the test becomes interactive again. Much of the benefit of test automation is lost.

You've created a hybrid test somewhere between an interactive test and an automatic test. Instead of letting a small number of testersbaby-sit a large number of tests and continually adding more tests, you'll havelarge number of testers looking at log files all day long. Half-automatedtests are certainly better than pure interactive testing but they fall far shortof where you can be. Work hard to make your tests completely automatic, including the determination of the pass or fail status.

Repeatable

An automated test also needs to be repeatable. A good test doesn't give you different results for three out of five test runs. If your tests aren't repeatable, break the tests down into smaller tests. Eventually you'll isolate the problem area and as a bonus, you’ll have new tests created for your test suite.

Don't forget about external dependencies either. You can rebuild and restock database tables cleanly before each test run with tools like Ant's SQL task or dbUnit (http://dbunit.sourceforge.net). A dirty database table can introduce all sorts of variation into a test run. (You may want to create a small but representative data set to load for your testing runs.)

Leverage Yourself

An automated test is a great way to leverage your experience and expertise. As an expert on your product, you probably know how to test partsof it in a way that few other people can. You can encode that knowledge in areusable format by creating an automated test. This makes your experienceavailable without diverting your attention. Sometimes a co-worker will run the tests, other times you will. In other cases, a program will run them.

Let these bits of your expertise exercise the product while you do other things, like go home on time, or stay late and solve problems thatare more interesting. These tests might run while you are coding or at homesleeping, but you are doing something else. Your tests are working in the background.

Getting Started

Sometimes people won’t install a CI system because they don't have tests ready to run in the system. There are enough benefits from fastcompile cycles to justify using Continuous Integration, so don’t wait. You don’twait to see a doctor until you’re not sick anymore, right? Having the CIsystem keep your compiles clean will free up some of the time needed to start writing tests as well.

I've also found that people are much more likely to write automated tests if they’re sure the tests will be used. By providing a CIsystem, you have a place to house your tests and run them immediately. This isthe best way I know to encourage test creation. People want to create things are used and this assures them the tests they create will run regularly.

The best way to get started with Continuous Integration is to start using an existing software package. I'm going to point you toCruiseControl on Source Forge (http://cruisecontrol.sf.net/).Since version 2.3 Cruise Control comes with an embedded servlet engine (Jetty- http://jetty.mortbay.com)and a sample project. You can download the project and see CC running in lessthan five minutes. Then, to add your own project, just copy the bundled example. It’s very easy to get started.

The CC team has a great write-up on how to run the binary release of CruiseControl. Visit http://cruisecontrol.sf.net/and click the "Getting Started" link on the left.

In Conclusion

The teams I know running smoothly and cleanly always have continuous integration in place. It's a practice I respect more everyday.

I’m seeing this practice overshadow all others. Teams that run smoothly use continuous integration. They respect the system instead oftolerating it, and the developers treat the notifications seriously. When thesystem says something is broken, these teams address the problem quickly. These teams insist on CI coverage from the first day of a new product.

Other shops, even those who are using a CI system but ignoring it, are very different. They live in turmoil. Heroic efforts are notthe exception but the rule. In fact, these teams always seem to be running behind. They always have a crisis issue to resolve or deadline to meet.

They work and live in a perpetual twilight of stress and problems. They’ve lived there so long that they think it’s the only way towrite software. Sadly, these teams tend to burn people out. I’ve been thereand it’s no fun. Creating software can be a great joy -and is- when doneright. CI can’t solve every problem, but it can remove several categories of problems that would otherwise clutter your day and slow you down.

If you’re not using a Continuous Integration system, try one out this week. Get a system installed and leave it running for one month. Atthe end of that month, turn it off if you don’t see the benefit.

Don’t be surprised if you find yourself missing the system the first day it’s gone. You might just become one of the developers whoinsists on continuous integration coverage on your new projects.

Resources:

Related Resources

Page 2   Back to the archive list

Page 1    Back to the archive list

Methods & Tools
is supported by

Testmatick.com
Software Testing
Magazine
The Scrum Expert