Methods & Tools Software Development Magazine

Software Development Magazine - Project Management, Programming, Software Testing

Scrum Expert - Articles, tools, videos, news and other resources on Agile, Scrum and Kanban

Click here to view the complete list of archived articles

This article was originally published in the Summer 2007 issue of Methods & Tools

An Agile Tool Selection Strategy for Web Testing Tools - Part 3

Lisa Crispin, http://lisa.crispin.home.att.net/

Be Open to Experimentation

Several months later, we had good coverage on the GUI side from our smoke test scripts, a safety net of unit tests for new code (plus the benefits of TDD), and used FitNesse tests to help guide development. We still had unfulfilled testing needs.

One difficulty was setting up test data. We had test databases, but creating data to achieve a specific scenario was often a tediousmanual task. When we hired a tester with extensive Perl experience who was alsointerested in learning Ruby, we decided to give WATIR a try. Although we alreadyhad a GUI test tool, our new WATIR scripts allowed us to quickly create anycombination of data we needed, making it much easier to do serious exploratorytesting. No doubt, we could have probably done the same thing with the tools wealready had, if we had spent the time researching how to do it. Having someonewith skills to get us all up to speed with WATIR scripts made that the path ofleast resistance, although we had to plan time to learn Ruby. (I enjoyed learning Ruby, and enjoyment is an important factor too!)

Another item missing from our toolbox was a load test tool. When we needed a way to do load testing a couple of years down the road, westarted doing some research. I asked the members of the agile-testing Yahoogroupfor load test tool suggestions. Recommendations included Httperf, Autobench,OpenWebLoad, Apache Bench, Apache JMeter and The Grinder. After consideringfactors such as scripting language, learning curve, result content andformatting, recommendations from other users, and compatibility with ourapplication, we budgeted time for trial runs of both JMeter and The Grinder. Wehad the best results with JMeter, a Java desktop application, and found that atool called Badboy helped us create the JMeter scripts. We liked the statisticsit generated, and how they were displayed. We were productive with JMeterquickly and were happy with our choice. As with all our tools, though, we’re always open to new developments on the tool front.

Even after you have test automation in place, there is so much development of new and existing tools that it pays to keep up with what’snew. Monitor mailing lists such as http://groups.yahoo.com/group/agile-testing,attend local user group meetings, read online and print publications (such asthis one!) and attend conferences when possible to catch the latest tool buzz.Recently a couple of our team members saw demos of Selenium, and theagile-testing mailing list had posts from happy Selenium users. It has somepotential advantages for us, so when we can budget time for it, we’ll check itout. We don’t intend to throw any existing tools away, since they’re stillworking for us. But if there’s a better way to automate particular types of tests, we’re open to trying it.

Your needs might change and evolve, as ours did. As team members come and go, your team’s skill set might change. When our WATIR expertmove on, my Java programmer teammates bought Ruby books and started learningmore about it. However, now there is a Java-based option, WATIJ, that might bemore appropriate to our needs (there is also a .NET version, WATIN). Part ofSelenium’s appeal is the ability to write the scripts in Java (which theprogrammers like) or Ruby (which I like). More and more tools, especially theopen source ones, support multiple languages and provide more flexibility.

Tools aren’t just for automating regression tests or helping with exploratory testing, either. Scripting languages such as Ruby canautomate all kinds of tedious tasks that teams need to do. Get ideas from books such as _Everyday Scripting with Ruby_ by Brian Marick.

Sometimes tools can be combined for even more advantages. Groovy scripts can be integrated into WebTest scripts to provide more flexibility. Selenium tests can be run from FitNesse. Those are just a couple of examples. Don’t limit your thinking to an individual tool’s features.

Successful Tool Implementation

What have our test tools done for our web application development? Just a few months after adopting JUnit, Canoo WebTest and FitNesse, we had a suite of regression tests that gave us a useful safety net, and our defect rate was going down dramatically. Today, three years later, we have increased our test numbers by a factor of ten or more. Our regression suites, running many times per day in two continuous builds, catch bugs at least a few times a week. Our rate of defects introduced during development is down by half. We have time and tools to do robust exploratory testing and load testing. Most importantly, but harder to measure, using our tests to drive development has resulted in features that delighted our customers.

My team put plenty of time into the research and tool adoption I’ve described here. Test automation is a big investment, but carefully done, it returns many times what you put into it. You need enough resources to first define what you need, then investigate your options, then to try out tools. Our team takes two weeks, twice a year, to devote to tasks such as researching new tools, and refactoring tests and code. This may seem like a luxury, but our management knows it helps us keep our technical debt to a minimum, so that we can improve our future productivity.

Depending what people have which skill sets, it may pay to pair people up to research or try out a test tool. A programmer and a testercould team up to try out a scripting language. A system administrator might helpdetermine if a tool can be integrated into the team’s build process. Havebrown bag sessions to brainstorm ideas, or start a book club to get ideas from publications.

Remember the ‘whole team’ approach. The team should come to a consensus on what tools to build, to try or to adopt. If a tool isn’tproducing expected benefits, the team should decide whether to try a newapproach or a different tool. More experienced members of the team can coachtheir less experienced coworkers to help everyone get up to speed on using thenew tool. You may need to involve experts from outside your team to help yousucceed with the new tool. People outside your team who need to use the tool, for example, to specify tests themselves, will need your help.

What If Your Team Isn’t Interested?!

This sounds very nice," you say, "but my team is so overloaded and busy, nobody has time to think about tools, and they don’tthink it’s important to automate tests." Not everyone has like-thinkingteam members. Or maybe you’re a tester on an isolated QA team that isn’t getting much support from programmers or other groups.

Don’t despair, just get creative. I once worked in a chaotic company developing a retail internet application. Despite encouragementfrom the development manager, the programmers automated few unit tests. Thecompany owned licenses for a vendor GUI test tool. We hired a tester withexpertise in that tool, who could automate some regression tests and teachothers how to use the tool. But the tool couldn’t address all our automation needs.

The web application was written in TCL, so there were several TCL developers. On various mailing lists, I ran across several people using TCLeffectively for test automation. I decided to teach myself enough TCL to createtest scripts for areas we couldn’t cover with the vendor tool. While thedevelopers weren’t interested in test automation, they were happy to help mewith my TCL coding problems. This shows how it pays to leverage the expertisearound you. Although far from an ideal situation, we automated enough regression tests to free up our time for useful exploratory testing.

People, not tools, make projects successful. It doesn’t matter what tools you use or develop, as long as they help you towards yourgoals. Collaborating to choose and use the right test tools for your team’ssituation helps allow all team members to do their best work. That’s the bottom line for test automation.

References

Related Resources

Page 2   Back to the archive list

Methods & Tools
is supported by

Testmatick.com
Software Testing
Magazine
The Scrum Expert