Click here to view the complete list of archived articles

This article was originally published in the Spring 2001 issue of Methods & Tools

---

Uncharted Territory: Introducing QA in a Web Startup

Lisa Crispin, http://lisacrispin.com/

Much has been written, in this newsletter and other publications, about the risks of e-Business applications. "Web-time" is a widely acknowledged phenomenon. We all agree that quality is imperative for an e-Business, as all the competition is just a click away. Unfortunately, most of us can agree that a Web startup is not an environment in which quality testing is typically found.

Development is fast and loose. Many developers are inexperienced. Marketing is pushing to be beat the competition to market. The rules change every day. An e-Business needs to respond immediately to market pressures. And an e-Business cannot afford poor performance or that big revenue drain, downtime.

I have now survived bringing quality assurance (QA) and testing to two startups - one an e-Business, TRIP.com, the other a software development shop, Tensegrent. By sharing my experiences, I will provide guidelines for successful implementation of QA and testing in the startup environment.

When I accepted the opportunity of being the first test engineer at TRIP.com, the startup consisted of 25 employees. The developers had produced some exciting applications and felt they were ready to "grow up and play with the big boys." The development team thought they were intellectually prepared to introduce standards and procedures.

In reality, development was frenetic, and the developers didn't have a clue as to how to stop and analyze their processes, much less how to impose discipline on them.

What's worse, the wide wild world of the Web was completely new to me. I truly felt lost in the wilderness without a map. My background was in testing traditional software in mid- to large-sized software companies. I had worked on every possible platform and operating system, tested databases and client/server software, but knew nothing about Internet applications. I knew my customers' needs. I was used to development cycles of several months - during which your typical Web application has gone through numerous incarnations.

Using the strategy I will outline below, I overcame these limitations. I collaborated with the software and product developers and marketing managers to implement development standards and project processes that build quality into the applications. TRIP.com now employs over two hundred people and has over four million registered customers. It was acquired in March, 2000 for $326 million by Galileo International Inc.

Recently I moved on to Tensegrent, getting in much earlier in the lifecycle of a startup - I was hired in the second month of the company's existence, the 10^th employee and the first Test Engineer. Tensegrent is devoted to developing high-quality software through the use of the eXtreme Programming (XP) technique. The good news was that XP involves intense unit and integration testing on the part of the developers. The bad news was that the role of the Test Engineer was not well-defined.

Any startup or DotCom is a work in progress. Even once we had a successful project process at TRIP.com, we faced continual challenges such as an inadequate test environment. Even when the whole company understands and is committed to the importance of quality assurance testing, unexpected events lead to surprises. The key is to keep plugging away at the following tasks:

• Get Buy-In
• Work Smart
• Define Processes

I. Get Buy-In

I won over my managers, developers, and marketing counterparts by following these tenets:

Identify a top manager in your organization who believes in your cause and will champion it. At TRIP.com, it was the Vice President of Web Development and Chief Cat Herder (yes, that really was her title). When I was hired, this person did not believe that five developers could keep one tester busy full time. But, in time, she became my biggest ally. She not only pushed the developers to work for quality, but she also lobbied the management team for testing resources. At Tensegrent, the staff was actively committed to quality, but I needed help from the General Manager to get them on track.

Partner with someone outside of your organization, such as a project or operations manager. Educate your ally to garner his or her help. At TRIP.com, we had a topnotch project manager who, once she understood what QA and testing would do for the company, did much of my job for me. She enforced processes such as document review and signoff, helped implement and police the defect tracking system, and tied up a million details involved with every big production launch. She became the prime channel of communication between marketing and development. At Tensegrent, our team was the whole company, at the beginning; the team captains were key to enforcing process.

Educate everyone you come into contact with at the company about software quality assurance: what it is and what it will do for them and the products. Early on at TRIP.com, I held a "Lunch 'n' learn" professional development seminar. The company bought lunch, so attendance was good. I explained why testing is essential and what it involves. At Tensegrent, I held a similar Brown Bag session where I demonstrated the automated test tool to the developers. Lots of meetings and one-on-one encounters are needed to get everyone on board and to establish priorities.

Support Information Systems, the group that administers the production site. These people (or person, if your company is really small) will benefit from not having their pager go off so much when applications are tested before being launched to production. The Vice President of Technology and the IS director at TRIP.com fully supported me and refused to launch any update that had not been fully tested. This kept the rest of the organization from steamrolling over me, giving me a chance to prove the value of testing. I don't bug IS for the things I can do myself, but I let them do their job. For example, at TRIP.com, I installed Y2K patches on the test machines, but IS controlled all the UNIX and NT account management.

Understand the developers' point of view. You may have young, brilliant developers who don't communicate in ways you are accustomed to. TRIP.com's original developers were mostly very young and inexperienced. Some had not finished high school!. Others weren't old enough to drink! The culture was anti-corporate, and they said what was on their minds. I found that if I listened, I learned, and they in turn were willing to listen to my ideas. I learned everything I now know about Web applications from the developers themselves.

Celebrate success. At TRIP.com, I presented a "Quality Hero" award each month to a developer who took exceptional measures to prevent defects and improve quality. The prize was just a Nerf gun and the developer's name engraved on a plaque, but it raised the visibility of high quality process and techniques. At Tensegrent, I try to make sure the celebratory end-of-iteration Foosball games occur AFTER acceptance testing is successfully completed!

Brainstorm with developers and others about problems that may not come out in testing. For example, I didn't know that if you change a URL, search engines may not be able to find your site. When we implemented a content management tool at TRIP.com that required changing every URL, this was important information!

II. Work Smart

Here's my advice for making the testing organization lean and mean. This is especially critical in an Extreme Programming environment or anywhere the ratio of developers to testers is high.

Evaluate tools. Put as much time as you can into tool evaluation, such as those for automated testing, defect tracking, and configuration management. Identify the vendors who can help you the most, and get as much information from them as you can. Ask fellow testers for their recommendations and experiences. Install new tools and try them out. Select tools that are appropriate for you and your company. It doesn't do any good to buy a tool you don't have time to learn how to use, especially if your testing team is small. I ended up choosing tools that are lesser known but still meet our needs. For example:

• For automated testing, both TRIP.com and Tensegrent use WebART (www.oclc.org/webart), an inexpensive, easy-to-learn, but powerful tool sold by OCLC Inc. (a non-profit company). The vendor gave me valuable advice and provided insights about Web testing. Pick everyone's brain, including vendors!
• For defect tracking at TRIP.com, we chose a Web-based tool, TeamTrack (now called TTrack), from a startup company called TeamShare . It, also was far less expensive than its competitors, but it was easy to implement and customize. At Tensegrent, which is a brand-new startup on a small budget, we use the free Bugzilla successfully.
• For configuration management, at TRIP.com we again turned to a smaller, innovative company which produces an inexpensive, easy to implement and learn yet robust tool, Perforce. At Tensegrent, we use freeware, CVS - it lacks some features, but the development team is small and can work around its drawbacks more easily.
• These tools won't necessarily meet your needs - just be open and creative when evaluating tools. Investigate alternatives - for example, if you create unit tests to reproduce each bug you find in testing, you may not even need a defect tracking system!

Design automated tests that work for you.

Automated acceptance tests at Tensegrent and TRIP.com use the following criteria to provide useful tests with minimum resources:

• Modular and self-verifying to keep up with the pace of development in "Web-time".
• Verify the minimum criteria for success. Make sure the developers write comprehensive unit tests. Acceptance tests can't cover every path through the code.
• Perform each function in one and only one place to minimize maintenance time.
• Contain modules that can be reused, even for unrelated projects
• Do the simplest thing that works. This XP value applies as much to testing as to coding.
• Report results in easy-to-read format. Create easy-to-read reports from your test results and post them so that everyone can monitor status and keep the project on track. The project team will gain confidence as they see the percentage of successful tests increases.

In addition, the developers try to design the software with testability in mind. This might mean building hooks into the application to help automate acceptance tests. Push as much functionality as possible to the backend, because it is much easier to automate tests against a backend than through a user interface.

Search the Web for resources. I would have quit TRIP.com after a week if I had not found an excellent Web site that points to information about testing Web applications and lists of tools. These sites, in turn, led me to more tools and information Here are some examples:

www.softwareqatest.com/index.html
Everything from basic definition and articles on how to test Web applications to comprehensive lists of Web tools to links to other informative sites.

Articles by Cem Kaner

Hire good help. It proved impossible at first to find experienced test engineers in our area, so at TRIP.com we hired bright but inexperienced people with the right qualities that make good test engineers: enthusiasm, dedication to the end user, and determination. A caveat: inexperienced testers who have no programming experience have a harder time learning a scripting language for an automated test tool. However, by using a combination of outside classes, hiring consultants and patient, one-on-one training, our testers learned UNIX, SQL, test scripting, HTML, configuration management, and other technical skills. You're going to spend money either way - paying high salaries for experienced test engineers, or training novice testers. Once you've turned them into pros, remember to keep them challenged, happy and well-compensated so other companies don't poach them!

Get input about quality from all departments in the company. At TRIP.com, I formed a quality board with members from sales, marketing, customer support and travel to gather fresh ideas about error prevention and prioritization of regression testing. Whenever a crisis occurred, we held a quality review panel where representatives from development and information systems heard short presentations from the people who experienced and fixed the problem. The panel studied the issues and recommended steps to prevent such problems from recurring. This was a big effort and to be truthful, it was difficult to get follow-through. But give it a try. We also held post-mortems after all major launches to see what lessons could be learned. By employing these methods, we learned some valuable lessons!

Insist on a test environment that is exact replica of, but is entirely independent from, production. You can't emulate a production load without the equivalent of production hardware and software. Since the production architecture is likely to change in response to increased traffic and other considerations, this is a moving target. The test environment will need to be updated in synch with the production environment. The architecture is key too. If the production servers are clustered, your testing had better be done in a clustered environment. If part of an application runs on a stand-alone machine, it must do so in your test environment. Establishing and keeping up development, test and production environments was a huge challenge. Even when the entire company is sold on the idea of a proper test environment, there are business and technical reasons (read: excuses) that get in the way of reproducing the production environment in for testing . Don't be complacent, and never give up. Make sure you have the best test environment you can get for each application going into production, and work actively with your information systems team to get the environment you really need. Even small applications can deceive you. For example, at TRIP.com we launched a simple application, SantaTracker, on Christmas Eve so kiddies could watch Santa's sleigh fly around the country. The test environment was broken, so we were not able to test on the same architecture that it was to run in production, but we weren't concerned. After all, it should have worked exactly like our regular FlightTracker application! Right? Wrong! It was a disaster!

In short: Dig your heels in and refuse to launch until some semblance of a test environment is established. Remember, it is harder to get the test environment once the new application is in production. Make it a requirement of release.

Learn about the development tools and processes. They present their own quality issues and offer some solutions, too. For example, the first version of our content management tool did not have any version control. It took more than a year to upgrade to the release that offered this capability, and even then it didn't enforce version control. We had to constantly police the process to ensure that developers version their code. The software on which our Java applications were based had complex configuration parameters we didn't fully understand when we first put it into production. We had tested our intelliTRIP product with a production load in terms of transactions per second, but never with a realistic number of concurrent users. As a result, the servers kept crashing on the first day we put it in production. If we had understood the configuration and the user session management parameters better in our Java-application management tool better, we could have prevented this problem. At Tensegrent, a deep understanding of the XP process has proved to be essential. A project can go off track in a matter of days. My job is to help keep it on track by participating in development as much as possible.

III. Define Processes

Figure 1: Sample Project Process Overview

Collaborate with your counterparts to formalize your processes:

Get control of the production environments. Work with your information systems team to create a production update procedure. When I started at TRIP.com, developers launched their own changes to production. It was hard to wean them away from this bad habit. Even after we thought we had implemented good production update procedures, we lacked the discipline to enforce it under pressure. For example, since we did not have good configuration management, it became impossible to build baselines of intelliTRIP, so developers would simply move new classes into production to fix problems. Only after two years were we able to require developers to build scripts and installation documentation before we accepted any software from development for testing. At Tensegrent, we started this good habit at the beginning. If you can get in on the very start of a start-up, implementing best practices is much easier.

Get involved from the beginning of each project. This is hard work. It forces you to juggle many tasks, but it is essential. See Figure 1 for a sample overview of a project process. Participate in all documentation reviews: Those for requirements, functional specifications, and design specifications. Make sure the documents are complete and clear. Look for ambiguities, gaps, lack of detail. All parties, including marketing, development, test, customer support, sales must agree on a vision for the product. This vision is a short phrase that describes the main thrust of the product. With intelliTRIP, development and test were told to produce a server-side version of the original client-side product as quickly as possible. Sales and marketing believed that the purpose of the product was to quickly locate fares from airline Websites. Since development and test wasn't told that part about quickly, and was focused only on time to market, we released the product even though we knew that is was sometimes slow to return results. This type of disconnect can be prevented by including a vision statement in the requirements.

Define quality. Work with marketing and product development to define quality for each product: Should the priority be good, fast, or cheap? (You can only have two of the three!) Even if you choose fast, don't sacrifice the process. At TRIP.com, we once implemented a promotion that marketing believed to be simple and wanted to rush to production. Since the product manager did not hold documentation reviews and get signoff, the HTML pages produced by the developers had to be changed three times. This took much longer than a documentation review meeting. There is no need to get bogged down in process either. If you find that is happening, change the process, or train people how to use it properly.

Document the internal processes of both test and development. You can't expect marketing and product development to follow best practices if you don't do it yourself. At TRIP.com, one of the development directors, with the help of the technical writer, led an effort to define and document the development process. An aside: No matter the size of the company, unless you are using a lightweight technology such as Extreme Programming, you need at least one experienced, skilled technical writer in your development organization. See Figure 2 for a sample Software Development process.

Figure 2. Sample Development Process

Enforce the process. If your QA team is large enough, dedicate one person to administering and enforcing configuration management and delivery of installation scripts and documentation. At TRIP.com, we expanded this Configuration Manager role to that of a deployment engineer who works closely with developers to produce the builds and installation procedures. Don't accept software to test if it is not accompanied by all the documentation and software you need to promote, test, and launch it to production.

Innovate! Look for new ways to present documentation such as functional specifications. Web applications require a new approach. You have creative people at your company who can help! Get input from as many different groups as you can. If your company really wants to get innovative, consider lightweight methodologies such as Extreme Programming (see www.extremeprogramming.org). These methodologies can meet the need to get to market quickly, accommodate rapidly changing requirements but still release a high quality product.

Summary - As You Grow

All companies change as they grow beyond the 'startup' size and environment. My team and I endured countless frustrations when fast growth at TRIP.com repeatedly led to temporary chaos. As your organization grows, educate new employees about project process and quality practices. Listen to them; take advantage of their fresh outlook and new ideas. Take the initiative. If a gap results from a re-organization, fill it yourself. For example, when we were temporarily without a project management function in the company, the TRIP.com development director and I set up a weekly tactical meeting with representatives from all departments so that everyone could stay informed and juggle resources. Quality assurance can be a frustrating job, especially in a Web startup. Pick your battles. Keep striving for better quality. Above all, enjoy the experience!

---

More Software Testing and Quality Knowledge

Software Testing Magazine

Software Quality Assurance Portal

---

Click here to view the complete list of archived articles