Managing Your Way through the Integration and Test Black Hole

Ellen George, PS&J Software Six Sigma

The Black Hole

How do you know how much effort to schedule for integration and test? Once in test, how do you know how much longer it will take? How do you know when you have found all the defects that you can find? When is integration and test really done? Often, integration and test is seen as a black hole. The product enters the phase, but no one seems to know when it will come out.

In order to understand how to accurately estimate integration and test effort and to develop a meaningful defect removal strategy, we need to first examine the relationships within the development and test cycle relative to defect injection and removal.

Integration and Test Processes

For simplicity, the discussion in this paper is limited to software only. However, the concepts can easily be extended to the systems engineering life cycle phases as well. In software, the major software development life cycle phases are requirements, architecture, design and code, ref. Figure 1, The Development and Test Cycle. In addition to creating product, these life cycle phases represent the primary mechanism for injecting defects into the product. Conversely, the major product validation life cycle phases are compilation, unit test, integration, and system test. The purpose of these life cycle phases is not only to validate the product, but also to remove the defects injected during development.

Figure 1 - Development and Test Phases

Each of these defect removal phases can be paired with a corresponding product development phase. The defect removal phases should be designed to find the unique category of defect injected during the corresponding development phase. Compile is used to find the syntactical defects injected during the coding phase. Unit Test is used to find the logic errors injected during Design. Integration should be used to find and fix the interface defects injected during Architecture, and finally, the purpose of System Test is to find Requirements defects.

Ultimately, the goals of integration and test are to:

• bring together the multiple pieces of a system
• find and fix defects that couldn't be found earlier
• ensure that the requirements of the system are met
• deliver a product of predictable quality that meets the business's quality objectives as well as the customer's quality expectations.

Estimating Integration and Test Effort

Many organizations estimate integration effort based solely on the number of test cases to be executed (integration effort = number of test cases x average time to run a test case). The problem with this method is that it misses the main cause of integration and test overruns... defects!! A better technique for estimating integration effort is to base it on the expected number of defects to be found in the product during integration and test plus the time it takes to run through a clean execution of all the test cases. Thus, integration effort = (number of predicted defects x average time to find and fix a defect) + (number of test cases x average time to run a test case).

In order to use this technique, you need to a mechanism to estimate the expected number of defects in the product and to determine the average cost of finding and fixing a defect during integration and during system test.

There are only three types of measurements required to manage the integration and test processes: effort, size and defects. "Effort" is the effort expended individually by each of the testing participants to find and fix the defects found in test. "Size" is the size of the work product being tested, often measured in lines of code (LOC). "Defects" is the quantity, type, injection phase, removal phase, and description of the defect. Although a defect measurement consists of several pieces of data, it is collected as a single measurement at a discrete point in time.

Using these three measures, we have all the data we need to estimate the number of defects expected to be found in integration and in test as well as the average cost to find and fix them.

Predicting Product Quality

Let's assume that the data collected tells us that developers inject approximately 40 defects per thousand lines of code (KLOC) in design and 60 defects per KLOC in code. Let's also assume that each defect removal step typically removes between 35% and 50% of the defects that are in the product. Using these measures, we can begin to model the product quality as shown in Table 1, Quality Matrix. Note the values in Table 1 are all referenced to 1 KLOC of new and changed code. These numbers are used for illustration only. Your own data may differ significantly.

Phase	Defects Remaining	Defects Injected	Defects Present	Defect Removal Yield	Defects Removed	Cost/ Defect
Design	0.0	40	40.0	0%	0.0	5 min
Design Inspection	40.0	0	40.0	50%	20.0	10 min
Code	20.0	60	80.0	0%	0.0	1 min
Compile	80.0	0	80.0	50%	30.0	1 min
Code Inspection	50.0	0	50.0	50%	25.0	5 min
Unit Test	25.0	0	25.0	50%	12.5	12 min
Integration Test	12.5	0	12.5	35%	4.4	5 hrs
System Test	8.1	0	8.13	35%	2.8	10 hrs
Customer	5.3

Table 1 Quality Matrix

To determine the average time to find and fix a defect in each of the defect removal phases, take the total effort spent in the defect removal step and divide by the total number of defects found in that phase. Assuming all defects that were found were also fixed, then the average time to find and fix a defect in integration would be the sum of the time spent by all testers in integration divided by the total number of defects found and fixed in integration. The same can be done for system test. This data can be documented in the quality matrix in the column titled "cost/defect". Note, the time spent writing the test cases and the time spent doing the final clean run through the test cases should be subtracted from the total integration or test effort before doing the calculation.

Clearly, the number of defects remaining in the product when it reaches integration will depend on the quality of the developed product as well as the adequacy of the defect removal steps leading up to integration. It is critically important for the developers and the testers to agree upon and set goals for defect injection and removal phases at the start of the development effort. These goals should be documented in a Quality Matrix. The effort estimate for integration and test can then be calculated by multiplying the number of defects expected to be found in integration and test by the average cost to find and fix a defect in those phases.

Quantitative Entry and Exit Criteria

What happens, however, if the quality of the product when delivered to integration does not meet the plan as documented in the Quality Matrix? If more defects enter integration than planned, then the estimated effort will not be adequate to complete integration and test.

Entry and exit criteria are used to "gate" the process. Before the product is allowed to exit a phase, the number of defects removed should be compared to the goals as documented in the Quality Matrix. A threshold can be set around the expected defect removal value. If the number of defects found in-phase falls within the expected range, then the product is allowed to move forward. If the number of defects found falls outside the expected range, then the product is prevented from moving forward.

For example, using the plan in Table 1, we expect to find 20 defects in design inspection. We can place a +/- 20% threshold around this value, resulting in an expectation of 16 to 24 defects per KLOC found in design inspection. If the actual number of unique defects found falls within these bounds, then the product moves forward to the coding phase. If the number of defects found in design inspection exceeds the upper bound, then the inspection moderator stops the product from moving forward and takes steps to determine if the inspection team did an exceptionally good inspection, or if the product was buggier than expected. Similarly, if the number of defects found in design inspection is less than expected, then the inspection moderator needs to determine if the team did a poor inspection, or if the product was of particularly high quality entering inspection.

One relatively quick mechanism for determining product and process quality can be made by asking another developer to spend a few minutes reviewing the fixed product. If they can find a defect in a short period of time, then the product is probably buggy. If the product is determined to be buggy, then corrective action needs to be taken before the product exits the phase. If a product that is known to be buggy is moved forward though the process, then each subsequent phase will have to spend more effort than originally planned and budgeted to remove the extra defects. When a product meets all of its phase exit criteria as it moves through the development process, then there is a high probability that it will enter integration with the planned number of defects waiting to be found.

Managing the Integration and Test Effort

Integration and test is managed by managing defects. The entry criteria into integration should be used to prevent poor quality product from moving forward. Only product that meets the entry criteria as specified in the Quality Matrix should be accepted. If product that is known to not meet the entry criteria must be accepted, then additional cost and schedule needs to be negotiated to adequately test the product. Once the product has been accepted and integration has begun, several techniques can be used to manage the defects that are found. A brief discussion will be provided on two such techniques, "Looking for Systemic Defects" and "Test vs. Rewrite"

Systemic Defects

As humans, we tend to be very repetitive in our actions. As a result, when we make a mistake once, it is probable that we have made the same mistake elsewhere. This human characteristic can be used to an advantage. Each time a defect that is found in integration and test, it should be used as an opportunity to look for additional occurrences of the same defect elsewhere in the product.

Each time a defect is found, a root cause analysis should be performed on the defect. When was the defect injected? How was it allowed to leak into integration? Where else in the code might a similar defect have been injected? By answering these questions and looking for additional occurrences of the defect, the tester can maximize the number of defects found through a single test failure. An automated search or targeted inspection can easily be used to search suspected code for duplicate defects.

Test vs. Rewrite

Defects tend to cluster. Actual data that we collected over 1200 modules illustrates this point in Figure 2, Defects Per Module. 70% of the defects were found in 20% of the modules. 50% of the defects were found in 10% of the modules. This data suggests that if we can identify the 20% of the modules that are the buggiest and take corrective action, we might be able to manage, or eliminate, 70% of the defects in the product.

Figure 2 Defects per Module

When a particularly buggy product is identified, appropriate corrective action could be to remove the module from the integration build and re-inspect it for additional defects. If too many defects continue to be found, then it might be more cost beneficial to rewrite it than to continue testing it.

Example

Assume Project A uses the quality plan specified in Table 1. Module X is currently in integration. Module X's size is 1 KLOC. According to the quality plan, a total of 4 to 5 defects are expected to be found in integration for a module of this size. 10% of the module's integration test cases have been run, and 3 defects have been found so far. We can assume that since 10% of the test cases have been run, that the 3 defects found represent 10% of the defects that will be found in integration. So, we calculate that Module X probably has a total of 30 defects that will be found in integration, compared to the plan of 4 to 5. As a tester, what should you do about this module? Continue to test it? Or send it back to development to be re-written?

Continue Testing

At 5 hours per defect, this module has already cost approximately 15 hours in integration. If we continue testing to find the remaining 27 defects, at 5 hours apiece, they will likely cost a total of 135 hours.

If we continue testing, then we can assume that:

• 30 defects will be found in integration
• we expected to find 35% of the total remaining defects in integration, so
• 86 defects entered integration (30 / 35% = 86)
• 56 defects will leak into system test (86 - 30 = 56)
• we expect to find 35% of the total remaining defects in system test, so
• 20 defects will be found in system test (56 x 35% = 20), and
• 35 defects will leak out to the customer (56 - 20 = 36).

The estimated cost of continued testing is 335 hours:

• 27 remaining defects x 5 hrs = 135 hours remaining in integration
• 20 defects x 10 hours = 200 hours remaining in system test
• 36 defects delivered to the production

Re-Write the Module

Assume a developer can design, code and unit test code at a rate of 20 LOC per hour. If the quality of the product meets the entry criteria for integration, then there will be approximately 5 defects found in integration and 3 defects found in system test.

The estimated cost of re-writing the module is 105 hours:

• 1000 LOC x 20 LOC/hour = 50 hours to re-write the module
• 5 defects x 5 hours = 25 hours to integrate the module
• 3 defects x 10 hours = 30 hours to system test it
• 5 defects delivered to the production

In this example, throwing away the module and re-writing it would yield a net saving of 230 hours over continuing to test it.

Summary

In order to successfully manage integration and test, it must be planned and managed as an integral part of the entire development lifecycle. The factor that most affects the effort required to complete integration and test is the number of defects expected to be found in these phases. This number comes as a direct result of the development process used to generate the code to be tested.

The key steps to a successful integration and test effort are:

• Collect basic measurement data: effort, size and defects. Use the data to create a Quality Matrix to baseline the organization's capability.
• Build a project specific quality plan based on the organizational Quality Matrix. Get agreement on quantitative phase entry andexit criteria from both developers and testers during project planning.
• Estimate time required for integration and test based on (number of expected defects) x (average time to find and fix a defect)
• Ensure that both developers and testers agree to deviations from the quality plan. Deviations from the plan should also result in negotiated adjustments to the budget and schedule.
• During integration and test execution, whenever a defect is found, use it as an opportunity to find as many additional defects as possible. Since people tend to be repetitive in their actions, it is very likely that the same defect will appear within the code in several more locations.
• Use defect identified in integration and test to understand how the defect was injected and how it was able to leak into the later life cycle phases before being found. Take corrective action to either prevent the category of defect from being injected again in the future or to contain it in the phase in which it was injected.
• When defect data on module indicates that it may be much buggier than planned, return the module to development for corrective action. Use calculations based on productivity rate, expected defects and time to find and fix a defect to determine the best course of action. Consider either a re-inspection of the code or to possibly re-writing the code from scratch.

---

Related Software Testing and Quality Assurance Resources

The Traditional Test Pyramid, Pitfalls and Anti-Patterns

Automated Testing Strategy for a Microservices Architecture

Metrics for Implementing Automated Software Testing

Software Testing Magazine

---

Click here to view the complete list of archived articles

This article was originally published in the Fall 2004 issue of Methods & Tools