Click here to view the complete list of archived articles

This article was originally published in the Winter 2009 issue of Methods & Tools

---

Does Process Improvement Really Pay Off?

Hans Sassenburg, Lucian Voinea
SE-CURE AG www.se-cure.ch, SolidSource BV www.solidsourceit.com

Introduction

Since the early nineties in the previous century, many organizations have substantially invested into software process improvement. Starting in the military industry, the concept of process improvement has nowadays been widely adopted in many other industry segments. It is one the few initiatives that have sustained over time, this in contrast to many hypes. CMMI as the primary reference model for process improvement has become widely known, both in industry and even academia. The widely accepted premise is that higher maturity will lead to better performance. Frequently, we are confronted with press release, stating "Company XYZ has reached CMMI Level 2". And sometimes even level 3 or higher. Level 3 means that processes at software or even system level have been standardized. Not only on paper, but also institutionalized in the daily way of working. Nothing wrong with this, but a justified and often raised question is what the pay off is. Does the quality of products increase? Has efficiency improved? Are products being brought to the market faster? We present some investigation results here.

Measurement in practice

In a series of assignments, the set was used to measure the performance capability of organizations. This implied the assessment of values for each indicator. Two important conclusions were drawn. Many low maturity organizations (CMMI Level 1) have the opinion that they lack quantitative data. In most cases, this is not true. Although not centrally stored, many sources of data can normally be identified. The challenge is then to identify these data sources and analyze them in order to obtain useful information. For the analysis of the software itself and its evolution over time, advanced but affordable tooling is available. Higher maturity organizations (CMMI Level 2 and higher) often have the opinion that they have access to rich sources of information. In most cases, the contrary is true. Despite many measurements, the conversion of data to useful management information is a weakness for many organizations. In addition, the availability of clearly defined measurement constructs and validation of measurement data before consolidation are exceptions. This leads to problems with respect to consistency, completeness and reliability. A possible explanation is that at level 2, measurements are mainly performed at project level. At the road towards higher maturity levels, it is tried to aggregate the data at organizational level. Issues are in that case the different project backgrounds and the lack of discipline to make project data centrally available and use it.

Correlation performance and process improvement

Back to the central question. Does a higher maturity level lead to increased performance? Our experiences so far reveal a disappointing picture. In the studies performed, no strong correlation could be found between performance, expressed in the 16 indicators, and maturity levels. In the first place, inefficient development processes were found in all organizations studied, irrespective of the maturity level. Too much time is spent on testing and defect removal instead of preventative activities. It even seems that the focus on reviews and inspections, examples of such preventative activities, is decreasing. This trend is confirmed by Capers Jones, who explains this phenomenon as ‚loss of organizational memory‘. Those managers that initiated improvement programs in the past have moved to other positions. Secondly, intermediate verifications of the quality of the software during development are at a low level. Although there is not a direct lack of measurements itself, converting the resulting data to useful management information and taking appropriate actions is missing. Examples are lack of clear insight in reached test coverage during various tests as well as underpinned predictions about the final defect density at release time. And these are primary indicators for reliability and maintainability. This all leads to situations where software is released when either the deadline has been reached or all critical defects have been removed. Intuition prevails, where numbers should be used. In summary, most projects are still managed by three indicators only: scheduled deadline, overall budget and removal of critical defects towards the end of the project. This is a narrow view on a multi-dimensional problem. Compare it with a car driving towards a destination under time pressure. There is no attention for efficient fuel use and preventative car maintenance. The consequences are well-known. One arrives late as a result of (avoidable) car repairs and the fuel bills are (unnecessarily) high.

Process improvement is not a goal

The conclusion that investing in process improvement is a waste of money would be wrong. Process improvement makes sense, as it standardizes development processes. This creates transparency with respect to roles, responsibilities, activities and deliverables. However, standardization is no guarantee for real performance improvements. That is why aiming at higher CMMI levels only is a wrong approach. The recommendation is to focus on a multi-dimensional assessment of the performance of an organization and derive improvements that make sense. A first step will be baselining the current performance using the best practice KPI set. In case measurements are not place, the first improvement actions are identified. As a second step, realistic target values must be determined for a given period of time. The gap between target and actual values is the basis for deriving improvement steps. It is hereby essential that the expected direction of each indicator is known. If changes in the right directions can be achieved, then a performance improvement is guaranteed. In addition, the positive effects are likely to show up in each category. This reduces or even eliminates the chances of sub optimization. The interesting fact is that improvements can only be achieved by changing the way of working. And of course, a validated improved way of working should be shared: considering standardization across projects becomes a logical process. This brings us to the conclusion of our story. Process improvement and standardization should not be a goal on its own. Real performance improvement results from taking a quantitative view on processes and products, and setting realistic improvement targets. If successful, higher CMMI levels will be a free gift in addition.

Best Practice KPI Set

Over the last years, the authors have developed a set of key performance indicators that characterize the software development process and the quality of the software. This multi-dimensional set of indicators has been derived from analysis of empirical benchmarking data and studies of multiple organizations from different industry segments and performing at different maturity levels. The final set of indicators has been selected using three different criteria. First, they must support project leaders and management in planning and monitoring projects. Secondly, they must enable the comparison of process and technology changes. Finally, they must enable the assessment of the performance capability of an organization. This means that an improved performance must reveal effects on each indicator and/or a combination of indicators. This is an important characteristic. It enables the assessment of current performance capability. But additionally, it allows the analysis of trends over time. It also allows for benchmarking with other projects, departments and external organizations. The combination of our analysis and research has resulted in a set of 16 indicators, categorized in four different dimensions (see figure below):

1. Project performance: What is the prediction of the performance of the project in terms of schedule, budget, resource availability and productivity?
2. Process efficiency: How efficient is the development process, seen from a cost of quality perspective?
3. Product scope: How large and stable is the scope of the planned effort in terms of feature and implementation size?
4. Product quality: What is the expected quality of the resulting product in terms of reliability and maintainability?

For each category, 4 indicators have been defined. Improved performance will normally have an effect on each of the individual indicators. See figure. Example: improved product quality can be achieved by:

• Reduction of the software (cyclomatic) complexity, hereby reducing the number of independent test paths. This positively influences reliability and maintainability.
• Increased test coverage, especially during unit testing, resulting in a lower defect density prior to integration and system testing. This positively influences reliability and maintainability.
• Reduction of defect density, by pro-actively decreasing the number of remaining defects after each development steps. This positively influences reliability.
• Increased removal efficiency, as relatively fewer defects are injected and more defects are detected and removed in each development step. This positively influences reliability.

These positive effects on software quality can only be accomplished by changing the conventional way of working. The most influential change will be the reduction of time and effort spent on testing and defect removal. This can for instance be achieved by a higher investment in preventative activities like inspections and reviews. The objective will be to finish the implementation phase with a high product quality. If successful, positive effects will also become visible for other indicators. Lead-time (schedule) and budget (effort) will be decreased and as a result, productivity will increase.

Related Methods & Tools articles

• Don't Write Another Process
• Assessing Readiness for (Software) Process Improvement
• Process Improvement – Is it a Lottery?

---

Back to the archive list