Click here to view the complete list of archived articles

This article was originally published in the Fall 2006 issue of Methods & Tools

---

Making an Incremental Transition to Software Product Line Practice - Part 3

Charles W. Krueger, http://www.biglever.com/
William A. Hetrick and Joseph G. Moore; http://www.engenio.com/

Our historical metrics have demonstrated a strong correlation between the number of branched files and the number of duplicate defectresolutions required. The reduction in file branches has led to a derivative return on investment – a reduction in clone defect resolutions.

Finally, at a strategic level, LSI Logic quickly gained increased confidence in their software product line development capabilitiesand, as a result, their ability to respond to more aggressively to expandingbusiness opportunities. As is typical with software product line transitions,the strategic returns on investment overshadow the tactical returns. In the twoyears since the initial transition, LSI Logic has scaled its firmware portfolio from 16 to 91 products, a factor of six increase.

Transition of the Organization

The new software product line development environment and transition to core assets facilitated the development of concurrent productreleases and also helped with the deployment and validation of new featuresacross multiple hardware platforms. However, after the first stage of thetransition, most of the development focus still remained product releasecentric. The second stage in the incremental transition was to start a shift indevelopment focus to core assets by restructuring the development organization.Developers on product release teams incrementally transitioned to a set of teamsdefined by collections of core asset components. The collections of core assets were grouped by affinities and service layers in the firmware architecture.

Each team in the new organizational structure has a core asset manager and a technical team leader. The core asset manager is responsiblefor making sure that their core assets provide the right capabilities at theright time on the product release roadmap. The core asset technical team lead isresponsible for implementing their core assets according to the architecturaland feature requirements on the product roadmap. Core asset managers and teamleads also have an internal focus with their team on core asset stewardship,or maintaining and optimizing the integrity of their core assets. Core assetstewardship includes tasks such as refactoring to optimize commonality andvariability, maintaining an appropriate balance for the requirements tradeoffsacross the entire product line, and searching for emerging abstractions in the variation points[4].

Asset team staffing was determined by existing and planned domain expertise needs of the clustered components and the asset evolutionrequirements for new features. This was a difficult transition culturally asdevelopers, who were accustomed to working on any file in the code base, werenow expected to request changes from the domain experts of other asset teams.

The investment for this increment was the planning to define the asset teams and training to educate team members, team leaders, anddevelopment managers in the roles and responsibilities of the asset team roles.

The return on that investment was better planning and a more controlled and efficient asset evolution. This has resulted in better designs and improved defect resolutions due to improved visibility into the product family and core asset requirements. Technical experts are making better generalized technical decisions for the assets they own, based on a deeper knowledge of those components. Having teams with ownership and deep expertise in a narrow collection of core assets ultimately reduced the rework in both designs and defect resolutions.

Although it has not been formally characterized yet, there is clear evidence that the feature development capacity within the organization has been increased by at least 100%. Rather than scrambling to reallocate overextending developers to meet production schedules, managers now have the luxury of choosing where to allocate this newly gained development capacity.

Transition of the Development Processes

One of the difficulties in transitioning the development team focus to an asset focus was that LSI Logic’s development process was release centric. The third stage in the transition was to align the development process with product family development. The process needed to include a mapping step from feature and release requirements to asset requirements. It needed to increase communication effectiveness of product family requirements at the asset evolution level. The process needed to allow for an asset assembly and feature or product validation function through the development cycle. LSI Logic’s process needs extended beyond compatibility with software product line practices by increasing the ability to respond to changing customer requirements as well as addressing the growing geographic distribution of the development staff.

LSI Logic made a significant investment to address its process needs by assembling a task force comprised of its Software Engineering Process Group (SEPG) along with management representation from other development and quality assurance roles. The task force held a face-to-face process summit with a clearly defined purpose of creating a software development process that addressed the needs of the company.

Amazingly enough, the group reached consensus at the summit on a process structure. Over the next few months, the process definition was refined and was incrementally put into practice. Changing the development process completed the shift of asset team focus to product family development, further reducing the development overhead by satisfying multiple component requirements with coordinated development efforts and leveraging the domain expertise of asset team developers.

Transition of Validation And Quality Assurance

The fourth incremental transition stage, currently in progress, is feature validation and quality assurance. The plan is to improve the product engineering capability of the organization by completing all feature requirements validation iteratively as part of the development work, and shifting the responsibility of the product certification group to the interoperability matrix of LSI Logic’s controller platforms, server systems, operating systems, network adapters, and network switches. This transition is a significant investment, affecting multiple cross functional groups and processes. The expected return from this investment is to move beyond efficiency improvements in development capacity to efficiency improvements in product production and release. For example, there are significant opportunities to reduce the time to complete final product validation, thereby further reducing the time-to-market for new products and new features.

Future Transition Work

The ultimate objective is to do software product line development at an optimal level of effectiveness and efficiency. As the four major transition stages reach completion, we begin to continually evolve and search out areas of waste and overhead, though these tend to be smaller and more focused optimizations rather than major transition efforts.

Conclusions and Lessons Learned

LSI Logic’s transition to software product line practice demonstrated that a large development organization with a large legacy code base can make the transition to software product line practice without a large upfront investment and without disrupting ongoing production schedules. By investing only 4 developer-months of effort upfront and 12 developer-months overall, LSI Logic incrementally transitioned 23 products – each comprising 1 million lines of code – and 135 developers to a sophisticated software product line practice. This is two orders of magnitude less than comparable transitions reported elsewhere.

By staging an incremental transition, small incremental investments very quickly yielded much larger incremental returns. These returns – in effort, time and money – could then be partially or fully reinvested to fuel the next incremental steps in the transition. Furthermore, the efficiency and effectiveness of the development organization constantly improved throughout the transition, demonstrating that development organizations do not need to take a hit in order to reap the benefits of software product line practice.

Stated a little more strongly, the LSI Logic transition refutes the conventional wisdom that it takes an upfront transition effort equivalent to developing 2 or 3 standalone products in order to achieve return on investment. LSI Logic achieved return on investment after an incremental investment of only 4 developer-months of effort. In contrast, the conventional wisdom of 2 or 3 products predicted the return on investment for LSI Logic to be 900 to 1350 developer-months, or 200 to 300 times greater than that actually experienced.

A big part of this success, we believe, is that LSI Logic made an explicit decision to transition to software product line practice utilizing the emerging knowledge, technology and best practices coming out of the software product line field. The data and experiences at LSI Logic are consistent with others that have utilized these emerging approaches [5,6].

Another of the lessons learned from this experience is that the transition in product line practice areas such as infrastructure, core assets, organizational structure, development processes, and QA practices do not have to occur in parallel. LSI Logic incrementally addressed these in sequential order, choosing to solve the problems with the largest potential return on investment first. With the sequential approach, after the transition in one area was complete, the need for transition in the next was more clearly illuminated. For example, after creating the production infrastructure and formal core assets, it was easier to argue why teams should be organized around core assets and limited to modifying their core assets rather than be organized around products and free to modify any source file.

The "people issues" are always the most difficult when imparting change in an organization[7]. A final lesson learned that wefound to be particularly important was that the incremental transition made oneaspect of the people issues easier than expected. By quickly and continuallyshowing benefit, it is much easier to quell the detractors. In fact, we foundthat the biggest skeptics and detractors quickly became the strongest advocates,once they experienced the benefits of software product line practice. By quicklyand incrementally showing return on investment, detractors have less time and opportunity to derail the initiative.

References

1. Paul Clements and Linda Northrop. 2001. Software Product Lines: Practice and Patterns, AddisonWesley, Reading, MA.
2. Davis Weiss and Chi Tau Robert Lai. 1999. Software Product-line Engineering. Addison-Wesley, Reading, MA.
3. Charles Krueger. Software Mass Customization. March 2006, Technical Report, BigLever Software, March 2006.
4. Charles Krueger and Dale Churchett. Eliciting Abstractions from a Software Product Line, in Proceedings of the OOPSLA 2002 PLEES International Workshop on Product Line Engineering. Seattle, Washington. November 2002, pages 43-48
5. Ross Buhrdorf, Dale Churchett, Charles Krueger. Salion’s Experience with a Reactive Software Product Line Approach. 5th International Workshop on Product Family Engineering. Nov 2003. Siena, Italy. Springer-Verlag LNCS 3014, p 315.
6. Charles Krueger. Variation Management for Software Product Lines, Software Product Lines 2^nd International Conference, SPLC 2, San Diego, CA, Aug 2002, Springer-Verlag LNCS 2379, p 257.
7. John Kotter. 2002. The Heart of Change, Harvard Business School Press. Cambridge, MA.

Page 2   Back to the archive list