Click here to view the complete list of archived articles

This article was originally published in the Fall 2006 issue of Methods & Tools

---

Making an Incremental Transition to Software Product Line Practice

Charles W. Krueger, http://www.biglever.com/
William A. Hetrick and Joseph G. Moore; http://www.engenio.com/

Abstract. LSI Logic made the transition to software product line practice in order to keep pace with growing business demand for itsproducts. By using an incremental transition strategy, LSI Logic avoided thetypical upfront adoption barrier – the equivalent development effort of 2 to 3standalone products – which in their case was projected to be 900 to 1350developer-months. LSI Logic discovered that by making an upfront investment ofonly 4 developer-months, they were able to start a chain reaction in which thetactical and strategic incremental returns quickly outpaced the incremental investments, making the transition pay for itself.

Introduction

The Engenio Storage Group division of LSI Logic faced a daunting challenge. The business demand for its RAID storage server products wasrapidly out pacing their ability to create, evolve and maintain the firmware forthose products. Software product line practice, which was just coming into viewat the time, seemed to be the answer. But there was an apparent barrier.Prevalent in the software product line literature at the time was the assumptionthat it requires an upfront investment of 2 to 3 products worth of developmenteffort in order to see return on those investments [1,2]. Based on metrics fromtwo recent deployments, LSI Logic’s projected upfront investment was 900 to1350 developer-months of effort, with a development staff of 100. There wasabsolutely no slack in LSI Logic’s contractually-dictated production schedulesto divert this amount of upfront effort to re-analyze and re-architectcontrollers, re-engineer and componentize assets, redesign productioninfrastructure, re-define processes, and re-organize management and development teams.

Rather than attempt to surmount this formidable upfront adoption barrier, LSI Logic chose instead to make an incremental transition intosoftware product line practice. Not incremental in the sense of prolonging thetime to make the large upfront investments and similarly prolonging the time tosee returns. But rather, incremental in the sense of making a series of smallinvestments, each of which would immediately yield the order-of-magnitudereturns that are characteristic of software product line practice.

LSI Logic discovered that with a relatively small upfront investment of 4 developer-months for its first incremental investment, thecumulative returns began to quickly outpace the cumulative investments in termsof time, effort and money. As a result, the return on investment was almostimmediate and the resulting "profits" in time, effort and money fromthe return on investment could be used to fuel the remaining incremental transition.

There are, of course, many different facets within a software development organization to consider when making an incremental transition tosoftware product lines. For example, Clements and Northrop characterize 29 key practiceareas that may be impacted by a software product line approach [1]. Any orall of these might be considered in an incremental transition. LSI Logic choseto incrementally address, in sequence, those facets that represented the primaryinefficiencies and bottlenecks in their development organization. By eliminatingthe inefficiencies and bottlenecks in the most critical facet, the next mostcritical product line problem in the sequence was exposed and targeted for the next increment.

To date, the sequence of incremental steps in the LSI Logic transition to software product line practice can be characterized by the following four primary stages:

• Transition the infrastructure from conventional configuration management and builds to first class software product line variation management, configuration management and automated production
• Transition from team organization by products to team organization by core assets
• Transition from development processes defined by product releases to development process defined by feature releases
• Transition from validation and quality assurance for individual products to validation and quality assurance for all of the software product line assets

Initial State of LSI Logic’s Development Practice

The Engenio Storage Group division in LSI Logic is in the business of providing feature-rich, high-performance, storage servers to major OEM vendors such as IBM, SGI, Cray, StorageTek and Teradata. Each OEM customer wants to take advantage of LSI Logic’s core competence in storage technology, but each one also wants unique and differentiated solutions. Thus, the need for efficient product line engineering is central to its business model. Also, the ability to extend its product line to attract and retain a growing customer base is key for establishing investor value.

The Controller Firmware team within LSI Logic’s Engenio Storage Group is the focal point for the evolution to a software product line. In recent years, this group has grown include to 210 firmware developers, working at four distinct geographic sites. The group currently provides firmware for 91 products, with about one million lines of embedded software going into each product. Approximately 80% of the code is common among all products.

In the late 1990s, the Engenio Storage Group Controller Firmware Development staff lived in far simpler world. The firmware was constructed as a single software build, applicable to all deployments. Any required product variability was resolved at runtime, using downloadable configuration data stored in non-volatile memory. Evolution of the firmware over time was managed using the concept of formal releases. These releases were managed in a disciplined manner, with a formal waterfall process guiding the way. Each release would transition through the stages of analysis, design, implementation and test. Once a release was complete and products were shipping to customers, work on the next release could begin. In retrospect, it is apparent that it was the sequential nature of the release cycle that kept the firmware development world simple and sane.

Market pressures eventually emerged that greatly disrupted the simple, sequential release-based perspective that had guided the firmwareteam through the nineties. The 21^st century brought increased successin the marketplace, but this success was accompanied by a furor of developmentdemands that could not be accommodated by sequential development cycles. Thetranquil world of sequential releases was replaced with the nonstop chaos of a half dozen intertwined and overlapping release cycles.

In the midst of the market-driven turbulence, the need for change within the engineering group at Engenio was not readily apparent.Instead, the evolution from a single product to a product line stealthily creptinto LSI Logic’s product development world. The initial portent of change wasthe introduction of a new, low-end, hardware platform. While the feature contentof the low-end product was to be the same and the bulk of the firmware code wasto be shared between the new low-end and existing mid-range products, there weresubstantial differences between the two hardware platforms. The introduction ofa second, substantially independent, development environment to support the newplatform was the initial solution for managing this variability. Although thismove solved the immediate problem of supporting the variation among the old and new products, it proved to be a less than ideal way to manage the commonality.

During the evolution from one to multiple products, the primary tool utilized by LSI Logic’s firmware group was a proven,well-integrated, configuration and problem tracking system. This system did anexcellent job in the role for which it was intended, but it created the tendencyfor product variability problems to be treated as configuration managementproblems. For example, variability to satisfy differing platform requirementswas managed by branched versions of the same source files. Similar branching wasintroduced to support the concurrent development of diverse features. The endresult of the evolution from a single product to a diversity of hardwareplatforms and firmware features was success in the marketplace, but chaos withinthe code base. Eventually, approximately 30% of the 3300 files in the LSI Logicsource code base had anywhere from 2 to 16 active branches under development.The resultant branching, merging, duplication and divergence became the dominanteffort in the firmware development team, leading to the realization that a change in development practice was imperative.

Page 2   Back to the archive list