Click here to view the complete list of archived articles

This article was originally published in the Fall 2007 issue of Methods & Tools

---

Mass Customizing Solutions

Jack Greenfield, Microsoft, http://blogs.msdn.com/jackgr/

Software Factories are a new paradigm, described in an award winning and best selling book [1], for bridging the gap between broad marketplatform technologies and custom solutions for business process automation inthe enterprise. Software Factories promise to bring efficient mass customizationto the enterprise application market by promoting the formation of supply chains used to provision software solutions.

The Customer Dilemma

Historically, there has been a large gap between the general purpose, horizontally oriented, broad market technologies supplied by softwarevendors and the specialized, vertically oriented, custom solutions developed toautomate proprietary business processes. The gap is created by a mismatchbetween the diversity of customer requirements and economies of scale in thesoftware industry. On the one hand, no two customers are identical. Each hasunique requirements that can only be completely satisfied by a custom solution.On the other hand, software vendors must target large homogeneous markets inorder to keep development costs within reason, essentially treating theircustomers as if they are identical. The gap is both a serious problem and a significant growth opportunity for the software industry.

Current Solution Strategies

Enterprises have tried to close the gap using a combination of third party products, especially packaged applications, and customdevelopment, often performed by external service providers. Neither of these solution strategies fully satisfies most customers.

Packaged Applications

Packaged application vendors like SAP, Oracle and Microsoft have successfully delivered custom solutions from predefined assets, includingrequirements, logical and technical architectures, implementation components,test suites, deployment topologies, operational facilities, maintenance plansand migration pathways, on a larger scale than had previously been seen in product based solutions in the enterprise market.

Unfortunately, packaged applications are not a panacea. While they can generally be customized to a great extent to address the needs ofindividual customers, the levels of customization they admit do not let thetypical customer fully follow their preferred proprietary business practices. Inorder to use a packaged application, the customer must generally adopt businesspractices mandated by the software. This forced conformity makes it hard for thecustomer to achieve competitive advantage over other companies running the same applications.

Another problem with packaged applications is that they are typically monolithic, meaning that they attempt to span the entire gap in asingle product based solution using closed architectures that are fullyaccessible only to the application vendor. These characteristics of the softwaremake it hard to integrate with products from other vendors, forcing the customerto depend on the packaged application provider to meet most or all of itsbusiness process automation needs, or to invest in clumsy integration strategiesthat often prove to be hard to use, slow, resource intensive, insecure, and hardto maintain as the packaged applications, the platform technologies, and the customer’s business needs change over time.

Enterprise customers are looking for a more flexible approach that better satisfies their requirements, enabling them to achieve competitiveadvantage through differentiation. They are also looking for more open and moremodular architectures that will give them more options in the marketplace, andthat will preserve their business information and their investments in businessprocess automation across changes in underlying products and technologies.

Custom Solution Development

With custom solution development, software is built to order, usually by third party service providers working closely with the customer’sinternal information technology organization. When carried out successfully,custom solution development produces software that fully satisfies the customer’sunique and proprietary requirements, enabling the customer to differentiate itself from competitors in the hope of achieving competitive advantage.

Unfortunately, custom solution development is often expensive, time consuming and error prone, and rarely delivers all of thefeatures originally planned. The resulting software is generally of lowerquality than commercially developed software products, in terms of usability,reliability, performance and security, and is often hard to operate, and hard to support and evolve as business requirements and technologies change.

Enterprise customers have explored many strategies for solving these problems, such as using widely marketed but marginally effectivemethodologies based on low fidelity general purpose modeling languages, and morerecently using agile development methods and off shoring. While these strategiescan yield some marginal gains, they cannot offer systemic improvements.

Most enterprises have tried multiple methodologies, and many are still looking for one that delivers consistently. Widely marketedmethodologies are generally either too thin and too weak to help the customermanage the challenges of developing software in the real world, or too heavy andtoo formal to accommodate the unpredictable nature of real world projects. Mostgive generic abstract advice rather than concrete guidance for a specific typeof deliverable. You get the same advice whether you’re building eBay orembedded software for an anti-lock braking system, although these domains obviously have very different requirements.

Most of the models used by these methodologies serve only as documentation, and then usually only in the early stages of projects, eitherbecause the modeling tools don’t generate production quality code, or because.they don’t stay synchronized with the code, and therefore cannot capture keydesign decisions. They are often thrown away when the code starts taking shape.

A Different Approach

Software Factories represent a different approach to bridging the gap between platform technologies and custom solutions. They combine thebest characteristics of the packaged application and custom solution developmentstrategies without the problems described above, enabling rapid, predictable andinexpensive delivery of highly customized product based solutions that meet theunique requirements of individual customers. They achieve these results bychanging the way individual suppliers build solution components, and bypromoting the formation of supply chains that distribute cost and risk acrossnetworks of suppliers who collaborate to produce a wide range of customsolutions tailored for a large number of individual consumers from a core set of products.

A Different Kind Of Methodology

Like other methodologies, Software Factories help developers follow known good practices and apply known good patterns. Unlike other methodologies, however, they recognize that different domains require different solution strategies. Consequently, instead of a one-size-fits-all approach, the methodology calls for the development of many factories, each providing guidance for a specific domain. Also, unlike other methodologies, which offer guidance from ivory tower experts, Software Factories assume that the people best qualified to provide the guidance are the practitioners who work in the target domain. For this reason, there are two parts to the methodology, one part for factory authors and another part for factory users.

In order to support the definition and differentiation of individual factories, the methodology starts by identifying and classifying architectural styles for frequently encountered families of solutions, such as web portals, smart clients and connected systems. It then captures information about each solution family, such as the artifacts needed to build its members, the technologies used and their placement within the architecture, required technology configurations, and best practices across the life cycle, including key decisions points and trade-offs. Factories containing only this type of information are called architectural factories. More specialized factories can also be created targeting common functions, such self service portals and ecommerce applications. These are called functional factories. Still more specialized industry factories can be created targeting specific industries, segments or solution areas, such as factories for banking self service portals, or online music stores.

Like other methodologies, Software factories describe processes that span part or all of the solution life cycle. Unlike other methodologies, however, Software Factories supply reusable assets to help users enact the processes. Some of these assets are documents, such as patterns, guidelines and check lists, some are tools, such as designers, wizards, and scripts, some are assets used by tools, such as templates, models and configuration files, and some are executables, such as libraries, frameworks and sample code.

The methodology is based on a familiar and time tested pattern for improving productivity and predictability. Known good patterns and practices in the target domain are harvested, refined and encoded into domain specific tools and runtimes. The tools are then used to rapidly select, adapt, configure, complete, assemble and generate solutions that use the runtimes. Familiar examples include graphical user interface builders and frameworks, web site development kits and platforms, and database management tools and servers.

Until recently, applying this pattern has been technically and economically viable only for broad market, horizontally oriented domains served by vendors offering general purpose tools and runtimes. Software factories change the market dynamics by making the pattern viable for much smaller, more vertically oriented domains, enabling vendors closer to the customer to supply special purpose tools and runtimes to much narrower market segments.

What Is A Software Factory?

A software factory is a specialized development and runtime environment that supplies an integrated set of special purpose assets encapsulating proven patterns and practices, including tools, processes and content. Examples of content assets include partial or prototypical life cycle artifacts, such as requirements, logical and technical architectures, test suites, deployment topologies, operational facilities, maintenance plans and migration pathways, and implementation artifacts, such as guidelines, patterns, code samples, templates, libraries, frameworks, models, and configuration files. These assets are used by solution builders to accelerate recurring tasks, such as process partitioning, contract definition, feature configuration, deployment validation, requirements tracing, and impact analysis, for a specific family of products or solutions.

The assets are delivered in a structured and installable package called a software factory template. The assets are usually customizable, and the organization of the template is designed to make it easy to select, adapt, configure, complete, assemble and parameterize the assets, enabling the factory to produce a wide range of solutions with varying features and operational qualities.

At the core of a factory is a description of the factory and its contents called a schema. Like a database schema, a software factory schema is a model that can be interpreted by people and tools. At the core of the model is an architecture framework for the target product family. (Architecture frameworks are described below.) It describes the assets that comprise the factory, how they are packaged, and how they are installed, customized and applied. The structure of the factory schema is illustrated by the following diagram.

How Do Software Factories Work?

Software Factories integrate four technologies to create a whole that is greater than the sum of its parts: software product lines, model driven development, guidance automation and architecture frameworks.

Software Product Lines

A software product line is a set of systems sharing a set of managed features that satisfy the specific needs of a particular market segment, developed from a common set of core assets in a prescribed way [2]. In a product line, new solutions are developed by assembling partial solutions and/or by configuring generic ones. Only the unique features of each solution must be specified because the common ones can be safely assumed.

In a software product line, some variations in requirements map predictably to variations in artifacts and processes. The decision to support personalization in a web service portal, for example, maps predictably to the addition of tables and columns in the database schema, to changes in web pages and page flow, to changes in business logic, to service contracts offered by components, and to new tasks or changes in tasks in development, testing, deployment, operations, maintenance and migration.

Software product line engineering practices are reported to reduce custom development by 40% to 80% for the typical solution. Lower gains are too small to economically support product line development and operation, while higher gains are nearly impossible to realize consistently in practice. The portions of the solution not covered by the product line are custom developed, often using a Request for Price Quotation (RPQ).

The asset base used in a product line is usually reverse engineered from existing systems or subsystems that are partially or wholly prototypical of the family of systems targeted by the product line. The assets are then adapted as necessary when new family members are developed, to address variations not yet anticipated by the product line. The revised assets are then customized and applied to produce the family member.

Guidance Automation

The goal of guidance automation is to help practitioners know what needs to be done at any given point in the life cycle of a given type of solution and to provide reusable assets, such as help topics, code samples, templates, wizards, workflows, libraries, frameworks and tools, that help them do what needs to be done. It is implemented using installable packages containing organized sets of integrated assets supporting commonly occurring use cases. The tool and content assets are contextualized by attaching them to relevant tasks in the processes and to relevant parts of the solution architecture. The tasks can then be scoped using pre and post conditions, so that the project workflow may vary subject to the conditions. This allows tasks to be activated or to come into focus only when relevant to the state of the solution, and to be deactivated or to go out of focus only when their exit criteria have been satisfied.

Model Driven Development

The goal of model driven development (MDD) is to automate life cycle tasks using metadata collected by models. In MDD, models are not used primarily as documentation, but as source artifacts that can be compiled to produce implementation components, or used by other tools to support trace, navigation, validation, analysis, refactoring, optimization and other operations. While MDD can be practiced by extending general purpose modeling languages like UML, it is most effective when practiced with highly focused Domain Specific Languages (DSLs) designed to solve specific problems, related to specific tasks, on specific platforms, and with custom tools developed to support them [3]. There are many familiar examples of DSLs, such as SQL, BEPL, HTML and WSDL. DSLs make solutions easier to understand and maintain, and generally improve agility by facilitating rapid iteration.

Architecture Frameworks

As noted by Peter Deutsch, interface design and functional factoring constitute the key intellectual content of software, and are far more difficult to create or recreate than code. Architecture frameworks capture that intellectual content using viewpoints that identify, separate and interrelate well-defined sets of stakeholder concerns. Well known examples of architecture frameworks include the Zachman grid [4] and the TOGAF grid [5]. Examples of stakeholders include business analysts, project managers, developers, and database administrators. Examples of viewpoints include business rules, business events, business information, business processes, user interface workflow, and database design. Examples of stakeholder concerns include how business rules are enforced by a given business process, or how a given physical database design supports a logical database design.

The software factory schema is an architecture framework. Unlike a grid, however, it is non-rectangular, allowing it to support nesting viewpoints, and to describe relationships among non-adjacent viewpoints. Also unlike a grid, it is specific to the solution family targeted by the factory, not generic. Finally, unlike a grid, the schema is dynamic, so that it can change to accommodate variations among the members of the solution family. For example, new viewpoints are added to the schema, and existing viewpoints are modified, when the Sales and Campaigns feature is added to an ecommerce application.

In addition to a set of stakeholder concerns, a viewpoint in a software factory schema defines a set of related artifacts relevant to those concerns, the activities that act upon the artifacts, and the assets used to perform the activities. The schema organizes the factory, and uses the relationships among viewpoints to integrate the activities, artifacts and assets across the software architecture and life cycle. For example, the relationships defined by the schema support operations like tracing features from requirements to implementation, navigating among solution artifacts, validating a deployment configuration across a set of connected systems, providing intellisense in user interface development from information captured during business information modeling, or generating a set of data access classes from a logical database design.

Using Software Factories

With a factory, projects start from well-defined baselines, instead of from scratch. Practitioners know what tasks need to be performed, what assets to use to perform each task, and how and why to use each asset. Guidance is available at their fingertips within the development environment. Junior developers follow guidance with minimal hand holding. Senior developers refine the guidance based on experience. The guidance is packaged and versioned, making it easier to gather feedback from users to improve its quality.

Development proceeds both top down, from viewpoints focusing on business goals and requirements and bottom up, from viewpoints focusing on implementation, testing, and deployment, toward viewpoints focusing on analysis and design. Tasks come into focus as the solution evolves, allowing the workflow to be driven by changes in circumstances, requirements and technologies, subject to correctness constraints, instead of following a prescriptive plan that may not be able to accommodate the realities of real world projects. Relationships between viewpoints are used to support the flow of development, through operations like trace, navigation, validation, analysis, refactoring, optimization and synchronization. In many cases, synchronization will involve generating some or all of the solution structure and contents targeted by viewpoints being synchronized. In other cases, it will involve editing the artifacts by hand to bring them fully into alignment.

A key feature of the methodology is that the products of two or more factories can be composed. Instead of a single factory that helps them build the ultimate deliverable in its entirety, users can work with multiple factories, each helping them build a portion of the ultimate deliverable.

The results are generally significant reduction in cost and time to market, significant improvements in quality attributes, and greater consistency from solution to solution. Risk is also reduced, as metric based estimation models can be constructed for the factory schema using data collected during solution development, making it easier accurately forecast budget, schedule, and resource requirements, and to analyze the impact of new requirements. The time and cost of training new developers is also reduced.

When it is time to maintain or evolve a solution developed using a factory, the metadata captured during its development is used to analyze the impact of changes in requirements and technologies, and orchestrate the necessary changes. It is also used when underlying technology evolves to identify customizations made during solution delivery that must be migrated to the new technology, and often to fully or partially automate the migration of affected artifacts.

Mass Customization

Like other methodologies, Software Factories seek to improve productivity and predictability within a single organization. Unlike other methodologies, however, they also seek to support mass customization by promoting the formation of supply chains.

Mass customization is the production of custom solutions on a large scale [6]. It is the result of bridging the gap between the diversity of customer requirements and economies of scale, and has been achieved in other industries, such as computers and automobiles.

For example, a well known automobile manufacturer builds about 800 cars and 1200 engines per day at one of their factories. Each unit is built entirely to order. Due to the large number of options and the large number of valid ways in which those options can be combined, there are over 1 trillion possible variants of every product. It takes about 4 years, on average, to produce 2 units with exactly the same features.

Using Supply Chains

Packaged applications and custom development are the prevailing strategies for mass customization in the software industry. As we have seen, both have succeeded to some extent, but both have significant shortcomings.

Supply chains provide a more efficient and more economically viable approach to mass customization. Instead of relying on a single product based solution from a single vendor, suppliers at every level of the supply chain can select the best offerings from other suppliers to use in their product offerings. Customers can select from among the offerings at the surface of the supply chain those that best satisfy their unique and proprietary requirements. Of course, these products can then be tailored further by the suppliers or by service providers to create solutions customized to the needs of the individual customer.

As the products, the platform technologies, and the customer’s business needs change over time, the supply chain creates competitive pressures among its members to either adapt their offerings or be displaced by other suppliers who will offer what the supply chain demands. With solution development distributed over a large number of potential suppliers, the supply chain is much more responsive than even the best single vendor could ever be to changing technologies and market conditions. Cost, risk and time to market are also reduced at every level of the supply chain by the distribution of solution development and the competitive pressures it creates, resulting in significant savings to the customer.

The Alignment Problem

Currently, supply chains do not form rapidly in the software industry, and those that do form are generally much shallower than supply chains seen in other industries. Analysis suggests that the root cause is misalignment among suppliers. Different suppliers generally take different approaches to development. They have different ways of defining requirements, different ways of organizing the software, different ways of using platform technologies, different ways of deploying system components, different ways of testing those components, and different ways of customizing system features. These differences, in turn, make it hard for others to integrate products from multiple suppliers to create solutions. As system integrators know, it is hard to design well formed solutions using product from multiple suppliers, hard to verify those solutions, hard to determine impact of changes in requirements or technologies, hard to determine what customizations were made for a given customer, hard to migrate customizations when the underlying products change, and hard to coordinate among suppliers to deliver custom features.

Supply Chain Formation

Software factories help to solve the alignment problem by making it easy to expose critical information about products in computationally convenient ways. This information helps suppliers to detect and address misalignment, and to globally optimize their supply chains.

Applying Factories

Replacing ad-hoc development with factory based development is the first step on the road to supply chains. Two important characteristics of factories, partitioning and assembly, make it technically feasible to distribute them across networks of cooperating organizations to facilitate the formation and operation of supply chains.

Factory Partitioning

A factory uses two interacting development processes. The first is the traditional development process by which solution developers build solutions for customers who use them to automate business processes, and who provide feedback to the solution developers, such as defect reports and feature requests. The second is a separate and more specialized development process by which factory developers build assets for solution developers who use them to build solutions, and who provide feedback to the factory developers, such as defect reports and feature requests. These processes are illustrated in the following diagram.

Factories can be partitioned to form supply chains along these processes. They can be partitioned horizontally by placing the factory development process and the solution development process in separate organizations. For example, an Independent Software Vendor (ISV) might provide a factory used by Systems Integrators (SIs) to build solutions for customers. Alternatively, an enterprise might provide a factory used by an offshore service provider to build solutions for the enterprise. Factories can also be partitioned vertically by composition or specialization across the boundaries between organizations. For example, a banking self service portal factory might be developed by an SI by specializing a self service portal factory offered by an ISV, which might in turn be developed by specializing a portal factory offered by a platform vendor. The two types of partitions are illustrated in the following diagram.

As mentioned earlier, the factory schema provides metadata used to organize tools, processes and content by viewpoint, and to integrate them across relationships among viewpoints using operations like trace, navigation, validation, analysis, refactoring, optimization and synchronization. When a solution factory is partitioned across multiple suppliers, its schema is distributed across the supply chain, and the operations over relationships among the viewpoints support interactions among the suppliers, such as requirements communication, solution configuration, logical and technical architecture alignment, component assembly, adaptation and orchestration, workflow integration, test suite integration, and distributed deployment, management, maintenance and migration.

Factory Assembly

Factories can be composed and specialized. Smaller factories can be composed to form larger factories, and more general factories can be specialized to form more specialized factories. For example, a factory for layered applications might be formed by composing factories for user interface, business logic and data access layers. A factory for retail banking self-service portals might be formed by specializing a factory for banking self-service portals, which might be formed by specializing a factory for self-service portals, which might be formed by specializing a factory for web portals.

Factory composition and specialization rely on the factory schema. Factory composition involves combining the viewpoints of the constituent factories, and factory specialization involves adding or removing viewpoints, and modifying viewpoints of the base factory by changing the artifacts produced, the activities that produce them, and the assets used to support and automate the activities.

Publishing Metadata

The second step on the road to supply chains is publishing information about key processes and artifacts, so that it can be consumed byother suppliers. This metadata is often readily available in the models used byfactories. Two kinds of information are particularly important in supply chain formation:

• Information about requirements, such as the information captured by feature models [8] can be used to reason about compatibility, variability, and dependencies among requirements.
• Information about architecture, such as the information captured by service contracts, service level agreements and deployment manifests, can be used to reason about how products from other suppliers should be deployed, what platform services they require, and how they can be adapted, assembled and configured.

Both types of information help suppliers identify products that they might incorporate from other suppliers, and what changes they mightrequest from them.

With any type of metadata, the key is making it readily available to other suppliers in computationally convenient ways. A relativelyrecent development that facilitates sharing information across organizationalboundaries is web service technology. Using web services, suppliers can exposedescriptions of externally facing artifacts and processes in a secure, reliableand platform independent manner. A promising application of web servicetechnology from a supply chain perspective is composing services hosted by othersuppliers to form new services, instead of composing components they supply toform new components. This kind of composition is commonly called a mash up, andenables some compelling scenarios, such as assembly by orchestration and automatic adaptation to address architectural mismatches.

Alignment and Optimization

Publishing metadata is a necessary condition for supply chain formation, but not a sufficient condition. The final step on the road to supplychains involves discovering and correcting misalignments among suppliers usingthe published metadata, and optimizing the resulting network. Aligned suppliershave agreed to use compatible technologies, and have identified effective waysto integrate their products. In the real world, these alliances are constantlyrenegotiated as technologies and business conditions change, and suppliers enter and leave the marketplace.

When the suppliers are aligned, we can turn our attention to optimizing the resulting supply chain. The goal of optimization is to eliminatebottlenecks that reduce the speed at which changes can propagate through thenetwork. Optimization generally involves replacing local responses to changes byindividual suppliers with collective responses by reasoning over publisheddependency information. Without optimization, suppliers individually recognizeand respond to changes in products from other suppliers as they encounter them.Each change therefore propagates incrementally from its source to its ultimateconsumers, one supplier at a time. With optimization, information about changesthat might affect large numbers of downstream suppliers can be published as thechanges arise, allowing downstream suppliers to anticipate and respond to thechanges before they arrive. This kind of optimization is common in supply chains in other industries, such as computers and automobiles.

Conclusion

The following diagram illustrates a supply chain for Product Lifecycle Management (PLM) applications assembled from factories. This kind ofindustry architecture and the efficient mass customization it enables is the ultimate goal of the Software Factories methodology.

While Software Factories provide the technical and architectural foundation required to enable the formation of supply chains inthe software industry, they are not enough to change the market dynamics bythemselves. Business models that promote and facilitate supply chain formationand that enable efficient supply chain operation are also required. A business model discussion is beyond the scope of this paper.

References

1. Jack Greenfield, Keith Short, Steve Cook, Stuart Kent. Software Factories: Assembling Applications with Patterns, Models, Frameworks, and Tools. Wiley. ISBN: 0-471-20284-3
2. Paul Clements and Linda Northrop. Software Product Lines: Practices and Patterns. Addison-Wesley Professional. ISBN-10: 0201703327
3. Steve Cook, Gareth Jones, Stuart Kent, Alan Cameron Wills. Domain-Specific Development with Visual Studio DSL Tools. Addison-Wesley Professional. ISBN-10: 0321398203
4. John Zachman . The Framework for Enterprise Architecture: Background,Description and Utility. Zachman Institute for Framework Advancement (ZIFA). Document ID: 810-231-0531
5. www.opengroup.org/togaf
6. Joseph Pine. Mass Customization: The New Frontier in Business Competition. Harvard Business School Press. ISBN: 0-87584-946-6
7. David Anderson. Build-to-Order & Mass Customization. CIM Press. ISBN: 1-878072-30-7
8. Krzysztof Czarnecki and Ulrich Eisenecker. Generative Programming: Methods, Tools, and Applications. Addison-Wesley Professional. ISBN-10: 0201309777

Back to the archive list