The Traditional Test Pyramid, Pitfalls and Anti-Patterns

Antoine Craske, La Redoute, https://laredoute.io/

A successful Test Strategy relies on well selected priorities, methods and tools aligned on the Business and Product imperatives. The Test Pyramids mental models are a common way to define and structure our test strategy, organizing the various possible combination of test techniques, tools, and automation effort. Do they apply to all models? How can we challenge them regarding their inherent assumptions? Do solutions and ways of working available nowadays could help revisit them? This is what we share in this article focused on Test Automation strategies.

Why automating tests in the first place?

To contextualize our reflection, we must step back regarding our objectives. The Test activity aims to support Business to reach a Flow state with fast, reliable and efficient software delivery. Thus, the test activity must also be fast, reliable and efficient.

This is where automated tests done right help to validate existing and new features in short release cycles. In the meanwhile, various market surveys, like the World Quality Report, do share the main pains linked with test automation, tooling, integration and ability to adapt to a constantly changing product.

Summarizing, test automation can bring a significant competitive advantage to the business in terms of software delivery, but still hard to achieve today.

Figure 1: The Traditional Testing Pyramid, and how it usually ends up. Source: https://dev.to/

Without technical and organisational constraints, the Traditional Test Pyramid focus the testing effort on the main business and customer facing value, from successful customer experience to a flexible architecture.

The definition and requirements of a Test Strategy

A properly defined Test Strategy is a key element of successful execution of the activity in its given context. It is usually bypass or partially done in organisations, resulting in missed opportunities for improvements in the delivery process, and thus business performance.

The requirements we define are the following ones:

• Qualitative: The main value resides in the capacity to validate a successful user experience, a flexible and testable software architecture.

• Accessible: Understandable, shared and accessible for all stakeholders, from business to operations.

• Reliable: Systematically executed during the delivery process and in production, with a flaky test ratio under 0.25%.

• Scalable: The organisation and execution can be scaled across the organisation, test management and execution. The target being to execute all tests at maximum parallelisation capacity with functional constraints.

• Maintainable: Flexible to adapt to the change of our product with minimal rework. Changing one behaviour must translate to one change on the test configuration, not more.

• Optimized: Optimize the investment in time on automation, execution to find most bugs at the lower cost. The number of defects found and not found by the process is a metric that can support this objective.

The Agile Testing Pyramid, in contradiction with our Strategy Objectives?

The Agile Test Automation Pyramid is referenced for a proper Testing Strategy implementation, in traditional and agile environments, challenging the limitations of the available alternatives.

The core motivations are linked to Agile objectives: reducing cycle-time, meeting frequent deadlines, rapid feedback, improved and co-located collaboration. Aligned on those imperatives, the pyramid recommends focusing mostly on unit tests done by developers and testers, to ensure rapid feedback in the cycle in order to secure the deadlines.

It also recommends limiting end-to-end and functional testing, being costly to develop and maintain, slow to execute, complex and unstable, as manual or unstable.

Figure 2: Mike Cohn, The Agile Testing Pyramid

Looking at the pyramid versus our objectives or validating business behaviour as early as possible, we can question this model mainly focused on opportunities provided by Agile models, and not on addressing business testing priorities.

We may prefer Unit and Integration tests for bad reasons

We understand all the points referred in the Agile Pyramid but focusing mainly on unit tests could be done for bad reasons. We will explore three different types: Organisational, Process and Tooling.

A first reason can be related to lack of maturity or budget for test activity and test engineer allocation, resulting in developers to validate their own development, and increasing the code base technical debt with lot of unit tests.

Another reason usually mention that tests are done too late in the process, thus having low value and slowing down the whole delivery. In fact, most of time the inclusion of test from the design - also known as shift-left pattern - is not done, explaining while tests are started too late in the process.

The last one, related to tooling, miss an automated testing solution fulfilling the automation requirements: flexible test configuration, fast test execution and feedback loop, integrated into CI/CD and system, accessible to the whole team including business.

Functional and end-to-end tests usually fail due to lack of Skills, Automation & Tooling

We can reverse the point of view of the previous point, from a functional test automation perspective. The difficulties raised in the ecosystem reside in the capacity to deliver a successful test automation strategy.

In a lot of cases, the tests are partially automated, resulting in costly tests, harder to maintain over time. Other experiences usually lack trained testers and software testing competencies, resulting in fragile tests, with manual data setting, lacking reuse, technical and design quality. This type of experience also results in a poor functional test automation experience.

Organisations also implement models that were not proven to scale or focused on the bad priority, thus increasing the tests complexity, execution time, maintenance. It usually resides in lack of design and analysis, and method selection like equivalent partitioning, boundary value analysis, decision-table testing, state transition, use-cases, … appropriated to each context. We get back to lack of proper investment and skill in the test activity.

The Traditional Test Pyramid assumptions can be challenged

The analysis of the different points of view and pyramid can make us challenge and assume the value of each type of automated testing layer.

Technique	Pros	Cons
End-to-End	Validated user-experience and process in end-to-end Accessible and understandable transversally, business to ops	To change and evolve as fast as software requirements Depending on test design, test data, platform, environments
Functional	Focused on business behaviour and use-cases Done by skilled testers, increase ratio of defects detection Accessible to business, developers and testers	To change and evolve as fast as software requirements Depending on test design, test data, platform, environments
Integration	Find integration problem before addressing functional tests Approximate dev and ops on integration	Dependent of services and environments Coverage and defects limited to integration Access usually limited to dev and ops
Unit	Can test architecture design, quality and flexibility Usually fast and reliable in CI, using mocking techniques	Lower defects detected being done by developers who also coded Without strategy, become too many, slow, costly to maintain Usually accessible mainly to developers

This view raises to the following open questions:

• How to implement end-to-end and functional tests meeting our test requirements?

• Do we need integration test with good end-to-end and functional tests?

• How to balance unit test effort and value? Focus them on architectural validation mainly?

Designing the Reversed Test Pyramid for a Test Automation Strategy

Our approach resides in applying the Traditional Pyramid, with an aggressive automation focus on end-to-end and functional testing, while limiting integration tests as possible, and keeping unit tests for architecture validation.

You can find practical information about our experience using the pyramid in this article using our open source testing solution, Cerberus.

Figure 3: La Redoute Testing Pyramid, with its main area of focus

Each test has its set of objectives, limiting duplicated effort on the test implementation and maintenance. An inherent assumption of this model is to invest in the testing area, meaning budget, skilled profiles, testing and evolution plan, valorising test within your organisation and careers. It can be translated in the metrics you follow in the cross-functional tests, adding the test strategy ones, number of defects, technical debt, adding testers early in projects as developers.

Tests need to be considered a part of the digital products you are delivering, so add their build and execution as part of the CI/CD chain, design them to be executed in production on a regular basis.

We also highly recommend shared dashboards for the various tests to be transparent and aligned with the stakeholders. Investing in an end-to-end and functional solution meeting the mentioned requirements, accessible to each member of the team, will also make a huge difference over-time.

Testing as a building block in the accelerated digital and more complex world

The evolving technological ecosystem is growing in complexity: distributed event-driven microservices, serverless functions, APIs, screen-less interactions, low-code and developer citizens. In this environment, we highly recommend investing in your Test strategy and implementation!

References

The Agile Testing Pyramid, The Agile Coach Journal, https://www.agilecoachjournal.com/2014-01-28/the-agile-testing-pyramid

The Forgotten Layer of the Test Automation Pyramid, Mike Cohn, https://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid 

The Practical Test Pyramid, Martin Flower, https://martinfowler.com/articles/practical-test-pyramid.html

This article was originally published on https://laredoute.io/blog/the-traditional-test-pyramid-pitfalls-and-anti-patterns/ is reproduced with permission from Uruit.

---

Related Software Testing and Quality Assurance Resources

Automated Testing Strategy for a Microservices Architecture

Metrics for Implementing Automated Software Testing

Software Testing Magazine

---

Click here to view the complete list of Methods & Tools articles

This article was originally published in February 2021