Performance Testing and Agile

V.M.Guruprasath, vmguruprasath [at] yahoo [dot] com

Introduction

With the advent of DevOps, there is an increasing trend of projects trying to adopt Agile model for their delivery. This article aims at performing a critical analysis of Agile performance testing and tries to answer the question: How Agile performance testing can fit into the current software delivery world?

The paper is divided into two parts

• The Pūrvapakṣa - The general myths prevalent about performance testing in Agile and

• The Uttara-pakṣa - My response to the general myths

The Pūrvapakṣa

In the Pūrvapakṣa section, we will discuss about the top five myths associated with Agile and performance testing.

Myth 1: Agile is cost effective

One of the common denominations that stands as an influencer for project managers taking a decision to move from Waterfall or any other delivery model to Agile is the cost effectiveness card. This cost effectiveness card arises from the notion that Agile squads are a smaller team comprising generally of 12-14 members. This notion is based/ derived from George A Miller's psychological review which is often interpreted to argue that the number of objects an average human can hold in working memory is 7 ± 2. So, a 9-member development team with a 25 percent top-up of testing team (functional and non-functional testing) and addition of operation team member/s to the squad, roughly becomes the squad size and hence the cost of the team becomes almost fixed. In some cases, the total squad size would be only 9 or even less.

Myth 2: Agile helps to reduce delays in time to market (TTM)

The early Waterfall model demanded the completion of the entire modules before it could be released to consumers. The Agile model provides sprint-based developments were the functionalities developed and tested in the earlier sprints could be released to customers and add-on features incrementally released thereby helping consumers to get an early feel of the application. This also helps to increase the feedback loops from the end-user thereby helping to fine tune the application.

Myth 3: SDET (Software Development Engineer in Test)

With the smaller team size, multi-tasking becomes the norm. This drives the need to have squad team members who are exposed to the full stack, like development (frontend/middleware/database), testing (functional and non-functional testing) and environment/ operations management. In a sportive cricketing term, it can be termed as the day of "All-rounders". The team that has more all-rounders are expected to be the most economical and fruitful team.

Myth 4: Co-location, the characteristics of Agile

The other key characteristic of Agile mode of delivery is the co-location of team members, mostly in the same conference room. This helps to reduce the feedback loops and in-turn helps to increase the co-bonding among the team members.

Myth 5: Reduced documentation - Storyboards, Epics and Cards.

Agile delivery is based on daily standups where epics, story cards, issues, risks are discussed daily across the detailed story boards, and actions assigned immediately thereby reducing the need for detailed documentations. The queries are addressed in the stand-ups, be it requirement clarifications, design clarifications, scope clarifications and so on. The reduced documentation is perceived to help team to concentrate on the core delivery, thereby improving the turnaround time for the deliverables.

The Uttara-pakṣa  

This section discusses the above myths and how they fare in the practical implementation

Myth 1: Agile is cost effective

Though it is considered that Agile is cost effective, what one can observe is that the cost overruns are higher in terms of Agile. With Agile, we are trying to achieve a sophisticated squad per module. Though the per module squad team members are within controllable units, the number of squads has to be increased later to cover the scope changes that come from early delivery feedback, missed functionalities, better performance improvements and so on. This would finally take the total number of resources equal or more than the traditional delivery models. A quicker release cycle also demands the availability of applications across different environments simultaneously thereby the need for additional environment resources. In the traditional model, the application build moves from Development environment (Dev) to System Test (ST)/ System Integration Test (SIT) / User Acceptance Test (UAT) to Performance Test environment (PTE) to Pre-production environment and then to Production environment in a sequential manner thereby the same environment team could be re-used. However, with the Agile and DevOps delivery, the movement is faster, thus the need for additional support from environments becomes a norm, especially the Performance Test environment that is production-like and needs production-like configuration to be implemented in shorter period. Though there are Continuous Integration/ Continuous Delivery and Continuous Deployment (CI/CD) and Kubernetes, the environment support is something that cannot be ruled out. One of the major flaws of the Agile model is that it is a developer friendly model. If we look at the developer to quality assurance (QA) team ratio, the proportion of QA team (both functional and non-functional) is very small compared to the overall team size. The team by its design has lesser priority for quality, leading to an expectation that the small testing team should cater to both functional and non-functional testing. In practical, functional and non-functional testing are two different spheres altogether. The result being defect slippages that gets added to the next iteration story boards. The key impacted area is the scope that can be effectively covered by the performance testing team, considering the shorter time duration and lesser resources available.

Myth 2: Agile helps to reduce delays in time to market (TTM)

What is perceived as a reduction in time to market is a decrease in scope of delivery. This decrease in the scope of delivery during the initial sprints marked against the timeline, practically provides less opportunity for the development team to concentrate on the overall application performance in the final integrated world of the interconnected modules and Application Program Interfaces (APIs). The delivery model becomes more focused on functionality being delivered per cycle, thereby the application performance activity gets a back seat and mostly are exception approved on the basis that the application would have lesser user base at the time of initial roll-out. However, when down the lane, the application user base is expanded, it becomes very difficult to fix the application performance problems. The common misunderstanding is that, the performance testing defects are like functional testing defects. Whereas in actual, performance testing and their defect fixes are complicated and require more time and effort to have them fixed. Even if we Shift Left and modularize the performance testing where application performance related defects are early identified, the development team is unable to concentrate or have enough bandwidth on performance defect fixes. This is because the key carrot for driving delivery is a working functionality. This doesn't mean the developers are voluntarily avoiding application performance considerations. However, it is the nature of the Agile model whose focus is more towards functional delivery in shorter time that give less importance to performance testing.

We know, when the time (schedule) is reduced the principle of project management states, either there is increase in cost or reduction in scope. In either case, it is the 'Quality' that gets compromised.

Myth 3: SDET (Software Development Engineer in Test)

Over the past decade the world was moving towards expertise in individual components: frontend, backend, functional testing, performance testing, security testing, operation acceptance testing, and so-on. Now, if there is a sudden reversal in the flow of direction and we need a full stack knowledgeable resource, it is going to be difficult to find them in market. Moreover, considering the testing realm, the functional testing and non-functional testing are different spheres altogether. The functional testing aims at the client side, whereas the performance testing aims at the server side of activities. The thought process required to do functional and non-functional are entirely different. The performance testing world by itself has its own set of deliverables that need to be addressed in a shorter timespan in the Agile world of delivery like, test environment, test data, script development, scenario shakeout, performance test executions/ re-executions, client and server-side analysis, network and infrastructure configuration triage/ support and so on. Hence the team members are going to get lesser time to concentrate on other fields. For a successful Agile implementation, it would be more appropriate to have individual team member who is expert in an area; to have tasks allocated to him/her according to the area of expertise and committed to fulfill their allocated task. Trespassing on different areas would only complicate things.

Myth 4: Co-location, the characteristics of Agile

To increase the feedback loops, reduce time to market and bring in the Squad-Sprint- Spirit, co-location becomes the norm. This would mean all team members should be ideally located at onsite. An all onsite team would have significant increase in the overall project costing structure. Considering the offshore model which is the cost saver for any project, co-location becomes a passé. Though there are concepts like Distributed Agile and video conferencing facilities, it doesn't get the same One-team feeling as with teams that are physically present together. Especially in the performance testing world, the test executions, defects and analysis need constant feedback from various stakeholders like development team, backend team, infrastructure, environment etc..,. Even in the traditional Waterfall model, co-location for performance testing execution and analysis is the norm. In reality, what one could see is that, due to the adoption of agility which is marketed to be cost saver, the performance test team who are last in the delivery cycle tends to get lesser share on the number of resources than required and the work also gets pushed to be delivered from offshore to save the overall cost. The results of which lands in communication gaps, delay in data availability and lesser knowledgeable information about the challenges faced by performance testing team reaching the higher management. This creates a false notion that performance testing is the bottleneck for delivery, however the real cause would be upstream delays and other factors that did not get their due attention.

Myth 5: Reduced documentation - Storyboards, Epics and Cards.

The current delivery model includes multi-vendor and multi-parties. With reduced documentation and shorter cycles, ambiguity on scope and requirements remains. One of the major challenges faced by performance testing team is on the availability of non-functional requirements. With business analysts time being mostly taken over to support development and functional testing streams, performance testing queries get swept under the carpet.

Conclusion

Here is a quick summary of the arguments of the Pūrvapakṣa and Uttara-pakṣa  

	Pūrvapakṣa	Uttara-pakṣa
Myth 1: Agile is cost effective	With small teams and fixed squad size, the cost is effective	Number of squads gets latter increased to cover delivery feedback, missed functionalities, address application performance improvements Insufficient ratio of development to testing team size to cover the complete landscape of End to End (E2E) performance testing Need for additional environment support resources.
Myth 2: Agile helps to reduce delays in time to market	Sprint-based developments were the functionalities developed could be early released to customers	Reduced scope, application functionality driven, thereby leading to less attention to fix performance related defects
Myth 3: SDET (Software Development Engineer in Test)	All-rounders improve team effectiveness	Should be more of collaboration of individual team members with responsible delivery on his/her field.
Myth 4: Co-location, the characteristics of Agile	Co-location facilitates co-bonding	With onshore-offshore model, co-location becomes a passé impacting the One Team effect.
Myth 5: Reduced documentation - Storyboards, Epics and Cards	Reduced documentation is perceived to help team to concentrate on the core delivery	With reduced documentation and shorter cycles there remains ambiguity on scope and requirements. With business analysts time being mostly spent on clearing the application development queries and less time for performance testing team.

With all due considerations, it can be evidently seen that the concept of Agile might work efficiently for a product-based organization where incremental changes are being made to an already established and quality proven product. Trying to implement an Agile model in a multi-party application delivery model is only trying to fit a square peg in a round hole.

The notion seems to be, by early integration of performance testing tools on application development IDEs (Integrated Development Environment) and a performance test execution would uncover all performance testing defects. This notion is flawed because the application performance not only gets affected by the application code, the application code is one of the factors that affects application performance. The other factors include - but are not limited to - bandwidth allocation, environment configurations, CPU and memory capacity, web and application server configurations, database indexing, database configurations, SQL execution plan, storage limitation, and so-on that could only be detected in a full load performance testing conducted in a production-like integrated performance test environment.

Solution for new Product/Application:

For multi-vendor, application development and testing engagements and new product development, an Agile and traditional combined performance testing model co-existing together would be the right approach that should be applied like this:

	Approach	Comments
Early performance testing	In sprint Agile/ DevOps at API level or individual component level.	Performance test team member part of the squad to early detect performance defects in the code Executed in N-1 sprint cycle
End to End performance testing	At the end of all sprints before delivery of full functionality or before full user base throttling load in the system.	Separate team conducting End to End integrated Performance Test on a production like environment

The delivery partners have to understand the need for this two-phased approach for performance testing and have enough budget and resource allocated so that we get short-term benefits and long-term benefits without compromising on 'Quality'.

---

Related Software Testing Resources

Software Testing Magazine

---

Click here to view the complete list of Methods & Tools articles

This article was originally published in October 2018