Eradicating Load Testing Errors - Part 1: Correlation Errors

Vadim Kleyzit, Max Kleyzit, StresStimulus, https://www.stresstimulus.com/

Beware of the nasty problem

You just installed a new load testing tool and are very excited to start testing the performance of your web application. You selected this tool because its training video showed how easy it is to use. Just navigate through a test scenario while the tool records it. Then the tool will emulate real world impact on the server infrastructure by replaying this scenario with as many virtual users (VUs) as you need and monitor how well your application performs under load. While this may seem pretty simple, testing may not go as you planned. The website under test can refuse to cooperate with the VUs. Errors may be thrown or functionality may seem to be broken. This means you are likely dealing with a correlation issue.

What this article is about: this two-part article presents a deep dive into dealing with correlation errors often found when load testing Web and mobile applications. Part 1 reviews performance testing errors and explains how to recognize correlation issues. Published in the next issue, part 2 will introduce a comprehensive method to fix these problems.

Who should read this: Developers, software quality assurance specialists and performance engineers who want to master performance testing and avoid typical mistakes in handling testing issues.

What is correlation? It is a common mechanism in web applications consisting of two aspects as shown in figure 1.

1. While communicating with multiple clients, the server generates dynamic values such as cookies, tokens or IDs that are included in responses. Every user session receives a unique set of values.
2. The server then expects to receive these values back in subsequent requests.

The name "correlation" points out that the dynamic values in the requests are related to the corresponding values in the previous responses. A web browser naturally correlates the dynamic values. In load testing tools however, the clients are virtual users and not browsers, so correlation must be handled using other means.

Why is correlation necessary? Here are several examples:

• Session ID. The server communicates with multiple clients at any given moment. Because the HTTP protocol is stateless, the server cannot determine which client sent a particular request by just using protocol metadata. Therefore, the server generates a unique ID when it establishes communication with this clien, to match requests to the client who sent them. The client should return this ID on all subsequent requests.

Figure1 - How Correlation Works

• Security tokens. The authentication process issues a security token that is sent back and forth between the server and client. Examples are Single Sign-on Security Token and AntiForgeryToken.

• Application context or state. In client/server systems, the server needs to have the application context of every user to process its requests correctly. The application context may include various data about past user activity. For scalability purposes, application context might not be stored on the server, but rather be included in all HTTP roundtrips. One example of a dynamic parameter that carries application state is the "__ViewState" parameter in ASP.NET applications. On every postback to the server, the web browser must return the same _ViewState value that it received in the previous response.

In all of these examples, if the clients fail to return the expected dynamic values generated on the server, the application will break.

How web browser implements correlation. The physical browser is designed to carry-over values from responses to the subsequent request thanks to several embedded mechanisms implementing correlation logic. Here are a few examples:

• Hyperlinks. Often time the server renders a web page with a hyperlink containing a query string with a dynamic parameter. When a user clicks on the link, the browser sends a request with the query string containing this parameter.
• Cookie correlation. The server sends cookie information to the client using the Set-Cookie header. The browser will include the cookie in every subsequent request to the server.
• Web form hidden fields. By default, the browser will submit hidden fields unchanged to the server when a user posts back the form.
• Programmatic correlation. The client's code can include the custom logic of storing dynamic server values and re-injecting them to requests when required. Such logic can be implemented in the webpage's JavaScript, or client-side code executed in Silverlight, Adobe Flex or another environment.

What is the problem with the virtual users? The essential component of load testing is realistically emulating user-base impact on a servers' infrastructure by instantiating hundreds or thousands of virtual users. A VU is a lightweight software object that generates HTTP traffic. Its main job is to be indistinguishable from a real user from the server's perspective. Unlike a real client, they are not designed to execute JavaScript or fully emulate the browser's behavior. Therefore, VUs do not implement correlation mechanisms. Instead, they just replay the traffic as it was recorded while using a small footprint, so hundreds or even thousands of users can be emulated from a single machine. Therefore, correlation must be created manually.

Who correlates your test? Some load testing tools can automatically create the required rules in the test configuration or script. This method is called autocorrelation. Not all errors, however, can be fixed this way, and the remaining issues must be addressed by applying manual correlation techniques. This article describes a comprehensive manual correlation method that simplifies it and makes it more efficient.

Ten reasons why correlation errors are tough

Among load testing errors, correlation errors are the toughest. Here is why:

1. Not obvious. If you are a novice in performance testing, it may not be obvious why correlation is necessary. It appears that the end-user knowledge of the application under test should be sufficient to record a scenario and run the test. Deep understanding of application internals seems to be unnecessary for successful load testing. After all, you are a tester, not a developer. The problem is that dynamic variables that are at the center of correlation are hidden from the end-user and live under the hood.

Side note: Does it mean that before testing application, you have to learn how it is designed? Fortunately, no. The comprehensive method of manual correlation described in Part 2 of this article will allow you to correlate applications of any complexity like a pro without spending time on learning application logic.

2. Correlate or not to correlate. Your test runs with errors. Does it mean that you need to correlate it? It depends on the type of errors you're dealing with. Broken correlation causes some but not all errors in the test. Non-correlation related errors cannot be fixed by correlation and vice versa. It puts correlation errors in a class by itself as you can see in load test error classification below. Unfortunately distinguishing them from other errors can be challenging as they look similarly.

3. Invisible errors. If you don't receive errors in your test, it can be bad news. While HTTP errors are easy to find because their response code is in the 400 or 500 range, some application errors can escape detection. If the VU incorrectly replayed business transaction messages, error pages return by the server can have a valid response code 200. Such error can be easily missed unless you monitor responses for specific known content indicating a failure. If you don't know that the problem exists, you don't know that it should be fixed.

4. Unclear goal. Since correlation errors are difficult to detect, it is also difficult to determine if the correlation rules that you created fixed the error.

5. Deal breaker. Correlation is not necessary to make your test better. It is necessary to make your test work. Running a performance test before all correlation issues are resolved is largely a waste of time. Because broken correlation causes errors in some transactions, they will fail and complete sooner. If such errors are not fixed, the average response times in the test report may be largely inaccurate. Relying on such test result can cause troubles for your team or clients.

6. Autocorrelation - false expectations? Some tools like Jmeter and most other open source tools do not offer autocorrelation. It means that even the most obvious dynamic variables like ViewState in ASP.NET websites must be correlated manually. High-quality commercial tools provide an autocorrelation feature. Does this mean that the correlation problem is solved? Not really. Some top-notch tools can reliably emulate standard behavior embedded in the web browser such as web forms or cookie correlation. But the biggest challenge is presented by applications where the dynamic variables are handled programmatically by a client-side JavaScript or other client runtimes such as Silverlight or Adobe Flex. Load testing tools would need to reverse engineer such client logic to crack down hidden correlation rules, and this is not easy. According to [1], the automatic correlation in load runner works in only 5% of cases and manual correlation is more advisable. If you expect that autocorrelation will resolve all errors, you may be disappointed as some of the errors have to be rectified manually.

7. Inconsistent result. To make things worse, you can never be sure that a script that successfully handles one application will also work with your next application or even the next version of the same application. Sometimes simple upgrading underlying framework to the next version will break your test script. Sometimes a script that worked last week stops working on Monday even though no changes were made on the server, except that over the weekend, the server was rebooted and some values that seemed to be static last week are actually dynamic keys that are re-generated on reboots. This unpredictability makes manual correlation even more inevitable.

8. Platform specific vs. systematic approach. There are some publications dedicated to correlation load tests of particular web frameworks or popular Enterprise applications, such as PeopleSoft, Oracle BI, Cognos reports, SharePoint and Microsoft CRM Dynamics. They list dynamic parameters belonging to these platforms and provide other helpful platform-specific hints. This may create an impression that the key to successful correlation is learning specific platforms. However, it diverts attention from focusing on a systematic and comprehensive approach that works in all environments.

9. Manual correlation is poorly defined. While there are some useful publications dedicated to correlation, they largely lack a systematic description of the manual correlation as a process. For example, a vague explanation like "It may take some searching to find, but careful study of a detailed log will eventually yield the session id" [2] provides some hints to the solution but not the solution itself.

10. Correlation-is-getting-harder trend. As web applications' versatility evolves over time, so does their complexity. Developers "hide" more often dynamic variables in nontraditional parts of the HTTP message like custom headers. An increasing variety of application protocols and encoding methods create even more challenges for correlation. Having a comprehensive approach to correlation becomes, even more, essential.

Simplifying the problem

Fortunately, you do not have to know how dynamic values are created or how they're used in the web application to understand and solve the correlation challenge. Instead, we can use a "black box approach" to simplify the problem by focusing exclusively on significant aspect and ignoring complicated and unessential details.

Three properties of the correlation:

• The server creates a dynamic value and includes it in response.
• The client sends back the same value in one or several subsequent requests.
• The value must change either across the VUs or across the iterations of replaying the test scenario.

Correlating the load test means to emulate such client behavior by creating correlation rules.

What is a correlation rule? It is an algorithm for extracting dynamic values from a server response and using it to substitute some recorded values in the subsequent requests. To create such rules you need to ascertain:

• which responses contain a dynamic value and how to find it;
• which requests should use this value and what search-and-replace pattern should apply.

Correlation rules should be added to the test configuration or test script so the VUs can correctly handle dynamic variables.

To start fixing correlation errors, you need to know how to distinguish them from other errors.

Load test errors

There are three types of errors that you may encounter in connection with load testing (Figure 2):

1. System Errors are caused by the application, framework, or network. They are not related to the test configuration. In an ideal world, these errors must be exposed during functional testing and addressed before the performance testing cycle. In reality, you may run into them while recording a test scenario or verifying the script. It is important to recognize them quickly and report to the system stakeholders. If you falsely mistake system errors for test errors and try to address them by tweaking the test configuration, you will be wasting time.

2. Performance Errors, like the previous group, are thrown by the system under test. However, they appear only on elevated load levels. Tracking down such errors is one of the goals of load testing. For example, a concurrency error happens when an application cannot handle more than N concurrent users. Once the load emulation exceeds N users, some transaction will fail. Performance errors should be addressed last. A detailed discussion about fixing them is outside of the scope of this article.

3. Load Test Errors are entailed by incorrect test configuration and should be fixed by correcting test script, configuration or execution. This group can be broken down into four categories:

a. Recording Errors are caused by using an incorrect or old recording. Here are three examples:

• If you record traffic from the web browser that has primed cache, some requests may be missing in the recording. To avoid such issues, before recording, always clear the browser cache or launch a browser in Private mode [3].
• If you started recording when the web browser displays one of the pages, your script would miss some initial actions, such as session initiation handshaking. When recording, always start a fresh browser instance.
• If after script recording, the application was modified, and this caused script verification errors, you need to either fix your script manually or re-record. Fixing it may include editing, repositioning, deleting existing or adding new requests.

Figure 2 - Load Test Error Classification

b. Test Environment Errors. The incorrect or incomplete configuration of the test environment can induce some errors. Here are two examples:

• After recording the scenario over HTTPS, replaying it with a single user, triggers multiple problems due to an issue with certificate. To fix it, install a certificate on the browser or mobile device.
• After recording the scenario, the application under test changed host, port or schema (HTTP to HTTPS and vice versa). It should be addressed by appropriate tool settings.

c. Correlation Errors are caused by missing correlation parameters.

d. Non-Correlation Errors are all remaining test errors.

The last two categories of errors are the most challenging. Let's review non-correlation errors first.

Non-Correlation Errors

These errors have to be resolve first to leave the toughest correlation errors for the end. Here are a few examples of non-correlation errors:

• Unique constraint. Your test scenario involves user entry of unique data. When replaying the script with a single user, the system will throw an error because of a unique constraint violation. These types of errors are easy to replicate by repeating the same user entry in the browser. To fix this issue, parameterize the recorded entry values to substitute them with unique values on replay. See the differences between parameterization and correlation below.
• User session limit. Some applications allow one session per authenticated user. If a real user tries to log in from a second browser, the system will either reject authentication or log him off in the first browser. When you replay a test scenario recorded on such application, it will work with one VU. However, it will break with two or more VUs. To resolve this issue, parameterize the recorded user credentials with an array of credentials, so every VU is treated as a unique user. (This is one of few non-correlation issues that does not prevent successful correlation and, therefore, can be addressed after correlation is complete.)
• Client dynamic parameters. In some applications, the client can also originate some dynamic values, such as a random GUID or a timestamp. Requests with such values must be parameterized to avoid replay errors.

Correlation vs. Parameterization. Parameterization helps to resolve some of the discussed non-correlation errors. It is also used for reasons not related to fixing errors. For example, you recorded a test scenario of a person entering her name and contact information to create a customer profile. To emulate such scenario more realistically, dynamic data that varies on different test iterations should replace the recorded values entered in the web form fields.

Similarly to correlation, parameterization replaces the recorded value with dynamic data. However unlike the former, the latter does not use values generated on the server [4]. Instead, it uses predefined data sets, random data, or other information originated on the client such as a timestamp.

To start fixing correlation errors, you need to identify them first. The next chapter describes how to do that.

Correlation Errors

If a stale recorded value is included in the request in place of a dynamic parameter during the test scenario playback, the server will encounter an application failure and will send back an appropriate response. It's hard to predict how a particular application will react to such correlation error. For example, it can return an abnormal HTTP response code in the 400 or 500 range that is detectable by the tool, but it may also try to inform a real user about the abnormality and return a valid HTTP response with a warning message, an alert image or text reporting an unhandled exception. It would be desirable to see the content of the error message in the web browser. Unfortunately, this is not feasible, as web browsers do not make correlation errors. So you don't know what content to expect in the responses, and therefore, it is virtually impossible to track down all such errors automatically by monitoring their content in the tool.

To overcome this challenge, we recommend using more broadly defined criteria to identify the responses potentially triggered by correlation errors.

Signs of potential correlation errors:

• Recorded response code was 200, but the replayed response code is in the 400 or 500 range.
• Recorded response code was 200, but the replayed response code is 302, or vice versa (Fig. 3)
• Recorded and replayed responses were redirected to different URLs.
• The size of the recorded and replayed response are substantially different (more that 50-100%).
• The expected transaction did not complete. For example, the file was not created, the email was not sent, or a database record was not created.

Figure 3 - Compare Recorded and Replayed Response

As much as responses to correlation errors vary from application to application, the resolution of correlation errors is not application-specific. This means that you do not need to know the application's design to efficiently resolve correlation errors and that a platform-independent correlation method can be developed.

Identifying potential correlation errors is the first step in fixing them. Published in the next issue of Methods & Tools, the part 2 of this article will describe step-by-step tool-independent correlation method that works on all platforms.

References

[1] Correlation in LoadRunner: Ultimate Guide. http://www.guru99.com/correlation-in-loadrunner-ultimate-guide.html#automatic

[2] Automated Test Script Correlation. http://www.testingperformance.org/performance-articles/correlation

[3] How do I set my browser to Incognito or Private mode? http://www.computerhope.com/issues/ch001378.htm

[4] What is the difference between parameterization and correlation in HP LoadRunner. http://learn-loadrunner.blogspot.in/2015/11/what-is-difference-between.html

---

Read the second part of this article: Eradicating Load Testing Errors - Part 2: Mastering Correlation

Related Methods & Tools articles

• Modeling the Real World for Load Testing Web Sites
• Testing Performance of Mobile Apps - Part 1: How Fast Can Angry Birds Run?
• Testing Performance of Mobile Apps - Part 2: A Walk on the Wild Server Side

---

Software Testing Resources

Software Testing Magazine

Load Testing Tools Directory

Software Testing Videos & Tutorials

---

Click here to view the complete list of archived articles

This article was originally published in the Spring 2016 issue of Methods & Tools