Archived PushToTest site

Calibration Test Methodology

PushToTest Logo

Calibration Test Methodology

Rapidly Determine The Scale Of A Performance Test


The PushToTest community often asks for the best method to determine the scalability of an application or service. Where is the starting point? How do I get started? And, how do I go from test use cases to actionable knowledge? PushToTest recommends Calibration Testing.

Calibration Testing methodology has three parts:
  1. Identify the test use cases. Define one or more user archetypes. The definition includes pages to visit, behaviors, and operational test data. Click here to read about user archetype definition methods.
  2. Identify the optimal throughput of the application and the test environment.
  3. Run the complete performance test and analyze the results
Step 2 requires a methodology to quickly determine the scale of the test. Start too high and the test delivers results that are a function of a performance bottleneck in the test equipment itself and not the application under test. Start too low and you test your application not really knowing the upper limits of performance. Calibration Testing relies on the operating characterists of an application with linear scalability.

Shows TPS increasing proportionally to increase in virtual users

Applications with linear scalability deliver increased throughput as the test increases the number of concurrently running users. In the above chart running 10 users delivers 2 transactions per second (TPS.) A transaction is successful operation of a test use case. Doubling the number of users doubles the throughput. At 20 users an application with linear scalability delivers 4 TPS.

Calibration Testing determines the peak throughput possible given the operating environment. Calibration testing increases concurrent user levels while keeping the equipment operating the application constant. The following chart identifies a Scalability Index for a fictitious application:

Throughput at 6 levels of virtual users

In the above fictitious example, 40 concurrent users is the optimal level for this application. When the calibration test goes to 80 users the TPS level actually decreases to 7 TPS. Above 80 users the application throughput continues to decrease.

In the above example, running 6 levels of tests is time consuming. When each test level takes 45 minutes to operate (10 minutes to start-up, 20 minutes to run, 15 minutes to ramp-down) the total test time to build the Scalability Index is 4 1/2 hours! Calibration Testing saves 20% or more time.

With limited performance testing time available, PushToTest recommends using the Calibration Testing methodology to rapidly identify the optimal test levels for the application under test.  Follow these steps:
  1. Determine a baseline application throughput goal. For example, 6,000 virtual user sessions in a 20 minute period or 5 TPS (6000 transactions / (20 minutes * 60 seconds) )

  2. Run tests at 50% (3,000 users,) 100% (6,000 users,) 150% (9,000 users) of the base level. Determine the Scalability Index by contrasting the TPS at each user level with an application with linear scalability. For example, the following chart contrasts application throughput performance at 3,000, 6,000, and 9,000 users to a linear scaling application.

    TPS results at 3K, 6K, 9K users

    The light blue shows the TPS levels that a linear scaling application produces. We calculate the light blue bars at each level of load's TPS value compared to the delta between the current level and the original 3K user test. In the above example the application reaches its top-level of scalability at 6,000 users. And even at that level the application should be producing throughput to the top of the blue bar!

  3. If the Baseline 1 results show the application reaches linear scalability then run a second base level at 150% of the Baseline 1 test. If the Baseline 1 results show the application is unable to reach linear scalability then run the second base level at 50% of the Baseline test. The above example, showed the application did not reach linear scalability. In this case the Baseline test 2 would be as follows:

    Second baseline showing TPS at lower levels of load


  4. Operate the performance test at 50%, 100%, 150% of the Baseline 2 levels.
Calibration Testing focuses on throughput measured from the client (load generation) equipment. Observing resource usage (CPU, Network, Memory) on both the client and the server while operating a Calibration Test helps diagnose performance issues in the test itself.

Test Experience

Likely Problem

What to Do Next

Increase user levels with a decrease in TPS

Check the average response time.

Run a test case comparing response times as user levels increase. Identify the least acceptable response time and work back from there.

Increase user levels with an increase in TPS

Test is not calibrated for high enough user levels and payload sizes.

Run calibration test to determine optimal Scalability Index and set correct user levels and payload-size levels.

Increase user levels with little change in TPS

user levels levels are set too high. Check CPU utilization if doing a stateless test.

Run another test with the user levels level reduced by 50%.

Server CPU at 95% utilization and increasing user levels levels increases TPS

This is probably a stateful system test.

Run a test case to determine the service-under-test Scalability Index.

Client CPU at 95% utilization and increasing user levels levels decreases the TPS

User levels levels are too high.

Add more load generating consumers.

Client CPU utilization at 15%, Server CPU utilization at 30% and increasing user levels levels barely changes TPS

Network is probably saturated.

Check network bandwidth utilization. Add network adaptors to server or consider a faster network.


Conclusion and Benefits

Calibration testing reduces the number of test levels needed to identify the scale of a performance tests. In the above example, the Calibration test determines the test levels by running 4 instead of 5 test levels. This is a 20% reduction in time to determine the scale of the test. Additionally, the calibration tests provide business managers with early load and performance testing and analysis inputs to business risk analysis.


PushToTest Logo

Additional documentation, product downloads and updates are at www.PushToTest.com. While the PushToTest testMaker software is distributed under an open-source license, the documenation remains (c) 2011 PushToTest. All rights reserved. PushToTest is a trademark of the PushToTest company.