XSTest - turning functional tests into load tests

Introduction

TestMaker is an open-source framework and utility for building intelligent test agents that check Web-enabled applications for scalability, performance, and function. TestMaker provides a Recorder to create test agent scripts for you and a graphical environment for building and running test agent scripts. These recorded test agent scripts test a Web-enabled application for functionality by making requests to the host and recording the success or failure and timing metrics of the response. XSTest takes these recorded test agent scripts and runs them multiple times concurrently to learn the scalability of the Web-enabled application as load increases. XSTest is distributed under the terms of an Apache-style free open-source license.

Functional Tests

Most tests begin with a simple functional test. The user asks "Does this part of the system work?" Later they may test the system for scaleability under increasing load, regression tests, or to monitor a system. For now the system manager or software developer wants a single test agent to run a Web-enabled application through a single series of steps and view the result.

TestMaker provides a Recorder feature that builds functional tests. The recorded test implements a TestCase class. The TestCase has a method to run the test and return results data. The results indicate if the test encountered errors, the timing metrics of the request and response, and the response from the host. TestMaker provides a graphical and command-line utility environment to run TestCases.

Load Tests

In a load test, the software developer, QA technician, and IT manager wants to learn how the Web-enabled application host handles requests from increasing large sets of concurrently running test agents. They seek answers to questions about data center readiness and software quality, including "Have I bought enough servers?" and "Is may application meeting user needs by responding quickly?" TestMaker provides the XSTest object framework to stage and analyze a load test to answer these questions.

Here the user operates the Recorder feature to create a TestCase object. Using the XSTest framework the user identifies a TestCase object and XSTest plans the test, stages the test, and calls the tally module to report the system's throughput as a Transactions Per Second (TPS) value. (Click here for a use case on the test scenario.)

Load testing delivers a single TPS value. For example, a load test states that at 50 concurrent virtual users the system performed 28 transactions per second (TPS). Scalability testing delivers a function of load tests over a series of increasing loads. For example, consider the following results of a scalability test. (This chart was developed by importing textual results from XSTest into a spreadsheet program that displays the results in graphical charts.)

The test scenario ran 4 series of test cases. In each test case the number of concurrently running test agents increased. It started with 50 concurrently running test agents and finished at 300 agents. After each test case concluded the test delivered a throughput measurement of the test case by stating the average number of transactions per second. For example, in the chart at 50 concurrent test agents the system handled 55 TPS. And, at 300 the system handled only 12 TPS.

There are any number of reasons that the throughput as measured in TPS of a test may decrease - or for that matter increase. To learn why your system performs this way we recommend you read Java Testing and Design: From Unit Tests to Automated Web Tests by Frank Cohen (Prentice Hall 2004.)

XSTest stages load tests and provides a textual scalability index report. XSTest takes a list of concurrent user levels to test against. XSTest calculates and operates the test cases automatically. Then it reports the results in throughput measurements.

Installation and Running

Download TestMaker. XSTest comes bundled with TestMaker. You will find XSTest and its documentation in TestMaker_home/xstest once you download and unpackage the TestMaker ZIP archive.

Please familiarize yourself with XSTest by opening the properties.py, master.py, and testcase.py files and browsing through their contents. XSTest consists of these files:

• properties.py - configuration settings for the test
• master.py - the main script that stages a scalability test. To begin a test open master.py in TestMaker and run it.
• testcase.py - code to call a functional test
• testbase.py - the binding code to turn the functional test in testcase.py into a scalability test
• functiontest.py - a test agent script built by TestMaker's Recorder that implements a functional test.
• tally.py - parses through the results log to show the service scalability index as measured in transactions per second.

Follow these steps to use XSTest:

1. Record a functional test agent script using TestMaker's Recorder feature. Name the test agent script functiontest.py. If you choose a different name then you will modify testcase.py.
   
   
2. If using TestMaker 4.0.8 or earlier then comment-out or delete the resulting functional test agent from the line that reads: Transactions-per-second Index Test to the bottom of the script. If you skip this task, XSTest will first execute the transactions-per-second index test before it executes the load test.
   
3. Open the properties.py file and modify the contents to suit your needs to test. It controls the number of concurrent test agents, the names of all the files, and many other options.
   
4. To run a test scenario open the master.py test agent script in TestMaker. Click the Run icon or type Control-R.
   
5. XSTest will display its progress in the output window in TestMaker.
   

By default, the functional test needs to be named functiontest.py and the test agent file must be in the XSTest directory. The testcase.py file controls the file and class name of the functional test.

Tally Reports

XSTest finishes a test by running the tally.py test agent script. Tally.py parses the recorded log file to determine the transactions per second value for the current level of testing.

By default properties.py names the file test.log. You can browse the contents of the test.log file to see the recorded results. It is a text file containing comma-delimited results data with one entry per completed functional test. For example, below is an example of the contents of the test.log file after a scenario runs.

   0,10553,3484816,14168064,3,0.5,15
   0,7483,2925216,14168064,9,0.5,15
   0,1658,2586952,14168064,14,0.5,15
   0,1071,3547176,14168064,13,0.5,15
   0,773,2527592,14168064,14,0.5,15
   0,20510,3665704,17272832,7,0.5,30
   0,7493,3466768,17272832,24,0.5,30
   0,10520,1508368,17272832,10,0.5,30
   0,17823,1863240,17272832,16,0.5,30

Each line contains the results of a single completion of the functional test. Commas delimit the following fields:

1. Result code, 1 = exception, 0 = ok
2. Time to complete a single test case in milliseconds
3. Java free memory (runtime.freeMemory()) in bytes
4. Total Java memory (runtime.totalMemory()) in bytes
5. Test agent ID
6. Test case duration in seconds
7. Number of concurrent virtual users in test case
   

The tally.py test agent parses through the test.log file and produces a scalability index that looks like this:

Results

CUs  TPS   Count Min  Max   Errors
15   1.90  57    773  15581 0
30   2.10  63    806  43471 0


Error count = 0
Total count = 120
% of errors = 0

Legend: CUs = Concurrent users, TPS = Transactions Per Second,
Min = Shortest test case time in milliseconds,
Max = Longest test case time in milliseconds,
Errors = Count of all errors

Done. Thank you for using XSTest. Further details
are available at http://www.pushtotest.com

The report shows that this test scenario learned the scalability index of the Web-application under test by running the test at 15 concurrent virtual users first and then at 30 concurrent virtual users. The properties.py file controls what levels of concurrent virtual users is included in the test scenario. The report shows a TPS value at 15 concurrent users is 1.9 transactions per second and at 30 concurrent users it is 2.10 transactions per second. While this is test scenario is limited to only two data points, we can surmise the following:

1. Even though the second test case employed twice the number of concurrently running test agents, the throughput as measured in transactions per second increased only 10%. This indicates that some part of the system has reached its maximum throughput and that we should investigate the components that process a functional transaction to find the bottleneck.The components to investigate may include network bandwidth, server CPU utilization, memory, scalability problems in the software.
   
   
2. Adding more test cases where additional levels of concurrent virtual users are applied will generate additional details about the scalability of the system. This test checked the system at 15 and 30 concurrent users. In our experience at PushToTest checking a system at 5 or more levels of concurrent users and for test periods of 10 minutes or more produces useful scalability information.

Chapter 12 of Java Testing and Design: From Unit Tests to Automated Web Tests by Frank Cohen (Prentice Hall 2004) provides extensive detail on how to analyze a scalability index report to turn the results into actionable knowledge.

Where to go next?

Download XSTest in TestMaker

Learn more about TestMaker

Read articles about testing in docs.pushtotest.com

Learn about support options for XSTest