Site MapAccessibilityContact Us

Call us at (855) 254-1164 to learn more about Enterprise

PushToTest - Open Source Test

Results Analysis and Reporting

Results Analysis and Reporting

The TestMaker Results Analysis Engine presents test results depending on the type of test: functional tests show a check list of step operations, load and performance tests have a choice of 1,000 or more charts and statistics, and business service monitors show a dashboard of monitor statistics and service up/down status. This chapter focuses on the results analysis possible from load and performance testing.

Overview

Analyze Test Results

Load Test Live Results

Log Contents

Results Setting and User Contents

Run Charts Manually

Pivot Tables, Dashboards, Correlation

Results Repository Choices


Analyze Test Results

TestMaker comes with everything you need to build tests, operate tests, and present the results graphically and with minimal training:

PushToTest designed TestMaker to operate a test and analyze the results for performance bottlenecks and remediation. PushToTest presents results depending on the type of test: functional tests show a check list of step operations, load and performance tests have a choice of 1000 or more charts and statistics, and business service monitors show a dashboard of monitor statistics and service up/down status.

  • Produces thousands of charts and reports showing root cause to performance bottlenecks and mitigation
  • TestMaker Results Analysis Engine produces reports from previous test runs
  • TestMaker records every transaction, step, and monitored resource timing value to a Results Database and Repository. Compatible with MySQL, Oracle, IBM DB2, and MS SQL Server
  • PushToTest offers Jasper Business Intelligence Suite as a TestMaker Enterprise option for the following features:
  • 10,000 charts from the test results
  • Report GUI Designer
  • Flash charts, maps, and widgets
  • Pivot tables, cubes, and dashboards

TestMaker uses a relational database (RDBMS) for its Repository. TestMaker logs every test operation, test step, and monitored resource to the Repository. TestMaker comes with the Apache Derby RDBMS. Derby is satisfactory for evaluating TestMaker and it is not acceptable for production environments. TestMaker supports MySQL, Oracle, IBM DB2, and MS SQL Server. See the Repository Configuration chapter for details.

TestMaker builds its own database schema to store the logs. The DB contains a catalog of scenarios (scenario, usecases, steps) and tables to store the results of executions (executions, transactions results, steps times). Find the schema here. Routine TestMaker operation does not require you to learn the schema. TestMaker automatically creates the tables and indexes according to the schema the first time TestMaker logs to the database.

TestMaker integrates JasperSoft iReports to analyze the test results and present charts and reports. Save reports to Excel files, Adobe Acrobat PDF, XML, HTML, and csv files.

Load Test Live Results

As TestMaker operates a load test scenario the Controller Panel displays a live view of the Scalability Index for the target service. TestMaker polls the TestNodes approximately every 10 seconds and updates the Real Time Scalability Index.

The Scalability Index is a very quick way to learn your application's ability to scale linearly. For example, the Scalability Index shows the application's throughput average for Transactions Per Second (TPS), measured by the TestNodes, at each concurrently running simulated user (CR) level. The following shows the live results chart for a load test scenario reporting .05 TPS at the first virtual user level, 0.9 TPS, and 0.12 TPS at the third user level:


Controller window show Load test results - scalability index

The higher the TPS the better. And the more TPS for each level concurrent users the better. In a system with linear scalability 4 concurrent running users should give 4 times the TPS as 1 concurrently running user. The ghosted scale behind the results columns shows where the application performance (in TPS) should be for linear scalability.

The Scalability Index appears as the default chart in load and performance tests because it helps identify server, network, and memory needs, it identifies the population of users that the application or service can handle, and it surfaces scalability and performance issues before your users and customers experience them. Click here to learn about the Scalability Index.

The live results chart provides several functions accessible using a right-click of the mouse on the chart. A pop-menu appears to save the chart to a PNG graphic file, zoom in / out, print and set preferences for the chart, including fonts, axis settings and titles.


Log Contents

TestMaker test operations generate useful data. TestMaker integrates Jasper to visualize the data into actionable knowledge, including the Scalability Index, Transaction Distribution Charts, and Resource Monitor Charts.

TestMaker logs a base data series:

1. Test Scenario (name)
2. Test Node (name)
3. Concurrent User levels (CRs)
4. Data Index (Message size)
5. Transaction (Use Case)
6. Sequence name
7. Step (run) name
8. Pass/Fail status 
9. Error Name and Details

See Java Enterprise Application Monitoring with Glassbox for details on performance bottleneck and functional issue montoring of Java enterprise applications.


Custom User Logging

TestMaker provides a custom user logging static method to start and end logging a step:

     PTTStepListener.startStep( "setUp" )
     ...

     PTTStepListener.endStep()

The method comes in the com.pushtotest.tool.PTTStepListener class.

Results Settings and User Controls

TestMaker provides a graphic interface for you to identify additional summary charts. From a load and performance Controller Window, click the Charts icon in the lower left corner of the controller panel.

Icon to get to Chart Settings

TestMaker displays a selection box to choose from a set of charts.

List of charts to create

TestMaker saves your selection in the TestScenario. TestMaker creates the selected charts at the end of a load test operation automatically.

Running Reports Manually

PushToTest TestMaker operates the chart definitions from the TestScenario at the end of a load test automatically. TestMaker provides access to results charts after the test operation. Choose the Results Analysis Chart Generator command in the TestMaker Console Tools drop-down menu.

List of charts to create, including how to select the execution date\

Click the Execution Dates button to choose from past operations of a test.

Shows test execution dates to choose

Click a Scenario, then click the Load! button. TestMaker opens a new window to display the required charts.

Windows showing all of the generated charts



Pivot tables, dashboards, correlation

PushToTest offers Jasper Business Intelligence Suite as a TestMaker Enterprise option for the following features:

  • 10,000 charts from the test results
  • Report GUI Designer
  • Flash charts, maps, and widgets
  • Pivot tables, cubes, and dashboards

Contact PushToTest for details on the BI suite.


Results Repository Choices

TestMaker uses a relational database (RDBMS) for its Repository. TestMaker logs every test operation, test step, and monitored resource. TestMaker comes with the Apache Derby RDBMS. Derby is satisfactory for evaluating TestMaker and it is not acceptable for production environments. TestMaker supports MySQL, Oracle, IBM DB2, and MS SQL Server.

Use the Tools menu, Preferences command to select the Repository tab. TestMaker uses JDBC drivers to communicate to the RDBMS. Internally, TestMaker uses the Hibernate library for object-to-relational data mapping and database schema creation. The Create DB button creates a new database and initializes the logging tables and schema. TestMaker includes Hibernate drivers for Derby, MySQL, and Oracle. Contact PushToTest Global Services for additional drivers.

Choose from the most popular charts



PushToTest logo

Additional documentation, product downloads and updates are at www.PushToTest.com. While the PushToTest TestMaker software is distributed under an open-source license, the documentation remains (c) 2011 PushToTest. All rights reserved. PushToTest is a trademark of the PushToTest company.