TestMaker Editor User Guide

The TestMaker Editor is a graphic tool for creating and changing TestScenarios. A TestScenario identifies the operating parameters for a functional test, load and performance test, and production monitor. Editor saves TestScenarios in an XML-formatted file. This User Guide describes the Editor and its many options.

For a quick start building tests of Ajax applications, SOAP and REST based Web services, Java-based unit tests, and Flex/Flash applications, consult the Tutorials.

Contents

• Editor Geography
• Working with Scenarios
• Scenario Mini Toolbar
• General Scenario Properties
• Working with Use Cases
• Configuring TestNodes
• Monitoring Scenarios
• Scenario Options
• More Scenario Options
• Scenario Notifications

Editor Geography

The editor window is divided into two parts. The left side shows a list of all the open test scenarios. In this image, three scenarios are open.

The right side of the window shows details for the selected scenario. In this image, Ajax_Load_Test.scenario is selected and its details are shown on the right.

When you run a scenario, it shows up in a separate window where you can see its progress. Here is an example scenario window.

Working with Scenarios

The editor supports standard file operations for scenarios through commands in the File menu:

• Use New Functional Test, New Load Test, or New Service Monitor Test to create a new scenario.
• Open... brings up a file chooser so you can open an existing scenario.
• Open Recent includes a submenu full of scenarios you have recently opened.
• Choose Save to write the current scenario to disk.
• Save As... is the same as Save, but you get to choose a new file name for the scenario.
• Execute Scenario runs the current scenario.
• Close Scenario removes the scenario from the editor window.

Editor updates TestScenarios created with TestMaker 5. When you open a version 5 scenario, editor displays an alert box asking if it may update the scenario. This update is required to run the test in TestMaker 6. Once updated, the scenario will not open in TestMaker 5.

A scenario is a complete definition of a functional test, load test and service monitor. A scenario document follows the XML Schema Definition (XSD) found at http://downloads.pushtotest.com/tm5/testscenario.xsd

Examples of TestScenario documents used in functional, load, scalability and performance tests plus service monitors are found in the Tutorials.

Scenario Mini Toolbar

Each open scenario in the editor window has its own mini toolbar with commonly used controls.

The controls in the mini toolbar are as follows:

closes the scenario so it will no longer be listed in the editor

saves any changes you have made to the scenario file

starts running the scenario.

pauses execution of the scenario

halts the execution of the scenario

brings up another window where you can control the charts that are generated for the scenario.

General Scenario Properties

The General tab contains overall settings for the scenario, test type, and meta data.

The Test Scenario Name is the display name of the scenario. This name is shown in the scenario window when you run the scenario. TestMaker uses the name in the Controller Panel, in the status Output Panel and in the results files.

Default Directory defines the path that will be used to find scripts, JAR files and other resources.

Click on the small triangle button to view and edit additional properties, like Author Name, Creator, Links, Versions, and Categories.

Test Type affects how TestMaker works with the scenario. Choose Functional Test, Load Test, or Production Monitor.

Different Kinds of Tests

The options below Test Type vary depending on the type you choose.

Functional Testing

For a Functional Test, you can choose how many times to Repeat this use case. Select Repeat for all data for TestMaker to repeat the test use case for every row of data the test use case accesses using a Data Production Library.

Load and Performance Test

Load tests operate the test use cases in one or more virtual users as many times as possible in a given amount of test time. By running a load test at several levels of virtual users, TestMaker identifies the Scalability Index of your application's scalability.

Choose one or more virtual user levels to operate the test. Click Add virtual user level to add a level. Select Test for all data for TestMaker to run the virtual user level while there is more data using a Data Production Library.

Production Monitor

For a Production Monitor, choose the interval between tests.

About Paths And The Default Directory

PushToTest TestMaker works well in a distributed environment where developers and QA engineers run tests within any directory structure (i.e. wherever they've checked out the project files). TestMaker TestScenario test orchestration documents determine the default directory path for test resources. The following options overriding the default directory and other elements in the TestScenario. For example, consider the following directory structure:

ScenarioDirectory
- Scenarios
- - MyScenario.xml
- Resources
- - resource.csv


TestMaker provides a {scenario} operator that resolves to the path of the TestScenario document. This operator may be used in the element, in elements, and in individual resource paths. For example, use the above resource.csv with

path="{scenario}/../Resources/resource.csv"

The following is also valid:

Defaultdirectory ="{scenario}/../" and path="Resources/resource.csv"


Working with Use Cases

The Use Cases tab is the place where you set up one or more test steps, options for each test step, and setup and teardown steps.

This example runs a single test (AddCompany.ds) with no setup or teardown.

The editor allows for sophisticated scenarios. TestMaker scenarios consist of one or more use cases, each of which can contain one or more test scripts. Each use case can have setup and teardown scripts. In addition the entire scenario can have global setup and global teardown scripts.

To create a test use case click the links, including Add Global Setup or Add Test. Each element appears graphically. Click on X to remove an element of the scenario.

The setup, teardown, and test elements in the scenario all point to scripts. TestMaker knows how to run many kinds of scripts, everything from Sahi and Selenium through to JRuby and Perl. The editor shows options for the script that are appropriate to the script type. For example, you get to choose a browser to run a Sahi script. You get to specify a class name for a Java language script.

Test Types

TestMaker provides support to run many test types. Use the Test Type menu to choose a test type. Every type of test provides fields for Test Name and Instance. TestMaker uses TestName when reporting the results. TestMaker uses Instance to reuse test objects from one test step to another. For example, a test with three steps (log in, search for products, log out) reuses the same Web browser instance when each step uses the same instance value.

Editor displays a special form for each test type:

Running Multiple Use Cases Concurrently

When you have more than one use case, TestMaker will run all the use cases concurrently as a sequence. For example, when running a load test of 100 virtual users, the test use case with 80% proportion runs in 80 of the 100 virtual users concurrently.

Adjust the percentage of time allotted to each use case by adjusting the Sequence proportion value.

Configuring TestNodes

The TestNodes tab is the place where you tell TestMaker where to run a scenario. You can specify any number of TestNodes, Grids, and Clouds. Read the Test Deployment chapter for details.

The TestNode has a human-readable Name and a URL where is it available. Monitor port is an optional value to determine the port number for PTTMonitor service running on the TestNode. PTTMonitor observes CPU, network, and memory utilization. PTTMonitor logs observed values to the repository for analysis in the results charts.

Deploy the test on a Grid or Cloud test environment by clicking Add Cloud.

TestMaker uses the Name value in the results charts. Type is Grid, Cloud, or EC2. Read the Test Deployment chapter for details.

Monitoring Scenarios

PTTMonitor is an agent based service that observes CPU, network, and memory utilization. PTTMonitor logs observed values to the repository for analysis in the results charts. The Monitoring tab identifies the location of PTTMonitor services running on the TestNodes and Hosts. A Host is a server running your application server or database server.

Use the Monitoring Tab to identify the location of Glassbox service agents. Glassbox observes memory leaks, thread deadlocks, slow database queries and other aspects of performance for Java Enterprise Applications. Read the Glassbox chapter for details.

Scenario Options

The Options tab controls test timing, logging, HtmlUnit headless Web browser, Selenium RC service, proxy service, NTLM authentication, bundles, and resources.

Options are grouped by function. The following sections describe the groups and specific options as appropriate. Many of the scenario options are self-explanatory. This section lists some of the options that need more description.

Timing

Load test time, per virtual user level sets the amount of time TestMaker operates a concurrently running simulated user level (CRLEVEL) in a load and performance test. This value does not include the time for setting up the test‚ when the TestNodes instantiate threads‚ or the time it takes to teardown the threads.

Sleep time at the end of a use case sets the duration TestMaker waits between running a concurrently running simulated user level (CRLEVEL) in a load and performance test.

Ramp-up speed sets the delay time between starting concurrently running users. TestMaker creates one Thread for each simulated user. This option facilitates pacing the amount of time it takes to stage a load and performance test.

Logging

To log to a database, specify the driver, the database URL, and credentials. Driver options are:

• Derby - the default for Apache Derby. Driver is Derby, URL is of the form jdbc:derby://host-name-or-url:1527/testmaker, and user name and password are credentials for the testmaker database.
• Oracle - Oracle RDBMS
• MySQL - MySQL RDBMS
  

Create the testmaker database using the TestMaker Tools menu, Preferences command. Choose the Repository tab, set the database input values and click the Create DB button.

Headless High-Speed Selenium Runner (SeleniumHtmlUnit)

Save received responses to temp directory - For each GET and POST issued from HtmlUnit to your Web application, HtmlUnit saves the server responses to a new file in the temp directory

Log to XML output - No longer supported

Supress JavaScript warnings - Hides warning messages when JavaScript functions throw exceptions or JavaScript resources fail to load

Supress CSS warnings - Hides warning messages when CSS resources fail to load

SeleniumHtmlunit Version - Identifies the HtmlUnit version to use when running the test

SeleniumHtmlunit log level - Log levels include Info, Debug, Error

Selenium

Selenium RC Service Enabled  - Automatically starts the integrated Selenium RC service in the TestNode at test run time.

Synchronized - Overrides the browser's ability to asynchronously load and operate JavaScript functions. This sometimes solves Ajax thread deadlocks.

HAR

TestMaker runs HTTP Archive (HAR) files. HAR makes HTTP protocol-level GET and POST requests to a Web application host to optimize Browser page element performance. TestMaker loads the resources for a Web page just as the browser does, in pools of concurrently running HTTP protocol connections. Thread Pool identifies the size of the pool.

Selenium Remote Control (Selenium RC)

Identifies the port numbers and Firefox profile template directory. Select Trust all SSL Certificates to automatically register new certificates for sites using HTTPS.

Proxy for Selenium Htmlunit, soapUI, designerscript and sahi

Identifies the network proxy service to use when connecting to external network resources.

NTLM Authentication

Identifies user credentials to connect to network resources that are protected by NTLM authentication.

Bundles

This section lists bundles that are associated with the test scenario. Click Add Bundle to add another bundle to the list. Fill in the bundle name and version. To remove a bundle, click the red X.

Resources

This section lists resources that are associated with the test scenario. Click Add Resource to add another bundle to the list. Fill in the bundle name and version. To remove a bundle, click the red X.

Scenario Notifications

Instead of tapping your foot waiting for scenarios to finish, use the Notifications tab to tell TestMaker to send you an email when it's finished. The Notifications tab is a list of emails that will be sent for a test scenario.

Fill in the email address and the subject line you'd like to see in the To and Subject fields. Choose an event from When, one of the following:

• Test starts
• Test ends
• Transaction fails means the email is sent if something goes wrong.
• Hourly means that TestMaker sends a message every hour while the test scenario is running, kind of like a very slow heartbeat.

You can set up as many notifications as you like. Click Add Notification to add another one to the list. To remove a notification, click on its red X.