Site MapAccessibilityContact Us

Call us at (855) 254-1164 to learn more about Enterprise

PushToTest - Open Source Test

TestMaker Editor

TestMaker Editor User Guide

The TestMaker Editor is a graphic tool for creating and changing TestScenarios. A TestScenario identifies the operating parameters for a functional test, load and performance test, and production monitor. Editor saves TestScenarios in an XML-formatted file. This User Guide describes the Editor and its many options.

For a quick start building tests of Ajax applications, SOAP and REST based Web services, Java-based unit tests, and Flex/Flash applications, consult the Tutorials.

Contents

Editor Geography

The editor window is divided into two parts. The left side shows a list of all the open test scenarios. In this image, three scenarios are open.

The right side of the window shows details for the selected scenario. In this image, Ajax_Load_Test.scenario is selected and its details are shown on the right.

Editor displaying three TestScenarios with the General tab selected

When you run a scenario, it shows up in a separate window where you can see its progress. Here is an example scenario window.

Controller window showing TestNode status as it runs a functional test

Working with Scenarios

The editor supports standard file operations for scenarios through commands in the File menu:

  • Use New Functional Test, New Load Test, or New Service Monitor Test to create a new scenario.
  • Open... brings up a file chooser so you can open an existing scenario.
  • Open Recent includes a submenu full of scenarios you have recently opened.
  • Choose Save to write the current scenario to disk.
  • Save As... is the same as Save, but you get to choose a new file name for the scenario.
  • Execute Scenario runs the current scenario.
  • Close Scenario removes the scenario from the editor window.

Editor updates TestScenarios created with TestMaker 5. When you open a version 5 scenario, editor displays an alert box asking if it may update the scenario. This update is required to run the test in TestMaker 6. Once updated, the scenario will not open in TestMaker 5.

A scenario is a complete definition of a functional test, load test and service monitor. A scenario document follows the XML Schema Definition (XSD) found at http://downloads.pushtotest.com/tm5/testscenario.xsd

Examples of TestScenario documents used in functional, load, scalability and performance tests plus service monitors are found in the Tutorials.

Scenario Mini Toolbar

Each open scenario in the editor window has its own mini toolbar with commonly used controls.

TestScenario selector in left side of Editor window highlighted

The controls in the mini toolbar are as follows:

Minibar close button closes the scenario so it will no longer be listed in the editor

Minibar save button saves any changes you have made to the scenario file

Minibar play button starts running the scenario.

Minibar pause button pauses execution of the scenario

Minibar stop button halts the execution of the scenario

Minibar chart settings button brings up another window where you can control the charts that are generated for the scenario.


General Scenario Properties

The General tab contains overall settings for the scenario, test type, and meta data.

General tab contents with a functional test selected

The Test Scenario Name is the display name of the scenario. This name is shown in the scenario window when you run the scenario. TestMaker uses the name in the Controller Panel, in the status Output Panel and in the results files.

Default Directory defines the path that will be used to find scripts, JAR files and other resources.

Click on the small triangle button to view and edit additional properties, like Author Name, Creator, Links, Versions, and Categories.

Test Type affects how TestMaker works with the scenario. Choose Functional Test, Load Test, or Production Monitor.

Different Kinds of Tests

The options below Test Type vary depending on the type you choose.

Functional Testing

For a Functional Test, you can choose how many times to Repeat this use case. Select Repeat for all data for TestMaker to repeat the test use case for every row of data the test use case accesses using a Data Production Library.

Functional test chosen in General tab

Load and Performance Test

Load tests operate the test use cases in one or more virtual users as many times as possible in a given amount of test time. By running a load test at several levels of virtual users, TestMaker identifies the Scalability Index of your application's scalability.

Choose one or more virtual user levels to operate the test. Click Add virtual user level to add a level. Select Test for all data for TestMaker to run the virtual user level while there is more data using a Data Production Library.

Load test selected

Production Monitor

For a Production Monitor, choose the interval between tests.


About Paths And The Default Directory

PushToTest TestMaker works well in a distributed environment where developers and QA engineers run tests within any directory structure (i.e. wherever they've checked out the project files). TestMaker TestScenario test orchestration documents determine the default directory path for test resources. The following options overriding the default directory and other elements in the TestScenario. For example, consider the following directory structure:

ScenarioDirectory
- Scenarios
- - MyScenario.xml
- Resources
- - resource.csv

TestMaker provides a {scenario} operator that resolves to the path of the TestScenario document. This operator may be used in the element, in elements, and in individual resource paths. For example, use the above resource.csv with

path="{scenario}/../Resources/resource.csv"

The following is also valid:

Defaultdirectory ="{scenario}/../" and path="Resources/resource.csv"

Working with Use Cases

The Use Cases tab is the place where you set up one or more test steps, options for each test step, and setup and teardown steps.

Use Cases tab chosen shows a Designer Script test step

This example runs a single test (AddCompany.ds) with no setup or teardown.

The editor allows for sophisticated scenarios. TestMaker scenarios consist of one or more use cases, each of which can contain one or more test scripts. Each use case can have setup and teardown scripts. In addition the entire scenario can have global setup and global teardown scripts.

To create a test use case click the links, including Add Global Setup or Add Test. Each element appears graphically. Click on X to remove an element of the scenario.

The setup, teardown, and test elements in the scenario all point to scripts. TestMaker knows how to run many kinds of scripts, everything from Sahi and Selenium through to JRuby and Perl. The editor shows options for the script that are appropriate to the script type. For example, you get to choose a browser to run a Sahi script. You get to specify a class name for a Java language script.

Test Types

TestMaker provides support to run many test types. Use the Test Type menu to choose a test type. Every type of test provides fields for Test Name and Instance. TestMaker uses TestName when reporting the results. TestMaker uses Instance to reuse test objects from one test step to another. For example, a test with three steps (log in, search for products, log out) reuses the same Web browser instance when each step uses the same instance value.

Editor displays a special form for each test type:

soapUI
Operates a TestCase in a soapUI or soapUI Pro project
Test Suite name of the Test Suite
TestCase name of the Test Case
Click the Browse button to choose the soapUI Project from the local file system
Selenium
Operates a Selenese Table Format (the output from Selenium IDE) using the HtmlUnit headless Web browser
Click the Browse button to choose the Selenese test script from the local file system
SeleniumRC
Operates a Selenese Table Format (the output from Selenium IDE) using Firefox, IE, Chrome, Safari, Opera
Click the Browse button to choose the Selenese test script from the local file system
Sahi
Operate a Sahi (JavaScript) test script in HtmlUnit, Firefox, IE, Chrome, Safari, Opera
Click the Browse button to choose the Sahi test script from the local file system
Designer Script
Operate a TestMaker Sahi (JavaScript) test script in HtmlUnit, Firefox, IE, Chrome, Safari, Opera
Click the Browse button to choose the Sahi test script from the local file system
Java
Instantiate and run a test object written in Java.
Click the Browse button to choose the JAR file containing the test from the local file system
Class identifies the class name. For example, com.pushtotest.test.MyTest
Method identifies the method to call within the class.
TestMaker uses the class, method, and arguments to determine the signature of the object to run. See the Java_JUnit_Tutorial for an example.
Jython
Instantiate and run a Jython module.
Click the Browse button to choose a .py file from the local file system
Class identifies the module name
Method identifies the method within the module to call
TestMaker stores Jython resources on the TestNode in the ./resources directory for handy access from Jython modules.
Command Line
Run a shell script in the local machine
See tutorials/miscellaneous/CommandLines for an example
Visual Basic
Run a Visual Basic (VB) script
See tutorials/miscellaneous/VisualBasic for an example
.NET (C#)
Run a .NET (C#) script
See tutorials/miscellaneous/DotNet for an example
HAR
Run an HTTP Archive (HAR) file. HAR makes HTTP protocol-level GET and POST requests to a Web application host to optimize Browser page element performance. TestMaker loads the resources for a Web page just as the browser does, in pools of concurrently running HTTP protocol connections.
See tutorials/HAR_Tutorial for an example
PHP
Run a PHP file
Click the Browse button to choose the .php file from the local file systems
Perl
Run a Perl file
Click the Browse button to choose the .perl file from the local file systems
JRuby
Run a Ruby file
Click the Browse button to choose the Ruby file from the local file systems
Groovy
Run a Groovy script
Click the Browse button to choose the Groovy file from the local file systems

    Running Multiple Use Cases Concurrently

    When you have more than one use case, TestMaker will run all the use cases concurrently as a sequence. For example, when running a load test of 100 virtual users, the test use case with 80% proportion runs in 80 of the 100 virtual users concurrently.

    Choosing the use case percentage in a sequence

    Adjust the percentage of time allotted to each use case by adjusting the Sequence proportion value.

    Configuring TestNodes

    The TestNodes tab is the place where you tell TestMaker where to run a scenario. You can specify any number of TestNodes, Grids, and Clouds. Read the Test Deployment chapter for details.


    TestNodes tab chosen to determine where TestMaker will run the test

    The TestNode has a human-readable Name and a URL where is it available. Monitor port is an optional value to determine the port number for PTTMonitor service running on the TestNode. PTTMonitor observes CPU, network, and memory utilization. PTTMonitor logs observed values to the repository for analysis in the results charts.

    Deploy the test on a Grid or Cloud test environment by clicking Add Cloud.

    Grid/Cloud entry in the TestNode tab

    TestMaker uses the Name value in the results charts. Type is Grid, Cloud, or EC2. Read the Test Deployment chapter for details.

    Monitoring Scenarios


    PTTMonitor is an agent based service that observes CPU, network, and memory utilization. PTTMonitor logs observed values to the repository for analysis in the results charts. The Monitoring tab identifies the location of PTTMonitor services running on the TestNodes and Hosts. A Host is a server running your application server or database server.

    Monitoring tab selected

    Use the Monitoring Tab to identify the location of Glassbox service agents. Glassbox observes memory leaks, thread deadlocks, slow database queries and other aspects of performance for Java Enterprise Applications. Read the Glassbox chapter for details.


    Scenario Options

    The Options tab controls test timing, logging, HtmlUnit headless Web browser, Selenium RC service, proxy service, NTLM authentication, bundles, and resources.

    Options tab selected

    Options are grouped by function. The following sections describe the groups and specific options as appropriate. Many of the scenario options are self-explanatory. This section lists some of the options that need more description.

    Timing

    Load test time, per virtual user level sets the amount of time TestMaker operates a concurrently running simulated user level (CRLEVEL) in a load and performance test. This value does not include the time for setting up the test‚ when the TestNodes instantiate threads‚ or the time it takes to teardown the threads.

    Sleep time at the end of a use case sets the duration TestMaker waits between running a concurrently running simulated user level (CRLEVEL) in a load and performance test.

    Ramp-up speed sets the delay time between starting concurrently running users. TestMaker creates one Thread for each simulated user. This option facilitates pacing the amount of time it takes to stage a load and performance test.

    Logging

    To log to a database, specify the driver, the database URL, and credentials. Driver options are:

    • Derby - the default for Apache Derby. Driver is Derby, URL is of the form jdbc:derby://host-name-or-url:1527/testmaker, and user name and password are credentials for the testmaker database.
    • Oracle - Oracle RDBMS
    • MySQL - MySQL RDBMS

    Create the testmaker database using the TestMaker Tools menu, Preferences command. Choose the Repository tab, set the database input values and click the Create DB button.


    Headless High-Speed Selenium Runner (SeleniumHtmlUnit)

    Save received responses to temp directory - For each GET and POST issued from HtmlUnit to your Web application, HtmlUnit saves the server responses to a new file in the temp directory

    Log to XML output - No longer supported

    Supress JavaScript warnings - Hides warning messages when JavaScript functions throw exceptions or JavaScript resources fail to load

    Supress CSS warnings - Hides warning messages when CSS resources fail to load

    SeleniumHtmlunit Version - Identifies the HtmlUnit version to use when running the test

    SeleniumHtmlunit log level - Log levels include Info, Debug, Error

    Selenium

    Selenium RC Service Enabled  - Automatically starts the integrated Selenium RC service in the TestNode at test run time.

    Synchronized - Overrides the browser's ability to asynchronously load and operate JavaScript functions. This sometimes solves Ajax thread deadlocks.

    HAR

    TestMaker runs HTTP Archive (HAR) files. HAR makes HTTP protocol-level GET and POST requests to a Web application host to optimize Browser page element performance. TestMaker loads the resources for a Web page just as the browser does, in pools of concurrently running HTTP protocol connections. Thread Pool identifies the size of the pool.

    Selenium Remote Control (Selenium RC)

    Identifies the port numbers and Firefox profile template directory. Select Trust all SSL Certificates to automatically register new certificates for sites using HTTPS.

    Proxy for Selenium Htmlunit, soapUI, designerscript and sahi

    Identifies the network proxy service to use when connecting to external network resources.

    NTLM Authentication

    Identifies user credentials to connect to network resources that are protected by NTLM authentication.

    Bundles

    This section lists bundles that are associated with the test scenario. Click Add Bundle to add another bundle to the list. Fill in the bundle name and version. To remove a bundle, click the red X.

    Resources

    This section lists resources that are associated with the test scenario. Click Add Resource to add another bundle to the list. Fill in the bundle name and version. To remove a bundle, click the red X.

    Scenario Notifications

    Instead of tapping your foot waiting for scenarios to finish, use the Notifications tab to tell TestMaker to send you an email when it's finished. The Notifications tab is a list of emails that will be sent for a test scenario.

    Notifications tab for email notification of test operations

    Fill in the email address and the subject line you'd like to see in the To and Subject fields. Choose an event from When, one of the following:

    • Test starts
    • Test ends
    • Transaction fails means the email is sent if something goes wrong.
    • Hourly means that TestMaker sends a message every hour while the test scenario is running, kind of like a very slow heartbeat.

    You can set up as many notifications as you like. Click Add Notification to add another one to the list. To remove a notification, click on its red X.

     

    PushToTest logo

    Additional documentation, product downloads and updates are at www.PushToTest.com. While the PushToTest TestMaker software is distributed under an open-source license, the documentation remains (c) 2011 PushToTest. All rights reserved. PushToTest is a trademark of the PushToTest company.