Archived PushToTest site

Introduction To TestMaker 6

Introduction to PushToTest TestMaker

This section of the TestMaker User Guide provides a quick start to learn the TestMaker platform, including concepts, test authoring, test deployment, and results analysis topics.


This section contains the following sections:




Learning TestMaker

Source Code and License

Supporting Operating Environments

Test Construction

Test Operation

Root Cause Analysis and Mitigation

Support Options

QuickStart - What Are You Here To Learn?

Pick a test topic and follow the quick start instructions:

  • Functional Testing for Smoke Tests, Regression Tests, Integration Tests - Use the TestMaker Designer to build a test script, play the functional test in a TestNode. Read the Ajax Test Tutorial for step-by-step instructions.
  • Load and Performance Testing - Use the TestMaker Editor to define the virtual user levels, use case, and results charts for a load test. Read the Web Test Tutorial for step-by-step instructions.
  • Production Monitoring - Use the TestMaker Editor to determine how frequently TestMaker operates a test. The Web Test Tutorial gives step-by-step instructions.
  • Web Application Testing - TestMaker supports Web 1.0 application testing and Rich Internet Application (RIA, using Ajax, Flex/Flash) testing. For Web 1.0 application testing PushToTest recommends using Selenium. Read the Selenium Tutorial for step-by-step instructions. For RIA testing PushToTest recommends using Sahi. Read the Web Test Tutorial for step-by-step instructions.
  • Flex/Flash Testing - TestMaker supports testing Flex and Flash backend testing using AMF protocols. Read the AMF Tutorial for step-by-step instructions. Use the TestMaker Designer to build a functional test of a Flex, Flash, and SWF component for smoke tests, regression tests, and integration tests. Read the Flex Test Tutorial for step-by-step instructions. Read the Flex/Flash User Guide for details on functional testing of SWF components.
  • Web Service (SOAP, REST) Testing - TestMaker supports testing SOAP-based Web Services with WSDL definitions and REST-based Web Services with WADL definitions. Build these tests using Eviware soapUI. Repurpose soapUI tests as functional tests, load and performance tests, and production monitors in TestMaker. Read the SOAP Test Tutorial for step-by-step instructions.
  • Web Page Object Optimization Testing - TestMaker supports Web page optimization using HTTP Archive (HAR) technology. Read the HAR Test Tutorial for step-by-step instructions.
  • Testing Using Java Unit Tests - TestMaker repurposes Java unit tests to be functional tests, load and performance tests, and production monitors. Read the Java Unit Tests Tutorial for step-by-step instructions.
  • Desktop Application Testing - PushToTest uses TestMaker to automate tests of desktop applications. However, TestMaker does not provide record/playback or desktop object recognition technology.


TestMaker comes with everything you need to build tests, operate tests, and present the results.

TestMaker console   

TestMaker Editor showing open TestScenario and Use Case tab

TestMaker Designer recording a data-driven test

TestMaker Designer

  Functional test controller panel


Load test controller panel

Business service monitor controller panel



PushToTest designed TestMaker around several testing concepts: Test orchestration, repurposing, record/playback, script-driven testing, data-driven testing, and cloud and grid testing.

Test Orchestration

Many organizations need a reliable way to deliver Internet-based applications. They provide software development services and hire testers to build and operate tests. It is common to see the following services in place:

  • Repository – to store and version control software assets, including source code, data, and tests.
  • Bug Tracker – to manage changes to the application software
  • Test Management – to manage requirements, to identify test coverage, manage test efforts, and provide management with business risk assessment inputs
  • Continuous Integration (CI) – to automate software builds and testing as developers commit changes to the repository
  • Monitoring – to close the loop by identifying an operating software applications compliance to its original requirements and operation

With these services in place the organization adds the most appropriate test authoring tool to the project. For example, soapUI works well for building test suites of SOAP and REST-based Web services, Sahi for building automated test scripts for Rich Internet Applications (RIA, using Ajax, Flex, Flash,) and JUnit tests written in Java for building reusable software test objects of Java and BPM applications.

It does not take long for the organization to move from manual testing to automated testing, to move on to data driven automated testing, and to repurpose their tests to be load and performance tests and production monitors. Organization's accomplish test automation by adopting a test orchestration platform

Application under test on the top and test services below

A Test Orchestration solution delivers the following tasks:

  • Test Operation Console – provides automation to the repository to store and version control test assets, including test scripts, operational test data, and test use case definitions. The Designer component provides record/playback capability to build test objects for Web, Ajax, and Flex based applications. The Console is a test runtime that deploys tests to one or more Test Injectors. The Console retrieves the test results and identifies the root cause to performance bottlenecks and functional issues. The Console delivers actionable knowledge in a standardized set of results analysis reports.
  • Test Injector – is a distributed test runner environment. The Test Injector provides test ScriptRunners to operate the tests in their native format, including Selenium, soapUI, Sahi, and unit tests written in Java, Ruby, Groovy, .NET, C#, and command-line shell scripts. The Test Injector Data Production Library (DPL) injects operational test data from relational database, comma separated value (CSV,) and data generation objects into tests. TestMaker calls a test injector a TestNode.
  • The Console and Injector environment is light weight enough to run on developer desktop development machines, test lab environments, virtual environments, cloud environments, and geographically distributed test environments.

Test orchestration components provide test automation to the complete software development lifecycle: Test Authoring, Test Orchestration, and Test Management.


TestMaker repurposes a single test script as a functional test, load and performance test, and business service monitor.

  • Functional Testing - Ensure that an application performs functions according to your design. For instance, a web-based email service delivers messages. The flow (test use case) of a test simulates the user writing a message and the test validating the message is sent correctly.
  • Integration, Compatibility and Regression Testing - Ensure the application is compatible with user’s operating environments, browsers and client software. For instance, a Web browser-based resource planning application needs to work successfully on Microsoft Internet Explorer and Firefox browsers.
  • Load and Performance Testing - Ensure an information system operates functions within an acceptable amount of time at various levels of  concurrent user load. A load and performance test identifies the Scalability Index ensuring the application achieves a Service Level Agreement (SLA) as more and more users operate the application concurrently.
  • Web Page Optimization Testing - HTTP Protocol level testing for optimization of Web pages.
  • Production Monitoring - Proof an organization’s application meets the terms of a Service Level Agreement (SLA.)
Record/Playback and Test Scripting

PushToTest built TestMaker for use by software developers, QA testers, and IT managers. These three groups often have a wide variety of skills, backgrounds, and skills. PushToTest provides many options to build tests:

  • PushToTest built the TestMaker Designer to record and playback test use cases quickly, easily, and graphically. Record/playback is often a good way to start.
  • When your tests require looping, conditionals, and complex data sets it may be time to move to script-driven tests written in Java, Jython, Ruby, Perl, PHP, and Groovy. TestMaker Designer exports recorded tests into these languages.
  • Use Selenium IDE to record tests in the Selenese table form and play them in TestMaker.
  • Use Sahi Recorder to record tests in the Sahi Javascript form and play them in TestMaker.
  • Use Firefox, Firebug, and NetExport to create an HTTP Archive (HAR) file. Play the HAR file in TestMaker to optimize Web pages.
  • Start from scratch and write a script or class that implements a JUnit TestCase in any of the supported languages.

This unique approach from PushToTest provides a new way to leverage any enterprise's software development, QA and IT efforts resulting in major cost-savings to the company.

Data Driven Testing

TestMaker makes it easy to transform tests into data-driven functional tests. TestMaker delivers Data Production Libraries (DPLs) to provide tests with operational test data from comma separated value (CSV) files, relational databases, and Web services. DPLs are used to provide tests with input data and validation data. DPLs also make it easy to reuse a test between development, staging, preproduction and production environments by simply changing the source of the DPL data at test time.

Cloud and Grid Testing

PushToTest TestMaker is a distributed test environment. TestMaker runs your tests on a Grid of your test equipment, in a Cloud Computing environment, or both. TestMaker Grid technology works in your QA lab, in the Cloud, or both. Identify a set of TestNodes running in your QA lab to TestMaker. TestMaker automatically selects the available TestNodes to deploy and operate your tests.

Application under test on the top and test services below

Component Approach

Many organizations pursue a “Test and Trash” methodology to achieve shortened software development lifecycles. For example, an organization may plan a quick product release with changes to 30% of an application in just 8 weeks. Without giving the change much thought, up to 30% of their recorded tests break! PushToTest recommends a component approach to building tests.

Test components perform specific test operations. We write or record tests as individuals components of test function. For example, a component operates the sign-in function of a private Web application. When the sign-in portion of the application changes, we only need to change the sign-in test and the rest of test continues to perform normally.

TestMaker supports the component approach in many ways: TestMaker Designer records reusable test objects, TestMaker Editor organizes test objects into Test Cases with multiple steps and parameterized test use cases, and TestMaker supports test script creation in a variety of dynamic languages (Java, Ruby, Perl, etc.) creates reusable software classes.

TestScenarios identify test use case and data source

TestMaker organizes components of a test into a functional test, load test, and production monitor using a TestScenario. For example, a test of an ecommerce application has a log-in step, search step, and order step. Use the TestMaker Designer to record a component for each step. The TestScenario identifies the recorded components in a test use case. When the log-in screen of the application changes only the log-in component needs to be updated.

TestScenarios define the dynamic operational test data production library (DPL) used by the test use case and components. DPLs inject test data from comma separated value (CSV) and relational database (RDBMS) sources at test run time. This facilitates reuse of components as the application moves from development, to QA regression testing, to load testing, and then production monitoring.

Share TestScenarios and test components among a team of testers, developers, and IT managers using source code repository services like Subversion (svn) and test management tools.

Learning TestMaker

PushToTest makes many resources available to help you learn TestMaker rapidly:

  • Tutorials, White Papers, and Presentations - Contains the latest tutorials, slide decks, white papers, and presentations from PushToTest. Most are presented by Frank Cohen, founder of PushToTest.
  • Screencasts - Interviews and presentations on the latest in Open Source Testing (OST) tools, techniques, and how-to instruction. The Learn To Be A Software Test Expert series shows practical tips and techniques from many experts in the PushToTest community.
  • Consulting Services
  • Case Studies - Read end-user case studies and examples of how the PushToTest open source community uses TestMaker.
  • Forums - Participate in the TestMaker online free community to get answers or help others find solutions.

Source Code and License

PushToTest distributes TestMaker in three ways:

  • GPL licensed Source Code - This is the "Build It Yourself" alternative. TestMaker is a standard Java application with large-scale integration to the supported subsystems. Prerequisites: Java development experience, Internet protocol experience, Javascript development experience, and HTML 5 development experience. PushToTest hosts support of the source code by hosting an open discussion in the Community Forums.

    Fisheye Source Code viewer at
    svn Source Code Repository at
  • TestMaker Community - PushToTest provides a pre-built, tested, and certified TestMaker Installer program. PushToTest distributes TestMaker Community under a free commercial license. TestMaker Community provides all fundamental application performance management and monitoring capabilities for Web and Rich Internet Applications (RIA.) TestMaker Community operates load tests up to 50 concurrent virtual users.
  • TestMaker Enterprise - Designed for organizations running large scale business-critical Web and Rich Internet Applications (RIA using Ajax, Flash, Flex,) Service Oriented Architecture (SOA,) and Business Process Management (BPM) and need world-class training, services and support. TestMaker Enterprise expands on TestMaker Community features, including data driven tests from relational databases, mock service development, Cloud and Grid testing, up to 100,000 results charts, and Flex/AMF testing. PushToTest distributes TestMaker Enterprise under a commercial license that costs between $20,000 and $50,000 USD per year.

Test Construction Considerations

TestMaker provides more than one way to accomplish an organization's test construction needs. This section describes TestMaker's test authoringapproaches and test scale, scripting, and record/playback features.

Test Authoring Approaches and Test Scale

As people who test information systems we arm our organizations with actionable knowledge: Should we release the new version? How many users can the Web site handle? What is the end-user experience when we make changes to our Rich Internet Application (RIA.) Since the answers are rarely 'yes' or 'no' in nature, why would be pick a test tool that offers only a single choice to accomplish a test?

For example, PushToTest Global Services provided load test services to a large organization. The organization needed to determine the response times end users get as a Web application services three increasing levels of concurrent requests: a small level at 500 requests per second, a medium level at 10,000 requests per second, and a large level at 50,000 requests per second.

The organization had been adding Javascript functions to the Web application and some pages have become heavyweight. Users notice long response times on these pages. Some browsers even have a difficult time processing the Javascript functions. This lead to high priority bug reports to the organization's technical support desk.

TestMaker has more than one way of accomplish the organization's test goals. For example, TestMaker repurposes the Selenium scripts written for functional testing to be used in load and performance testing. Within TestMaker is a SeleniumHtmlunit script running engine that operates Selenium scripts using the HtmlUnit browser.

HtmlUnit's footprint in a machine is smaller than Internet Explorer, Firefox, Chrome, and Safari. TestMaker runs more test scripts concurrently as virtual users per TestNode, the load injector component of TestMaker. There are a few cases where HtmlUnit is not be able to simulate the browser, even as the Javascript scripting grows more complex. In these cases, TestMaker provides alternatives:

  • Run the Selenium Scripts in TestMaker using the integrated SeleniumRC script runner. This runs the Selenium script in Explorer, Firefox, Chrome, Safari, and Opera. Tests achieve the needed level of load by using the cloud as a source of hundreds to thousands of TestNodes. TestMaker provides the Cloud Testing ability built-in. Configure how many TestNodes we need in the TestMaker Editor.
  • Convert the scripts to Java unit tests using the TestMaker Designer and rewrite the blocking parts of the test  using the HTTP client package. This approach simulates the calls to the server and it is the lighest-weight of all agents as no Javascript or HTML parsing is involved. Of course, this approach takes a little longer to create the tests than using a record/playback tool.
  • Record an HTTP Archive (HAR) file in the Firefox browser and play the HAR file back in TestMaker. Create a HAR file using the popular Firebug debugger utility for Firefox. TestMaker operates an HTTP protocol-level test using the HAR file as input to a test. TestMaker loads the resources for a Web page just as the browser does, in pools of concurrently running HTTP protocol connections. TestMaker shows the test operation - including drill-down on how long each resource in the page takes to load - using the TestMaker Results Analysis charting and reporting engine.

These choices show TestMaker's usefullness when Selenium by itself does not fit the bill or is too heavy for the TestNode to operate the test at high concurrent virtual user levels. We do not need to rewrite the whole test, just the part of the test where we hit a road block, as the others can be used as-is. We use TestMaker orchestration to make the Selenium and Java unit tests play as a whole in a test use case.

Each of these options have advantages and disadvantages, and an impact on the scale of the machines you will need to operate the test:

TestMaker Approach
Advantages and Disadvantages
Protocol Level Testing
Large volume of virtual users per TestNode (1000+ VUs)
Script-level coding effort for Web apps with client-side business logic
Extra coding to test Web page resources (CSS, Javascript, Images)
Test message-oriented services (BPM, MOM, ESB)
1000+ virtual users per TestNode
HTTP Archive (HAR)
Large volume of virtual users per TestNode (1000+ VUs)
Scriptless test authoring, create HAR files using point-and-click Firefox with Firebug
Automatic testing of Web page resources (CSS, Javascript, Images)
Does not support Ajax, Javascript functions within Web pages
1000+ virtual users per TestNode
HTMLUnit (SeleniumHtmlUnit, SahiHtmlUnit)
Scriptless test authoring using record/playback
Operates client-side business logic automatically
Small volume of virtual users (50 VUs)
Possibly extra test coding for Javascript incompatibility
20-200 virtual users per TestNode
Browser Functional (Selenese RC TestNode or Sahi with Browser)
Browser compatibility testing with the same test script
Visual test operation for test debugging
Very small volume of virtual users (2 VUs)
Slow test operation
2-4 virtual users per TestNode

Script-driven Testing in Java, Jython, Ruby

TestMaker supports script-driven testing in addition to record/playback testing. Script-driven testing is for times when tests need to use conditional execution, looping, branching and complex operational test data sets. TestMaker provides support of script-driven testing with user custom logging, compatibility with the Data Production Library system, and Results Analysis Engine support. TestMaker supports tests written in Java, Jython, Ruby, Groovy, C#, Visual Basic (VB,) Perl, PHP, and Bin/Bash. Read the Script-Driven Testing chapter for details.

Record-Playback Testing in Selenium, soapUI, Sahi

TestMaker repurposes test scripts created in popular open source testing tools:

  • Selenium tests written in Java, Jython, Ruby
  • Selenese table format tests created with Selenium IDE
  • SOAP and REST-based service tests written in soapUI
  • Flex component tests written in TestMaker Designer Script
  • Web and Rich Internet Application (RIA, using Ajax) tests written in TestMaker Designer Script
  • Browser page component tests written in HTTP Archive (HAR)

TestMaker provides an open Executable Object Provider (EOP) framework for adding adding test types to the test runtime environment.PushToTest engineers take 2-3 days to write and test a new EOP. PushToTest Global Services is glad to help you build an EOP.

TestMaker Editor and TestScenarios

TestMaker Editor is a graphic utility for building and configuring TestScenarios. A TestScenario determines the operating instructions to operate a test in a distributed grid and cloud of TestNode test injectors.

TestMaker Editor showing open TestScenario and Use Case tab

Tabs in the Editor control the following:

  • General - test name, type of test, and details. For example, set to a Functional Test the General tab inputs the number of times to repeat operation of the test, including repeat for all data in a given data source
  • Use Cases - defines the tests to operate as steps in a use case, the data sources for each test, and the global set-up and tear-down functions
  • TestNodes - defines the test injector(s) to operate the test, including a grid and cloud test environment
  • Monitoring - defines the location of one or more PTTMonitors, agent-based resource monitoring agents. PTTMonitor observes CPU, Network, and Memoryutilization as TestMaker operates a test
  • Options - defines timing settings for tests, logging parameters, network proxy settings, and options for the subsystems
  • Notifications - defines email message settings for TestMaker to send when starting, stopping, or operating a test

Editor saves TestScenarios to the local file system as XML-formatted files.

TestMaker Object Designer

TestMaker Object Designer is a graphical utility for creating test object components. Designer will be familiar to Selenium IDE and HP QuickTest Pro (QTP) users. Designer has the following features:

  • Designer has object recognizers for Web, Rich Internet Applications (RIA, using Ajax,) and Flex/Flash components.
  • Designer runs in Windows, Linux, and Mac operating environments.
  • Designer records tests in Internet Explorer (IE,) Chrome, Firefox, Safari, and Opera.
  • Designer records tests in Selenium, Sahi, and Flex test script types.
  • Designer uses integrated Sahi network proxy recorded technology to record test objects. 
  • Designer builds data-driven tests using the TestMaker Data Production Library (DPL) system.
  • Designer exports tests to Selenium in Java, Jython, Ruby, Perl, PHP, and Groovy format, Sahi in Javascript, and Selenese table-format files.

TestMaker Designer recording a data-driven test

Designer outputs Designer Script (.ds) files. Designer Script files are XML-formatted files.

Test Operation

TestMaker is a distributed testing environment. TestMaker operates tests in grid and cloud environments. Special care must be paid to ensure your tests are scaled to the server equipment running the test injectors. TestMaker features integration with Continuous Integration (CI) services like Bamboo, Hudson, Cruise Control, and Maven.

Deploying Tests To The Cloud and Grid

TestMaker Console is a desktop application that runs on Windows XP, Vista, Windows 7, Linux, and Mac OS X operating environments. The Console operates tests in a cloud and grid of test injectors called TestNodes. The TestNode operates test use cases that implement real-life scenarios against the Application Under Test (AUT.)

Console sends tests to TestNodes. Results get logged to a RDBMS for charting and presentation

As the test operates TestMaker records every transaction, step, and monitored resource timing value to a central Results Database and Repository. TestMaker provides a Results Analysis Engine to produce 400 or more charts. The resulting reports surface performance bottlenecks and functional issues.

Console is operated by a human operator, from a continuous integration (CI) service, and from a build environment like Maven.

Calibration Testing and Hardware Requirements

TestMaker operates tests on inexpensive and plentiful servers that run the TestNode test injector service. TestNode software is 100% Java and runs on servers that run Java. See the PushToTest site to download a TestNode installer program.

Special care must be paid to ensure your tests are scaled to the server equipment running the test injectors. Run too many virtual users on a TestNode and the test results will include performance bottlenecks in the server running the TestNode. PushToTest recommends its Calibration Testing Methodology to determine the scale of the test running on a TestNode.

A TestMaker user's approach to test construction impacts the scale of a test running on a TestNode. See the Test Authoring section for details.

PushToTest Monitor (PTTMonitor) is an agent based system. It monitors CPU, Network, and Memory resource usage as TestMaker operates tests. TestMaker uses the monitored datato correlate to the front-end results.

TestMaker logs every test operation, test step, and monitored resource to a relational database (RDBMS) service. TestMaker ships with the Apache Derby RDBMS. Derby is satisfactory for evaluating TestMaker and it is not acceptable for production environments. TestMaker supports MySQL, Oracle, IBM DB2, and MS SQL Server.

Continuous Integration and Testing

TestMaker features integration with Continuous Integration (CI) services like Bamboo, Hudson, Cruise Control, and Maven. See the Installation and Integration chapter for details.

Root Cause Analysis and Mitigation

We do all this effort of test construction and test operation in pursuit of delivering actionalable knowledge to our organization. Is the application ready for customers? Ready for users?TestMaker's Results Analysis Engine surfaces performance bottlenecks and functional issues.

Results Analysis Engine

As the test operates TestMaker records every transaction, step, and monitored resource timing value to a Results Database and Repository. TestMaker's Results Analysis Engine produces thousands of charts. Reports surface performance bottlenecks and functional issues. The Results Analysis Engine produces restults charts based on the type of test. For example, when operating a functional test TestMaker operates the test use case with 1 virtual users. The Results Analysis engine shows the test operation - sucess or failure and duration in milliseconds - for each step of the test.

Controller window for functional test showing check mars and duration for each step of the test

When operating a load test TestMaker operates the test use case over-and-over with the number of virtual users defined in the TestScenario. The load test goal is to surface a Scalability Index - a ranking of how close the application under test performance is to linear scalability. An application with linear scalability will correctly serve 100 concurrent users - with the same average response time - as 1,000 concurrent users.

Controller window for a load test, showing a Scalability Index chart

Results Analysis Engine combines the select charts into one report.

Choose from the most popular charts

When operating a Monitor, TestMaker runs the test use case in 1 virtual user periodically. For example, configure a TestScenario to operate a Monitor every 30 minutes. Many times TestMaker users will leave a Monitor up and running for the IT operations team to get a quick view of the health of the application.

Controller window for a Production Monitor, showing pass fail over time

See the Results Analysis chapter for details.

Comparing Results from Previous Test Runs

TestMaker Results Analysis Engine produces reports from previous test runs. Access the Results Analysis Engine from the Tools menu bar, Results Analysis Chart Generator command. 

Selecting charts from a list

Click Execution Dates to see a list of test results to chart. Choose which execution of a test to chart.

Choose a test run from the repository

Results Analysis Engine displays a window showing the required charts.

Choose from the most popular charts

See the Results Analysis chapter for details.

Pivot tables, dashboards, correlation

PushToTest offers Jasper Business Intelligence Suite as a TestMaker Enterprise option for the following features:

  • 10,000 charts from the test results
  • Report GUI Designer
  • Flash charts, maps, and widgets
  • Pivot tables, cubes, and dashboards

Contact PushToTest for details on the BI suite.

Results Repository Choices

TestMaker uses a relational database (RDBMS) for its Repository. TestMaker logs every test operation, test step, and monitored resource. TestMaker comes with the Apache Derby RDBMS. Derby is satisfactory for evaluating TestMaker and it is not acceptable for production environments. TestMaker supports MySQL, Oracle, IBM DB2, and MS SQL Server.

Use the Tools menu, Preferences command to select the Repository tab. TestMaker uses JDBC drivers to communicate to the RDBMS. Internally, TestMaker uses the Hibernate library for object-to-relational data mapping and database schema creation. The Create DB button creates a new database and initializes the logging tables and schema. TestMaker includes Hibernate drivers for Derby, MySQL, and Oracle. Contact PushToTest Global Services for additional drivers.

Choose from the most popular charts

Support Options

PushToTest provides a variety of support options to deliver your test results reliably and quickly and to be successful with TestMaker.Use the Tools menu, Preferences command to select the Repository tab. TestMaker uses JDBC drivers to communicate to the RDBMS. Internally, TestMaker uses the Hibernate library for object-to-relational data mapping and database schema creation. The Create DB button creates a new database and initializes the logging tables and schema. TestMaker includes Hibernate drivers for Derby, MySQL, and Oracle. Contact PushToTest Global Services for additional drivers.

PushToTest provides Open Source Test (OST) technology and services to help you and your organization succeed. We are often asked: "What skills and background does it take for my team to be successful with PushToTest TestMaker?" PushToTest recommends matching Open Source Test (OST) tools and methodology to your team's skills, budget the right amount of time for a test, and having a backup plan in case your team needs help.

Match The Tools and Methodology To Your Team's Skills

The goal of most organizations is to predictably provide information services to customers, vendors, employees and partners. Many times this means developing test skills in your own team (internally or outsourced to an off-shore partner).

Test With Your Internal Team

Train your internal team to stand-up on their own feet to generate and operate tests and understand the results. Identify the key roles and the individuals that will fill those roles. To be successful with PushToTest your team will need to fill the following roles:

  1. 1. Application Domain Expert - An engineer that understands the features, functions and flow of the service and application under test. This engineer provides operational test data as input to the test and assertion data to validate the service responses.
  2. 2. Test Engineer - An engineer that transforms the service application features, functions and flow into a unit test. The unit test uses the native protocols of the service and application, makes requests to the application and validates the response. The engineer creates the unit test using a record / playback tool, scripting language or JAVA class.
  3. 3. Engineering Support Technician - A technician that applies patches to the service and application under test, solves network, firewall and security issues and provides installation and operational support to operate PushToTest TestMaker.
  4. 4. Manager - A decision maker that decides
    a) How much test coverage is satisfactory to the organization,
    b) Decides if the metrics gathered from the testing effort meet the organization's requirements for satisfactory service delivery,
    c) Staffs the testing effort with employees and budgets for tools, training and support.

Sometimes your organization is missing the skills to be successful or they need the most rapid jumpstart available to success. PushToTest recommends:

Test With An Off-Shore Outsource Partner

Many organizations take advantage of off-shore engineering services to staff their development and test efforts. To utilize all the resources effectively, we recommend having a local internal team with an Application Domain Expert, Test Engineer, Engineering Support Technician and Manager. This group can then provide mentoring and technical support services to the test engineering resources of the off-shore partner.

The internal team should develop the initial test use cases and unit tests, train the off-shore partner’s people to run the tests with the TestNodes provided (load test injector services) remotely. Train them to build their own unit tests with your test requirements. PushToTest TestMaker is a distributed test environment where a TestMaker console orchestrates test operation on a distributed network of TestNodes. This is ideal for outsourced development teams, which operate tests remotely on your network of TestNodes.

PushToTest provides free mission-critical resources TestMaker open-source software, self-guided tutorials and community support for your organization to succeed. PushToTest recommends you budget 3 - 4 weeks for your team to develop the required skills on their own to succeed.

Outsource Your Test Projects

Many organizations apply the expert test resources of an outsource test services company. This is often the fastest way to build, operate and analyze your information systems for functionality, load and performance scalability and on-going service delivery SLA compliance through service monitoring.

PushToTest recommends engaging PushToTest Global Services consulting. Use PushToTest’s test experts to solve your test needs. After the consulting engagement your organization keeps the PushToTest TestMaker platform for continued use. Several organizations have succeeded by outsourcing their test projects, including PepsiCo, Measurfed Progress, and The Jackson Laboratory.

Budget The Right Amount of Time

PushToTest TestMaker is a platform for testing a wide variety of software applications. PushToTest recommends budgeting an appropriate amount of time for your team to learn the platform. The following table summarizes previous experiences PushToTest had in working with customers:

3 to 10 days
Train Your Team: Sometimes your organization is missing the skills to be successful or they need a rapid jumpstart to succeed. PushToTest recommends a 3-Day Training Course on PushToTest TestMaker complemented with 5 or more days of PushToTest Global Service Consulting to help your Engineering Support Technician install and configure PushToTest and help your Test Engineers build and operate their first test. In addition, a support agreement to provide your team with access to PushToTest’s Technical Support through email, phone, instant messenger and Web conference.
2 to 3 Days
Identify the Issues in Your Application: For instance, identify broken and invalid HTML in your web pages, servlet redirection issues, URL parameters, cookies for session management, custom security protocol headers and security tokens.
1 to 2 Days
Identify Application Dependencies: Tests frequently succeed or fail based on the availability and accurate function of your application’s dependencies on other applications and services.
3 or more Days
Team Performance Speed: The speed your team implements changes in your QA, Pre-production and Production Environments will impact test development and operation. For instance, one PushToTest customer took 6 hours to implement a change to fix one broken HTML tag for its log-in page.
1 to 2 Days
Remote Access and Assistance: Be prepared to provide remote access to one or more services, either through web desktop sharing, public service access or VPN access.

Off-Shore Test Development Service Available

Did you know that PushToTest offers offshore test development and support at very competitive rates?

- Test development (scripting)
- Selenium, soapUI, TestGen4Web, HTMLUnit, TestMaker and others
- On-going test maintenance
- Tool augmentation, enhancement, and integration
- Integrate testing into your build environment
- New feature development
- Transform your tests into data-driven test

To learn more see:

Pay-As-You-Go OnDemand Test Services Available

Pay-as-you-go testing is now available from PushToTest. Have a test to run but you do not have the budget to buy an entire test package? PushToTest OnDemand is now available! PushToTest TestOnDemand (TOD) provides organizations with test solutions on an as-needed, on-demand basis. TestOnDemand delivers a global test capability that is fully scriptable, fully functional, multi-protocol, and a complete solution to meet your performance test needs.

- Test development (scripting)
- Selenium, soapUI, Sahi, HTMLUnit, TestMaker and others
- On-going test maintenance
- Tool augmentation, enhancement, and integration
- Integrate testing into your build environment
- New feature development
- Transform your tests into data-driven test

To learn more see:

QuickStart Packages Available

PushToTest provides a complete suite of software and services to helpyou succeed in surfacing performance issues in your IT systems. PushToTest software provides an integrated solution for testing and monitoring your services, including Web applications, Web Services, Service Oriented Architecture (SOA,) and Ajax. Through our unique approach our goal is to overcome the common hurdles to achieving effective IT testing, monitoring and management.

For example, the Proof Of Concept (POC) QuickStart is ideal for medium and large size organizations that need to:

  • Quickly and easily conduct a Proof Of Concept (POC) test of the function and scalability of a Web service, SOA, Ajax, and SOAP-based Web service using the PushToTest Platform
  • Certify an SOA, Web service, and Ajax application is ready-for-deployment
  • Test a SOAP-based Web service for functionality and regression
  • With the Proof Of Concept (POC) QuickStart you get everything you need to immediate develop and operate tests:
  • PushToTest TestMaker Platform to author functional tests, includes browser record/playback test authoring, script-driven authoring, and data-driven test authoring
  • Up to 200 Simulated concurrently running users for load and performance tests
  • Eviware soapUI Pro to author tests of SOAP-based Web services. soapUI Pro gives you everything soapUI provides plus: Test Coverage Reports, Mock Services, Virtualized Services, XPath Operators, Datasources, and Groovy Script Repository functions.
  • Distributed TestNode Environment to operate tests on your laptop, in a test lab, and globally
  • Results analysis engine to do root-cause analysis and mitigation
  • Unlimited number of test controllers
  • Bronze Support of soapUI Pro and the PushToTest environment via telephone, email, instant messenger, and Web conference
  • On-site training for soapUI Pro and the PushToTest environment. Advanced training available.

What If My Team Just Doesn't Get TestMaker To Work?

If your team is not able to build and operate tests within 5-7 business days, then we recommend you contact PushToTest. PushToTest offers expert services to get teams up-and-running with tests quickly and efficiently. We have been providing this kind of help to organizations for 10 years. Sometimes your team just needs a little more training, othertimes your team needs a jump-start where PushToTest Services builds your first test as a model for future tests. PushToTest even offers a complete "TurnKey" option where PushToTest experts build, operate, and analyze your application and service performance and optimize the service for better performance and scalability.

Please click here to tell us about your challenges and receive our free response.

Additional documentation, product downloads and updates are at While the PushToTest testMaker software is distributed under an open-source license, the documenation remains (c) 2011 PushToTest. All rights reserved. PushToTest is a trademark of the PushToTest company.