Cloud Testing with TestMaker

PushToTest TestMaker is a distributed test environment. TestMaker runs your tests on your test equipment, in a Cloud Computing environment, or both. fc

TestMaker 5.3 introduces specific commands to support automatic Cloud Testing. For example, identify a cloud testing service like Amazon EC2 in a TestScenario. TestMaker creates the TestNodes in EC2 instances, runs the test, retrieves the results, and takes down the EC2 instances. All in a "lights out" manner for full Cloud Test automation.

PushToTest provides Cloud Test OnDemand services to host your tests.

Learning About Cloud Testing and TestMaker

We recommend the following learning resources.

• Introduction to TestMaker Cloud Testing screencast
• Take a Cloud Testing Training Class offered by PushToTest Global Services
• Request a live demonstration of TestMaker Cloud Testing

Features and Benefits

Cloud Computing provides an huge new benefit to Open Source Test solutions. Cloud Testing enables the following:

• Run At User Levels Never Before Possible - Building a test lab used to be prohibitively costly for medium and large scale load tests. Cloud computing delivers bandwidth and computing power to achieve tests in the millions-of-users levels at commodity prices.
• Run From Multiple Locations Around The World - Cloud Testing environments are a world-wide phenomenon. TestMaker runs tests in multiple Cloud Testing environments. This makes it possible to manage performance from your users locations.
• Pay As You Go, Pay Only For What You Need - Cloud Testing eliminates the cost of building and maintaining a test lab for load and performance testing. No longer is there a need to provision expensive and difficult to manage quality test labs. PushToTest Cloud Testing lets you pay as you operate tests.
• Test In Your Environment, In A Cloud Environment, Or Both - PushToTest delivers ultimate flexibility.

Architecture

PushToTest TestMaker is a distributed test environment. TestMaker is a console and runtime to operate functional tests, load and performance tests and business service monitors in a distributed network of TestNodes. Install and run the tests on TestNodes in your QA lab, in a Cloud Test Provider, or a combination of both.

The PushToTest technology runs on Cloud Test environments from PushToTest OnDemand, Amazon Web Services (EC2,) GoGrid, and RackSpace. PushToTest supports Cloud services for developer, including Collabnet Cubit. PushToTest provides integrated clients to Amazon EC2 and Collabnet Cubit.

How Does It Work?

PushToTest TestMaker uses a TestScenario to define the orchestration of a test. TestScenario defines the locations of the TestNodes to operate a test using the following notation:

<testnodes>

<node location="http://localhost:8080/TestNetwork/ws/TestNode" name="localhost"/>

</testnodes>

Running a test on multiple TestNodes in your QA lab requires additional <node> entries. For example, the following shows a TestScenario to run a test on 5 TestNodes in your QA lab.

<testnodes>

<node location="http://qa-lab-node-1:8080/TestNetwork/ws/TestNode" name="node-1"/>

<node location="http://qa-lab-node-2:8080/TestNetwork/ws/TestNode" name="node-2"/>

<node location="http://qa-lab-node-3:8080/TestNetwork/ws/TestNode" name="node-3"/>

<node location="http://qa-lab-node-4:8080/TestNetwork/ws/TestNode" name="node-4"/>

<node location="http://qa-lab-node-5:8080/TestNetwork/ws/TestNode" name="node-5"/>

</testnodes>

TestMaker Cloud Testing extends this notation to define a Cloud Testing environment. For example, the following shows a TestScenario that runs a test of 40 TestNodes in the Amazon EC2 environment.

<testnodes>

<cloud

config="{scenario}/ec2.config"

type="ec2"

name="ec2-node-"

amount="40"

instantiate="all"

terminate="all"

reboot="before" />

</testnodes>

The above notation tells TestMaker to provision 40 TestNodes on the Amazon EC2 service. TestMaker will instantiate the TestNodes at the beginning of the test and then terminate the TestNode instances after the test finishes.

TestMaker communicates with Amazon EC2 using configuration stored in a file named ec2.config. Create this file using the integrated EC2 Client for TestMaker. The configuration file stores your Amazon security keys and EC2 specific instructions for running the TestNode.

PushToTest believes the flexibility and power of this approach to Cloud Testing is unmatched. For example, Cloud Testing means you never have to list and name the TestNodes again. Create groups of TestNode server hardware based on hardware and operating environment combinations. For example, use a cloud to create Rack1 with dual CPU and 2 gigabytes of memory running Windows and Rack2 with single CPU and 1 gigabyte of memory running Linux.

Combine multiple cloud test environments. For example, the following shows a TestScenario that operates the test in multiple locations.

<testnodes>

<cloud

config="{scenario}/ec2-westernUSA.config"

type="ec2"

name="ec2-node-west-"

amount="20"

instantiate="all"

terminate="all"

reboot="before" />

<cloud

config="{scenario}/ec2-easternUSA.config"

type="ec2"

name="ec2-node-east-"

amount="20"

instantiate="all"

terminate="all"

reboot="before" />

<node location="http://qa-lab-node-1:8080/TestNetwork/ws/TestNode" name="node-1"/>

<node location="http://qa-lab-node-2:8080/TestNetwork/ws/TestNode" name="node-2"/>

<node location="http://qa-lab-node-3:8080/TestNetwork/ws/TestNode" name="node-3"/>

<node location="http://qa-lab-node-4:8080/TestNetwork/ws/TestNode" name="node-4"/>

<node location="http://qa-lab-node-5:8080/TestNetwork/ws/TestNode" name="node-5"/>

</testnodes>

The above example runs a test from two cloud provider environments and in your QA lab.

Use the same flexible notation to run tests on multiple types of Cloud Test equipment. For example, create a test using inexpensive Amazon EC2 c1.small instances that then run the test on more expensive c1.medium instances.

Cloud Testing Options and Syntax

TestMaker provides the following syntax and options to run tests in a Cloud Testing environment.

<cloud

config="{scenario}/myconfiguration.config"

type="ec2"

name="cloud-name"

amount="20"

instantiate="all"

terminate="all"

reboot="before" />

Cloud Environment Clients

TestMaker provides a Cloud Test client to identify cloud provider specific security tokens, names, and parameters. TestMaker provides integrated clients for Amazon EC2 and Collabnet Cubit Cloud Testing environments.

EC2 Cloud Client for TestMaker

Access the EC2 Cloud Client in the TestMaker Tools drop-down menu EC2 Client command. The client appears in its own window.

The File drop-down menu controls opening and saving configuration files. The Tools drop-down menu provides functions to manage TestNode instances in the Amazon EC2 service, including options to view a list of running instances, start and stop instances, and reboot instances.

Collabnet Lab Management (Cubit) Client for TestMaker

Collabnet Lab Management (formerly Cubit) enables IT users to configure software development and operations environments, upload software assets, set up and tear down lab environments, and manage projects. Lab Management integrates with TeamForge (formerly Source Forge Enterprise) for Application Lifecycle Management (ALM.) Lab Management provisions from a pool of virtual and physical servers in your private corporate data center or a public cloud, including Amazon EC2.

Access the Collabnet Lab Management Client for TestMaker in the TestMaker Tools drop-down menu Collabnet Cubit command. The client appears in its own window. The File drop-down menu controls opening and saving configuration files. The Tools drop-down menu provides functions to manage TestNode instances in the Cubit service, including options to view a list of running instances, start and stop instances, and reboot instances.

Amazon EC2 AMI Profile for TestNode and PTTMonitor

PushToTest provides public Amazon Machine Imagine (AMI) files for the TestNode/PTTMonitor. The profiles are completely configured TestNode and PTTMonitor instances. The profiles are stored in publicly available buckets in the S3 system. Search for PushToTest in S3.

Resources

Consider the following resources to help you:

PushToTest Test OnDemand Service

TestMaker Cloud Testing User Support Forum