Archived PushToTest site

Automatic Test Deployment To A Grid or Cloud Environment

Automatic Test Deployment To A Grid or Cloud Environment

TestMaker runs tests on your test equipment, in PushToTest´s OnDemand Cloud Computing environment, or both! Test to millions of virtual users for very little cost.

Register for new screencasts and webinar announcements

The Camtasia Studio video content presented here requires a more recent version of the Adobe Flash Player. If you are you using a browser with JavaScript disabled please enable it now. Otherwise, please update your version of the free Flash Player by downloading here.

Watch new screencasts

Automatic Test Deployment

To A Grid or Cloud Environment

Download TestMaker and Run Selenium Tests  Watch the Tutorial Screencast

Hi. This is Frank Cohen. I’m the CEO and founder at PushToTest, the open source test automation solutions company.

We’re very happy to introduce TestMaker 5.3 this week and I’m glad that you’ve decided to spend a few minutes of your time to learn more about the cloud testing features that are built into TestMaker 5.3.

If you go to the website, you’re able to download TestMaker 5.3, and we provide the product in two different versions. One is the free open source Community version that lets you do things like repurposing Selenium tests into load tests, using soapUI to test web services, and even to repurpose Java unit tests or unit tests written in any other language like Ruby or Groovy or Python into functional tests, load and performance tests, or even into business service monitors.

Here on the PushToTest site, we show you the difference between the Community version and the Enterprise product. The Community edition is enough for you to do load testing, service monitoring, and functional testing. It can also turn tests that have already been created into data-driven tests using things like comma-separated value files. We even include the PushToTest Monitor to correlate CPU, network, memory utilization, and other resource monitoring to the tests that are operating. And the Community edition provides about 25 different charts and reports. Of course, the Community is responsible for giving you support and you have access to all of the documentation on the product.

The Enterprise product builds on the Community edition in that not only it provides all of the Community features, but it also incorporates Selenium, TestGen4Web, soapUI Pro, Windmill, and Mozmill support. The data-driven testing is expanded to include getting data from relational databases and from data-generating objects. We even include now the Glassbox technology to do root cause analysis and mitigation. You can do mock service creation. We provide cloud testing environments to do Test OnDemand, which is what I’m going to talk about in just a moment. And also, we now include testing of Flash and Flex applications using Flex AMF Protocol Handler.

The TestMaker environment is a distributed testing environment, meaning that you’re actually running the test on a set of distributed web services that we call TestNodes. This kind of distributed test environment has always been ready for cloud environments and now what we see is that with the cloud computing providers getting their pricing down to essentially commodity pricing that we’re able to run tests at user levels never before possible.

I remember just nine months ago begging Sun Microsystems to let me use one of their QA labs to run a test at 10,000 users. Well, now with the cloud testing environment capability built into TestMaker, I’m able to avoid the expensive charges of building my own test lab and instead run on a cloud computing environment provider, and now I can achieve tests in the millions of user levels at very inexpensive rates.

What I’m also able to do with cloud testing is to be able to run the tests from multiple locations around the world. So if I wanted to compare say the performance of my application running from Europe and running from mainland United States, I can run in two different cloud test environments and compare the results.

The other big advantage is that now I can pay as I go and I only pay for what I am actually using. So I no longer have to bear the cost of building and maintaining a test lab for doing performance testing. Instead, now I can just pay as I go and operate the test.

And then lastly, many applications are written to run behind the firewall but some are also written to run on the public internet. With the new cloud testing capability in TestMaker 5.3 I can test in my own environment, in the cloud environment, or both, and that gives me ultimate flexibility.

Let me show you a little architectural drawing about how this works. The PushToTest TestMaker console installs as a Java Swing-based desktop application and so that means I can install the PushToTest TestMaker console on my Windows machine, on my Macintosh, on my Linux machine, on my Solaris machine, you name it. It will run anywhere Java runs.

The console acts as a test run time by communicating over SOAP protocols to a web service that we call a TestNode. The TestNode has various different script runners built into it. So one script runner for example will run Selenium recorded tests as a functional test, a load test, or as a service monitor.

Download TestMaker and Run Selenium Tests  Watch the Tutorial Screencast

Another script runner will do the same thing using soapUI and then of course we have script runners for all of the other types of tests that we support, including unit testing with unit tests written in Java or a variety of other languages.

So your test gets moved from the PushToTest console into the TestNode and then it’s operated from there, and the test then interacts with the application that’s running on say a Java application server. It’s not limited to just Java app servers. It can also be .NET or C or Apache or PHP. You name it.

Well, what we’ve done with TestMaker 5.3 is to allow you to provision the TestNodes to be in a cloud provider. So for example, the PushToTest OnDemand service now partners with Amazon Web Services, GoGrid and Rackspace and others, and that allows you to run the TestNodes within these distributed cloud computing environment providers.

So the advantage of this architecture is now I could run my TestNodes from within my own QA lab or I could run them within the cloud test provider; and of course, if I am running it within a cloud test provider, I can just identify the number of TestNode instances I wanted. Like let’s say I wanted to just run two and then later on 200 and then 2000.

So let me show you an example of this working. Here is the PushToTest TestMaker console and the console itself like I said is a Swing-based desktop application. Up on the top of the user interface are commands or controls for opening a test, creating new load test, service monitor. In the middle then is Emacs style text editor for working with configuration files.

And then when I open a test itself, it opens as this graphical window. This graphical controller then lets me run the test, pause it or stop it. I can edit the parameters or the orchestration instructions of the test. And this test actually is already finished running on one of the TestNodes and you can see the results of the real time scalability index. As the test was operating, it’s showing me the number of transactions per second that the application was able to accomplish at various different levels of load, so moving from say one user to two users to four users.

If I take a look at the orchestration instructions for this test, it’s using a simple test scenario, XML definition, and the orchestration instructions tell you for example where the TestNodes are that you want to run the test on.

So it used to be the case that if we wanted to run on multiple TestNodes that we would insert multiple node entries and that was great. But you know, when you’re working in a cloud environment, there’s no reason why you can’t tell the cloud, “Hey, I need X number of TestNodes and please run the test on them.” So what we’ve done is to create a new set of tags for this orchestration instruction that deals specifically with our cloud testing support, and what this is going to do is say, “Hey, I want you to run the test by provisioning a certain number of clouds.” So we can name what the instances are going to be. We can identify the number of TestNodes that we want created. In this case, I’m telling it 40.

And then the next three tags are just incredibly powerful. So for example, the instantiated tag will tell TestMaker to go to the cloud provider and create X number of instances on the TestNodes. You can also tell it instead of instantiating all of them from scratch see what TestNodes are already up and running on that cloud provider and make use of the ones that aren’t in use.

After the test runs, then the terminate tag will tell TestMaker what to do with the TestNodes that are operating the test. For example, you could tell TestMaker to take all of them down. And then you could also tell it instead of taking them all down, just reboot them and get them ready for the next test. What this means is that I can do automatic initiation and termination of the test injectors in the cloud automatically and I can do all of this lights out.

Now, we’re not limited to running a test just in one cloud at a time. As a matter of fact, if I scroll down a little bit here, you can see that this is a definition where we’re going to run a test on the western United States datacenter of Amazon EC2 Web Services, and at the same time run we’re going to run on the eastern datacenter that Amazon has running. And when we see the test results, we’ll be able to compare which datacenter was able to operate more tests against the application.

You might have noticed also that there is a configuration file tag here and what this does is refer to an external configuration file that you can create using the built-in cloud configurator utilities that are built into TestMaker.

Download TestMaker and Run Selenium Tests  Watch the Tutorial Screencast

So here in the tools dropdown menu for example is the cloud configurator for working with Amazon Web Services. The cloud client itself works with this configuration file so you can open and save them. And then for Amazon, this acts as a client to the Amazon web service by identifying your account name and then the access key and the security access key that Amazon will provide for your account.

Under machine profile configuration, you can then identify an Amazon machine image or an AMI file, identify what type of machine you want to run this instance on. Amazon gives you a choice between kind of small, medium, and large sizes. And then of course, we fully support the security group settings that Amazon supports to that you can have dynamic configuration of firewall settings.

We’ve also included under the tools dropdown menu the capability of kind of doing a pre-flight check of the test by making sure that your test configuration actually works with the cloud provider. What we can also do is check to make sure that all of the machines have started and to identify what the details are for each machine, like for example what their public DNS name might be.

We can also print out or display a dynamic list of the functional machines that are ready to run the test and then we have various controls here for starting and stopping and rebooting the individual machines that are running.

So to give you a summary of what we’ve seen, the new cloud testing support in TestMaker 5.3 is groundbreaking. The benefits are that you’ll be able to greatly reduce the cost to operate your test. You’d be able to pay as you go by using the test equipment as you run the test and not having to pay for an expensive quality lab.

You’ll be able to scale up your test to new levels of virtual users that were never before available. You could even go up to millions of users. TestMaker does all of the work including initiation and termination of the test injectors in the cloud automatically. We also provide now a Test OnDemand service where PushToTest provides you with the cloud environment and can provide with all of the service and support to get that environment up and running very quickly.

TestMaker 5.3 features support for Amazon EC2 Web Services, GoGrid, and CollabNet Lab Manager which previously was named Cubit. We expect the Rackspace support to come online very shortly.

So with that, I’m glad to be able to present to you the new cloud testing features that are in TestMaker 5.3 and we hope that you’ll download the product and start your own evaluation of TestMaker test automation.

Thank you for your kind attention.

Download TestMaker and Run Selenium Tests  Watch the Tutorial Screencast


While The Screencast Loads Consider Joining...

Join the PushToTest social network of testers, developers, business managers, and architects for these free services:

- 6 Free Webinars on Testing Best Practices Each Month
- 4 Free How-To Articles Each Month
- New Improvements To The Open Source Test Tools Every Week

Join 22,100 subscribers today. Complete this form. You may unregister (opt-out) at any time.