This week I was the speaker at SkillsMatter.com in London. I gave a talk on the PushToTest test methodology and free open-source test automation tools to check Web applications, Web services, and Ajax applications for function, scalability, and performance. SkillsMatter is the new European partner to PushToTest to offer training on our products. In June I am back in London teaching our popular Open-Source Test Automation Bootcamp.

Harry Wood attended my talk and posted a blog entry about his experience at http://www.harrywood.co.uk/blog/2008/04/11/unit-testing-soa-and-mule-talks

-Frank

We've been making very good progress in building TestMaker 5.2. I'm feeling like a proud father and want to show off my child's cleverness. In TestMaker 5 we introduced a new system of ScriptRunners. This is an extensible facility to operating externally created functional unit tests within a PushToTest TestMaker test. We bundled Eviware soapUI and SpikeSource TestGen4Web with TestMaker and provided ScriptRunners for each. As a result of the latest work TestMaker tests share data between ScriptRunners through a Data Production Library (DPL.) For instance, a functional test first makes a SOAP-based Web service request to learn product information codes and then makes a Web application (HTTP) request to make sure customers are finding the products.



PushToTest orchestrates a Functional test by defining the operating parameters and use cases in a TestScenario document in XML format, as illustrated above.

The test identifies a PushToTest Data Production Library (DPL) that will be the exchange medium between the soapUI TestCase and the TestGen4Web recorded test. The soapUI TestCase reads the data value for the SOAP request from a Properties file in the soapUI project. The soapUI TestCase receives the SOAP response and saves the response parameter to the DPL.

The TestScenario instructs PushToTest to then run the recorded TestGen4Web unit test. The test uses the DPL data from the soapUI test to make an HTTP Get request to the Web host. That concludes the functional test.

I am happy to make the above example available for immediate free download at http://downloads.pushtotest.com/tm5/EndToEndTestExample.zip

You will need the latest TestMaker 5.2 (pre-alpha) version to operate the example. See the faq for instructions on getting the code.

Enjoy.

-Frank

I recently received an email asking:

"Is it possible to use PushtoTest to build a test of a 

Web Application that uses Flash?"

It is possible to test Flash applications using PushToTest. We are currently helping FordDirect.com do this. It does not come without some extra work. For instance, by itself there is no record/playback capability to build tests of Flash animations. However, there is a programmatic way to discover the content of ActionScript variables and to activate functions by doing the equivalent of pressing a button or using Flash controls. PushToTest professional support services will instruct you on the approach, how to instrument your Flash animations for testing, and teach our test methodology and testing platform.

-Frank

PushToTest version 5.2 (which should be out this Spring) will feature Selenium integrated into the PushToTest test automation platform. See http://bugs.pushtotest.com/ticket/136 for details.) Selenium users will create functional tests of Web and Ajax applications using the Selenium plug-in to MS Internet Explorer, Firefox, Mozilla, and other browsers. PushToTest users have two choices to run Selenium tests in the PushToTest test automation environment:

1) PushToTest 5.2 will include a Selenium test runner. This runner instantiates a Web browser in a TestNode (our distributed test environment,) operates the Selenium test, and reports the results. This is a powerful choice to run Selenium tests as functional and regression tests, and business service monitors.

2) PushToTest 5.2 includes a Selenium-to-Java and Selenium-to-Jython transformation utility. The utility creates a Java class (or Jython script) that implements the test steps using the HTMLUnit framework. HTMLUnit acts like a browser, including operating Ajax applications using Rhino (the JavaScript engine from the Apache Mozilla/Firefox browsers.) The is a powerful choice to run Selenium tests as load and performance tests.

For both of the above options, Selenium tests receive dynamic operational data at runtime from PushToTest Data Production Libraries (DPLs.) For instance, a Selenium test that operates a sign-in page receives the account number and password from a DPL that accesses data from a comma-separated-value file (or a relational database or custom DPL.)

Additionally, Selenium tests produce results data that PushToTest Results Analysis engine renders into hundreds of charts. And, these charts correlate Selenium test operation to resource utilization (CPU, Network, Memory, Threads) in the back-end server.

We have an "alpha" quality version of the Selenium-to-Jython transformation utility to give to you today. This is from a contribution from Dominique and Olivier at Denali. All you need to do is ask.

-Frank

I often get asked questions like "What does PushToTest do?" PushToTest is a test automation solutions business. We surface scalability and performance problems in Web-based information systems. Our PushToTest solution drives a Web system like a real user and correlates the activity to identify broken functions and hot spots of performance issues. The same system then operates tests periodically through the day to monitor an enterprise's services and provide proof-of-service reports. See our "white paper", http://downloads.pushtotest.com/PushToTest_WhitePaper.pdf for a more complete answer. I often get asked a follow-up question that is usually asked like this: "I was wondering if you could provide me a sample of the outputs the system generates (or can generate)." The PushToTest software produces a summary report that shows the Scalability Index to the application, the resource utilization as the test operated, and the performance as load increased. This summary report is an HTML document and is easily shared among your team members. "Click here", http://downloads.pushtotest.com/tm5/ExampleSummaryReport/index.html for an example summary report. I often also get asked about the PushToTest methodology and techniques to find and solve performance issues. The question is often asked like this: "Once you run a test and the results are not acceptable then how do I leverage the PushToTest tools to find the code that is running slow?" PushToTest version 5.1 keeps track of the steps for each transaction. For instance, the following snippet of a TestScenario implements a test use case where four steps are required for each transaction.

<test>
   <run name="Open Account" testclass="com.pushtotest.WebAppTest"
method="testOpenAccount" langtype="java" />

   <run name="Transfer Funds" testclass="com.pushtotest.WebAppTest"
method="testTransfer" langtype="java" />

   <run name="Check Transfer" testclass="com.pushtotest.WebAppTest"
method="testValidateTransfer" langtype="java" />

   <run name="Close Account" testclass="com.pushtotest.WebAppTest"
method="testCloseAccount" langtype="java" />

</test>

In the above test PushToTest operates four steps: Open Account, Transfer Funds, Check Transfer, and Close Account. PushToTest keeps track of the total transaction time and the time for each step. The time of each transaction and step is stored in the transaction log file. When a particular step takes longer than expected you can break open the transaction logs for the application server and learn what was going on at that point in time. We have been hard at work over the past two months building an Enhanced Results Analysis engine. The new feature will appear in PushToTest version 5.2. It is already working and we are happy to provide it to anyone that "asks", http://www.pushtotest.com/ask_a_question. The new engine makes it much easier to visualize broken functions and hot spots to performance. Consider the Step Report that visually charts the average duration of each step across 10 equal periods of time of a test. The reporting is great but that still leaves the correlation effort up to a software engineer. And, it is a manual process to go from PushToTest logs to the application server logs. Our planned solution is to incorporate "Glassbox", http://www.glassbox.com into PushToTest version 5.2. Glassbox watches a Java application server for performance hotspots as a test operates and identifies the objects and database connection level issues causing the hotspots of performance. "Click here", http://bugs.pushtotest.com/ticket/61 to learn about the integration project. To learn more about Glassbox "click here", http://www.glassbox.com/glassbox/Download.do?mediaItemId=173 to read the Glassbox user guide. One final question I usually hear goes like this: "Where exactly are your developers located? In other words when you leverage out-of-country developers, where are they and if the customer wanted US based developers can you accommodate that." PushToTest outsources its customer engineering to our Costa Rica-based partner "Avantica", http://www.avantica.net. Costa Rica is a 4-5 hour flight to the United States and Canada. The country is on Central Time (GMT -6) and their English is very good. We normally need about one week to plan for our engineers to be in your office. -Frank

Last year PushToTest built a test runner for soapUI and bundled soapUI in the PushToTest version 5 distribution. The advantage to Java developers is three fold:

1) soapUI test suites run in a distributed PushToTest test environment - a grid of test machines each turning out SOAP requests to your application.

2) The PushToTest environment dynamically provides operational data to the soapUI test suites. For instance, the data could be sign-in information or product part numbers. The data production libraries (DPLs) get data from a relational database, from a comma-separated-value file, and from service calls dynamically.

3) PushToTest provides a results analysis engine and performance comparison utility to identify problem test suite performance and problem service calls. Additionally, while the test suites operate the PushToTest Monitor watches CPU, Network, and Memory utilization of the test machines and the target hosting the application and correlates these back to the test suite operation to surface the problem datacenter components.

We increasingly see upper management at our customers look at consumer hits like Facebook, Flikr, Google Maps, and tell their engineers "We want that!" At the same time Java developers in 2008 have good tools to service enable their classes. The interface may be SOAP, or any number of others including REST, Ajax, JMS to a message queue, RMI, etc.

We are seeing customers use PushToTest in service environments. They tend to start service enabling a few classes and wind-up with a grid of interconnected services. While soapUI by itself is a terrific tool for working with individual services, the nature of service enabling Java classes means most organizations need a test methodology, patterns, and tools that work in a grid of services.

Customer success at testing in a grid environment is the motivation behind a new partnership that eviware and PushToTest announced this week. Java developers benefit when two open-source 100% Java projects partner to solve service testing challenges and help us test in a grid environment.

-Frank

We are getting very close to the PushToTest 5.1 release - hopefully this week! New datasheet and white paper downloads are available to explain PushToTest and open-source test automation:

PushToTest 2-Page DataSheet

PushToTest White Paper: The Importance of Test Automation

Both are in Adobe Acrobat (PDF) format.

-Frank

I am happy to report that TestMaker 5.1 is available for immediate download at http://www.pushtotest.com. The new version fixes many bugs and delivers useful enhancements.

We are making this Release Candidate available to get user feedback on the changes. I anticipate we will make 2 additional release candidates to solve bugs and make additional improvements. We will be using http://bugs.pushtotest.com to track the changes.

Thank you to everyone that participated in this release!

-Frank

--

TestMaker 5.1 Changes

• TestMaker 5.0 ships with TestGen4Web 0.49 that fails to install on Firefox 2.0.0.7 or later. TestMaker now ships with TestGen4Web 0.50.2-PTT to be fully compatible with Firefox 2.0.0.7 and later.
• TestMaker 5.1 changes the way TestGen4Web scripts are run in the TestNode environment. The new script runner uses TestGen4Web's htmlunit-interpreter package. This improves TestMaker's ability to natively run TestGen4Web recorded unit tests.
• TestMaker 5.1 offers many minor improvements and bug fixes to the results and reporting capabilities. The charts for a load test now include a legend of concurrent user (CR) levels. We fixed an issue where load tests greater than 12 hours show negative horizontal axis values in the resource distribution charts. We removed the load test summary report from the output panel. We added new options to the TestScenario definition for a load test to automatically save all charts to the results directory and create an HTML-based summary report.
• We fixed many problems with installing and running TestMaker on Mac OS X, including an installation problem that prevented the Resource Monitor (PTTMonitor) from running.
• We updated the documention by correcting many spelling and grammer problems and including new sections for QA Testers and IT management. Additionally, we added new documentation to show the options to log results data directly to a Relational Database Management System (RDBMS)and added documentation to show how to set TestMaker and TestNode memory settings.
• We did an overhaul of the TestMaker classpath and supporting Java Archive Resource (JAR) files.
• TestMaker runs on Windows XP, Windows 2003 Server, and Windows Vista. TestMaker 5.1 and later is no longer compatible with the old Windows 2000 and NT operating environments because of a bug in Windows that limits the size of a classpath.
• TestMaker 5.0 shipped with a typo in the element in a TestScenario. The element was incorrectly named (notice the extra s.) TestMaker 5.1 accepts the correct and incorrect versions.
• Added the File menu -> Open Scenario command to open TestScenarios.
• Added new capability to TestScenarios to control timeout values for usecases (unit tests) that take too long to operate.
• Added an option for the transaction distribution charts to use a moving data average instead of inserting every data point.

A complete list of changes and their associated bug reports is found at: http://www.pushtotest.com/Docs/vreleasenotes

TestMaker test scenarios generate a lot of useful data. TestMaker 5.0 introduced standard charts to visualize the data into actionable knowledge, including the Scalability Index, Transaction Distribution Charts, and Resource Monitor Charts. TestMaker 5.0 also introduces a result log archiving system and a Performance Comparison Utility to generate charts to compare results between operations of test scenarios.

TestMaker 5.2 will introduce new results analysis functions:

1) Transaction logs will include the "step" times for the operations within a

2) A new charting function will summarize logged results data in new and flexible ways.

TestMaker 5 creates a lot of data that can be put into a chart. Here are the base data series:

• Test Scenario (name)
• Test Node (name)
• Concurrent User levels (CRs)
• Data Index (Message size)
• Transaction (Use Case)
• Sequence name
• Step (run) name
• Pass/Fail status

I put together a new presentation to introduce you to the Enhanced Results Analysis functions. Download the presentation at http://downloads.pushtotest.com/tm5/PushToTest_ResultsAnalysis_20071220.pdf

-Frank

Last week PushToTest hosted the first of a new series of 2-Day Hands-On seminars on Test Automation. The event sold out and the room was packed with eager students. Some traveled from Michigan to the seminar in Silicon Valley, California.

Here are some of my notes from the seminar:

1) The students at the seminar were from a rich cross section of backgrounds and experiences. We had software architects, software developers, QA testers, database administrators, and project managers. This shows me that test automation is a multi-discipline effort. It is not enough anymore to throw testing over the wall to a bunch of testers and not worry about the results.

2) TestMaker is in better shape that I had thought. In the days that lead up to the seminar I was concerned that the usability in TestMaker would make the hands-on portions difficult to follow. I was happy to none of the students ran into any roadblocks that kept them from completed the workbook of tasks. (This is not to say that we can put off usability improvements in TestMaker. Usability is a big issue in my eyes.)

3) The students universally told me they are not finding commercial test automation tools that are appropriate for Web application, Service Oriented Architecture (SOA,) Ajax, Web service, and Web 2.0 testing. As a result they often have one tool for Web application testing and another tool for Ajax testing.

4) Many students liked the informal style of the presentations. They were looking to gain an understanding of testing in the 21st century and take home tactics and methodology they will use in their own environment.

5) Several of the students are working in SOA environments and were interested to learn about TestMaker integration with registry/repository products. See here for details

The next seminar events are in London, New York, and then back in Silicon Valley. For details check http://www.pushtotest.com/Docs/training/seminarform.

-Frank

I have been looking into functional test development tools for Windows desktop GUI testing. There are 101+ click-and-keyboard recording utilities out there but they all suffer from not being able to handle any change in the environment. We saw one large hotel/casino in Las Vegas pour $150,000 into script development only to have to throw away the scripts when they applied a patch to one of their applications. Our preference is to have an object-oriented framework to write test scripts that drive an application programatically.

So far I found two promising technologies for Windows desktop applications. pywinauto is a Python (not Jython) based framework for programatically driving a Windows application. It has a very nice object-oriented approach. For example, consider the following code.

from pywinauto import application
app = application.Application.start("notepad.exe")
app.notepad.TypeKeys("%FX")

app.Notepad.MenuSelect("File->SaveAs")
app.SaveAs.ComboBox5.Select("UTF-8")
app.SaveAs.edit1.SetText("Example-utf8.txt")
app.SaveAs.Save.Click()

The downside to pywinauto is it does not currently support some of the custom controls in Microsoft office.

Another promising approach is to combine IronPython and Microsoft UI Automation. UI Automation is Microsoft's latest attempt to provide a single API to programatically access desktop applications.

The downside to both of these approaches is the lack of an available GUI-based recorder. That may be something PushToTest will need to write.

What tools have you looked into? Please post a comment below.

-Frank

The expert group working on an enhancement to Java that will make dynamic scripting languages easier to code and more efficient to run met this week. The JSR 292 group has been quietly considering designs for a new Java byte code instruction called "invokedynamic".

Java 1.6 introduced the JSR 223 ScriptEngine API that makes it possible for TestMaker to support multiple languages to write test suites. With the ScriptEngine TestMaker users are able to write tests in Java, Jython, JRuby, Groovy, PHP, and about 15 more languages.

Some of the supported languages are on-the-fly byte-code compilers. For instance, Jython is the Python scripting language that runs in the Java Virtual Machine (JVM.) At runtime Jython converts a Python script into Java Byte Code and runs the script in the JVM.

For the on-the-fly byte-code compiler script engines, the current JVM lacks an efficient way to dynamically dispatch calls to methods. Enhancing the JVM with an "invokedynamic" instruction will make it much easier to write this kind of script engine.

Originally the JSR 292 project was lead by Gilad Bracha, an expert in object-oriented programming languages, in particular modularity, reflection and type systems. Gilad left Sun last year and John Rose, took the project and has been running with it.

JSR 292 will make significant changes to the JVM instruction architecture. The expert group wants public review of the changes early and is producing an EDR (early draft review required by JCP). To make the current Proof of Concept design public, the expert group needs to informally agree that the design shows a direction that is worth explaining and improving - sometimes called a "red face test."

The original JSR 292 language includes not only invokedynamic but also some sort of class modification or extension. Based on recent experience, the expert groups believes something is needed beyond invokedynamic to support some sort of lightweight behavioral extension (method handles, autonomous methods, etc.)

The group is planning to start an OpenJDK open source sub-project to help create the Reference Implementation (RI) for JSR 292. The group plans to meet again soon but there is no published calendar of milestones or dates.

This is very good news for Java and Dynamic Scripting.

-Frank

PushToTest TestMaker provides three ways to create a unit test that may be then run as a functional test, load and scalability test, and service monitor:

• Use TestGen4Web to record a Web application
• Use the Network Proxy Recorder to watch a browser interact with the host
• Write a unit test using one of the many supported languages

The TestGen4Web utility does not support Web 2.0 and AJAX very well. It does not model the operations in a JavaScript that may include making asynchronous calls to the back-end.

We recommend using TestMaker's scripting support to test AJAX applications. In TestMaker transform the TestGen4Web recorded file into a Jython script (use the Tools menu -> Import TestGen4Web) and change it to emit the HTTP calls to the target host that the JavaScript in your app is doing. Although you may need to learn Jython or one of the other support languages, you will not have to get a PHD degree.

We are investigating a way to use Selenium and HTMLUnit and Rhino to record a test, transform it into a TestMaker test agent script, and then run it in the TestMaker environment.

With our current workload it may be a few months before this is done. We would be happy if a company or individual would like to provide an engineer for 2-3 weeks and we could get this done right away.

-Frank

PushToTest is partnering with ReplicateTech to provide virtual and dynamic testbeds using PushToTest TestMaker 5. Our first step in this partnership is to develop knowledge of how TestMaker operates in the ReplicateTech environment. Today we completed installation of the PushToTest TestMaker 5 environment and the Sun Blueprints Petstore 2 (beta) onto the ReplicateTech lab.

Sun has very much moved the Petstore forward from the current 1.3 version. Petstore 2 is an entirely new application that delivers a rich Internet application using Web 2.0 technologies: AJAX, RSS, mashups, and client-side animation.

We are going to create a set of TestMaker test scripts to test the Ajax capabilities of Petstore 2. We should be ready in a day or two with these scripts to conduct our first scalability test.

Our plan is to use the "snapshot" capability to record the initial state of the Petstore and its Derby database. Then to operate the test at 10 levels of concurrent requests and 3 varieties of sequences. ReplicateTech uses VMWare so the snapshot capabilities will make rolling-back the Petstore to a known initial state easy.

I will be keeping a diary of the experience and use it to write an article for one of the on-line developer communities and to be incorporated into the TestMaker documentation.

Feel free to post your comment about this project.

-Frank

Jython 2.2 is the first major overhaul of the Jython dynamic scripting language in 4 years. Jython 2.2 implements the internal architecture of Python 2.2 and includes numerous features from Python 2.3.

Jython is a Java implementation of the Python language. It allows users to compile Python source code to Java byte codes, and run the resulting bytecodes on any Java Virtual Machine.

Jython 2.2 has support for most of Python 2.2 and numerous features from Python 2.3. The new release - the first major overhaul in 4 years - includes many major changes:

• new-style classes
• Java Collections integration
• PEP 302 implementation
• iterators
• generators
• __future__ division
• support for running on JDK1.5 and 1.6
• new installer
• a significant number of features to bring Jython in line with CPython

The Jython download is at: http://sourceforge.net/project/showfiles.php?group_id=12867

The Jython community meets at: http://www.jython.org

-Frank

The Server Side ran our announcement of TestMaker 5 today at:

http://www.theserverside.com/news/thread.tss?thread_id=46598

We also issued a press release on the new product. Download the press release

If you are so inclined, please post a reply asking a question or telling of your experience using TestMaker. The more you post the more attention we get.

Thanks.

-Frank

I gave a TechTalk at Google yesterday on TestMaker 5. I showed how TestMaker reuses unit tests as functional tests, load tests, and service monitors. Google recorded the TechTalk and makes the video available at:

http://video.google.co.uk/videoplay?docid=-4427578441647658650

-Frank

The new TestMaker 5.0 is now available for immediate download! The new release is the full and complete TestMaker 5 product.

http://www.pushtotest.com/Downloads/registrationForm

-Frank

I put together a 2-page sales sheet on TestMaker 5 to help our salespeople. The sales sheet gives the high-level benefits-orientated features found in TestMaker 5.0.

Download a 2-Page Datasheet on TestMaker, (1.9 Mbyte, Adobe Acrobat format.)

Thanks!

-Frank

PushToTest plans to release TestMaker 5 this week. We wanted to move from CVS to Subversion after the TestMaker 5 release. A blog by Charlie Nutter at Sun's JRuby group makes me wonder if we should move to Mercurial instead. Any feedback, opinions, questions are welcome.

Please see: http://headius.blogspot.com/2007/08/business-case-for-supporting-jython.html

-Frank