Matt at Appcelerator pointed me to a discussion of Automated Testing of Appcelerator With Selenium on the Appcelerator user forums. I am very interested in ways to use Selenium for testing Appcelerator applications.

From TestMaker's perspective a Selenium test is a unit test of a Web application or Ajax application. TestMaker already provides Test Runners for TestGen4Web, soapUI, and the Java 6 ScriptEngine support for unit tests written in Java, Jython, JRuby, Groovy, and a bunch of others.

It seems to me that the forum members have already hit upon some of Selenium's limitations and design problems:

a) Selenium was not really developed for Ajax, and vice versa. Ajax frameworks like Appcelerator implement their own eventing mechanism and Selenium has no idea of how this works. For instance, when an Appcelerator log-in page changes the DOM to reveal site content Selenium IDE does not notice the change. Selenium IDE will sometimes record a 'click' when 'clickAndWait' is appropriate. I find myself adding 'waitForElementPresent' commands that look for the logout element in the user interface to appear to signal that we've successfully logged-in. As a tester I need to resolve the differences between the Selenese model of my application and the actual application state. This is not fun, error prone, and difficult to teach. Appcelerator does a good job of providing quality element ID values. Consequently I find myself in Firebug and Firefox a lot more than I should to get my Selenium test to run properly.

b) Selenium has no apparent way to provide a test with operational data. I am in the process of writing Data Production Libraries (DPLs) for Selenium IDE as a user extension. My first DPL reads from comma-separated-value (csv) flat files. The DPL opens a file and provides getNext and getPrevious methods to retrieve data from the csv file. I had to write this entirely in JavaScript and it was not fun. I will contribute these to the Selenium project in the next few weeks.

c) Selenium IDE recorded test scripts need to be exported into a scripting language format to implement business test flows. Conditionals, looping, and branching are possible using community contributions. For example, the Selenium Wiki includes a flow control extension.

For the past month I have been working on a design that enables TestMaker to operate Selenium tests. Here is what we have in mind:

1) Browser compatibility tests using Selenium IDE and Selenium RC.

2) Load and performance testing using Selenese scripts and TestMaker.

3) Service monitoring using Selenese scripts and TestMaker.

If this interests you then please participate! The enhancement ticket at design notes for Selenium in TestMaker are at http://bugs.pushtotest.com/ticket/136

-Frank

I will be giving a talk at STARWest on testing Ajax applications using Open Source Test Tools in September 2008.

Here is the updated abstract for the talk:

Ajax developers and testers are challenged to develop unit tests, functional tests, and load and performance tests in a time when Ajax, REST, and Web application development is expanding greatly. Frank Cohen, the leading authority on testing Ajax applications, will explain his methodology to find and solve scalability, performance, and reliability issues in Ajax applications. Frank will show how to apply the methodology using Selenium, soapUI, TestGen4Web, PushToTest, and many other open-source test projects. Frank will show hands-on lessons using Ajax applications created with Appcelerator and Google Widget Toolkit (GWT) frameworks. Frank will show how to use Firebug and Firefox to identify and instrument user interface elements for testing, he will show how to construct a functional unit test to perform a test business flow, how to provide the test with operational data at test run time, how to validate the test responses, and how to automate the entire test.

-Frank

I am planning to attend the Redmonk Unconference at CommunityOne on Monday. The Unconference is at Moscone Center in San Francisco on May 5, 2008. I will propose myself to talk about data-driven Ajax performance testing using Selenium and PushToTest.

Last year's Unconference was terrific.  Instead of going to another track planned by Sun to hear from their marketing folks or a JavaOne conference sponsor, the Unconference topics are proposed by attendees and the selection is made by a democratic vote at the beginning of the day. The Unconference worked out splendidly. I am very much looking forward to the Unconference!

Please join me and I will be happy to take you around and introduce you to everyone I know.

-Frank

We've been making very good progress in building TestMaker 5.2. I'm feeling like a proud father and want to show off my child's cleverness. In TestMaker 5 we introduced a new system of ScriptRunners. This is an extensible facility to operating externally created functional unit tests within a PushToTest TestMaker test. We bundled Eviware soapUI and SpikeSource TestGen4Web with TestMaker and provided ScriptRunners for each. As a result of the latest work TestMaker tests share data between ScriptRunners through a Data Production Library (DPL.) For instance, a functional test first makes a SOAP-based Web service request to learn product information codes and then makes a Web application (HTTP) request to make sure customers are finding the products.



PushToTest orchestrates a Functional test by defining the operating parameters and use cases in a TestScenario document in XML format, as illustrated above.

The test identifies a PushToTest Data Production Library (DPL) that will be the exchange medium between the soapUI TestCase and the TestGen4Web recorded test. The soapUI TestCase reads the data value for the SOAP request from a Properties file in the soapUI project. The soapUI TestCase receives the SOAP response and saves the response parameter to the DPL.

The TestScenario instructs PushToTest to then run the recorded TestGen4Web unit test. The test uses the DPL data from the soapUI test to make an HTTP Get request to the Web host. That concludes the functional test.

I am happy to make the above example available for immediate free download at http://downloads.pushtotest.com/tm5/EndToEndTestExample.zip

You will need the latest TestMaker 5.2 (pre-alpha) version to operate the example. See the faq for instructions on getting the code.

Enjoy.

-Frank