Data Production Library
May 15, 2008
Selenium's Ajax limitations and new designs
Filed Under:
Matt at Appcelerator
pointed me to a discussion of Automated
Testing of Appcelerator With Selenium on the Appcelerator
user forums. I am very interested in ways to use Selenium for testing
Appcelerator applications.
From TestMaker's perspective a Selenium test is a unit test of a Web application or Ajax application. TestMaker already provides Test Runners for TestGen4Web, soapUI, and the Java 6 ScriptEngine support for unit tests written in Java, Jython, JRuby, Groovy, and a bunch of others.
It seems to me that the forum members have already hit upon some of Selenium's limitations and design problems:
a) Selenium was not really developed for Ajax, and vice versa. Ajax frameworks like Appcelerator implement their own eventing mechanism and Selenium has no idea of how this works. For instance, when an Appcelerator log-in page changes the DOM to reveal site content Selenium IDE does not notice the change. Selenium IDE will sometimes record a 'click' when 'clickAndWait' is appropriate. I find myself adding 'waitForElementPresent' commands that look for the logout element in the user interface to appear to signal that we've successfully logged-in. As a tester I need to resolve the differences between the Selenese model of my application and the actual application state. This is not fun, error prone, and difficult to teach. Appcelerator does a good job of providing quality element ID values. Consequently I find myself in Firebug and Firefox a lot more than I should to get my Selenium test to run properly.
b) Selenium has no apparent way to provide a test with operational data. I am in the process of writing Data Production Libraries (DPLs) for Selenium IDE as a user extension. My first DPL reads from comma-separated-value (csv) flat files. The DPL opens a file and provides getNext and getPrevious methods to retrieve data from the csv file. I had to write this entirely in JavaScript and it was not fun. I will contribute these to the Selenium project in the next few weeks.
c) Selenium IDE recorded test scripts need to be exported into a scripting language format to implement business test flows. Conditionals, looping, and branching are possible using community contributions. For example, the Selenium Wiki includes a flow control extension.
For the past month I have been working on a design that enables TestMaker to operate Selenium tests. Here is what we have in mind:
1) Browser compatibility tests using Selenium IDE and Selenium RC.
2) Load and performance testing using Selenese scripts and TestMaker.
3) Service monitoring using Selenese scripts and TestMaker.
If this interests you then please participate! The enhancement ticket at design notes for Selenium in TestMaker are at http://bugs.pushtotest.com/ticket/136
-Frank
From TestMaker's perspective a Selenium test is a unit test of a Web application or Ajax application. TestMaker already provides Test Runners for TestGen4Web, soapUI, and the Java 6 ScriptEngine support for unit tests written in Java, Jython, JRuby, Groovy, and a bunch of others.
It seems to me that the forum members have already hit upon some of Selenium's limitations and design problems:
a) Selenium was not really developed for Ajax, and vice versa. Ajax frameworks like Appcelerator implement their own eventing mechanism and Selenium has no idea of how this works. For instance, when an Appcelerator log-in page changes the DOM to reveal site content Selenium IDE does not notice the change. Selenium IDE will sometimes record a 'click' when 'clickAndWait' is appropriate. I find myself adding 'waitForElementPresent' commands that look for the logout element in the user interface to appear to signal that we've successfully logged-in. As a tester I need to resolve the differences between the Selenese model of my application and the actual application state. This is not fun, error prone, and difficult to teach. Appcelerator does a good job of providing quality element ID values. Consequently I find myself in Firebug and Firefox a lot more than I should to get my Selenium test to run properly.
b) Selenium has no apparent way to provide a test with operational data. I am in the process of writing Data Production Libraries (DPLs) for Selenium IDE as a user extension. My first DPL reads from comma-separated-value (csv) flat files. The DPL opens a file and provides getNext and getPrevious methods to retrieve data from the csv file. I had to write this entirely in JavaScript and it was not fun. I will contribute these to the Selenium project in the next few weeks.
c) Selenium IDE recorded test scripts need to be exported into a scripting language format to implement business test flows. Conditionals, looping, and branching are possible using community contributions. For example, the Selenium Wiki includes a flow control extension.
For the past month I have been working on a design that enables TestMaker to operate Selenium tests. Here is what we have in mind:
1) Browser compatibility tests using Selenium IDE and Selenium RC.
2) Load and performance testing using Selenese scripts and TestMaker.
3) Service monitoring using Selenese scripts and TestMaker.
If this interests you then please participate! The enhancement ticket at design notes for Selenium in TestMaker are at http://bugs.pushtotest.com/ticket/136
-Frank
May 14, 2008
Testing Ajax applications
Filed Under:
I will be giving a talk at STARWest
on testing Ajax applications using Open Source Test Tools in September
2008.
Here is the updated abstract for the talk:
Ajax developers and testers are challenged to develop unit tests, functional tests, and load and performance tests in a time when Ajax, REST, and Web application development is expanding greatly. Frank Cohen, the leading authority on testing Ajax applications, will explain his methodology to find and solve scalability, performance, and reliability issues in Ajax applications. Frank will show how to apply the methodology using Selenium, soapUI, TestGen4Web, PushToTest, and many other open-source test projects. Frank will show hands-on lessons using Ajax applications created with Appcelerator and Google Widget Toolkit (GWT) frameworks. Frank will show how to use Firebug and Firefox to identify and instrument user interface elements for testing, he will show how to construct a functional unit test to perform a test business flow, how to provide the test with operational data at test run time, how to validate the test responses, and how to automate the entire test.
-Frank
Here is the updated abstract for the talk:
Ajax developers and testers are challenged to develop unit tests, functional tests, and load and performance tests in a time when Ajax, REST, and Web application development is expanding greatly. Frank Cohen, the leading authority on testing Ajax applications, will explain his methodology to find and solve scalability, performance, and reliability issues in Ajax applications. Frank will show how to apply the methodology using Selenium, soapUI, TestGen4Web, PushToTest, and many other open-source test projects. Frank will show hands-on lessons using Ajax applications created with Appcelerator and Google Widget Toolkit (GWT) frameworks. Frank will show how to use Firebug and Firefox to identify and instrument user interface elements for testing, he will show how to construct a functional unit test to perform a test business flow, how to provide the test with operational data at test run time, how to validate the test responses, and how to automate the entire test.
-Frank
May 01, 2008
Unconference, Unconference, Unconference
Filed Under:
I am planning to attend the Redmonk
Unconference at CommunityOne on Monday. The Unconference is
at Moscone Center in San Francisco on May 5, 2008. I will propose
myself to talk about data-driven Ajax performance testing using
Selenium and PushToTest.
Last year's Unconference was terrific. Instead of going to another track planned by Sun to hear from their marketing folks or a JavaOne conference sponsor, the Unconference topics are proposed by attendees and the selection is made by a democratic vote at the beginning of the day. The Unconference worked out splendidly. I am very much looking forward to the Unconference!
Please join me and I will be happy to take you around and introduce you to everyone I know.
-Frank
Last year's Unconference was terrific. Instead of going to another track planned by Sun to hear from their marketing folks or a JavaOne conference sponsor, the Unconference topics are proposed by attendees and the selection is made by a democratic vote at the beginning of the day. The Unconference worked out splendidly. I am very much looking forward to the Unconference!
Please join me and I will be happy to take you around and introduce you to everyone I know.
-Frank
Apr 07, 2008
End To End Testing
Filed Under:
An example of using soapUI, TestGen4Web and PushToTest to automate a test of a Web service and Web application functional test.
We've been making very good progress in building TestMaker 5.2. I'm
feeling like a proud father and want to show off my child's cleverness.
In TestMaker 5 we introduced a new system of ScriptRunners. This is an
extensible facility to operating externally created functional unit
tests within a PushToTest TestMaker test. We bundled Eviware soapUI
and SpikeSource
TestGen4Web with TestMaker and provided ScriptRunners for
each. As a result of the latest work TestMaker tests share data between
ScriptRunners through a Data Production Library (DPL.) For instance, a
functional test first makes a SOAP-based Web service request to learn
product information codes and then makes a Web application (HTTP)
request to make sure customers are finding the products.

PushToTest orchestrates a Functional test by defining the operating parameters and use cases in a TestScenario document in XML format, as illustrated above.
The test identifies a PushToTest Data Production Library (DPL) that will be the exchange medium between the soapUI TestCase and the TestGen4Web recorded test. The soapUI TestCase reads the data value for the SOAP request from a Properties file in the soapUI project. The soapUI TestCase receives the SOAP response and saves the response parameter to the DPL.
The TestScenario instructs PushToTest to then run the recorded TestGen4Web unit test. The test uses the DPL data from the soapUI test to make an HTTP Get request to the Web host. That concludes the functional test.
I am happy to make the above example available for immediate free download at http://downloads.pushtotest.com/tm5/EndToEndTestExample.zip
You will need the latest TestMaker 5.2 (pre-alpha) version to operate the example. See the faq for instructions on getting the code.
Enjoy.
-Frank

PushToTest orchestrates a Functional test by defining the operating parameters and use cases in a TestScenario document in XML format, as illustrated above.
The test identifies a PushToTest Data Production Library (DPL) that will be the exchange medium between the soapUI TestCase and the TestGen4Web recorded test. The soapUI TestCase reads the data value for the SOAP request from a Properties file in the soapUI project. The soapUI TestCase receives the SOAP response and saves the response parameter to the DPL.
The TestScenario instructs PushToTest to then run the recorded TestGen4Web unit test. The test uses the DPL data from the soapUI test to make an HTTP Get request to the Web host. That concludes the functional test.
I am happy to make the above example available for immediate free download at http://downloads.pushtotest.com/tm5/EndToEndTestExample.zip
You will need the latest TestMaker 5.2 (pre-alpha) version to operate the example. See the faq for instructions on getting the code.
Enjoy.
-Frank


