Open Source (OSS)
May 15, 2008
Selenium's Ajax limitations and new designs
From TestMaker's perspective a Selenium test is a unit test of a Web application or Ajax application. TestMaker already provides Test Runners for TestGen4Web, soapUI, and the Java 6 ScriptEngine support for unit tests written in Java, Jython, JRuby, Groovy, and a bunch of others.
It seems to me that the forum members have already hit upon some of Selenium's limitations and design problems:
a) Selenium was not really developed for Ajax, and vice versa. Ajax frameworks like Appcelerator implement their own eventing mechanism and Selenium has no idea of how this works. For instance, when an Appcelerator log-in page changes the DOM to reveal site content Selenium IDE does not notice the change. Selenium IDE will sometimes record a 'click' when 'clickAndWait' is appropriate. I find myself adding 'waitForElementPresent' commands that look for the logout element in the user interface to appear to signal that we've successfully logged-in. As a tester I need to resolve the differences between the Selenese model of my application and the actual application state. This is not fun, error prone, and difficult to teach. Appcelerator does a good job of providing quality element ID values. Consequently I find myself in Firebug and Firefox a lot more than I should to get my Selenium test to run properly.
b) Selenium has no apparent way to provide a test with operational data. I am in the process of writing Data Production Libraries (DPLs) for Selenium IDE as a user extension. My first DPL reads from comma-separated-value (csv) flat files. The DPL opens a file and provides getNext and getPrevious methods to retrieve data from the csv file. I had to write this entirely in JavaScript and it was not fun. I will contribute these to the Selenium project in the next few weeks.
c) Selenium IDE recorded test scripts need to be exported into a scripting language format to implement business test flows. Conditionals, looping, and branching are possible using community contributions. For example, the Selenium Wiki includes a flow control extension.
For the past month I have been working on a design that enables TestMaker to operate Selenium tests. Here is what we have in mind:
1) Browser compatibility tests using Selenium IDE and Selenium RC.
2) Load and performance testing using Selenese scripts and TestMaker.
3) Service monitoring using Selenese scripts and TestMaker.
If this interests you then please participate! The enhancement ticket at design notes for Selenium in TestMaker are at http://bugs.pushtotest.com/ticket/136
-Frank
May 14, 2008
Testing Ajax applications
Here is the updated abstract for the talk:
Ajax developers and testers are challenged to develop unit tests, functional tests, and load and performance tests in a time when Ajax, REST, and Web application development is expanding greatly. Frank Cohen, the leading authority on testing Ajax applications, will explain his methodology to find and solve scalability, performance, and reliability issues in Ajax applications. Frank will show how to apply the methodology using Selenium, soapUI, TestGen4Web, PushToTest, and many other open-source test projects. Frank will show hands-on lessons using Ajax applications created with Appcelerator and Google Widget Toolkit (GWT) frameworks. Frank will show how to use Firebug and Firefox to identify and instrument user interface elements for testing, he will show how to construct a functional unit test to perform a test business flow, how to provide the test with operational data at test run time, how to validate the test responses, and how to automate the entire test.
-Frank
May 13, 2008
JavaOne 2007 RedStripe Research Report Now Available
http://downloads.pushtotest.com/200805/PushToTest_JavaOne2008_Redstripe.pdf
Topics covered include: Open Source Test Alliance, OpenSolaris, Mac OS X, Virtual OS Support, Twitter, JavaFX, EJB Container Changes, Jython and Dynamic Scripting.
-Frank
May 11, 2008
JavaOne Pavilion Finds
Krugle is a source code search engine. It spiders your organizations source code repositories and lets you track code definitions. For instance, when you fix a bug in a well used, and shared, code library Krugle shows the other projects that use the same bug-laden code.
Style Intelligence has 101+ dashboards, reports, scorecards, and other visualizations of data.
Skyway Software makes an open-source application development environment. You build everything in their Eclipse plug-in. It's wonderfully well design user interface makes construction of apps very easy. I'm going to look into this one with the hope that I can build a bunch of sample code for PushToTest TestMaker users quickly.
Determyne is an open-source transaction-level performance monitoring solution for JEE applications. I'll be investigating this to incorporate into TestMaker.
-Frank
May 08, 2008
JavaOne 2008 Report
http://www.theserverside.com/news/thread.tss?thread_id=49332
-Frank
Apr 24, 2008
JavaOne in 2 weeks
Open Source Test Automation vendors Eviware (maker of soapUI) and PushToTest (maker of TestMaker) will have some exiciting news to announce at the JavaOne conference. The announcement will leverage our abilities to deliver SOA testing best practices using open source tools and methods.
I am finding that customers are increasingly turning to Open Source solutions for their testing needs. We recently wont such large customers as AMD, Amazon, Ford, The Jackson Laboratory and TVGuide. With 160,000 users, PushToTest delivers a single affordable platform to accomplish functional testing, load and performance testing, and business service monitoring. PushToTest integrates soapUI for fast and easy authoring of Web service test suites to run on a set of distributed PushToTest load test injector TestNodes.
PushToTest and Eviware will be exhibiting in the SOA Village in the JavaOne Exhibition Hall in booth 425.
I am available to meet you on Wed., May 7 or Thurs., May 8 at the show. Please let me know.
-Frank
Apr 22, 2008
TIBCO TUCON keynote
If you are attending TUCON then please let me know and I will be glad to meet with you personally.
-Frank
Apr 11, 2008
The Reviews Are In
They like me, they really like me.
Harry Wood attended my talk and posted a blog entry about his experience at http://www.harrywood.co.uk/blog/2008/04/11/unit-testing-soa-and-mule-talks
-Frank
Learning More About ESBs and SOA Development at Mulecon
Last week I attended the first day Mulecon, the Mule user conference.
The presentations varied between Mule staff talking about Mule and the product roadmap and customers talking about their use of Mule. The Mule staff seemed to be singing from the same songbook: Mule claims 13,000 deployments of Mule ESB, Mule 2.0 is not compatible with 1.4 but the migration should not be too difficult as the concepts are the same, Mule Enterprise Edition, Saturn and Mule HQ releases coming late this year.
A Room Full of Architects
It did not take long for me to realize that Mulecon delegates are mostly software architects. Typical questions I heard goes like this:"Your company inherits a big bank and needs to decide between JMS and Advanced Message Queue Protocol (AMQP) for a universal protocol for your bank. Which would you pick?"
and
"How do you empirically come to a model that will let you uncover the performance characteristics and measurements?"
These are the kinds of discussions that make my wife immediately yawn and roll her eyes! Geeks, there's a lot of 'em in this room.
ESB, Grid, AMQP Approaches
Conversations about hub-and-spoke, virtual circuits, and bus architecture seem to be over. Just about everyone I met expects a heterogeneous architecture where parts of a system are hub-and-spoke and others are bus-based. Easy service interfaces now exist between each type of architecture using daemons and local brokers. Jahan Moreh, VP of Engineering at U1, had a very basic message "Provided the underlying products are good you won't have a problem."Jahan was more sanguine about guaranteed deliver and performance. In essence "performance and guaranteed delivery are in conflict." Choose one but don't expect both. I very much want to do primary research to determine a standards way to profile performance in a guaranteed delivery setting. Hopefully one of our customers will commission a PushToTest study. (hint, hint.)
I learn about AMQP at Mulecon. AMQP is a protocol-level standards initiative to provide a messaging infrastructure. Jahan told me "AMQP tries to do something very important but may not be attainable. Because they are at the protocol level they may wind up being extensions on the side. I would bet on the JMS side."
POJOs Rule, But Where Is BPM?
Eugene Ciurana of LeapFrog gave a talk about LeapFrog's use of Mule. I just saw Eugene at TheServerSide.com Java Symposium in Las Vegas last week. Boy does Eugene get around!LeapFrog is marketing a child's toy that features downloadable content – not bad for products that must cost less than a few dollars to manufacture! The backend system to support this effort uses ESB technology to provide flexibility to LeapFrog's software developers and reliability in operating the service and applications.
Many of the question-and-answer sessions at Mulecon include a question on Business Process Management (BPM) such as: "What are you doing for BPM in your architecture?" Eugene took one of these.
This question echoes what I had learned in Las Vegas: Java architects and developers are frustrated looking for Business Process Management (BPM) standards and tools. Brian Sletten's talk at TSSJS was titled "Avoiding ESBs" but could have been better described as "I'm sick and tired of waiting for vendors to give me a decent Business Process Management (BPM) platform!"
At Mulecon, Rory dela Pax of Biogen Idec, told me that "Mule is good at doing some workflow. But, all of us are struggling with BPM. We do a separation of skills strategy rather than overloading the ESB with these tasks."
Over the years I have seen most of the platform vendors try to provide the Java development community with set of products, best practices, framework, and code to implement applications that deliver business flows. I am specifically remembering the failures of JBI, JEMs, BPEL, SCA, and WS-*. None have come together to offer a standard. I sense a lot of anger and disappointment at all the failed attempts. Mostly disappointment from me.
Eugene answered the BPM question by telling the delegates that LeapFrog pushes all of its development to Plain Old Java Objects (POJOs.) They actively separate every application from each other. LeapFrog then uses Mule routers to define the inputs and outputs. Eugene said "We are not doing any transactional data at Leapfrog. Our support is for the toy devices." He added that Leapfrog has a separate side of the IT house for Oracle tools and BPEL.
Mule and Clustering
Rory dela Pax of Biogen Idec gave a brief talk on their migration from BEA WebLogic Integration (WLI) to Mule. The migration came about like many companies adopting an open-source project: A developer mentioned it during a water break conversation. The environment was ripe: Biogen Idec found WLI to be "a bit on the heavy side, unstable, and has not scaled well." They were also leery of the rumors of the Oracle buy-out since late 2006. They found that Mule has a one-for-one match in capabilities, plus they see Mule as lightweight, stable, and scalable.I can understand lightweight and stable, but the scalability claim seemed to need some proof. Rory tells me they achieve scalability by using WebLogic Server (WLS) 8.1 SP6's clustering. They deploy their Mule application as a WAR file in a cluster of JVMs in WLS. There are no separate JVMs in their environment. I imagine the gaul the WLI product manager must feel at being swapped out. :-)
Still, there is a lot that Biogen Idec needs to learn about clustering. They still need to define the best way to cluster. Rory told the delegates "We don't know how to do that." They would like to define clustering methods in environments where message sizes grow larger. Right now their application uses simple verb+link combinations. The incoming request is to approve an invoice and the response contains a link to bring the user's browser into the application.
Rory told me some of the downsides to using this WLS clustering approach. For example, they use Quartz as a scheduler. In this WLS clustering set-up changing the Quartz schedules means redeploying the application to the cluster.
Moving Up To Mule 2.0
Mule 1.4 uses DTDs to define pretty much everything – settings, configuration, deployments. Mule 2.0 delivers XML Schema based configuration that leverages Spring's "Extensible XML Authoring." The result is less cumbersome class names because of namespace support. They are updating their Eclipse support to provide auto-completion and context-specific help. Mule 2.0 features transport-specific endpoints and connectors and now everything is done through typed properties. There is a lot of extensibility because each module/transport comes with its own schema. Mule 2.0 lets you implemtn your own schema too.Some additional changes in Mule 2.0: CXF supercedes XFire (XFire is still available,) there is a new expression evaluator framework, streaming improvements, auto-transformation, and lots of bug fixes. Plus, Mule 2.0 has 30% more unit tests over 1.4. Mule uses Spring to implement Beans. Mule uses session and entity (data) beans inside the Mule container or through proxies to external data services.
Mulecon left me a few questions:
- How important was the Spring-based approach to building services to Mule's popularity and success?
- Is anyone, I mean anyone on Earth, using OpenESB from Sun? None of the Mule users had any experience with it.
- How are Mule and Mule users testing Mule for scalability and performance? Mule 2 provides hundreds of unit tests but apparently no performance tests, at least none that I could find.
- What are the tradeoffs of an ESB versus Gigispaces, Tangosol and other grid environments?
-Frank
Apr 02, 2008
The Next Step in Unit Testing, Java, and SOA
I'm giving a free presentation at Skills Matter in London, UK, on Monday night
The session is free and I will be happy to answer your questions. For details and directions
see: http://www.skillsmatter.com/event/java-jee/the-next-step-in-unit-testing-and-java
Feb 23, 2008
Selenium and PushToTest Plans
PushToTest version 5.2 (which should be out this Spring) will feature Selenium integrated into the PushToTest test automation platform. See http://bugs.pushtotest.com/ticket/136 for details.) Selenium users will create functional tests of Web and Ajax applications using the Selenium plug-in to MS Internet Explorer, Firefox, Mozilla, and other browsers. PushToTest users have two choices to run Selenium tests in the PushToTest test automation environment:
1) PushToTest 5.2 will include a Selenium test runner. This runner instantiates a Web browser in a TestNode (our distributed test environment,) operates the Selenium test, and reports the results. This is a powerful choice to run Selenium tests as functional and regression tests, and business service monitors.
2) PushToTest 5.2 includes a Selenium-to-Java and Selenium-to-Jython transformation utility. The utility creates a Java class (or Jython script) that implements the test steps using the HTMLUnit framework. HTMLUnit acts like a browser, including operating Ajax applications using Rhino (the JavaScript engine from the Apache Mozilla/Firefox browsers.) The is a powerful choice to run Selenium tests as load and performance tests.
For both of the above options, Selenium tests receive dynamic operational data at runtime from PushToTest Data Production Libraries (DPLs.) For instance, a Selenium test that operates a sign-in page receives the account number and password from a DPL that accesses data from a comma-separated-value file (or a relational database or custom DPL.)
Additionally, Selenium tests produce results data that PushToTest Results Analysis engine renders into hundreds of charts. And, these charts correlate Selenium test operation to resource utilization (CPU, Network, Memory, Threads) in the back-end server.
We have an "alpha" quality version of the Selenium-to-Jython transformation utility to give to you today. This is from a contribution from Dominique and Olivier at Denali. All you need to do is ask.
-Frank
Feb 19, 2008
Selenium Meet-up on Monday Feb. 25
The Selenium founders are organizing a user meeting at Google's Mountain View, California office on Monday, February 25. Among other things they will be describing the push to release Selenium 1.0 and the roadmap and internals of Selenium 2.0.
Details on the Selenium user meeting are at
http://selenium.openqa.org/meetup.jsp
-Frank
Comparing Oracle, IBM, BEA, and TIBCO for SOA
This morning I am glad to introduce you to a new set of resources to help surface scalability and performance issues in Service Oriented Architecture (SOA.) The SOA Knowledge and Performance Kit is a free open-source resource to show you what it really takes to build services using today's leading SOA development platforms.
The Kit delivers an SOA use case design, source code to the implementations of the use case on Oracle, IBM, BEA, and TIBCO platforms, developer journals describing our experiences step-by-step, a Total Cost of Ownership (TCO) calculator, and performance and scalability tests that leverage the PushToTest test automation platform.
PushToTest looked below the surface-level marketing claims to understand the skill sets, domain expertise, and specialization that it takes to be successful. We discovered a wide variety of experiences: some highly successful and others rife with challenges. We measured the amount of developer effort to implement an SOA use case and summarize the findings in the following total cost of ownership (TCO) comparison chart.

This morning we opened the doors on a new center within the PushToTest Web site for you to learn more about SOA, learn about the new composition approach to SOA development and deployment, and download the kits.
SOA Knowledge and Performance Kit
-Frank
Feb 05, 2008
soapUI + PushToTest Moves Java Developers Towards The Grid
Last year PushToTest built a test runner for soapUI and bundled soapUI in the PushToTest version 5 distribution. The advantage to Java developers is three fold:
1) soapUI test suites run in a distributed PushToTest test environment - a grid of test machines each turning out SOAP requests to your application.
2) The PushToTest environment dynamically provides operational data to the soapUI test suites. For instance, the data could be sign-in information or product part numbers. The data production libraries (DPLs) get data from a relational database, from a comma-separated-value file, and from service calls dynamically.
3) PushToTest provides a results analysis engine and performance comparison utility to identify problem test suite performance and problem service calls. Additionally, while the test suites operate the PushToTest Monitor watches CPU, Network, and Memory utilization of the test machines and the target hosting the application and correlates these back to the test suite operation to surface the problem datacenter components.
We increasingly see upper management at our customers look at consumer hits like Facebook, Flikr, Google Maps, and tell their engineers "We want that!" At the same time Java developers in 2008 have good tools to service enable their classes. The interface may be SOAP, or any number of others including REST, Ajax, JMS to a message queue, RMI, etc.
We are seeing customers use PushToTest in service environments. They tend to start service enabling a few classes and wind-up with a grid of interconnected services. While soapUI by itself is a terrific tool for working with individual services, the nature of service enabling Java classes means most organizations need a test methodology, patterns, and tools that work in a grid of services.
Customer success at testing in a grid environment is the motivation behind a new partnership that eviware and PushToTest announced this week. Java developers benefit when two open-source 100% Java projects partner to solve service testing challenges and help us test in a grid environment.
-Frank
Feb 04, 2008
New PushToTest Datasheet and Whitepaper
We are getting very close to the PushToTest 5.1 release - hopefully this week! New datasheet and white paper downloads are available to explain PushToTest and open-source test automation:
PushToTest White Paper: The Importance of Test Automation
Both are in Adobe Acrobat (PDF) format.
-Frank
Feb 01, 2008
Open-Source In Sheep's Clothing
Dan Farber of ZDNet posted a good article that talks about user experiences while tryign to understand open-source product offerings.
In the article he talks about a user's experience coming to grips with SugarCRM's open-source offering not meeting the user's needs. I posted a reply that you can read here. Below I a copy of what I wrote.
-Frank
--
Hi Dan and Larry: Thanks for bringing the ZD communities' attention to this issue. My company PushToTest is an open-source test automation company and we frequently need to explain our dual license model to users and prospective customers.
There are a lot of different open-source business models. I will be at the Open-Source Think Tank (http://thinktank.olliancegroup.com/) next week to meet a hundred or more fellow CEOs of open-source companies to talk about this. I want to make sure your readers know that there is no single truth when it comes to knowing exactly what you get when a company says it is "open source."
By the way, Simon Phipps, the Chief Open Source Officer at Sun Microsystems, led a very heated conversation about open source last year. Simon put forward that there needs to be a renewed effort at OSI (http://www.opensource.org/ ) to shame people, groups, and businesses that are pushing the open source marketplace in bad ways. SugarCRM came up in that it is a company that claims to be open source but does not allow companies to change and branch because of their attribution requirements. Simon says we need a group like OSI to shame people that aren't really playing by the open source rules.
I personally don't like the idea of shaming people, but there does need to be some easy way to identify the types of open-source licenses to make it easier for users and organizations to make informed decisions.
For PushToTest we publish our SOA, Ajax, Web application and Web Service test automation platform under a dual license. If you want to build your own test platform then download the source code – under a GPL v2 license – and build it yourself! We make a pre-compiled, tested, and ready-to-install version available that comes with a commercial license that is free for use up to 200 concurrent users in a load test, or up to 10 business service monitors. If you want PushToTest professional support – and our necks for you to strangle – or to run at more than 200 users then we sell a commercial license. There are no functional differences between the GPL source code and the commercial product.
-Frank Cohen http://www.pushtotest.com
Jan 29, 2008
soapUI, Selenium, PushToTest Bootcamp
A lot of QA testers are coming to the realization that testing applications built on Java, Database, and Application Server tools can't get them from here-to-there when using the old-and-trusted record/playback test tools. The problem often starts when a database instance needs to service client Web interfaces with dynamic data at the same time as the middle tier is serving Ajax (XML, String, and JSON) data for rich browser interfaces, and REST and SOAP interfaces with transactional data.
New tools have emerged over the past year - such as soapUI, Selenium, and PushToTest - that are designed for Web application, Web service, and Ajax testing. Pulling these tools and platforms together into something that the average database developer, application engineer, and tester can use requires a lot of effort.
To meet this need, PushToTest is now offering a Bootcamp to train technical folks how to leverage soapUI, Selenium, and PushToTest. Details are found at http://bootcamp.pushtotest.com
-Frank
Jan 22, 2008
A Repository Confluence
Every once in a while a confluence of ideas happens around open-source test automation. The confluence of this week is:
- Apache's announcement of Jackrabbit 1.4
- Zephyr's announcement that their private beta of a new collaborative test automation platform uses Jackrabbit
- Jimmy Foulkes (one of our architects) push for PTTWeb to use JSR 170 and specifically Jackrabbit as our new repository for the upcoming PushToTest collaborative test automation platform.
All of these happened this week!
PushToTest is being used by hundreds of thousands of developers now. We often hear users and customers asking for a collaborative test environment that features a repository of tests and a scheduler. We are working on these under the codename PTTWeb. The first piece of the project is the repository.
This is perhaps the most important module of PTTWeb because it provides all the storage functionality required by the other modules. Will serve as the storage repository for test scenarios, resources and scripts as well as test results.
In general this will include the storage of the following areas:
- Test resources: test related artifacts using a file system metaphor
- Test results: all test data results associated to the executed jobs from the different test nodes
- Security: all information related to security access like users, roles, permissions, etc.
- Scheduling: job and its execution scheduling information
In order to avoid spending time on writing code to provide this functionality it is necessary to find an Open Source product/project that provides this service. The Java Community process has a specification for Content Repositories under the JSR 170. There are several Open Source implementations of this JSR. One possible implementation is the Apache Jackrabbit project that fulfills this JSR as well as adding several more features of its own. Jackrabbit is in use by several important projects like JBoss portal, Magnolia Content management system, jLibrary, Sun's OpenPortal Project.
Most providers for this JSR can provide RDMS implementation for the underlying data store. A MySql database could provide the actual storage capabilities for the repository although a file system based repository is much faster. The idea is that the repository should emulate a typical hierarchical file system with similar access permission typical of a file system. Attention will be placed in the design to the appropriate structure of this repository and to allow the definition of the required metadata associated with the possible resources to be kept there.
A general search function will be provided on this repository. For those resources of text type the actual contents of the resource can be searched, otherwise the search will be performed on the actual names, attributes and repository structure for the repository.
The design document for PTTWeb is available for your review, feedback and ideas here.
-Frank
Jan 21, 2008
Enhanced Results Analysis design
Here is a preview of how the new interface works.

Custom reports appear below the Real Time Scalability Report. The user adds a new custom report by clicking the + icon. A dialog window appears to let the user choose the chart parameters. You will see a dialog that looks like the following:

The user enters the Definition Name, chooses the X Axis, Y Axis, Series, and Graph Type settings, and clicks the Save button. TestMaker saves the parameters in the <charts> element in the current TestScenario file. The dialog closes and the panel for the TestScenario shows the new chart definition. The following illustrates the user interface after the user adds a new chart definition.
The design enables the user to reuse chart definitions among TestScenarios and among users. The left side of the dialog offers a list of existing chart definitions. The list appears in three sections: chart definitions in the current TestScenario, chart definitions in the TestMaker properties file, and chart definitions in the PushToTest Repository. The Repository is an upcoming feature in PushToTest.
The "downward triangle" icon displays a pop-up list with commands to save a copy of the chart definition to the TestMaker Properties file. This makes the chart definition available to be reused in other TestScenarios. The pop-up also includes a command to save a copy of the chart definition to the Repository.
Pretty cool stuff!
-Frank
Jan 16, 2008
Oracle buys BEA, Sun buys MySQL
Rod Johnson of Spring fame has posted The Power of Adoption: Why No Company is Big Enough to Deny Developers What They Want to talk about two big software company acquisitions that were announced today. Oracle announced it is buying BEA. And Sun announced it bought MySQL.
For BEA, I was hoping they would succeed at combining the power of JRockit with AquaLogic to build scalable workflow (ala BPM) services. Instead, we users and developers will be waiting for years to see WebLogic and AquaLogic integrated into the Oracle Fusion Middleware stack.
For MySQL, I was hoping for an IPO. My company, PushToTest, is an open-source software publisher and I was looking to MySQL to show me the way to go really big. $1 Billion just does not sound like that much considering Oracle is currently worth $113 Billion. Actually a lot of us in the OSS space were looking for the MySQL IPO as a proof-point that OSS investments are a good thing generally.
To the good people at BEA and MySQL I send a heart felt congratulations. As a user and developer I wonder what the future holds?
-Frank Cohen


