I gave the presentation at last night's eBIG user group meeting in Oakland, California. This is a Java user group and I felt very much at home. The group was very small (about 10 people) and they made up for size with the quality of their questions and attention. This was the first public demonstration of PushToTest TestMaker Version 5. TestMaker 5 got a very good reception from the group.

• They understood the power of being able to build a test on their development machines and giving it to a QA person to stage a load test.
• They liked the Data Production Library (DPL) system.
• Two of the attendees said they tried TestMaker 4 and gave up on it because of its complexity. Both said they will download TestMaker 5 and try again.
• They liked the idea of dynamic scripting support but were really wanting to implement their tests in Java.

And I took feedback and suggestions on how to further improve the feature set.

• Enhanced root-cause analysis using Glassbox, a utility that uses JMX to gather information from a running application to do root cause analysis. They strongly recommended integrating Glassbox into the Monitor system and results charts.
• Logging to a database, they want us to log directly into a database table and for us to publish the schema so they can write custom reports (using Jasper and others). They also suggested that we include the Derby DB in TestMaker. They also pointed me to DB4O.com.
• Better results analysis, two of the attendees were from large companies that have budgets for testing tools, so they compared TestMaker to Mercury LoadRunner. I heard comments like "The only thing I like about LoadRunner is its results analysis, you can slice and dice the data in so many ways." I told them about the Performance Comparison Utility and they said that would be very helpful.
• Agent-less monitoring, they asked why we would do CPU/Net/Memory monitoring with an agent versus just using an SNMP trap. They don't want to install agents for monitoring.
• TestNode option to run on JDK 1.5, they have run into problems using JAX-WS on JDK 1.6, specifically the XML parsing has incompatibilities. They would like an option to run a TestNode with JDK 1.5 and only operate Java or Jython tests in this mode. So no ScriptEngine support in exchange for running their existing unit tests written in Java on a TestNode.

I'll put the above items into http://bugs.pushtotest.com as enhancements.

-Frank