A short list of companies at the JavaOne Pavilion that look interesting to me:

Krugle is a source code search engine. It spiders your organizations source code repositories and lets you track code definitions. For instance, when you fix a bug in a well used, and shared, code library Krugle shows the other projects that use the same bug-laden code.

Style Intelligence has 101+ dashboards, reports, scorecards, and other visualizations of data.

Skyway Software makes an open-source application development environment. You build everything in their Eclipse plug-in. It's wonderfully well design user interface makes construction of apps very easy. I'm going to look into this one with the hope that I can build a bunch of sample code for PushToTest TestMaker users quickly.

Determyne is an open-source transaction-level performance monitoring solution for JEE applications. I'll be investigating this to incorporate into TestMaker.

-Frank

This week I was the speaker at SkillsMatter.com in London. I gave a talk on the PushToTest test methodology and free open-source test automation tools to check Web applications, Web services, and Ajax applications for function, scalability, and performance. SkillsMatter is the new European partner to PushToTest to offer training on our products. In June I am back in London teaching our popular Open-Source Test Automation Bootcamp.

Harry Wood attended my talk and posted a blog entry about his experience at http://www.harrywood.co.uk/blog/2008/04/11/unit-testing-soa-and-mule-talks

-Frank

HTMLUnit is an open-source library that implements APIs that act like a modern JavaScript-enabled browser. It is ideal for building unit tests of browser-based Ajax-style applications. We use HTMLUnit in PushToTest to test Ajax, Web, and JavaScript in browser-based applications. PushToTest turns HTMLUnit's functional tests of a Web site into a load and performance test and a Business Service Monitor (BSM.)

I'm happy to spread the news that the HTMLUnit team release version 2.0 today!

The Change Log notes many improvements and bug fixes, including migration to Java 5, implementation of org.w3c.dom.*, better support for incorrect html code, large improvements in JavaScript support (GWT 1.4 tests now pass, and a bunch of bug fixes.

I opened an enhancement ticket to upgrade PushToTest to HTMLUnit 2.0.

Congratulations HTMLUnit team!

-Frank

We've been making very good progress in building TestMaker 5.2. I'm feeling like a proud father and want to show off my child's cleverness. In TestMaker 5 we introduced a new system of ScriptRunners. This is an extensible facility to operating externally created functional unit tests within a PushToTest TestMaker test. We bundled Eviware soapUI and SpikeSource TestGen4Web with TestMaker and provided ScriptRunners for each. As a result of the latest work TestMaker tests share data between ScriptRunners through a Data Production Library (DPL.) For instance, a functional test first makes a SOAP-based Web service request to learn product information codes and then makes a Web application (HTTP) request to make sure customers are finding the products.



PushToTest orchestrates a Functional test by defining the operating parameters and use cases in a TestScenario document in XML format, as illustrated above.

The test identifies a PushToTest Data Production Library (DPL) that will be the exchange medium between the soapUI TestCase and the TestGen4Web recorded test. The soapUI TestCase reads the data value for the SOAP request from a Properties file in the soapUI project. The soapUI TestCase receives the SOAP response and saves the response parameter to the DPL.

The TestScenario instructs PushToTest to then run the recorded TestGen4Web unit test. The test uses the DPL data from the soapUI test to make an HTTP Get request to the Web host. That concludes the functional test.

I am happy to make the above example available for immediate free download at http://downloads.pushtotest.com/tm5/EndToEndTestExample.zip

You will need the latest TestMaker 5.2 (pre-alpha) version to operate the example. See the faq for instructions on getting the code.

Enjoy.

-Frank

I recently received an email asking:

"Is it possible to use PushtoTest to build a test of a 

Web Application that uses Flash?"

It is possible to test Flash applications using PushToTest. We are currently helping FordDirect.com do this. It does not come without some extra work. For instance, by itself there is no record/playback capability to build tests of Flash animations. However, there is a programmatic way to discover the content of ActionScript variables and to activate functions by doing the equivalent of pressing a button or using Flash controls. PushToTest professional support services will instruct you on the approach, how to instrument your Flash animations for testing, and teach our test methodology and testing platform.

-Frank

Over the past year PushToTest observed a trend among the 160,000 people in our open-source community: Software developers, QA testers, and IT managers are looking for a better way to develop, orchestrate, deploy, and manage services. The Composition Approach to Building Large-Scale SOA introduces a new service composition approach to building SOA services, explains composition, and provides a methodology and test kit to evaluate today's tools for developer productivity and ease of deployment/management.

Last week, TIBCO hosted me on a Webcast to talk about Service Virtualization, Service Composition, and the SOA Knowledge Kit. The Webcast is now available at:

http://media.tibco.com/video/tibco_031808/index.html

 

-Frank

PushToTest version 5.2 (which should be out this Spring) will feature Selenium integrated into the PushToTest test automation platform. See http://bugs.pushtotest.com/ticket/136 for details.) Selenium users will create functional tests of Web and Ajax applications using the Selenium plug-in to MS Internet Explorer, Firefox, Mozilla, and other browsers. PushToTest users have two choices to run Selenium tests in the PushToTest test automation environment:

1) PushToTest 5.2 will include a Selenium test runner. This runner instantiates a Web browser in a TestNode (our distributed test environment,) operates the Selenium test, and reports the results. This is a powerful choice to run Selenium tests as functional and regression tests, and business service monitors.

2) PushToTest 5.2 includes a Selenium-to-Java and Selenium-to-Jython transformation utility. The utility creates a Java class (or Jython script) that implements the test steps using the HTMLUnit framework. HTMLUnit acts like a browser, including operating Ajax applications using Rhino (the JavaScript engine from the Apache Mozilla/Firefox browsers.) The is a powerful choice to run Selenium tests as load and performance tests.

For both of the above options, Selenium tests receive dynamic operational data at runtime from PushToTest Data Production Libraries (DPLs.) For instance, a Selenium test that operates a sign-in page receives the account number and password from a DPL that accesses data from a comma-separated-value file (or a relational database or custom DPL.)

Additionally, Selenium tests produce results data that PushToTest Results Analysis engine renders into hundreds of charts. And, these charts correlate Selenium test operation to resource utilization (CPU, Network, Memory, Threads) in the back-end server.

We have an "alpha" quality version of the Selenium-to-Jython transformation utility to give to you today. This is from a contribution from Dominique and Olivier at Denali. All you need to do is ask.

-Frank

I often get asked questions like "What does PushToTest do?" PushToTest is a test automation solutions business. We surface scalability and performance problems in Web-based information systems. Our PushToTest solution drives a Web system like a real user and correlates the activity to identify broken functions and hot spots of performance issues. The same system then operates tests periodically through the day to monitor an enterprise's services and provide proof-of-service reports. See our "white paper", http://downloads.pushtotest.com/PushToTest_WhitePaper.pdf for a more complete answer. I often get asked a follow-up question that is usually asked like this: "I was wondering if you could provide me a sample of the outputs the system generates (or can generate)." The PushToTest software produces a summary report that shows the Scalability Index to the application, the resource utilization as the test operated, and the performance as load increased. This summary report is an HTML document and is easily shared among your team members. "Click here", http://downloads.pushtotest.com/tm5/ExampleSummaryReport/index.html for an example summary report. I often also get asked about the PushToTest methodology and techniques to find and solve performance issues. The question is often asked like this: "Once you run a test and the results are not acceptable then how do I leverage the PushToTest tools to find the code that is running slow?" PushToTest version 5.1 keeps track of the steps for each transaction. For instance, the following snippet of a TestScenario implements a test use case where four steps are required for each transaction.

<test>
   <run name="Open Account" testclass="com.pushtotest.WebAppTest"
method="testOpenAccount" langtype="java" />

   <run name="Transfer Funds" testclass="com.pushtotest.WebAppTest"
method="testTransfer" langtype="java" />

   <run name="Check Transfer" testclass="com.pushtotest.WebAppTest"
method="testValidateTransfer" langtype="java" />

   <run name="Close Account" testclass="com.pushtotest.WebAppTest"
method="testCloseAccount" langtype="java" />

</test>

In the above test PushToTest operates four steps: Open Account, Transfer Funds, Check Transfer, and Close Account. PushToTest keeps track of the total transaction time and the time for each step. The time of each transaction and step is stored in the transaction log file. When a particular step takes longer than expected you can break open the transaction logs for the application server and learn what was going on at that point in time. We have been hard at work over the past two months building an Enhanced Results Analysis engine. The new feature will appear in PushToTest version 5.2. It is already working and we are happy to provide it to anyone that "asks", http://www.pushtotest.com/ask_a_question. The new engine makes it much easier to visualize broken functions and hot spots to performance. Consider the Step Report that visually charts the average duration of each step across 10 equal periods of time of a test. The reporting is great but that still leaves the correlation effort up to a software engineer. And, it is a manual process to go from PushToTest logs to the application server logs. Our planned solution is to incorporate "Glassbox", http://www.glassbox.com into PushToTest version 5.2. Glassbox watches a Java application server for performance hotspots as a test operates and identifies the objects and database connection level issues causing the hotspots of performance. "Click here", http://bugs.pushtotest.com/ticket/61 to learn about the integration project. To learn more about Glassbox "click here", http://www.glassbox.com/glassbox/Download.do?mediaItemId=173 to read the Glassbox user guide. One final question I usually hear goes like this: "Where exactly are your developers located? In other words when you leverage out-of-country developers, where are they and if the customer wanted US based developers can you accommodate that." PushToTest outsources its customer engineering to our Costa Rica-based partner "Avantica", http://www.avantica.net. Costa Rica is a 4-5 hour flight to the United States and Canada. The country is on Central Time (GMT -6) and their English is very good. We normally need about one week to plan for our engineers to be in your office. -Frank

This morning I am glad to introduce you to a new set of resources to help surface scalability and performance issues in Service Oriented Architecture (SOA.) The SOA Knowledge and Performance Kit is a free open-source resource to show you what it really takes to build services using today's leading SOA development platforms.

The Kit delivers an SOA use case design, source code to the implementations of the use case on Oracle, IBM, BEA, and TIBCO platforms, developer journals describing our experiences step-by-step, a Total Cost of Ownership (TCO) calculator, and performance and scalability tests that leverage the PushToTest test automation platform.

PushToTest looked below the surface-level marketing claims to understand the skill sets, domain expertise, and specialization that it takes to be successful. We discovered a wide variety of experiences: some highly successful and others rife with challenges. We measured the amount of developer effort to implement an SOA use case and summarize the findings in the following total cost of ownership (TCO) comparison chart.

This morning we opened the doors on a new center within the PushToTest Web site for you to learn more about SOA, learn about the new composition approach to SOA development and deployment, and download the kits.

SOA Knowledge and Performance Kit

-Frank

Last year PushToTest built a test runner for soapUI and bundled soapUI in the PushToTest version 5 distribution. The advantage to Java developers is three fold:

1) soapUI test suites run in a distributed PushToTest test environment - a grid of test machines each turning out SOAP requests to your application.

2) The PushToTest environment dynamically provides operational data to the soapUI test suites. For instance, the data could be sign-in information or product part numbers. The data production libraries (DPLs) get data from a relational database, from a comma-separated-value file, and from service calls dynamically.

3) PushToTest provides a results analysis engine and performance comparison utility to identify problem test suite performance and problem service calls. Additionally, while the test suites operate the PushToTest Monitor watches CPU, Network, and Memory utilization of the test machines and the target hosting the application and correlates these back to the test suite operation to surface the problem datacenter components.

We increasingly see upper management at our customers look at consumer hits like Facebook, Flikr, Google Maps, and tell their engineers "We want that!" At the same time Java developers in 2008 have good tools to service enable their classes. The interface may be SOAP, or any number of others including REST, Ajax, JMS to a message queue, RMI, etc.

We are seeing customers use PushToTest in service environments. They tend to start service enabling a few classes and wind-up with a grid of interconnected services. While soapUI by itself is a terrific tool for working with individual services, the nature of service enabling Java classes means most organizations need a test methodology, patterns, and tools that work in a grid of services.

Customer success at testing in a grid environment is the motivation behind a new partnership that eviware and PushToTest announced this week. Java developers benefit when two open-source 100% Java projects partner to solve service testing challenges and help us test in a grid environment.

-Frank

We are getting very close to the PushToTest 5.1 release - hopefully this week! New datasheet and white paper downloads are available to explain PushToTest and open-source test automation:

PushToTest 2-Page DataSheet

PushToTest White Paper: The Importance of Test Automation

Both are in Adobe Acrobat (PDF) format.

-Frank

Todd Bradfute on September 6, 2007 wrote to me:

I've drunk the Eclipse cool-aid. What do you think about having a
PushToTest plug-in to Eclipse? Maybe it's not the top priority for your
engineers, but that IDE feels so much cleaner than the Netbeans Interface
that PushToTest v5 uses.

Then earlier this month a medium-size prospective customer sent us a request-for-proposal that includes using PushToTest as an Eclipse plug-in. There is nothing quite like the scent of a sale to get us motivated! :-)

William Martinez drafted a good first design of a PushToTest Eclipse Plugin. You will find the document at:

http://downloads.pushtotest.com/tm5/PushToTest_EclipsePlugin_Design.pdf

We are seeking your feedback on the design, our assumptions, and how this new plug-in would make you more productive.

-Frank

Every once in a while a confluence of ideas happens around open-source test automation. The confluence of this week is:

• Apache's announcement of Jackrabbit 1.4
• Zephyr's announcement that their private beta of a new collaborative test automation platform uses Jackrabbit
• Jimmy Foulkes (one of our architects) push for PTTWeb to use JSR 170 and specifically Jackrabbit as our new repository for the upcoming PushToTest collaborative test automation platform.

All of these happened this week!

PushToTest is being used by hundreds of thousands of developers now. We often hear users and customers asking for a collaborative test environment that features a repository of tests and a scheduler. We are working on these under the codename PTTWeb. The first piece of the project is the repository.

This is perhaps the most important module of PTTWeb because it provides all the storage functionality required by the other modules. Will serve as the storage repository for test scenarios, resources and scripts as well as test results.

In general this will include the storage of the following areas:

• Test resources: test related artifacts using a file system metaphor
• Test results: all test data results associated to the executed jobs from the different test nodes
• Security: all information related to security access like users, roles, permissions, etc.
• Scheduling: job and its execution scheduling information

In order to avoid spending time on writing code to provide this functionality it is necessary to find an Open Source product/project that provides this service. The Java Community process has a specification for Content Repositories under the JSR 170. There are several Open Source implementations of this JSR. One possible implementation is the Apache Jackrabbit project that fulfills this JSR as well as adding several more features of its own. Jackrabbit is in use by several important projects like JBoss portal, Magnolia Content management system, jLibrary, Sun's OpenPortal Project.

Most providers for this JSR can provide RDMS implementation for the underlying data store. A MySql database could provide the actual storage capabilities for the repository although a file system based repository is much faster. The idea is that the repository should emulate a typical hierarchical file system with similar access permission typical of a file system. Attention will be placed in the design to the appropriate structure of this repository and to allow the definition of the required metadata associated with the possible resources to be kept there.

A general search function will be provided on this repository. For those resources of text type the actual contents of the resource can be searched, otherwise the search will be performed on the actual names, attributes and repository structure for the repository.

The design document for PTTWeb is available for your review, feedback and ideas here.

-Frank

We are making outstanding progress on building the Enhanced Results Analysis engine for PushToTest TestMaker. The internal work is almost done and we are now concentrating on the user interface and integration. We put together a user interface design for your review, feedback, and comments. Feel free to download the design document to learn more.

Here is a preview of how the new interface works.


Custom reports appear below the Real Time Scalability Report. The user adds a new custom report by clicking the + icon. A dialog window appears to let the user choose the chart parameters. You will see a dialog that looks like the following:



The user enters the Definition Name, chooses the X Axis, Y Axis, Series, and Graph Type settings, and clicks the Save button. TestMaker saves the parameters in the <charts> element in the current TestScenario file. The dialog closes and the panel for the TestScenario shows the new chart definition. The following illustrates the user interface after the user adds a new chart definition.

The design enables the user to reuse chart definitions among TestScenarios and among users. The left side of the dialog offers a list of existing chart definitions. The list appears in three sections: chart definitions in the current TestScenario, chart definitions in the TestMaker properties file, and chart definitions in the PushToTest Repository. The Repository is an upcoming feature in PushToTest.

The "downward triangle" icon displays a pop-up list with commands to save a copy of the chart definition to the TestMaker Properties file. This makes the chart definition available to be reused in other TestScenarios. The pop-up also includes a command to save a copy of the chart definition to the Repository.

Pretty cool stuff!

-Frank

William Martinez is one of our architects on TestMaker. He said to me recently, "You know, the name TestMaker is kind of different from what it actually does. You should call it PushToTest." This resonates with something I have been thinking about for a while.

We released the first TestMaker (called Load) in 2001. At the time, TestMaker was a test automation construction set and lived within an Integrated Development Environment (IDE) to build tests that would run on a network of test machines. TestMaker has grown to be a test creation utility, a console to operate the tests, and it even has its own SOAP-based service interface to operate tests. It makes sense to call this PushToTest.

In 2008 we will transition from TestMaker to PushToTest.

-Frank

I am happy to report that TestMaker 5.1 is available for immediate download at http://www.pushtotest.com. The new version fixes many bugs and delivers useful enhancements.

We are making this Release Candidate available to get user feedback on the changes. I anticipate we will make 2 additional release candidates to solve bugs and make additional improvements. We will be using http://bugs.pushtotest.com to track the changes.

Thank you to everyone that participated in this release!

-Frank

--

TestMaker 5.1 Changes

• TestMaker 5.0 ships with TestGen4Web 0.49 that fails to install on Firefox 2.0.0.7 or later. TestMaker now ships with TestGen4Web 0.50.2-PTT to be fully compatible with Firefox 2.0.0.7 and later.
• TestMaker 5.1 changes the way TestGen4Web scripts are run in the TestNode environment. The new script runner uses TestGen4Web's htmlunit-interpreter package. This improves TestMaker's ability to natively run TestGen4Web recorded unit tests.
• TestMaker 5.1 offers many minor improvements and bug fixes to the results and reporting capabilities. The charts for a load test now include a legend of concurrent user (CR) levels. We fixed an issue where load tests greater than 12 hours show negative horizontal axis values in the resource distribution charts. We removed the load test summary report from the output panel. We added new options to the TestScenario definition for a load test to automatically save all charts to the results directory and create an HTML-based summary report.
• We fixed many problems with installing and running TestMaker on Mac OS X, including an installation problem that prevented the Resource Monitor (PTTMonitor) from running.
• We updated the documention by correcting many spelling and grammer problems and including new sections for QA Testers and IT management. Additionally, we added new documentation to show the options to log results data directly to a Relational Database Management System (RDBMS)and added documentation to show how to set TestMaker and TestNode memory settings.
• We did an overhaul of the TestMaker classpath and supporting Java Archive Resource (JAR) files.
• TestMaker runs on Windows XP, Windows 2003 Server, and Windows Vista. TestMaker 5.1 and later is no longer compatible with the old Windows 2000 and NT operating environments because of a bug in Windows that limits the size of a classpath.
• TestMaker 5.0 shipped with a typo in the element in a TestScenario. The element was incorrectly named (notice the extra s.) TestMaker 5.1 accepts the correct and incorrect versions.
• Added the File menu -> Open Scenario command to open TestScenarios.
• Added new capability to TestScenarios to control timeout values for usecases (unit tests) that take too long to operate.
• Added an option for the transaction distribution charts to use a moving data average instead of inserting every data point.

A complete list of changes and their associated bug reports is found at: http://www.pushtotest.com/Docs/vreleasenotes

TestMaker test scenarios generate a lot of useful data. TestMaker 5.0 introduced standard charts to visualize the data into actionable knowledge, including the Scalability Index, Transaction Distribution Charts, and Resource Monitor Charts. TestMaker 5.0 also introduces a result log archiving system and a Performance Comparison Utility to generate charts to compare results between operations of test scenarios.

TestMaker 5.2 will introduce new results analysis functions:

1) Transaction logs will include the "step" times for the operations within a

2) A new charting function will summarize logged results data in new and flexible ways.

TestMaker 5 creates a lot of data that can be put into a chart. Here are the base data series:

• Test Scenario (name)
• Test Node (name)
• Concurrent User levels (CRs)
• Data Index (Message size)
• Transaction (Use Case)
• Sequence name
• Step (run) name
• Pass/Fail status

I put together a new presentation to introduce you to the Enhanced Results Analysis functions. Download the presentation at http://downloads.pushtotest.com/tm5/PushToTest_ResultsAnalysis_20071220.pdf

-Frank

While WSDL is often derided for being simply awful, it does provide developers with a standards-based programmatic way to know how to invoke services. The Java developer community is finally seeing some advancement in the tools to work with WSDL. For instance, it is possible to look at a WSDL document and compare it to the unit tests of the services. The comparison shows the amount of test coverage a developer has on the deployed services.

Some tools such as Eviware's soapUI 2.0 are delivering code coverage estimates while developers use the tool to build test suites. Other tools like BEA WebLogic Workshop 10.3 (now in preview release) are focusing on graphic utilities to work with WSDL.

Maybe these tools will extend SOAP-based Web service viability as a universal remoting capability for a little while longer. If it isn't SOAP and WSDL, then what is it? REST, XML-RPC, FastInfoset, what?

-Frank

Last week PushToTest hosted the first of a new series of 2-Day Hands-On seminars on Test Automation. The event sold out and the room was packed with eager students. Some traveled from Michigan to the seminar in Silicon Valley, California.

Here are some of my notes from the seminar:

1) The students at the seminar were from a rich cross section of backgrounds and experiences. We had software architects, software developers, QA testers, database administrators, and project managers. This shows me that test automation is a multi-discipline effort. It is not enough anymore to throw testing over the wall to a bunch of testers and not worry about the results.

2) TestMaker is in better shape that I had thought. In the days that lead up to the seminar I was concerned that the usability in TestMaker would make the hands-on portions difficult to follow. I was happy to none of the students ran into any roadblocks that kept them from completed the workbook of tasks. (This is not to say that we can put off usability improvements in TestMaker. Usability is a big issue in my eyes.)

3) The students universally told me they are not finding commercial test automation tools that are appropriate for Web application, Service Oriented Architecture (SOA,) Ajax, Web service, and Web 2.0 testing. As a result they often have one tool for Web application testing and another tool for Ajax testing.

4) Many students liked the informal style of the presentations. They were looking to gain an understanding of testing in the 21st century and take home tactics and methodology they will use in their own environment.

5) Several of the students are working in SOA environments and were interested to learn about TestMaker integration with registry/repository products. See here for details

The next seminar events are in London, New York, and then back in Silicon Valley. For details check http://www.pushtotest.com/Docs/training/seminarform.

-Frank

BEA announced WebLogic Server 10.3 Tech Preview is now available.

http://commerce.bea.com/products/weblogicplatform/weblogic_prod_fam.jsp

Here is my take on WLS 10.3 preview:

"Lightweight" Server - They reduced the footprint to make it a smaller download. The new version strips out Workshop IDE, various JVMs, Upgrade Wizards, Database Drivers, and the console optional. The installer downloads these optional items as the user determines.

Iterative Development and faster startup - BEA is finally making it easier and faster to do iterative development - where you make changes in your code and redeploy over-and-over quickly. In my opinion JBoss owes a large part of its early success to delivering iterative development to developers.

Web Service stack - WLS 10.3 comes with a new SOAP stack for JAX-RPC (J2EE 1.4) and JAX-WS (Java EE 5) Web Services.

SCA Support - Still in the "coming soon" category. SCA should be a good alternative (to EJBs) deployment description language for services running on WLS. BEA is still working on how they will package SCA support.

Spring 2.0.2 now, 2.1 by GA

Java 6 support, finally!

Dojo client support with a publish/subscribe mechanism. It sounds a lot like DWR

C# JMS client - Enables .NET applications to be a client to the BEA JMS message bus, including persistent messaging.

SAML 2.0 support (new) and an update to SAML 1.1 for Web-based security.

Seems like a good minor upgrade to a good application server.

-Frank