A short list of companies at the JavaOne Pavilion that look interesting to me:

Krugle is a source code search engine. It spiders your organizations source code repositories and lets you track code definitions. For instance, when you fix a bug in a well used, and shared, code library Krugle shows the other projects that use the same bug-laden code.

Style Intelligence has 101+ dashboards, reports, scorecards, and other visualizations of data.

Skyway Software makes an open-source application development environment. You build everything in their Eclipse plug-in. It's wonderfully well design user interface makes construction of apps very easy. I'm going to look into this one with the hope that I can build a bunch of sample code for PushToTest TestMaker users quickly.

Determyne is an open-source transaction-level performance monitoring solution for JEE applications. I'll be investigating this to incorporate into TestMaker.

-Frank

JavaOne 2008 struggles to retain its importance. Read my full report at:

http://www.theserverside.com/news/thread.tss?thread_id=49332

-Frank

Jeremy Geelan of Sys-Con interviewed me at the AjaxWorld conference last month. He asked me about trends I am seeing in Web 2.0 and Ajax development at enterprise IT shops. I also talk about what its like for software developers and testers to work in Ajax environments.

http://www.sys-con.tv/read/540772.htm

-Frank

Java developers and QA testers are challenged to develop automated tests in a time when Ajax, REST, Service Oriented Architecture (SOA), and Web application development is expanding greatly. Open Source Test Automation has emerged as a more affordable and flexible option to the traditional vendors.

Open Source Test Automation vendors Eviware (maker of soapUI)  and PushToTest (maker of TestMaker) will have some exiciting news to announce at the JavaOne conference. The announcement will leverage our abilities to deliver SOA testing best practices using open source tools and methods.

I am finding that customers are increasingly turning to Open Source solutions for their testing needs. We recently wont such large customers as AMD, Amazon, Ford, The Jackson Laboratory and TVGuide. With 160,000 users, PushToTest delivers a single affordable platform to accomplish functional testing, load and performance testing, and business service monitoring. PushToTest integrates soapUI for fast and easy authoring of Web service test suites to run on a set of distributed PushToTest load test injector TestNodes.

PushToTest and Eviware will be exhibiting in the SOA Village in the JavaOne Exhibition Hall in booth 425.

I am available to meet you on Wed., May 7 or Thurs., May 8 at the show.  Please let me know.

-Frank

I'll be giving a keynote talk on SOA Development In The Real World at TIBCO's user conference next week. This will be based on the work PushToTest did to surface the developer productivity and performance differences between TIBCO, BEA, Oracle, and IBM platforms for SOA development. All of the research is available under a free open-source license at http://soakit.pushtotest.com.

If you are attending TUCON then please let me know and I will be glad to meet with you personally.

-Frank

This week I was the speaker at SkillsMatter.com in London. I gave a talk on the PushToTest test methodology and free open-source test automation tools to check Web applications, Web services, and Ajax applications for function, scalability, and performance. SkillsMatter is the new European partner to PushToTest to offer training on our products. In June I am back in London teaching our popular Open-Source Test Automation Bootcamp.

Harry Wood attended my talk and posted a blog entry about his experience at http://www.harrywood.co.uk/blog/2008/04/11/unit-testing-soa-and-mule-talks

-Frank

We've been making very good progress in building TestMaker 5.2. I'm feeling like a proud father and want to show off my child's cleverness. In TestMaker 5 we introduced a new system of ScriptRunners. This is an extensible facility to operating externally created functional unit tests within a PushToTest TestMaker test. We bundled Eviware soapUI and SpikeSource TestGen4Web with TestMaker and provided ScriptRunners for each. As a result of the latest work TestMaker tests share data between ScriptRunners through a Data Production Library (DPL.) For instance, a functional test first makes a SOAP-based Web service request to learn product information codes and then makes a Web application (HTTP) request to make sure customers are finding the products.



PushToTest orchestrates a Functional test by defining the operating parameters and use cases in a TestScenario document in XML format, as illustrated above.

The test identifies a PushToTest Data Production Library (DPL) that will be the exchange medium between the soapUI TestCase and the TestGen4Web recorded test. The soapUI TestCase reads the data value for the SOAP request from a Properties file in the soapUI project. The soapUI TestCase receives the SOAP response and saves the response parameter to the DPL.

The TestScenario instructs PushToTest to then run the recorded TestGen4Web unit test. The test uses the DPL data from the soapUI test to make an HTTP Get request to the Web host. That concludes the functional test.

I am happy to make the above example available for immediate free download at http://downloads.pushtotest.com/tm5/EndToEndTestExample.zip

You will need the latest TestMaker 5.2 (pre-alpha) version to operate the example. See the faq for instructions on getting the code.

Enjoy.

-Frank

I recently received an email asking:

"Is it possible to use PushtoTest to build a test of a 

Web Application that uses Flash?"

It is possible to test Flash applications using PushToTest. We are currently helping FordDirect.com do this. It does not come without some extra work. For instance, by itself there is no record/playback capability to build tests of Flash animations. However, there is a programmatic way to discover the content of ActionScript variables and to activate functions by doing the equivalent of pressing a button or using Flash controls. PushToTest professional support services will instruct you on the approach, how to instrument your Flash animations for testing, and teach our test methodology and testing platform.

-Frank

I was surfing the Web to find an answer to a customer questions and I ran across 

PushToTest version 5.2 (which should be out this Spring) will feature Selenium integrated into the PushToTest test automation platform. See http://bugs.pushtotest.com/ticket/136 for details.) Selenium users will create functional tests of Web and Ajax applications using the Selenium plug-in to MS Internet Explorer, Firefox, Mozilla, and other browsers. PushToTest users have two choices to run Selenium tests in the PushToTest test automation environment:

1) PushToTest 5.2 will include a Selenium test runner. This runner instantiates a Web browser in a TestNode (our distributed test environment,) operates the Selenium test, and reports the results. This is a powerful choice to run Selenium tests as functional and regression tests, and business service monitors.

2) PushToTest 5.2 includes a Selenium-to-Java and Selenium-to-Jython transformation utility. The utility creates a Java class (or Jython script) that implements the test steps using the HTMLUnit framework. HTMLUnit acts like a browser, including operating Ajax applications using Rhino (the JavaScript engine from the Apache Mozilla/Firefox browsers.) The is a powerful choice to run Selenium tests as load and performance tests.

For both of the above options, Selenium tests receive dynamic operational data at runtime from PushToTest Data Production Libraries (DPLs.) For instance, a Selenium test that operates a sign-in page receives the account number and password from a DPL that accesses data from a comma-separated-value file (or a relational database or custom DPL.)

Additionally, Selenium tests produce results data that PushToTest Results Analysis engine renders into hundreds of charts. And, these charts correlate Selenium test operation to resource utilization (CPU, Network, Memory, Threads) in the back-end server.

We have an "alpha" quality version of the Selenium-to-Jython transformation utility to give to you today. This is from a contribution from Dominique and Olivier at Denali. All you need to do is ask.

-Frank

I often get asked questions like "What does PushToTest do?" PushToTest is a test automation solutions business. We surface scalability and performance problems in Web-based information systems. Our PushToTest solution drives a Web system like a real user and correlates the activity to identify broken functions and hot spots of performance issues. The same system then operates tests periodically through the day to monitor an enterprise's services and provide proof-of-service reports. See our "white paper", http://downloads.pushtotest.com/PushToTest_WhitePaper.pdf for a more complete answer. I often get asked a follow-up question that is usually asked like this: "I was wondering if you could provide me a sample of the outputs the system generates (or can generate)." The PushToTest software produces a summary report that shows the Scalability Index to the application, the resource utilization as the test operated, and the performance as load increased. This summary report is an HTML document and is easily shared among your team members. "Click here", http://downloads.pushtotest.com/tm5/ExampleSummaryReport/index.html for an example summary report. I often also get asked about the PushToTest methodology and techniques to find and solve performance issues. The question is often asked like this: "Once you run a test and the results are not acceptable then how do I leverage the PushToTest tools to find the code that is running slow?" PushToTest version 5.1 keeps track of the steps for each transaction. For instance, the following snippet of a TestScenario implements a test use case where four steps are required for each transaction.

<test>
   <run name="Open Account" testclass="com.pushtotest.WebAppTest"
method="testOpenAccount" langtype="java" />

   <run name="Transfer Funds" testclass="com.pushtotest.WebAppTest"
method="testTransfer" langtype="java" />

   <run name="Check Transfer" testclass="com.pushtotest.WebAppTest"
method="testValidateTransfer" langtype="java" />

   <run name="Close Account" testclass="com.pushtotest.WebAppTest"
method="testCloseAccount" langtype="java" />

</test>

In the above test PushToTest operates four steps: Open Account, Transfer Funds, Check Transfer, and Close Account. PushToTest keeps track of the total transaction time and the time for each step. The time of each transaction and step is stored in the transaction log file. When a particular step takes longer than expected you can break open the transaction logs for the application server and learn what was going on at that point in time. We have been hard at work over the past two months building an Enhanced Results Analysis engine. The new feature will appear in PushToTest version 5.2. It is already working and we are happy to provide it to anyone that "asks", http://www.pushtotest.com/ask_a_question. The new engine makes it much easier to visualize broken functions and hot spots to performance. Consider the Step Report that visually charts the average duration of each step across 10 equal periods of time of a test. The reporting is great but that still leaves the correlation effort up to a software engineer. And, it is a manual process to go from PushToTest logs to the application server logs. Our planned solution is to incorporate "Glassbox", http://www.glassbox.com into PushToTest version 5.2. Glassbox watches a Java application server for performance hotspots as a test operates and identifies the objects and database connection level issues causing the hotspots of performance. "Click here", http://bugs.pushtotest.com/ticket/61 to learn about the integration project. To learn more about Glassbox "click here", http://www.glassbox.com/glassbox/Download.do?mediaItemId=173 to read the Glassbox user guide. One final question I usually hear goes like this: "Where exactly are your developers located? In other words when you leverage out-of-country developers, where are they and if the customer wanted US based developers can you accommodate that." PushToTest outsources its customer engineering to our Costa Rica-based partner "Avantica", http://www.avantica.net. Costa Rica is a 4-5 hour flight to the United States and Canada. The country is on Central Time (GMT -6) and their English is very good. We normally need about one week to plan for our engineers to be in your office. -Frank

This morning I am glad to introduce you to a new set of resources to help surface scalability and performance issues in Service Oriented Architecture (SOA.) The SOA Knowledge and Performance Kit is a free open-source resource to show you what it really takes to build services using today's leading SOA development platforms.

The Kit delivers an SOA use case design, source code to the implementations of the use case on Oracle, IBM, BEA, and TIBCO platforms, developer journals describing our experiences step-by-step, a Total Cost of Ownership (TCO) calculator, and performance and scalability tests that leverage the PushToTest test automation platform.

PushToTest looked below the surface-level marketing claims to understand the skill sets, domain expertise, and specialization that it takes to be successful. We discovered a wide variety of experiences: some highly successful and others rife with challenges. We measured the amount of developer effort to implement an SOA use case and summarize the findings in the following total cost of ownership (TCO) comparison chart.

This morning we opened the doors on a new center within the PushToTest Web site for you to learn more about SOA, learn about the new composition approach to SOA development and deployment, and download the kits.

SOA Knowledge and Performance Kit

-Frank

Last year PushToTest built a test runner for soapUI and bundled soapUI in the PushToTest version 5 distribution. The advantage to Java developers is three fold:

1) soapUI test suites run in a distributed PushToTest test environment - a grid of test machines each turning out SOAP requests to your application.

2) The PushToTest environment dynamically provides operational data to the soapUI test suites. For instance, the data could be sign-in information or product part numbers. The data production libraries (DPLs) get data from a relational database, from a comma-separated-value file, and from service calls dynamically.

3) PushToTest provides a results analysis engine and performance comparison utility to identify problem test suite performance and problem service calls. Additionally, while the test suites operate the PushToTest Monitor watches CPU, Network, and Memory utilization of the test machines and the target hosting the application and correlates these back to the test suite operation to surface the problem datacenter components.

We increasingly see upper management at our customers look at consumer hits like Facebook, Flikr, Google Maps, and tell their engineers "We want that!" At the same time Java developers in 2008 have good tools to service enable their classes. The interface may be SOAP, or any number of others including REST, Ajax, JMS to a message queue, RMI, etc.

We are seeing customers use PushToTest in service environments. They tend to start service enabling a few classes and wind-up with a grid of interconnected services. While soapUI by itself is a terrific tool for working with individual services, the nature of service enabling Java classes means most organizations need a test methodology, patterns, and tools that work in a grid of services.

Customer success at testing in a grid environment is the motivation behind a new partnership that eviware and PushToTest announced this week. Java developers benefit when two open-source 100% Java projects partner to solve service testing challenges and help us test in a grid environment.

-Frank

We are getting very close to the PushToTest 5.1 release - hopefully this week! New datasheet and white paper downloads are available to explain PushToTest and open-source test automation:

PushToTest 2-Page DataSheet

PushToTest White Paper: The Importance of Test Automation

Both are in Adobe Acrobat (PDF) format.

-Frank

Dan Farber of ZDNet posted a good article that talks about user experiences while tryign to understand open-source product offerings.

In the article he talks about a user's experience coming to grips with SugarCRM's open-source offering not meeting the user's needs. I posted a reply that you can read here. Below I a copy of what I wrote.

-Frank

--

Hi Dan and Larry: Thanks for bringing the ZD communities' attention to this issue. My company PushToTest is an open-source test automation company and we frequently need to explain our dual license model to users and prospective customers.

There are a lot of different open-source business models. I will be at the Open-Source Think Tank (http://thinktank.olliancegroup.com/) next week to meet a hundred or more fellow CEOs of open-source companies to talk about this. I want to make sure your readers know that there is no single truth when it comes to knowing exactly what you get when a company says it is "open source."

By the way, Simon Phipps, the Chief Open Source Officer at Sun Microsystems, led a very heated conversation about open source last year. Simon put forward that there needs to be a renewed effort at OSI (http://www.opensource.org/ ) to shame people, groups, and businesses that are pushing the open source marketplace in bad ways. SugarCRM came up in that it is a company that claims to be open source but does not allow companies to change and branch because of their attribution requirements. Simon says we need a group like OSI to shame people that aren't really playing by the open source rules.

I personally don't like the idea of shaming people, but there does need to be some easy way to identify the types of open-source licenses to make it easier for users and organizations to make informed decisions.

For PushToTest we publish our SOA, Ajax, Web application and Web Service test automation platform under a dual license. If you want to build your own test platform then download the source code – under a GPL v2 license – and build it yourself! We make a pre-compiled, tested, and ready-to-install version available that comes with a commercial license that is free for use up to 200 concurrent users in a load test, or up to 10 business service monitors. If you want PushToTest professional support – and our necks for you to strangle – or to run at more than 200 users then we sell a commercial license. There are no functional differences between the GPL source code and the commercial product.

-Frank Cohen http://www.pushtotest.com

A lot of QA testers are coming to the realization that testing applications built on Java, Database, and Application Server tools can't get them from here-to-there when using the old-and-trusted record/playback test tools. The problem often starts when a database instance needs to service client Web interfaces with dynamic data at the same time as the middle tier is serving Ajax (XML, String, and JSON) data for rich browser interfaces, and REST and SOAP interfaces with transactional data.

New tools have emerged over the past year - such as soapUI, Selenium, and PushToTest - that are designed for Web application, Web service, and Ajax testing. Pulling these tools and platforms together into something that the average database developer, application engineer, and tester can use requires a lot of effort.

To meet this need, PushToTest is now offering a Bootcamp to train technical folks how to leverage soapUI, Selenium, and PushToTest. Details are found at http://bootcamp.pushtotest.com

-Frank

Todd Bradfute on September 6, 2007 wrote to me:

I've drunk the Eclipse cool-aid. What do you think about having a
PushToTest plug-in to Eclipse? Maybe it's not the top priority for your
engineers, but that IDE feels so much cleaner than the Netbeans Interface
that PushToTest v5 uses.

Then earlier this month a medium-size prospective customer sent us a request-for-proposal that includes using PushToTest as an Eclipse plug-in. There is nothing quite like the scent of a sale to get us motivated! :-)

William Martinez drafted a good first design of a PushToTest Eclipse Plugin. You will find the document at:

http://downloads.pushtotest.com/tm5/PushToTest_EclipsePlugin_Design.pdf

We are seeking your feedback on the design, our assumptions, and how this new plug-in would make you more productive.

-Frank

PushToTest celebrates its 6th year of operation in 2008! It has been very pleasing to see the renewed interest at medium and large enterprises for software testing. A lot of credit to this interest is due to the experts, on-line communities, and media/editors that are following software test automation. From time-to-time PushToTest users (and customers) ask where they can look for information on software test automation.

I put together a list of the people I turn to to get the word out about PushToTest. I am publishing this list starting today at:

http://www.pushtotest.com/Docs/software-test-automation-media-list

The list is by no means complete. For instance, Google Analytics credits visits to our Web site from 245 referring sites! If there is someone you know that deserves to be on the list then please let me know.

-Frank

Every once in a while a confluence of ideas happens around open-source test automation. The confluence of this week is:

• Apache's announcement of Jackrabbit 1.4
• Zephyr's announcement that their private beta of a new collaborative test automation platform uses Jackrabbit
• Jimmy Foulkes (one of our architects) push for PTTWeb to use JSR 170 and specifically Jackrabbit as our new repository for the upcoming PushToTest collaborative test automation platform.

All of these happened this week!

PushToTest is being used by hundreds of thousands of developers now. We often hear users and customers asking for a collaborative test environment that features a repository of tests and a scheduler. We are working on these under the codename PTTWeb. The first piece of the project is the repository.

This is perhaps the most important module of PTTWeb because it provides all the storage functionality required by the other modules. Will serve as the storage repository for test scenarios, resources and scripts as well as test results.

In general this will include the storage of the following areas:

• Test resources: test related artifacts using a file system metaphor
• Test results: all test data results associated to the executed jobs from the different test nodes
• Security: all information related to security access like users, roles, permissions, etc.
• Scheduling: job and its execution scheduling information

In order to avoid spending time on writing code to provide this functionality it is necessary to find an Open Source product/project that provides this service. The Java Community process has a specification for Content Repositories under the JSR 170. There are several Open Source implementations of this JSR. One possible implementation is the Apache Jackrabbit project that fulfills this JSR as well as adding several more features of its own. Jackrabbit is in use by several important projects like JBoss portal, Magnolia Content management system, jLibrary, Sun's OpenPortal Project.

Most providers for this JSR can provide RDMS implementation for the underlying data store. A MySql database could provide the actual storage capabilities for the repository although a file system based repository is much faster. The idea is that the repository should emulate a typical hierarchical file system with similar access permission typical of a file system. Attention will be placed in the design to the appropriate structure of this repository and to allow the definition of the required metadata associated with the possible resources to be kept there.

A general search function will be provided on this repository. For those resources of text type the actual contents of the resource can be searched, otherwise the search will be performed on the actual names, attributes and repository structure for the repository.

The design document for PTTWeb is available for your review, feedback and ideas here.

-Frank

We are making outstanding progress on building the Enhanced Results Analysis engine for PushToTest TestMaker. The internal work is almost done and we are now concentrating on the user interface and integration. We put together a user interface design for your review, feedback, and comments. Feel free to download the design document to learn more.

Here is a preview of how the new interface works.


Custom reports appear below the Real Time Scalability Report. The user adds a new custom report by clicking the + icon. A dialog window appears to let the user choose the chart parameters. You will see a dialog that looks like the following:



The user enters the Definition Name, chooses the X Axis, Y Axis, Series, and Graph Type settings, and clicks the Save button. TestMaker saves the parameters in the <charts> element in the current TestScenario file. The dialog closes and the panel for the TestScenario shows the new chart definition. The following illustrates the user interface after the user adds a new chart definition.

The design enables the user to reuse chart definitions among TestScenarios and among users. The left side of the dialog offers a list of existing chart definitions. The list appears in three sections: chart definitions in the current TestScenario, chart definitions in the TestMaker properties file, and chart definitions in the PushToTest Repository. The Repository is an upcoming feature in PushToTest.

The "downward triangle" icon displays a pop-up list with commands to save a copy of the chart definition to the TestMaker Properties file. This makes the chart definition available to be reused in other TestScenarios. The pop-up also includes a command to save a copy of the chart definition to the Repository.

Pretty cool stuff!

-Frank