Open Source Test

We get the following three questions a lot from perspective users and customers. Here are my stock answers:

1. I am a bit confused about the license. Can the client use the GPL-d system without commercial license? Or is the commercial license mandatory? Also, is there a difference between the commercial and the GPL-d system?

PushToTest distributes pre-built and ready-to-run TestMaker under a simple free commercial license. The license limits you to run performance tests up to 200 concurrent simulated users. We sell licenses for additional simulated users.

PushToTest also distributes the source code to TestMaker under a GPL version 2 free open-source license. The code does not come with the installer and some other options. A comparison is found at http://www.pushtotest.com/Downloads/compare

Most of our users download the pre-built version. They don't have the technical skills necessary to compile, install, operate, and support a test platform. That's our value add to the software source code. And, we generally cost a fraction of the commercial test tools.

2. Do you know whether pushtotest has ever been used for WebMethods? If so, do you have a reference or evaluation or article? My client mainly uses the messaging capabilities of WebMethods (jdbc-adapter, MQ-adapter and trading networks).

We are partners with Software AG. We provide a Software As A Service (SAAS) interface between the WebMethods X-Registry. We also use TestMaker to test ESB and BPM products for scalability and performance. See http://soakit.pushtotest.com for details.

3. Is the monitoring tool part of PushToTest or do you use something like Glassbox?

Monitoring is part of PushToTest TestMaker. Our monitors observe CPU, Network, and Memory utilization and correlated the findings back to the test operation automatically. Plus, we are working on Glassbox integration. See http://bugs.pushtotest.com/ticket/61. We also integrate with DBA InfoPower's database root cause analysis and mitigation service.

We will be renovating the Web site shortly to address these questions.

-Frank

I've been doing some thinking lately about how we build and test applications. I'm no agile expert but I am impressed with the concepts of test-first and iterative development. Waterfall seems like it is designed to kill developers, testers, and projects by its nature.

I've never really understood the test-first thing in its entirety. In an SOA environment, it seems to me that test-first would put the emphasis on defining interfaces first and building tests against the interface (both good things.) The interface is just an interface. So I wonder if having some of the application logic implemented behind the interface is required to really do test-first right.

It seems to me that some sort of enhancement of agile could help here. One where I build the application and the test at the same time and iteratively.

I am trying this out using an open-source Ajax development framework (Appcelerator) and my test framework (TestMaker.) This is just a bunch of notes right now. I'm hoping to have something coded in the next 2 weeks. Here's how it would work:

Appcelerator uses non-standard tags in HTML to implement Ajax functions. For example:

<script src="javascripts/appcelerator.js"
 type="text/javascript"></script>

This provides Appcelerator's function library to a page.

<input type="text" id="username" validator="required" decorator="required" fieldset="login"/>

When the page loads Appcelerator provides Ajax functions for validator, decorator, and fieldset attributes of the <input> tag.

<input type="button" value="login" activators="username,password" id="button"
on="click then r:login.request" fieldset="login"/>

The fieldset attribute is used to link <input> elements (or divs, spans) for the purpose of sending a message. The example above links the username to the submit button. When the user clicks submit the "on" attribute tells Appcelerator to click the submit button and send a login.request message to the server handling the log-in. (Appcelerator implements the server side component and the messaging for you.)

This kind of tag framework approach to application composition and development is wide spread and makes application development easier. In the above example, I didn't have to do any JavaScript work to deliver an Ajax enabled log-in page.

My thought is to extend the tags to include test definition tags. For instance:

<script src="javascripts/testmaker.js" type="text/javascript"></script>

<input type="text" id="username" validator="required" decorator="required" fieldset="login" test_input_data="dpl:mycsvfile field:username"/>

<input type="button" value="login" activators="username,password" id="button" on="click then r:login.request" fieldset="login"
test_verify="verifyTitle:Main menu"
/>

In the example above, when the page loads the testmaker.js functions respond to the test_input_data tag by reading from a comma-separated-value (csv) file using a Data Production Library (DPL.) The first row of the csv file defines field titles. TestMaker sets the username input field with the username value in the csv file.

When I want to test the page I use a test runner that loads the page and clicks the login button. The response to the login must be a page with a title of "Main menu". Otherwise, the test throws an exception and I could that as a failed functional test.

What I'm after here is a way to build the tests as we build the application.

I would love your feedback, thoughts, and ideas on this.

-Frank

Jeremy Geelen of SysCon TV interviewed me at the JavaOne 2008 conference. A streaming video of the interview is at: http://tv.sys-con.com/node/674567.

In the interview we talk about moves in the Open Source Test (OST) world to build a competitive test and operational management product stack. The stack provides record/playback, test scripting, functional testing, load and performance testing, and business service monitoring. The advantage of this stack is that it works in all environments: developers use it to turn unit tests into functional tests, testers use it to turn functional tests into load and performance tests, and IT managers turn functional tests into business service monitors. All in the same product!

I will be in Los Angeles/Anaheim next week to teach the Open Source Test Workshop. Tuesday, September 30, 2008 from 8 am to 12 pm. This is a free event! Here are the details:

Software developers, QA testers, and IT managers are challenged to rapidly build and test Ajax, Service Oriented Architecture (SOA,) and Rich Internet Applications (RIA) in a time when schedules are short, budgets are tight, standards are few, and much of this is new technology! Find out why AMD, Ford, and The Jackson Laboratory succeed with open source build and test software as a more affordable and flexible option to the traditional test vendors.

Learn how you can leverage open-source RIA application server technology from Appcelerator to rapidly build Ajax applications. Learn how you can leverage open-source testing tools (Selenium, soapUI, PushToTest, TestGen4Web, HTMLUnit) for functional testing, load and performance testing, and business service monitoring, with more flexibility than traditional solutions provide.

Here is the agenda:

Module 1: The why, what, and how of open-source build and test tools vs. traditional tools
Module 2: Building applications with Rich Internet Application technology without learning JavaScript
Module 3: Testing using record/playback and scripting tools
Module 4: Identifying and solving performance bottlenecks in Rich Internet Applications (Web 2.0 and Ajax)
Module 5: Service Oriented Architecture (SOA) testing
Questions and Answers

Register Now To Guarantee Your Seat At The Workshop

Six Questions to Consider Before Building a SOA Testing Team

SOA Expert Mike Kavis interviewed Randy Rice, Jim Giddings, and me for an article in CIO Magazine. The article discusses the challenges in testing and operational management of Service Oriented Architecture (SOA.)

Here is an excerpt:

What is the biggest change that SOA brings to testing?

RR: Randy states that "accessibility to services" or the "headless nature of services" creates new challenges for testers. Testers will have to test many services without the aid of user interfaces and must perform rigorous testing to validate these services. Randy says that testing services reminds him of the challenges of testing drivers which require test harnesses and testing frameworks.

FC: Frank talks about how services are "always available" which is different from n-tier architectures where "you are in control". With SOA you have services calling other services and in the world of mashups, testers do not always have knowledge of how these services will be used before deployment. Scale and performance are also high on Frank's list of challenges.

JG: Jim says you should expect frequent testing cycles much like agile projects and a test driven development methodology should be employed. Jim states that testing governance becomes critical in an SOA. Like his peers, Jim discusses the challenges of the headless nature of services and performance testing.

Read the rest of the article.

A business partner asked us recently: Where is PushToTest's primary audience? And, what are their job functions? We use Google Analytics to track our Web site activity. Google reports the following for August 16 to September 15:

United States, 3,315, 36.39%
India, 1,033, 11.34%
United Kingdom, 492, 5.40%
Germany, 386, 4.24%
Canada, 293, 3.22%
France, 260, 2.85%
Brazil, 204, 2.24%
Australia, 175, 1.92%
Costa Rica, 170, 1.87%
Italy, 166, 1.82%

I wish someone would invite me down to India for a couple of weeks to do a consulting project, teach the PushToTest methodology, and let me see the country.

I did a quick query of our registration list. Of the people that describe their job function:

      654  Test\QA Engineer                                   
      403  Developer                                          
      257  Project\Program Manager                            
      256  Application\IT Architect                           
      201  QA Manager                                         
      166  Consultant                                         
      121  Student\Faculty                                    
      112  Performance Engineer                               
       98  Director\VP - Application Development\IT           
       98  Business\IT Analyst                                
       94  Partner                                            
       86  Executive\C-Level                                  
       85  Portfolio Manager                                  
       81  Process Manager                                    
       76  Director\VP - Quality Assurance                    
       76  Other                                              
       75  Product Manager                                    
       64  Director\VP - Project Management Office            
       50  VAR\OEM Partner                                    

When I look at page view activity I see an equal number of software developers, QA testers, and IT managers. Earlier this year I spoke at the Selenium User meeting. I asked for a show of hands on the questions:

How many of you are software developers that also test?
How many of you are testers that also write software?

90% of the attendees answered 'yes' to the above questions. Maybe the message we've been yelling for so long is taking effect! And it a very good way!

-Frank

We're packaging a minor bug fix and feature enhancement release. This will be TestMaker 5.2.1.

The following is a list of changes:

#289 Selenium waitForCondition support
#265 Data-driven Tests Need hasMore() function
#272 Installer does not install Monitor For windows (0 bytes)
#295 TestMaker must not erase TestScenario comments
#296 Selenium RC not working with Firefox 3
#237 PTTMonitor throws NPE on second call to stop function
#239 Exception trace prevents Summary report

Changes are found at http://bugs.pushtotest.com.

Thanks for all those that provided suggestions, patches, and feedback!

-Frank

We are building Selenium support into PushToTest TestMaker 5.2. The idea is to support Selenium IDE in Firefox for authoring and replay of tests in the browser. TestMaker will support Selenium RC to test Web applications across browser and operating environment versions. TestMaker will reuse Selenese tests in TestMaker's distributed test environment using HTMLUnit and Rhino as load and performance tests and business service monitors. Selenium tests have full access to operational test data using TestMaker's Data Production Library (DPL) system. We posted details at http://selenium.pushtotest.com.

For Selenium users this means they have a new option for Selenium training and support. PushToTest offers Selenium training courses and professional support agreements. The move gives organizations new uses for their Selenium tests as load and performance tests and service monitors.

The new Selenium IDE and HTMLUnit/Rhino support is already in the code base and will appear in TestMaker 5.2 beta 6 next week. The code is already in svn at http://dev.pushtotest.com/svn/tm5.

We added Selenium RC too. Tests may run in our TestNetwork grid in a similar fashion to Selenium Grid.

We would appreciate any feedback on our approach.

Frank Cohen
CEO and Founder
PushToTest

A bunch of small things today:

We're making good progress towards shipping TestMaker 5.2. Click here to view the remaining change list.

Building Selenium support for testing Ajax has taken us the past 4-5 weeks. We've found a number of problems and solved most of them. We found problems in the SeleniumHTMLUnit, HTMLUnit, and NekoHTML. We will contribute back all of the changes back to their respective projects.

Jeff Haynie of Appcelerator is extrodinarily fast at finding JavaScript and HTMLUnit problems!

Our engineering community are "heads-down" to get a 5.2 release out by Friday August 22, 2008.

We switched to svn: http://dev.pushtotest.com/svn/tm5

We still need to add a solution to the link="locator" library for SeleniumHTMLUnit.

TestMaker 5.2 will also support Selenium RC!

Watching the kids return to school and the Summer slowly slipping into Fall. Got my first issue of Powder magazine

-Frank

I will be speaking at STARWest, the software test automation conference, and PushToTest will be exhibiting at the conference. STARWest runs September 30 to October 3, 2008 at the Disneyland Hotel in Anaheim, California. PushToTest will be in booth 44.

I will be giving a talk on Open Source Test titled "Getting Open Source Test Tools to Work Together". Budgets are crunched and yet we still need to test. Open Source Test (OST) tools make test creation easy and efficient. Web, SOA, SOAP, and Ajax protocols are on the move and these tools provide much needed support testers.

We're planning to give away a new open-source test platform on a shiny new memory stick (pcmcia). I would be glad to send you one in the mail if you cannot attend the conference. Send me an email at frank@pushtotest.com with your address. I'll do my best to send these outside of the USA.

-Frank

Watch the SearchSoftwareQuality Webcast on architecting SOA for performance. The Webcast is presented by Frank Cohen of PushToTest.

Software developers and testers are challenged to develop unit tests, functional tests, and load tests in a time when AJAX, REST, Service Oriented Architecture, and Web application development is expanding greatly. Frank Cohen, the leading authority on testing Web, AJAX and SOA applications, will share a proven methodology to find and solve scalability, performance, and reliability issues. Frank will show how to apply the methodology using TestGen4Web, Selenium, TestMaker, Glassbox, soapUI, and many other open-source test automation projects to develop better AJAX applications.

View the WebCast here.

Download the slide deck at http://downloads.pushtotest.com/200808/SSQ_SOADesign.pdf

Congratulations to Vinay Srini and SpikeSource on shipping TestGen4Web 1.0. The new software makes it easier to record functional tests of Web applications and fixes bugs. TestGen4Web 1.0 will appear in TestMaker 5.2. We worked closely with Vinay to make sure TestGen4Web tests fully support TestMaker Data Production Libraries (DPLs) to provide tests with operational test data at test runtime, to support the latest HTMLUnit to better support Ajax tests, to support Firefox 3.0, and to fix problems where the new Assert-Text-Exists function was not throwing exceptions correctly. -Frank

I am very happy to announce that Peter Schumacher joined PushToTest as Director of Professional Services. Peter participated in the TestMaker User Community since 2001. He was very early to see the potential to repurpose unit tests as functional tests, load and performance tests, and business service monitors using TestMaker. That wasn't an easy transition as Peter is one of the original LoadRunner power users. In 1994 Schumacher was Mercury Interactive’s first service partner for LoadRunner for SAP and SAP BASIS. Peter will lead the new Professional Services group to deliver PushToTest expertise at building, testing, managing, and monitoring SOA, Ajax, and Web applications. Peter is a peter@pushtotest.com and works out of our new Austin, Texas office.

Official announcement is at:
http://downloads.pushtotest.com/Schumacher_Announcement.pdf

-Frank

I am attending Dr. Dobb's Architecture and Design Conference in Chicago this week. Attendence is about 1000 software architects, developers and designers. The sessions have been all over the map: Some terrific like Brett Schuchert's talk on Test-Driven Concurrency, some a waste of time like the Waterfall versus Agile debate. It did not make sense to have Terry "Mr. UML" Quatrani arguing for Agile and Scott "Mr. Agile" Ambler arguing for Waterfall. I would like to hear how people are using agile in off-short development environments.

Brett showed examples to introduce concurrency and thread deadlock problems when writing threaded Java applications. Brett recommends ConTest to expose and eliminate concurrency-related bugs in parallel and distributed Java applications. He showed ConTest scheduling thread execution to uncover race conditions, deadlocks, and synchronization problems. Brett thinks the Java developer community in 5 years will mostly be doing concurrency programming using Scala or Haskell instead of using the built-in Java concurrency functions. ScalaTest is a unit test framework for testing Scala applications for concurrency.

I gave two talks at the conference. Download the slides to the talks here:

Architecting Services for Testability, (Acrobat PDF format, 8.7 Mbytes)

Architecting SOA, BPM, and CEP Applications Today, (Acrobat PDF format, 4.6 Mbytes)

Before my Testability session I asked conference delegates their opinions of the conference. Here is what I heard:

• Wish the presentations were more on the practical side. How do I do this? Sessions that present the theory. What I want is more of the "Here is how we did it."
• We're here because we agree on test driven development. So tell us how you are practicing test driven development.
• Need something that compiles, need something that I can mock this up, not trivia side.
• More examples would be great.
• Tutorials gave practical examples,
• Monday's sessions were good length, tues-wed were shorter.
• Lesson's learned (T-Mobile and Bank of America) SOA experiences. Talk to to the people that actually did the work.

Justin Gordon introduced himself. He recently developed the Dependent Object Framework (DOF.) DOF enables jUnit testing and test driven development against code that depends on objects that are persisted (e.g., database), more efficiently than alternative techniques such as SQL scripts.

I'm glad I attended. Dr. Dobb's is a good conference. I hope I get to come back for the next one.

-Frank

Norm Dressler send the following message this morning. I feel like a proud father! -Frank

--

I wanted to extend a 'thanks' to the PushToTest team and other open source developers/users on this list who assisted me in my testing endeavors.

We have successfully completed our scalability tests on a SOAP interface server and identified bottlenecks that allowed our system to scale from about 10 tps to just over 50 tps with a clear path on ways to improve that even further.  I suspect I will be promoting TestMaker in future projects as the opportunity arises.

Again thanks--

Norm Dressler
Senior Technical Architect
Emergis - A Telus Company

Back in 2006 I wrote a 500-page book titled Java Testing and Design. It was my first solo book and the experience almost killed my marriage! The reviews were pretty consistent: "It isn't the 'web-testing bible' so many programmers have been looking for, but it is proof by example that automated testing of web-based applications doesn't have to be any harder than any other kind of testing." So much has happened in the software industry since I wrote that book. I'm thinking it might be a good time to write a new book on Web testing. I am seeking input, feedback, and thoughts. How would a book on Web testing would help you, your organization, and the world? Click here to let me know your thoughts. Thanks.

-Frank

Once a month I make a recommittment to build PushToTest as a company. I'm proud of what we accomplished in the past month. We substantially finished TestMaker 5.2, delivered solutions and patches to customers, and designed several new features. In the next 4 weeks I expect we will finish the Selenium functions and launch TestMaker 5.2.

In the next 2-3 months I expect we will begin work with a new customer to test their Appcelerator-based Ajax application. This likely means many bug patches and minor feature upgrades to TestMaker 5.2. I also expect we will begin work on TestMaker 5.5. The new software will feature a Mercury LoadRunner Script Runner and Repository/Scheduler functions.

I'm proud to recommit to PushToTest's success.

-Frank