Functional Test

Jeremy Geelen of SysCon TV interviewed me at the JavaOne 2008 conference. A streaming video of the interview is at: http://tv.sys-con.com/node/674567.

In the interview we talk about moves in the Open Source Test (OST) world to build a competitive test and operational management product stack. The stack provides record/playback, test scripting, functional testing, load and performance testing, and business service monitoring. The advantage of this stack is that it works in all environments: developers use it to turn unit tests into functional tests, testers use it to turn functional tests into load and performance tests, and IT managers turn functional tests into business service monitors. All in the same product!

I will be speaking at STARWest, the software test automation conference, and PushToTest will be exhibiting at the conference. STARWest runs September 30 to October 3, 2008 at the Disneyland Hotel in Anaheim, California. PushToTest will be in booth 44.

I will be giving a talk on Open Source Test titled "Getting Open Source Test Tools to Work Together". Budgets are crunched and yet we still need to test. Open Source Test (OST) tools make test creation easy and efficient. Web, SOA, SOAP, and Ajax protocols are on the move and these tools provide much needed support testers.

We're planning to give away a new open-source test platform on a shiny new memory stick (pcmcia). I would be glad to send you one in the mail if you cannot attend the conference. Send me an email at frank@pushtotest.com with your address. I'll do my best to send these outside of the USA.

-Frank

Watch the SearchSoftwareQuality Webcast on architecting SOA for performance. The Webcast is presented by Frank Cohen of PushToTest.

Software developers and testers are challenged to develop unit tests, functional tests, and load tests in a time when AJAX, REST, Service Oriented Architecture, and Web application development is expanding greatly. Frank Cohen, the leading authority on testing Web, AJAX and SOA applications, will share a proven methodology to find and solve scalability, performance, and reliability issues. Frank will show how to apply the methodology using TestGen4Web, Selenium, TestMaker, Glassbox, soapUI, and many other open-source test automation projects to develop better AJAX applications.

View the WebCast here.

Download the slide deck at http://downloads.pushtotest.com/200808/SSQ_SOADesign.pdf

Congratulations to Vinay Srini and SpikeSource on shipping TestGen4Web 1.0. The new software makes it easier to record functional tests of Web applications and fixes bugs. TestGen4Web 1.0 will appear in TestMaker 5.2. We worked closely with Vinay to make sure TestGen4Web tests fully support TestMaker Data Production Libraries (DPLs) to provide tests with operational test data at test runtime, to support the latest HTMLUnit to better support Ajax tests, to support Firefox 3.0, and to fix problems where the new Assert-Text-Exists function was not throwing exceptions correctly. -Frank

The software industry is moving forward again to provide new ways to build software. In the 1980's Apple and Microsoft produced operating systems that included features to display standard user interface elements to the desktop screen. Before that if you wanted to display a menu bar you wrote code to draw the menus to the screen and more code to let the user click a menu command. In the past 2 years Yahoo, Google, and Appcelerator began publishing user interface frameworks for browser-based applications.

Today, Yahoo, Google, and Appcelerator publish excellent software development kits to help organizations improve the usability of their Web applications. For instance, I added a menu bar to one of my Web applications by dropping-in a Google Web Toolkit (GWT) widget. The process took less than an hour.

Writing code is only half of the effort it takes to deliver high quality Web services. Service delivery requires testing: functional testing to ensure the application works, load and performance testing to ensure we have enough servers and bandwidth, and service monitoring to ensure we achieve a service level agreement.

Unfortunately, none of the new toolkits comes with a test system. Once you add a widget to your application, how do you test the application? From this perspective, Yahoo User Interface (YUI) and GWT are viruses that spread into your applications. While they help you get more functionality into your application they are little help at service delivery.

Some of the toolkits include support for unit testing and functional testing. For instance, GWT includes a unit test framework based on jUnit. Developers write jUnit test cases in Java to test asynchronous messaging between a widget and the back-end server. Setya has a good introduction to unit testing in GWT here. Developers may use test automation from PushToTest or The Grinder to automate operation of these test cases as load and performance tests. Additionally, some toolkits refer developers to use Selenium for functional testing at the browser user interface level.

Selenium is no cure-all for GWT testing. It is complex, requires learning its command syntax and language, has no sense of the Ajax event model, no operational data production to deliver test data, and has no sense of the schemas used to message the back-end. (PushToTest is working on these issues.)

I am calling for the toolkit publishers to deliver a complete solution. I urge the publishers to deliver their excellent frameworks, the cool widgets, and a powerful test framework and utility. The test framework understands and supports the event model of their widgets, the message schemas, and the widgets are fully instrumented with published APIs to make test authoring easy and efficient.

-Frank

I had an interesting chat with Charles Boyd of the Checkpoint project and The Frontside Software. We discussed ideas on how PushToTest TestMaker could use Checkpoint to provide browser profiles when playing recorded tests from Selenium or TestGen4Web. Here is an edited transcript of the discussion:

Frank Cohen (PushToTest)
I found checkpoint mentioned on a blog about open source test tools at http://www.pushtotest.com/thecohenblog/2008/07/07/four-open-source-test-projects. PushToTest TestMaker is an open source test tool for checking Web apps, SOA, Web services, and Ajax apps for function, load, and monitoring.
A lot of our users started their test careers using record/playback technology.
We provide TestGen4Web and Selenium as record utilities. We play-back using HTMLUnit and Rhino to get the recorded tests to be load tests, functional tests, and business service monitors.
Your work with Rhino is interesting.

Charles Boyd (checkpoint)
Thanks, I use rhino for so much in my day-to-day, it's ridiculous... not just crosscheck
so how do you handle your javascript/ajax stuff now? does httpunit have rhino built in?
or rather, does it simulate the browser host environment?

Frank
We take Selenium and TestGen4Web test files as input, and run them with a simple proxy class to HMTLUnit. HTMLUnit does the interaction with Rhino.

Charles Boyd (checkpoint)
I see, so, for example HTMLUnit can parse <script> tags, and it provides objects like XmlHttpRequest and friends...

Frank
yes, for TestGen4Web we cover all the test action parameters, for Selenium it's about 80%.

Charles Boyd (checkpoint)
can you record on any browser? or only for specific ones.

Frank
only firefox

Charles Boyd (checkpoint)
ok, so would you're thinking that we might be able to take the crosscheck browser environments and use those when running the test suites?

Frank
that's a limitation of testgen4web and selenium. we get questions from customers about authoring tests in IE

Charles Boyd (checkpoint)
Is it that they want to author their tests there, or that they want them to check IE behavior?

Frank
our platform is unique in that we reuse unit tests as functional test, load and perf tests, and service monitors.

Charles Boyd (checkpoint)
how do you turn them into service monitors?

Frank
if they are doing functional testing where they need verify that an app works well across multiple browsers, then we recommend they set up a farm of OS/Browser combinations using Selenium Grid.
They are playing back in the target browser and OS environment.
PushToTest is a distributed environment: the tests run in services called TestNodes. (kind of like a load injector)
when they play a test in a TestNode we need them to use HTMLUnit and Rhino.
In a TestNode they run the test as a functional test (meaning 1 simulated user once through), as a load test (multiple simulated users at increasing levels), or service monitors (1 user running the test periodically.)

Charles Boyd (checkpoint)
hmmm, I'm wondering how you could use the crosscheck browsers environments. It seams, if you're using htmlunit, I'm not sure how it would be possible to integrate the to Rhino runtimes.
because if HTMLUnit defines XMLHttpRequest, and so does crosscheck, which one wins?

Frank
i haven't looked at the crosscheck source yet. maybe you could tell me what it does.

Charles Boyd (checkpoint)
well, first off, it's mainly for unit testing javascript libraries
so it doesn't need a web server, or a network or anything like that
but as you know, the javascript language itself is very minimal, the only objects it defines are Object, Array, Date, String, and a couple of other top-level constructors
everything else is provide by the host environment: Node, Document, document, Event, Window, etc...
crosscheck implements those objects in either java or javascript (depending on the object)
but it has an implementation of those objects for each browser type
at the beginning of each test, it bootstraps the host environment, defines all the objects, and then executes the test code
an interesting note is that it also throws away the entire host environment after each test, so there is no global state that could hang around and interfere with subsequent tests.
when you run a test suite, you specify which browsers you want to target, and it will repeat the same installation, run, dispose sequence for as many hosts as you want to run the test.
so, for example, there is no XMLHttpRequest constructor in IE6, so in a crosscheck test, the same code:
alert(XMLHttpRequest)
would output [Object]
in mozilla but 'undefined' in IE6
other things, like if you have an html element 'el'
el.cssText = null is no problem in mozilla
but in IE it will throw an error

Frank
When crosscheck bootstraps the host environment, are we talking about IE or Firefox itself?

Charles Boyd (checkpoint)
no
we implemented by hand the host environment
crosscheck has an implementation of w3c.Node
w3c.Event
and those implementations are different for IE and mozilla

Frank
wow. that sounds like a ton of work

Charles Boyd (checkpoint)
yes, it is

Frank
it seems to me that PushToTest users would benefit by identifying a crosscheck profile when they want to play their recorded tests.
we have a simple XML-based orchestration document (called a TestScenario)

<run name="CoffeeTest" testclass="BrewBizCoffeeTest. selenium"
                  method="selenium" langtype="selenium">
</run>

this runs a selenium test using our proxy to HTMLUnit and Rhino.
I could see us doing something like:

<run name="CoffeeTest" testclass="BrewBizCoffeeTest. selenium"
                 method="selenium" langtype="selenium" profile="Crosscheck_IE">
</run>

Our proxy would then instantiate Crosscheck with the identified profile
The proxy would call Crosscheck APIs, instead of HTMlUnit APIs.

Charles Boyd (checkpoint)
ah, ok
crosscheck's original purpose was that of a unit testing library, so it lacks some functional testing features. like it doesn't interpret script tags when inserting things into the DOM, and it only like XHTML

Frank
That may be a problem. Have you tried running it against any of the more complicated Ajax pages: Yahoo User Inter (YUI), Google Wdigets, Appcelerator?

Charles Boyd (checkpoint)
only jquery and prototype. We add things to it as we come across it, or we receive updates, but it is not extensive yet.

Charles Boyd (checkpoint)
we've played around with implementing TagSoup

Frank
i like tagsoup and jtidy, both are robust in the terrible html they parse

Charles Boyd (checkpoint)
so how long has PushToTest been around?

Frank
we got started in 2001
we're up to 6 full time people (2 sales/marketing, 1 account exec, 3 engineers)
customers include: AMD, Ford, Perficient, Jackson Labs
We sell support, training, consulting around a commercial version of our open source product.
how about checkpoint?

Charles Boyd (checkpoint)
crosscheck? we've been at it since around 2005. Most of the currently functionality was in place by 2006, and we've been just adding to it piecemeal ever since. It's not our bread and butter so much as an open source project which is an auxillary to our consulting business.
Our company is actually The Frontside Software
there are three of us, all engineers.
I've been planning to do some major rework of crosscheck this fall, just so that it can do alot of the things we've been talking about.
If you want/need any help with the rhino/javascript/browser quirks end of things, regardless of whether you use crosscheck or not, we'd be happy to help.

Frank
great

Charles Boyd (checkpoint)
ok, what is your esitmated release date?

Frank
Early August 2008

This was a nice, long weekend here in California. We celebrated Independence day on Friday. It seems like everyone is on vacation. I had a chance to see what other open-source test (OST) projects are working on. Here are four of interest:

• Crosscheck is an open source testing framework for verifying your in-browser Javascript code against several browser profiles. Crosscheck uses Rhino and provides profiles for Internet Explorer, Firefox, and others. I could see offering Crosscheck in PushToTest when we incorporate Selenium, HTMLUnit, and Rhino for Ajax testing.
  
  
• Cobertura is an open source tool that measures test coverage by instrumenting a code base and watching which lines of code are and are not executed as the test suite runs. Operates only against Java code. The utility is great. The project seems to be dormant for the past year.
  
  
• We're heading into Ajaxland with PushToTest TestMaker. I found Peter-Paul Koch's site QuirksMode. This is a treasure for the frequent times when I find Javascript that runs on one browser but not another.
  
  
• John Resig wrote a blog entry last year on better ways of automating tests and ways to bring JavaScript-style DOM/HTML interaction to the server-side. Having a way to bring this popular idiom to established problem sets seemed like a lot of fun.: automated testing, screen scraping, Web application development.
  
  

Enjoy.

-Frank

We've been moving quickly with TestMaker 5.2. TestMaker 5.2 has a new face to functional tests. The controller panel for a functional test immediately logs to a new dashboard. The panel shows pass/fail status and duration to operate each step of a Functional TestScenario operation. This enhancement supports ScriptRunner execution of soapUI, TestGen4Web, and Selenium tests. For instance, each Assertion step in a soapUI TestSuite logs to PushToTest TestMaker as a step within a transaction.



It's very cool! (If I do say so for the team!)

-Frank