Selenium Load Test and Monitoring Webinar

Hi everybody this is Frank Cohen and I’ll be your host for today’s Selenium Workshop. I hope that it’s a really rewarding experience for you. We’ve decided to host a workshop specifically on Selenium after being contacted by Dzone. Dzone is a pretty popular community for the Java Development Community and I’d say somewhat for our testers as well.

Dzone was working on an update to the RefCard what they do for Selenium. I wrote the first one and they were working on a second one for Selenium 2.0 and they said, “Hey, you know we’d like you to be sponsors of this thing.” And of course we really didn’t do the technical voting on it and so I did. And it struck me at the time that we should be doing a workshop specifically for Selenium. So, that’s what we’ve put together for today.

About PushToTest

Our mission is basically to provide testing to organizations so that we can deliver a more reliable world. We make it very much easier to move from manual to automated testing by using Open Source Testing Tools and we provide support for advanced testing including advanced software development and methodologies like Scrum and Agile, but we also do low testing. And basically we’re an Open Source company and we believe that an Open Source Testing or OST for short, because it delivers the benefits to everybody and everybody can use these tools immediately without having to go through sales cycle and approvals and all of that, just download it and start using it.

Moving From Manual to Automated Testing

Part of the reason that we got so interested in Selenium is because we saw a bunch of people taking Manual Testing techniques and starting to automate them using Selenium. So, you know in a Manual Testing Process you have kind of a 4-step process typically. The first is you’re given a new piece of software or a new service or a new website, and you do some exploration. The second step then would be to write a test plan then the third step would be to run these test steps and fourth are to recommend changes based on the results that you see.

And you know your role within a company is unique and that nobody else is going to do this kind of effort to be able to actually run the test and then to recommend some changes. The problem with this is that if you go through this loop over and over and over again for each release or each update to the software, you wind up with all sorts of bad side effects, exhaustion, fatigue, insanity, hair loss, divorce, you name it. So, you know what I’m finding is that there is a merging, an Open Source Testing Architecture which kind of glues together all of the Test Authoring tools that are available in the Open Source space.

Open Source Testing (OST) For Everyone

These are tools like, Selenium and Sahi and SoapUI, and then I’m writing classes of objects in structures like fitness and as a unit test, and at the same time most of the organizations that we work for would have already made an investment in some development services like -- maybe a source code or a repository like subversion or continuous integration environment, maybe a Bug Tracker and a Monitoring tool. But at the same time, we as testers are basically asked to do kind of a combination of Functional Testing, Load Testing and Production Monitoring.

OST Architecture

And all of these can be a manual effort unless we adapt it, something which I find is kind of that missing middle tier. This is what I would call a Test Orchestrations Tier and the idea here is that in a Test Orchestration Tier you’d have two kinds of components, (1) would be a Test Operation Console, and that console would give you the capability of doing record and playback to design your test, you know kind of visually. You’d also have Test Deployment unto a grid of machines that would run the test or maybe a cloud testing environment.

You’d have a repository to store all the results and then also a business intelligence tool to analyze the results into your charts that would surface the performance bottlenecks, and at the same time this console would have a second -- or Test Orchestration, they would have a second tier which is the Test Injector. And this would have a script runner to run your Selenium Test and then also of course a way to data-enable those tests, without having to change the test themselves or recode them.

And then of course having some sort of monitoring capability built into these means that, the console as it operates your Selenium scripts could observe what’s going on in the backend and be able to correlate those backend efforts to the test that’s operating in the frontend of your application. So, I call this Test Orchestration, and basically when you add Test Orchestration to your environment lined up with this really cool architecture that keeps us from having to get into that kind of fatigue situation.

The cool thing about an Open Source Testing environment like that is that you could plug-in multiple different ways of interoperating with the Application Under Test and Selenium is an excellent choice for that. So, if you have a web application or even a Flex-based application, using the Selenium system to create test that automate the operation of a test, in this case there is -- Selenium is a great choice.

Why Do We Need Test Orchestration?

The thing to keep in mind though is that if you’re going to bump into issues where you’re going to need to accomplish one kind of test and Selenium was really written for different kind of test. And for example when I talk to the Selenium Core Developers, a lot of times they’ll come back to me and say, “Hey look, Selenium is not really about testing as much as it is about browser automation.”So, there aren’t any kind of classical test concepts built-in to Selenium itself.

So, there’s nothing in Selenium to tell you like to compare the results of the test that you ran last week to this week. That would be up to your Test Orchestration System, and if you’re coming into Selenium from -- upon a proprietary product like HP’s QTP product, what you’ll find is that QTP is part of an overall platform that gives you all of those different types of Test Orchestration features where Selenium is just about browser automation.

Selenium Browser Automation

So, Selenium itself comes in four basic pieces. There’s Selenium Core which is the code that implements the Selenium Domain Specific Language and then there’s Selenium RC which is the point of Selenium that basically injects some JavaScript into your browser window and then runs the Selenium Test steps that you have created in a Selenium script. The Selenium Grid which then kind of makes Selenium RC work in a grid of machines, so if you had say 5 or 10 blades you can install Selenium RC, and then Selenium Grid would take care of distributing your test to those 5 or 10 machines running the test and retuning the results.

We also have a visual or a graphic tool called Selenium IDE which is a simple record and playback tool. It lets you record Selenium scripts in the Firefox browser. And then you have the Selenium Domain Specific Language which in my mind, it might actually -- the Selenium Language or Selenese as it’s known, I think it’s the single greatest part of the Selenium platform. It’s such a great little language and that I’ve been able to use it literally dozens of times now and I haven’t found a situation where I really just couldn’t get from here to there in terms of building a test script using the Selenium Language because it’s so well thought out. I really like it.

Selenium Architecture

The Selenium Architecture for the current Selenium, I’ve drawn this little picture on the screen here.


Basically you write a Selenium script in a language like Java or Ruby or Groovy, they support for about 8 different languages. The test that you’re writing is going to be calling the Selenium base class and that class act as a client to be able to send commands over to the Selenium RC service. So, Selenium RC service acts as like a mini server. It’s up and running and basically it can handle launching a browser, opening a window, and then injecting some code that it has called the Browserbot.

The Browserbot is written in JavaScript and so it’s running within a browser page. The browser page itself has an iFrame that’s running the Browserbot and then inside that iFrame is another frame which runs the Application Under Test. So, when you tell a Selenium Client to open a certain page what it’s going to do is use Selenium RC to communicate to that Browserbot, the Browserbot is then going to actually execute the open command to open your application inside that internal frame.

So, when application is up and running you’ll see -- say in Firefox or Chrome or IE, your application will come up, you won’t actually see the Browserbot, it doesn’t have a user interface and all of these will be hidden for you inside this frame. So, the Application Under Test of course has it’s own portion of the DOM for the overall page, and then of course any JavaScript that the Application Under Test loads will be running within this page. And so because the JavaScript within the Application Under Test is running in the same page as the Browserbot that means that you could send JavaScript over to the Browserbot, you know like your own Javascript function and run that and then that Javascript function can interoperate with the Javascript running in the page.

So, it’s really quite a sophisticated technique and architecture that comes with the Selenium system. The killer for this architecture though is what happens when the browser requires some human intervention. So, like imagine that you’ve written a test script that moves from a security -- an insecure page when the browser brings up a modal dialog box that says, “Hey do you approve of moving from a secure to an insecure page?” When that modal dialog box comes up it stops any operation of JavaScript running in that page including the Browserbot so… Until some human comes along and clicks on that “OK” button your test has basically stopped.

Selenium 2

So, to resolve that issue, instead of -- the Selenium project has now started to work on Selenium 2 which is also known as WebDriver. In this -- in Selenium 2 you’ve got the client talking to the Selenium RC service and then the RC service is going to talk to the browser’s API. So, there’s no Browserbot kind of injected into the page and consequently the Browser API can work around problems like where the browser requires human intervention.

Selenium 2 is currently in Beta, and you know I’d like to say, “Oh you know, well, everybody should be turning their attention to it.” But Selenium 1 took something like 2 years to get out of Beta and so it might be quite a long time before Selenium 2 is actually shipping, but it’s worthwhile to take a look at now, the Beta is quite functional. let me just pause this moment because we’re about 15 minutes into the workshop and I wanted to make sure that this was on target for what you are expecting. And if you have any questions then please just type them into the questions box inside the go to webinar window. I’m happy to answer them.

Selenium Language basics

The Selenium system delivers a Domain Specific Language that gives you about 200 different commands, and it normally takes about a day for most testers and most developers to learn the Selenium Language. The 200 commands are usually broken down into kind of 5 major groups. You have commands that basically work at a page level, so you can tell Selenium to open a page to select a frame, to select a window, and then any of these page commands then work at a page level.

When you want to work with elements within each of those pages, there’s a whole set of Element Commands. So these are commands like “type” where you are able to enter characters into like a text field but there’s also elements -- there are also commands like “click,” where click will like click a button or click an element.

Element and Validation Commands

Validation is different than Assertions. They’re basically the same name but like verified text present is different than a cert text present, and that Validation Commands will log an error but continue processing the rest of the Selenium script. Whereas an Assertion Command that fails will throw an exception and will stop operating that task. Then the fifth type of command is basically around Storage Commands they -- Selenium implements it’s own like mini little hash map where you’re able to store and retrieve values from the page that you’re operating the task against.

I find that these are pretty worthless because if you’re going to be coding in a high level language like Java or Ruby, there’s much better storage and retrieval commands around data in those high level languages than anything you needed to use within a Selenium context. I see we got a couple of questions. Kamala asks, “Yeah, I believe Selenium is good for web apps only that is the application on web browser only.” Yeah, that’s true Kamala. Really the focus of Selenium is on web applications.

Now, there are extensions to Selenium that I will be talking about for Flex, but you know those are still basically web applications. There are tools like QTP they’re disappointed to see that you can’t use Selenium for like application testing like desktop apps but it works great for web applications. So, Said says, “We’re having an issue recording JWT base screen using Selenium, please cover that in your session.” Again, I have a demonstration of that Said and let’s see, Kamala says, “Yeah, yeah.” So, yup that’s true.

Selenium Locators

In in addition to the 200 or so commands each of those commands often times will need to locate elements within the page and the page is basically constructed in the browser of a Document Object Model or a DOM. It looks kind of like a hierarchical tree where you’ve got kind of a parent or a root element and then all of the elements that you see within the page are in a hierarchical tree order down to the tree from the parent.

So, if you wanted to identify a particular element within a page, you have a bunch of different Selenium Locators to choose from. For example you could just use their generic identifiers, so if you had a form in a webpage titled “loginForm” you could just type that as the locator. If the loginForm in the Html of your page used an ID attribute, then you could say id=loginForm and then Selenium is smart enough to look for all of the elements within your page that have an ID attribute equal to loginForm.

XPath

Now of course, the attribute doesn’t need to be named “ID,” you could also use for example an attribute named “name” and that would have a value of loginForm too. You can also use XPath expressions and this is a very powerful way for you to identify elements within a page but it is XPath 1.0 the spec from the World Wide Web Consortium. There’s no support of XPath 2.0 and XQuery unfortunately. So, XPath might be powerful but you know for the true XPath aficionados it’s kind of disappointing that there isn’t XPath 2 support.

Link Locators

You can also use Link Locators. Imagine that you had an tag, a basic link in the page titled “Continue” that user would normally click on as a link. Well, with Selenium you can say link=Continue and then Selenium is smart enough to look for the text “Continue” in the link on the page.

DOM APIs

Kamala asked the question, “Until now, I use Firebug to identify the elements in Firefox but Firebug won’t work for IE.” That’s true. So, Firebug is this great toolkit, if you go to GetFirebug you’d be able to download it and install it into Firefox. The equivalent of Firebug on IE is called “DevTools” and it’s available from Microsoft itself. If you go to the msdn.com site and just search for DevTools you’d be able to download it. It installs into IE8 and IE9 that gives you function that’s very similar to Firebug.

Chris says, “Dev Tools?” Yup, yup Chris I’m right there for you. I’d say probably the toughest thing about Selenium is that it was written initially before Ajax became so popular. Ajax itself is a technique for using a JavaScript code and Document Object Model to be able to create dynamically running webpage. A lot of times these webpages don’t load another page, so a lot of the Selenium commands are kind of useless in an Ajax setting. For example, there’s a command called Click and Wait in Selenium, that will click a button and then wait for the page to reload.

Ajax Web Objects

In an Ajax environment a lot of times that click will run a JavaScript function which will modify the page without having to reload the page. So, the Click and Wait part of that click command really doesn’t even function in an Ajax setting and that’s because Selenium Tests themselves are really event oriented, they’re not kind of timing oriented. I remember seeing HP’s QTP product for the first time and realizing that it was not only trying to find out what was going on, on the screen but it was also timing out the clicks of the mouse that I was doing. And so, it would automatically add pause commands, and Selenium really -- one of the worst practices for using Selenium is to add pause commands to your script because it’s very easy for the Selenium Test to lose synchronization with the application that’s being tested.

The other thing to keep in mind about this is, you know there’s -- Ajax introduces literally hundreds of thousands of new web objects and those objects aren’t easily recognized by anything and including HP’s QTP product. So, in an Ajax web object if that complicated data structures, you got Dynamic Values meaning that the values themselves can change as you’re operating the test. You also have Asynchronous Events meaning that JavaScript could be running and changing the page that you’re operating the test against while you’re operating the test. And there’s no synchronized like event manager in an Ajax app. All of the Ajax’s JavaScript functions run concurrently to one another, they’re all running in concurrent threads.

So, let me show you an example of this and I’ll show you what we’ve done to solve these issues. Now, let’s see Kamala says, “DevTools good.” Homa asked the question, “Hi. Does it work with Flash or Flex-based apps? Can this be used for Flash objects?” Yes, Selenium does have extensions for Flex and Flash, and I’ll be talking about that in just a moment.

Event Handling and Ajax

This is the Firefox browser and I’m running this sample Ajax application locally. And you can see they’re trying to like emulate kind of a desktop metaphor, you’ve got a start button here, you’ve got different icons, and the icons you can actually drag around. Those icons will open. When you double click on them, they’ll open up windows and the windows themselves you can drag around. The windows have like a window pane kind of ordering so you can have one obscure the other. When you want to add something you can click here and I can add like -- yeah, I’ll add PushToTest to the listing here worth $500 of share, and that’s 100% improvement.

So, if I click here under Save you can see that this application has now inserted an entry into this row. All of this is being done within the browser using a whole bunch of JavaScript and that JavaScript is responding to my mouse clicks as though it’s a desktop application. So, I can pretty easily create a test of this Ajax application using Selenium, and I’ll do that using the Selenium ID Tool. So, I’ve installed the Selenium ID Tool here and it appears under that Tools menu and when I open it up it comes up with its own window, and the window looks like this.

And a Selenium ID Tool is cool and that it gives you a very simple record and playback capabilities. So, I’ll say that I want to create a new test case, I’ll click here on this button and make sure that it’s in record mode, and now when I click around the interface like this, you’ll notice that Selenium IDE is recognizing what I’m recording so I could see -- well, this would be Sencha and they’re doing even better than PushToTest, it’s 105%.

I can even right click on these elements, this helper pop-up and it will appear, and it will show me the different Selenium commands that are available. So, for example I could say assert that the text is present on this page Intel Corporation. When I do that, then the Selenium ID tool has inserted this extra command. Now, once I’ve created a test like this, I can also play it back, I can click here and it will run through the different commands that I’ve created and it’ll show me the status of the operation of each of those commands and notice that I’ve got already an error here.

The error is saying that ext-gen613 is not found. Now, if you recall that I said that Ajax applications can change their name -the names of any of the objects that it creates within that DOM dynamically, so in this case between the time that I recorded the script and when I played it back the JavaScript running this page has changed one of the element’s names and now I no longer have it found.

Accessing Elements

Another option then instead of using a Link Locator that has an ID like this would be to use an XPath expression defined that value instead, so if I click over here and then click on this little targeted form, I can see that Selenium IDE will show me some of the other ways of addressing or accessing or locating that element, and so I can use the ID value but I can also use an XPath expression. And there are a couple of XPath expressions. One would use attributes, the other would use relative positions, and then the absolute position.

So, if I click on this and ran again, you’d see that the test gets further and that it is able to -- using XPath 2 identify this element and move on further in the test. So, the thing that I like about Selenium IDE is that it makes it so easy for me to be able to record and playback a test, but there are some pretty significant downsides to this. First of all, the test themselves are completely procedural. There’s no like looping or conditional execution. There’s no easy way to change this depending on what the outcome is of the test. The second thing is I’m limited to only record and playback within the Firefox environment, so there’s no happiness for us if we’re IE users or Chrome users for that matter -- or Safari.

And then the third thing is that, really for me to be successful with Selenium requires me to get into a coding mode and so for that I would export a test case here. Let’s say I exported into Java as a Unit Test I’m going to save it here as a Workshop.Java and now if I go back out to the file system and I tell it, “Hey, I want you to open this.” Let’s see. And there’s Workshop -- sorry I just need to -- there’s Workshop.Java. So, if I open this now I can see that I’ve got a base of Java class and the class itself is pretty straightforward.

You know I’ve got a Selenium Base Class called Selenese Test Case, that’s going to bind in the Selenium client code automatically so I can identify where does Selenium -- I’m sorry, which browser for the Selenium RC Service to use and then what the base URL was going to be, and then after that I’ve got all of the Selenium commands implemented as Method calls within a Selenium Base Class. So, I can say open, click. I can say type. I can do my verification commands in here like here’s that assert -- or I’m sorry, assert that a text is present.

So, all of these makes a nice Unit Test but you can see that I’ve had to kind of go from record and playback into scripting, and I’ll tell you my experience with the kind of tester community is that testers come from all sorts of every backgrounds, and unless you really know how to code Selenium is going to be pretty difficult proposition for you. And the other thing that I find is that the Selenium team itself really they’re very code oriented, so all of the really cool stuff like release notes and how to articles and things like that are really oriented around how to do it in Java or Ruby as a language.

Let me step back over here. What I’m showing you so far is that Selenium scripts themselves can certainly be adapted to using Ajax, but it really requires you to understand that these Selenium scripts are event oriented and that it’s up to you to insert all of the extra code, all of the extra Selenium commands that are going to keep your test in synchronization with the Application Under Test and that can be pretty tough.

Our Recommendations

Let me just give you a couple of kind of Do’s and Don’ts. In terms of Do, what we recommend is that you create Unit Test that operate at a component level. So, if you’re writing a test that has like a login task and the login task has to fill in a login ID, a login password and then click a submit button, then what we would recommend is that you write that as a component or as essentially a Unit Test that would then be called in sequence to make up a use-case. By doing this component approach to using Selenium then when the test -- or when the Application Under Test changes then you’re only having to do maintenance on the components that change.

We also recommend very much to do -- to go from the recording style of Selenium IDE directly into scripts, those scripts would be Java or Ruby or Groovy, or any of the other supported languages that Selenium supports. What we don’t recommend you do is hang around forever in the Selenium IDE tool because of its obvious limitations. What we also recommend you do is, do use Dynamic Locators instead of locators that locate by ID value, because Dynamic Locators like things using XPath expressions or CSS Selectors, those are going to much more resilient to changes within your application. We also recommend that you work with your development team and explain to them the kind of automation that you need from their work.

Developers, often times will be able to add attributes, be able to add kind of clues within their webpages that will become very handy and very timesaving so if you -- for you, when you’re writing these Selenium scripts. And then like you were seeing in the last slide, do make your test event oriented and not time oriented. So, for example in the Don’t category here, don’t add pause commands to your Selenium Test because you never know where the Selenium scripts are going to run, it could be on a machine that’s running really fast or a machine that has a really slow processor. And when you use these pause commands, a lot of times they’re not very accurate to pause the exact amount of time that you’ve identified.

We also say don’t use Static Values, so if there are -- when you’re writing your test scripts, make sure that you take an object or an approach to writing them, that is each unit test that you write should store and maintain its own values, and you should be using good coding practices like getters and setters for addressing those values. We also say don’t use the Storage Commands in Selenium because they’re much better choices if you’re using a high level language like Java or Ruby, than to use the storage and retrieval commands in Selenium.

We also say don’t practice Test and Trash. I mean this is a methodology that we kind of jokingly refer to and this is what happens when you’re working for a company who says, “Well, you only got a week to test this new application.” And like the third of it has changed, and when you’re thrown into that kind of environment you might think like, “Oh, I just have to get these test written as fast as I can and then I’ll never use them again.”

What we’d recommend is instead approach these Selenium scripts as though they’re software and that they need to be version controlled and maintained, and I think you’ll do much better over the long run. And also, don’t expect the same. So any of the Selenium scripts that you’re writing, you shouldn’t expect that the test itself -- that the Application Under Test itself is going to remain the same. It’s always going to change.

Flash And Flex Testing

There were questions about Flex and Flash testing with Selenium. And basically the Selenium Language got started as a browser automation system, but it really never gave that much thought to like how -- what would happen within components that are running within the browser. So, I’d say probably the most popular of these kind of components would be Flash and Flex components, but you also have like applets, you know like a -- if you have ever tried to test Oracle forms based applications, those forms are actually applets running, and so Selenium, you know being a browser automation API is extensible to work within those components running within the page but now without some effort.

So, there are a bunch of different efforts underway to kind of extends Selenium’s language so that you’re able to also automate the Flex and Flash applications, and I’ve highlighted a couple of them here. There’s this Flex UI and Flash UI projects. These are both Open Source extensions to the Selenium RC service. It basically extends Selenium RC clients to add the flash command languages or command methods. So, I’ve included a little snippet of code here. This would be something that would tell the Flex application component that’s running within the particular page to execute a “type” method.

So, you’re typing the value “2” into this Flex component and you’re indentifying the target within the Flex component using this “at” method which will look for an action script method that’s named “arg1,” and then you can use this of course to automate the other elements within that Flex component, and eventually do things like click which will execute some action script. You can even do assertions which are Selenium like or Selenium friendly where you’re saying “Okay, here’s a basic JUnit-style assertion” or you’re saying, “Okay, read from the Flex component from the result field and assert that it’s equal to the value of 5.”

Flex Automation API

There’s another way of going -- so, the thing about this extension to Selenium RC client is that it implements its own language here for you to construct these steps of the test. Adobe a year and a half ago introduced a new -- its own testing API called the Adobe Flex Automation API, that implements its own language and that language is very similar to Selenium. You have like type and click and/or commands that work on a Flex component level. So, there’s a popular tool called FlexMonkey which if you just did a simple Google search you’ll find FlexMonkey, that gives you a Selenium ID kind of -- or a Selenium ID like a user interface for recording and playing back test of Flex and SWF components.

So, what a very ingenious person did is to come up with Flexmonkium which is implementing a bridge between the FlexMonkey system and Selenium. So, with Flexmonkium you have the same kind of commands but now you’re using the Adobe Flex Automation API command set itself, and so it’s much more kind of a standard approach.

I see a couple more questions came in. Homa asked the question, “Can we run these test scripts in batch on a daily basis for regression testing?” Homa, yes, you can and so I’m going to show you how TestMaker implements that kind of Test Orchestration. By itself Selenium doesn’t do anything like that kind of test automation. it’s only interested in browser automation, TestMaker perform PushToTest is a free Open Source Test will -- to do that kind of batch processing of Selenium scripts.

Said asked a question, “Sometimes events take more time during playback and the test failed due to timeouts. How do we make that event event-based rather than time-based?” So, there’s a couple of ways of doing it. In the Selenium command language there are various wait commands. So, you could say, “Wait for element present.” When you do that then Selenium will wait for up to a predetermined timeout value. I think the default is 30 seconds for that element to become part of the DOM, and so when you’re doing that, that synchronizes the test to what’s going on within the JavaScript running in that Ajax page.

TestMaker And Selenium

Homa asked a question about you know, “What kind of a test automation and how to run this?” PushToTest has its own product called TestMaker and TestMaker is available for free download from the PushToTest site. It’s nice and it’s under a GPL license, and it comes with a great tutorial that shows you how to use Selenium, the integrated built-in Selenium support within TestMaker. The TestMaker console itself is a desktop application and it is easily configured to run within a continuous integration environment.

So, if you had something like Hudson or if you have like cron job that would kickoff a test once a night then the TestMaker would receive that call. The console would identify the Selenium Test. It would send the Selenium Test over to a web service called the TestNote. The Selenium Test then either runs with Selenium RC which is built in to the TestNote, and now would fire up Chrome or Internet Explorer or and of the other browsers, and operate the test against the Application Under Test. The TestNote keeps track of how long each step of the Selenium Test takes to run and then it returns the results back to the console for charting.

And so, you can develop these beautiful charts and identify pass, fail, success, how long each test step took and then you can even compare the operation of one test to, say the operation of the test a week ago and compare if the results have changed. TestMaker also includes a library that PushToTest has created called SeleniumHtmlUnit. Htmlunit is a headless web browser and it’s built-in to the TestNode, so those Selenium Test then can run in this headless browser and interoperate with the Application Under Test.

The advantage of Html Unit is that it’s headless and so there’s no problem of modal dialog box popping up and stopping your test. Html Unit also is incredibly efficient with using memory in CPU so… Whereas you can typically only get a few copies of browser running in a machine, running these TestNotes, you can get literally hundreds of copies of Html Unit running concurrently in the same machine. And so HtmlUnit is very appropriate for repurposing these Selenium scripts to do load and performance test or production monitors. So, the SeleniumHtmlUnit library -- I meant add run time, transforms that Selenium script into Html Unit commands and runs the Application Under Test. All of this is packaged into the TestMaker system and it’s available to you for free under a GPL license.

TestMaker Object Designer

The thing that occurred to us about a year ago is that there was a big transition underway for people using some of the proprietary test tools like HP QTP over to Open Source Test Tools and a lot of times they were asking questions like, “Well, why can’t I record in Internet Explorer?” So, what we did is to basically make our single biggest investment into writing a new code and we’ve created the TestMaker Object Designer, and I’d like to show you an example of that designer. This will come in TestMaker 6 which is now in Beta. We’re working on our first release candidate which we hope will be available from the PushToTest site next week.

The TestMaker Object Designer is all about making a Point-and-Click Functional Test Record and Playback tool but it works in all of the browsers. So, you can record tests in IE or Chrome or Safari, Firefox, Opera and that’s just the ones that we certify against, but pretty confident that this technology will work in all browsers. We also, in Object Designer provide Drag-and-Drop Data Driven Testing capability, that is if you had a CSV file or relational database query and you wanted to map that data into your Selenium scripts it should be as easy as just Drag-and-Drop and I’ll show you an example of that.

The TestMaker Object Designer creates Selenium scripts itself, and those Selenium scripts could be in Java or Ruby or Groovy, all of the supported languages of Selenium are supported in the designer. But it also creates test scripts in the Sahi language. Sahi is an up and coming competitor to Selenium and Sahi is really good at testing Ajax applications, it’s much further ahead in terms of its functions and features and compatibility to recognize those Ajax objects within a page and to automate their testing.

The designer will also record and playback Flex-based applications. And so let me give you a quick view on how this works. So, I’m running on a Mac system here but I could’ve just as easily installed TestMaker onto Windows or a Linux machine. Here is the basic console window and in here I could create a new load test, new functional test, a new service monitor, and when I create a new test I can bring up the designer, which looks like this. Let me -- and the designer itself -- we created it to kind of give you the same kind of Selenium IDE feel and also if you’re moving from HP’s QTP product this should look somewhat familiar.

So, here I can identify that I’ll create a new test object. I can name this so I’ll say this will be my login object. And now what it can do is click over here under record, and I’ve got a selection of different browsers that are installed on my local machine. So, I have Firefox and Safari and I can even use Flex Testing from here. If I had installed Chrome on this machine then I’d see Chrome in this list, if I was running on a Windows machine, I’d see IE as one of the browsers that I could record from.

In this case, I’ll be recording using the Firefox browser and I’ll tell it that I want to load the -- that I want to test the PushToTest.com site, click here on “OK,” and what this will do is bring up a copy of the Firefox browser and you can see it’s loading up in the background here, I’m just dragging this over here and the -- and as I operate the application, then the designer will be creating the test script for me. So here I clicked on this and click over here, and you can see that over here the designer is recording the different steps of that command.

Now, if I wanted to change from save in Sahi command language to Selenium, I’ll just do that and now here are the Selenium commands. I have of course full access to all of the Selenium commands here, so here’s a quick list of all of those commands. If you wanted to data enable this test, it’s really pretty simple. I just go over here to the data production library and I go under this data tab. I can browse out to the local file system and select like a CSV file, and here are the different elements of that CSV file. And now if I wanted to just drag over here and drop, I’m dropping in a value which is mapping name for a company name.

And so when I playback this test, this test is going to read from this row and pullout that value. I can also change this to be -- say the price value, and as you can see directly from here I can add rows and add columns and delete. So, once I’m done recording the test I can click end recording and then if I wanted to playback the test I’d click here, and notice now I can have my selection of which browser I want to playback the test in.

TestMaker Object Designer For Ajax

So, let me just show you a quick example of what this looks like for that Ajax application. What I’ll do is go over here under Add Company and what you might be able to see is that we have various different steps to this test, they’re all in Selenium. Now, when I played back this test, I can play it back in the Firefox browser like this. And so, I can do all these great functional testing immediately within the designer environment. There’s that getting the data from that CSV file and away you go. You can also insert breakpoints here, and one of the cool things for playback is when I tell it to playback I cannot only playback in the browser but I could also use that HtmlUnit browser.

Now, the cool thing here is that HtmlUnit will be operating this test heedlessly, right? Well, what happens if we want to debug our test script, we can also tell the designer to show us each step of the test as it operates an HtmlUnit using one of the available browsers, and so I can tell it to run the test. Right now it’s actually firing up the HtmlUnit browser and you’ll see this little indicator here showing what HtmlUnit is doing. When I get to my breakpoint then what HtmlUnit will do is open up the browser that I chose and show me a view of that. I can just click here. So, now this is going to open up Firefox and show me what HtmlUnit is seeing in that application.

So we have a couple of questions. Let’s see, Chris asked a question, “Are you going to support Selenium 2 with TestMaker?” Yes, we are. We will be fully supporting Selenium 2 and WebDriver both in the TestMaker runtime environment as well as this designer environment. We designed it in a way so that you can easily create test scripts of Selenium or Sahi, and then when Selenium 2 goes fine and then we’ll add that as a script type.

We have all of 2 minutes, so I’m going to do a 1 minute demo of record and playback of Flex-based applications within the same designer. All we have to do here is -- say the name of the SWF component like this. This brings up the Flex component that I identified and then as I am clicking through the items here, notice that the designer is recording the Flex Automation API commands for me to be able to playback this test. And now if I end this recording and go to playback, it will run the test back in that Flex component. It’s a pretty cool stuff, isn’t it?

I had a couple of more slides to show you, just to give you some resources to work with and then we’ll be ending the workshop. And of course all of the slides that I’ve been using are going to be sent to you as soon as you replied to this little 3 question questionnaire that we’re going to send you right after the workshop. So I’m going to put and leave up the Resources page here to show you some of the ways that you can learn more about Selenium. And for example, seleniumhq.org, that’s kind of the center of the universe for the Selenium downloads and for the documentation.

PushToTest has its own Selenium resource center. If you go to selenium.pushtotest.com you’ll find the Dzone RefCardz. You’ll find the architecture documents for Selenium. You’ll find how TestMaker supports Selenium. If you want to download TestMaker with the Selenium support built in just go this URL pushtotest.com/products-comparison .

So, where to go from here is that when you -- when we completed this workshop then you’ll receive and e-mail message with the 3 or 4 questions in it, and if you replied to that evaluation form then we’ll send you the slides. You can also watch a screen cast of this presentation. We’ve been recording at pushtotest.com/screencastcentral. We are also hosting a much larger workshop called the Open Source Test Workshop which covers testing of web applications, Ajax apps, web services and Flex that’s called the Open Source Test Workshop and if you go to workshop.pushtotest.com you’ll find all of the details, and of course PushToTest is a commercial Open Source company, and if any of these has peeked your curiosity to find that about licenses, trainings, support and consulting services please contact Troy Amyett. Troy is our sales director. He can be reached at This e-mail address is being protected from spambots. You need JavaScript enabled to view it or by telephone at 512-750-8769 here in the United States.

So, I have a time for just a few more questions. So, let me answer them and then we’ll be wrapping up this workshop. So, let’s see, ARtest asked the question, “How many headless browsers can you open?” Well, a lot depends on the scope and size of the test that you’re running like with the Unit Test, but on average it can generate between 150 to 200 virtual users each running their own HtmlUnit headless browser in a single machine.

I just blogged about this, the same question -- yeah, here it is. So, if you look on the pushtotest.com site on the right side here it says, “Latest News and Blogs.” How to specify hardware to run test? We’ll identify the -- what our methodology is for identifying how many of these headless browsers can run within a TestNote machine. Let’s see. Vanua asked a question, “Are you going to support Page Objects Pattern and where to download the 6.0 Beta?” So, Vanua, it’s -- go to the pushtotest.com site and in a couple of days you’ll find there’s going to be a posting there that has the Beta download location and then next week we’ll have the release candidates up and ready for you to download.

Said asked a question, “How do we cater” -- let’s see, “How do we cater Windows pop-ups in Selenium? For example windows pop-up to upload a file from the local machine.” Said, there are some things that are browser specific and there’s file upload commands are browser specific, so if you look on the seleniumhq.org site there’s a documentation tab and that has a section on how to handle browser specific test like the file pop-up. Basically, in each of the browser types you’re going to have different set of Selenium commands, some of them are really ugly and others are pretty nice and sophisticated.

Let’s see, AR asked the question, “Can we record -- can the recorded code be edited?” Well, yeah, of course, everything that you record you can edit within the designer. The designer even supports conditionals if then variables and Data Driven Testing. Let’s see, Kamala asked a question, “The Object Designer is that the same as Selenium IDE?” No. Kamala, actually Selenium IDE is officially part of the Selenium project and the TestMaker Object Designer is part of TestMaker and it’s available also under an Open Source license, so it doesn’t cost anything but it is separate from the Selenium IDE tool.

The Selenium guys still haven’t seen the designer and haven’t formed any decisions about whether a designer will be incorporated into the Selenium Project. I think the chances are probably not, but it doesn’t mean that it isn’t useful and available just like Selenium is. Kamala asked a question, “How do we go for Data Driven Testing in TestMaker, simply using CSV files as input?” So, Kamala TestMaker supports the CSV files, relational database queries, you can write data generating objects that share data from like one call of a test to another, and I urge you to attend the Open Source Test Workshop because we show all about TestMaker and how it works, whereas today’s workshop was primary focused on Selenium.

I think our time is up. Well, here, there’s a couple of more questions. So, Monique asked the question, “How can we open Selenium ID test than TestMaker?” Well, Monique, attend the Open Source Test Workshop tomorrow and you’ll see exactly how it’s done. Also in -- let’s see. On the PushToTest site there’s the homepage. What you’ll find is a tutorial here, it’s titled Selenium Tutorial and this shows you step by step, how to build a test of a rich internet application using Selenium and then running it within the TestMaker environment. So, all of that is available to you from the PushToTest site.