Archived PushToTest site

Testing With Selenium

Testing with Selenium

PushToTest TestMaker integrates the Selenium testing platform. And, the result is Ajax testing joy.

Overview

This chapter contains the following sections:

Architecture

Features and Benefits

Tutorials

Browser Compatibility Testing (Selenium RC)

Scriptless Selenium Test Development

Data Driven Tests

Selenium IDE

SeleniumHtmlUnit Selenese Reference

Unimplemented Selenese Commands

DZone RefCard

Tutorials

Debugging Techniques

Selenium Project Documentation

Open Source Test Workshop

High Speed Selenium Test Operation (SeleniumHtmlUnit)

Results Analysis

Scaling Selenium Tests in the Grid and Cloud

Continuous Integration

Alternative To HP QuickTest Professional(tm) (QTP)

The Future: TestMaker IDE Preview


Introduction

Global enterprises are challenged to surface performance and load issues, regression and functional issues, and to accomplish business service production monitoring in Rich Internet Application (RIA) environments that use Ajax technology. Selenium is a popular and widely used open-source test development tool. Selenium enables testers to record functional tests of applications in Selenium IDE, a Firefox-based record/playback test development environment. Playback these tests in Firefox, Internet Explorer, Chrome, Opera, Safari and other browsers.

Features and Benefits

TestMaker integrates Selenium to provide the following features and benefits:

  • TestMaker repurposes a single Selenium test as a functional test, load and performance test, and business service monitor. Details
  • Scale-up your Selenium tests to operate in your QA lab, in the PushToTest OnDemand Cloud Test Environment, or both! Details
  • TestMaker data-enables Selenium tests for test-driven development. Details
  • Dramatically reduce test operating costs. TestMaker efficiently operates multiple Seleniums test in a grid concurrently. Details.
  • TestMaker runs Selenium tests for browser compatibility testing of your Web application. Details.
  • Produce actionable knowledge by analyizing the results into 350 or more performance and functional charts. Details.
  • Transform Selenium tests into Java unit tests for additional features: looping, variables, conditionals
  • Run At User Levels Never Before Possible - Building a test lab used to be prohibitively costly for medium and large scale load tests. Cloud computing delivers bandwidth and computing power to achieve tests in the millions-of-users levels at commodity prices.

Repurposing Methodology and Architecture

PushToTest Repurposing Methodology applied to Selenium enables many types of testing:

Functional Testing in the Firefox Web Browser - Record test scripts using Selenium IDE, a plug-in to the Firefox browser. Play the test scripts in the Firefox browser. 

Browser Compatibility Tests - Play recorded Selenium tests in Internet Explorer, Safari, Opera, Chrome, and other browsers. Play the tests on a grid of machines with multiple operating environments and multiple browser versions.

Functional Testing - Play recorded Selenium tests as smoke tests, regression tests, and integration tests.

Load and Performance Testing - Determine the Scalability Index of your application by observing the throughput of your application at increasing levels of concurrently running virtual users. Each virtual user operates a test use case with one or more Selenium tests.

Production Monitoring - Prove compliance with a Service Level Agreement (SLA) by periodically running a Selenium test script. TestMaker sends notification emails when the Selenium script fails or encounters an error condition.

Stress Testing - Understand the behavior as your application comes under unexpected high load conditions.

In each of the above types of tests TestMaker repurposes a single Selenium test script. The TestMaker architecture make repurposing possible.

PushToTest TestMaker operates Selenium tests as load and performance tests by running the tests in one or moreTestNodes. The TestNode operates the test in a browser (such as Internet Explorer, Safari, Opera, Chrome) or using HTMLUnit. HTMLUnit is an open-source headless browser framework that provides a programmatic simulation of a Web browser. HTMLUnit uses Rhino, the Mozilla JavaScript engine, to run JavaScript and Ajax programs. This approach makes it easy to run hundreds to thousands of simulated users in a load and performance test. The alternative would be costly test runs that start a browser for each simulated user.

TestMaker makes it easy to pivot from one type of testing to another. For example, functional testing is often the time to ensure the test scripts work as expected. With a click of a button in the TestMaker Editor a functional test becomes a load and performance test. While loading the application under test with several virtual users your test may uncover another functional issue. Pivoting back to functional testing is easy with TestMaker.

Once you record a test, TestMaker repurposes the test to be a functional test, and load and performance test, and a production monitor.

Load test control panel window


Often TestMaker users will following this methodology:

a) Record a Selenium test in Selenium IDE using Firefox browser. Save the test in the Selenese file format. Confirm the test runs by playing the test against the application in the Firefox browser.

b) Run the Selenium test in a TestNode in the Firefox browser. Create a TestMaker TestScenario using the TestMaker Editor. Identify one or more Selenium tests to run in a test use case in the Editor's Use Case tab. Use the Selenium RC test type and identify *firefox as the browser. TestMaker sends the Selenium test to the TestNode. The TestNode launches Firefox and plays the test. The TestNode records the time it takes for each step of the Selenium test to operate. TestMaker plots a set of results charts to identify functional issues in the test and application.

c) Run the Selenium test in a TestNode in the HtmlUnit browser. Same operation but this time using the headless HtmlUnit browser.

d) Run the test as a Load and Performance test. Operates the Selenium test at several levels of load. Identifies the Scalability Index of the application. Surfaces performance bottlenecks and functional issues.

e) Run the test as a Production Monitor. Operates the Selenium test periodically to prove compliance with an SLA and to surface functional issues.


Tutorials

Please read the tutorial on Selenium testing in TestMaker. The tutorial shows step-by-step instructions to accomplish functional testing, load and performance testing, and business service monitoring of Selenium tests in TestMaker.

Tutorial: Building Tests of Rich Internet Applications Using Ajax and Selenium

These additional resources are available:

Screencast: Watch the 8 Minute Screencast on TestMaker

DZone RefCard: Introduction To Selenium

Selenium Documentation Project

Open Source Test Workshop


Open Source Test Workshop

This Open Source Test Workshop is for QA testers, software developers, and network and server managers needing visibility, tools, and methodology into all the demand for IT. This is a deep-technical Workshop to answer your issues:

  • TestMaker repurposes a single Selenium test as a functional test, load and performance test, and business service monitor. Details
  • How To Use Selenium in Rich Internet Application (RIA, using Ajax) Environments
  • How To Build Test Suites of SOAP and REST-based Applications using soapUI
  • How To Data Enable Selenium Tests Using TestMaker
  • Testing in Flash/Flex Environments Using AMF Protocols
  • Integrating TestMaker Tests Into Your Continuous Integration Environment
  • How To Analyze Test Results Into Actionable Knowledge
  • Using Glassbox and DynaTrace for Root Cause Analysis

Sign-up for the Workshop.

Data Driven Test Development

A PushToTest TestMaker Data Product Library (DPL) delivers operational data to tests at test runtime. For instance, a recorded Selenium test of a Web-based log-in page retrieves the user name and password values from a comma separated value (CSV) data file as TestMaker operates the test. Tests use DPL provided data not only as input values but also to validate the application's function.

The TestMaker DPL system provides a simple and easy way to data-enable tests. A DPL does not require programming or scripting. And it is a flexible framework that permits you to add your own DPLs to meet your organization's specific needs.

Read The TestMaker DPL Guide

Results Analysis and Reports for Root Cause Analysis and Mitigation

TestMaker records the time it takes to operate each step in a Selenium test and the overall time to process the test use case (transaction.) TestMaker's Results Analysis Engine summarizes the transaction logs into a set of charts and reports.

Read The Results Analysis Guide

Scaling Selenium Tests in the Grid and Cloud

PushToTest TestMaker is a distributed test environment. TestMaker runs Selenium tests on a Grid of your test equipment, in a Cloud Computing environment, or both.

Read The Test Deployment Guide

Continuous Integration and Selenium

TestMaker is an ideal environment for automating Selenium test operation from a Continuous Integration environment. The CI platform starts a TestScenario test and TestMaker automatically provisions and deploys tests to the available TestNodes in your own Grid or in a Cloud Testing environment.

Read The Test Deployment Guide

Test Debugging Techniques

TestMaker provides techniques and functions to debug test scripts.

savePage( path ) Command

TestMaker implements SavePage as a special Selenense command in SeleniumHtmlUnit to output HtmlUnit's current view of the Web page to a file. savePage also saves CSS and image data used in the page. Add savePage to a Selenese test script:

Selenium IDE does not implement savePage. You will see an error message. Run the Selenese test in TestMaker to operate the savePage command.

HtmlUnit Options

Use the TestMaker Editor, Options tab, to activate HtmlUnit debugging features.



printStats() Command

TestMaker ships with multiple versions of the HtmlUnit browser. printStats is a special Selenese command in SeleniumHtmlUnit to output statistics about the HtmlUnit browser version. Add printStats to a Selenese test script:


        printStats
       
       


When you run a TestScenario look at the Output panel to view the statistics:

version="2.9.1800"
....
Specification-Version: 2.9-SNAPSHOT
Specification-Vendor: Gargoyle Software Inc.
Implementation-Title: HtmlUnit
Implementation-Version: 2.9-SNAPSHOT
....

The Open Source Alternative To HP QuickTest Professional(tm) (QTP)

Many testers, developers and IT managers ask PushToTest to compare HP QuickTest Professional (QTP) to PushToTest TestMaker with Selenium IDE. PushToTest provides open source test automation solutions for organizations needing to improve the reliability and reduce service interruptions in their Web applications, Rich Internet Applications (RIA, using Ajax, Flex, Flash,) Web services, and Business Process Management services. PushToTest is the Open Source Test (OST) expert for dozens of tools. This comparison document helps organizations looking for an inexpensive open source solution that works with their latest applications.

Download the full report: TestMaker_Selenium_QTP Comparison (528 K bytes, Adobe Acrobat PDF format)

Here is a high-level summary of the full report:

HP QTP 10 TestMaker 5.4 with Selenium 1.0
Technical experience required to be successful alt altalt
Web 1.0 Record/Playback alt alt
Rich Internet Application (RIA, Using Ajax) Support alt
Drill down to root cause of functional issues/crashes altalt alt
Record Tests MS IE 6, 7, 8 Firefox 2, 3
Playback Tests MS IE 6, 7, 8 MS IE 5, 6, 7, 8, Firefox 2, 3, Opera, Safari, Chrome
Data-driven Testing alt alt
Reusable Test Components alt
Agile software development lifecycle (SDLC) support alt
Add-ons available altalt alt
Language Support VBScript Selenese, Java, Ruby, Perl, PHP, Python, C#, Groovy
Functional Testing, Load Testing, Service Monitoring from one test script alt
Run tests in the Cloud, in your QA lab, or both alt
Price $6000 USD Per Seat, Plus $1500 Annual Maintenance Free (GPL 2, Apache 2)



Selenium Command Language Reference (Selenese)

Selenium implements a domain specific language (DSL) named Selenese. PushToTest implements the SeleniumHtmlUnit object library in TestMaker to operate Selenese commands in HtmlUnit. Here is a reference to the Selenese commands and a list of unumplemented Selenese commands in SeleniumHtmlUnit.

Selenium Command Reference

The following is a reference to all of the Selenese commands.


Command
Wait Command Target Value
Selenium Control
setTimeout milliseconds
setMouseSpeed number of pixels
setMouseSpeedAndWait
setSpeed milliseconds
setSpeedAndWait
addLocationStrategy strategyName functionDef
addLocationStrategyandWait
allowNativeXpath boolean
allowNativeXpathAndWait
ignoreAttributesWithoutValue boolean
ignoreAttributesWithoutValueAndWait
assignId locator identifier
assignIdAndWait
captureEntirePageScreenShot filename, kwargs
captureEntirePageScreenShotAndWait
echo message
pause milliseconds
runScript javascript
runScriptAndWait
waitForCondition javascript milliseconds
waitForPageToLoad milliseconds
waitForPopUp windowID milliseconds
fireEvent locator eventName
fireEventAndWait
Browser Operations
open url
openAndWait
openWindow url windowID
openWindowAndWait
goBack
goBackAndWait
refresh
refreshAndWait
close
deleteCookie name path
deleteCookieandWait
deleteAllVisibleCookies
deleteAllVisibleCookiesAndWait
setBrowserLogLevel logLevel
setBrowserLogLevelAndWait
Cookie Operations
createCookie nameValuePair optionString
createCookieAndWait
deleteCookie name optionString
deleteCookieAndWait
deleteAllVisibleCookies
deleteAllVisibleCookiesAndWait
Popup Box Operations
answerOnNextPrompt answer
answerOnNextPromptAndWait
chooseCancelOnNextConfirmation
chooseCancelOnNextConfirmationAndWait
chooseOkOnNextConfirmation
chooseOkOnNextConfirmationAndWait
Checkbox & Radio Buttons
check locator
checkAndWait
uncheck locator
uncheckAndWait
Lists & Dropdowns
addSelection locator optionLocator
addSelectionAndWait
removeSelection
removeSelectionAndWait locator optionLocator
removeAllSelections
removeAllSelectionsAndWait locator
Edit Fields
type locator value
typeAndWait
typeKeys locator value
typeKeysAndWait
setCursorPosition locator position
setCursorPositionAndWait
Keyboard Operations
keyDown locator keySequence
keyDownAndWait
keyPress locator keySequence
keyPressAndWait
keyUp locator keySequence
keyUpAndWait
altKeyDown
altKeyDownAndWait
altKeyUp
altKeyUpAndWait
controlKeyDown
controlKeyDownAndWait
controlKeyUp
controlKeyUpAndWait
metaKeyDown
metaKeyDownAndWait
metaKeyUp
metaKeyUpAndWait
shiftKeyDown
shiftKeyDownAndWait
shiftKeyUp
shiftKeyUpAndWait
Mouse Operations
click locator
clickAndWait
clickAt locator coordString
clickAtAndWait
doubleClick locator
doubleClickAndWait
doubleClickAt locator coordString
doubleClickAtAndWait
contextMenu locator
contextMenuAndWait
contextMenuAt locator coordString
contextMenuAtAndWait
mouseDown locator
mouseDownAndWait
mouseDownA locator coordString
mouseDownAtAndWait
mouseMove locator
mouseMoveAndWait
mouseMoveAt locator coordString
mouseMoveAtAndWait
mouseOut locator
mouseOutAndWait
mouseOver locator coordString
mouseOverAndWait
mouseUp locator
mouseUpAndWait
mouseUpAt locator coordString
mouseUpAtAndWait
dragAndDrop locator movementString
dragAndDropAndWait
dragAndDropToObject sourceLocator destLocator
dragAndDropToObjectAndWait
Form Operations
submit formLocator
submitAndWait
Windows/Element Selection
select locator optionLocator
select AndWait
selectFrame locator
selectWindow windowID
focus locator
focusAndWait
highlight locator
highlightAndWait
windowFocus
windowFocus AndWait
windowMaximize
windowMaximizeAndWait
Selenium Accessors
store expression variableName
storeAlert variableName
assertAlert pattern
assertNotAlert pattern
verifyAlert pattern
verifyNotAlert pattern
waitForAlert pattern
waitForNotAlert pattern
storeAlertPresent variableName
assertAlertPresent pattern
assertNotAlert Present pattern
verifyAlertPresent pattern
verifyNotAlertPresent pattern
waitForAlertPresent pattern
waitForNotAlertPresent pattern
storeAllButtons variableName
assertAllButtons pattern
assertNotAllButtons pattern
verifyAllButtons pattern
verifyNotAllButtons pattern
waitForAllButtons pattern
waitForNotAllButtons pattern
storeAllFields variableName
assertAllFields pattern
assertNotAllFields pattern
verifyAllFields pattern
verifyNotAllFields pattern
waitForAllFields pattern
waitForNotAllFields pattern
storeAllLinks variableName
assertAllLinks pattern
assertNotAllLinks pattern
verifyAllLinks pattern
verifyNotAllLinks pattern
waitForAllLinks pattern
waitForNotAllLinks pattern
storeAllWindowIds variableName
assertAllWindowIds pattern
assertNotAllWindowIds pattern
verifyAllWindowIds pattern
verifyNotAllWindowIds pattern
waitForAllWindowIds pattern
waitForNotAllWindowIds pattern
storeAllWindowNames variableName
assertAllWindowNames pattern
assertNotAllWindowNames pattern
verifyAllWindowNames pattern
verifyNotAllWindowNames pattern
waitForAllWindowNames pattern
waitForNotAllWindowNames pattern
storeAllWindowTitles variableName
assertAllWindowTitles pattern
assertNotAllWindowTitles pattern
verifyAllWindowTitles pattern
verifyNotAllWindowTitles pattern
waitForAllWindowTitles pattern
waitForNotAllWindowTitles pattern
storeAttribute attributeLocator variableName
assertAttribute attributeLocator pattern
assertNotAttribute attributeLocator pattern
verifyAttribute attributeLocator pattern
verifyNotAttribute attributeLocator pattern
waitForAttribute attributeLocator pattern
waitForNotAttribute attributeLocator pattern
storeAttributeFromAllWindows attributeName variableName
assertAttributeFromAllWindows attributeName pattern
assertNotAttributeFromAllWindows attributeName pattern
verifyAttributeFromAllWindows attributeName pattern
verifyNotAttributeFromAllWindows attributeName pattern
waitForAttributeFromAllWindows attributeName pattern
waitForNotAttributeFromAllWindows attributeName pattern
storeBodyText variableName
assertBodyText pattern
assertNotBodyText pattern
verifyBodyText pattern
verifyNotBodyText pattern
waitForBodyText pattern
waitForNotBodyText pattern
storeChecked variableName
assertChecked pattern
assertNotChecked pattern
verifyChecked pattern
verifyNotChecked pattern
waitForChecked pattern

Unimplemented Selenese Commands

SeleniumHTMLUnit is a TestMaker object library that translates Selenese commands on-to-fly and runs the commands using the HTMLUnit headless browser. Some Selenese commands do not make sense in a TestMaker context. For example, TestMaker provides cookie management commands of its own. The following is a list of Selenese commands not implemented in SeleniumHTMLUnit. TestMaker throws a NotImplemented exception when encountering one of the following commands:

altKeyDown

getAlert

getTable

mouseMoveAt

altKeyUp

getAllButtons

getWhetherThisFrameMatchFrameExpression

mouseUpAt

answerOnNextPrompt

getAllFields

getWhetherThisWindowMatchWindowExpression

openWindow

captureScreenshot

getAllLinks

getWhetherThisWindowMatchWindowExpression

refresh

chooseCancelOnNextConfirmation

getAllWindowIds

getXpathCount

setBrowserLogLevel

chooseOkOnNextConfirmation

getAllWindowNames

highlight

setContext

clickAt

getAllWindowTitles

isAlertPresent

setCursorPosition

close

getAttributeFromAllWindows

isConfirmationPresent

setMouseSpeed

controlKeyDown

getBodyText

isEditable

shiftKeyDown

controlKeyUp

getConfirmation

isOrdered

shiftKeyUp

controlKeyUp

getCookie

isPromptPresent

start

createCookie

getCursorPosition

keyDown

submit

deleteCookie

getElementHeight

keyPress

typeKeys

doubleClick

getElementIndex

keyUp

waitForFrameToLoad

doubleClickAt

getElementPositionLeft

metaKeyDown

waitForPopUp

dragAndDrop

getElementPositionTop

metaKeyUp

windowFocus

dragAndDropToObject

getElementWidth

mouseDownAt

windowMaximize

dragdrop

getExpression

mouseMove

 

fireEvent

getLocation

   
 

getMouseSpeed

   
 

getPrompt

   

Questions And Answers 

PushToTest, the open source application performance management and load test solution company, participates in the Selenium project. Many times PushToTest's response to a customer issue is in the form of an improvement to Selenium iteself. The following questions and answers originally appeared in DZone.

What exactly is Selenium?

Selenium is a combination of tools and a domain specific language (DSL) to build and operate functional, smoke tests, and integration tests of Web applications, including Rich Internet Applications (RIA, using Ajax.) Selenium provides a record/playback tool that plugs-into the Firefox browser. The recordings use the Selenese DSL to express the steps of a test. Selenium plays-back the test steps in the browser using a Javascript-based Browserbot daemon. Selenium Remote Control (RC) runs the test in Internet Explorer, Firefox, Opera, Safari, Chrome and other browsers. Selenium Grid makes it easy to run these tests in a grid of test machines.

How long has the project been around? And how long have you been using it?

Selenium had its start at ThoughtWorks, a system integrator and global consulting firm. The core developers released Selenium into an open source distribution in 2006. PushToTest began using Selenium since 2007. PushToTest was looking for a way to repurpose Selenium tests as load and performance tests, and business service monitors when Olivier Dony and Dominique de Waleffe contributed a TestMaker package to run Selenium tests using the HtmlUnit headless high performance browser. PushToTest enhanced this work to make the Selenium tests data-driven, to provide root cause analysis and results analysis functions, to provide compatibility to test Ajax applications, and to run these tests in a Cloud testing Environment. This has been hugely beneficial to organizations needing to reduce costs and take advantage of Selenium's easy-of-test-authoring features.

What are the basic steps involved in getting setup with Selenium? (jars/server setup etc)

Setup to record and playback tests of Web browser-based applications using Selenium takes about 5 minutes. The Selenium IDE plugs into Firefox as a standard add-on using XPI installation techniques. To playback the tests in Internet Explorer, Opera, Safari, Chrome and other browsers requires installation and configuration of Selenium RC. Selenium RC uses a daemon architecture. Start the daemon using a simple command-line call to start Java and run the Selenium RC JAR package. Selenium RC installations take an hour or less. Selenium Grid installs Selenium RC services on a farm of servers.

What language is used to write the tests?

Selenium implements a domain specific language (DSL) for testing Web applications. For example, the DSL implements commands to click a button, type characters into a test field, and wait until text appears in the page. Selenium implements the DSL as class library in Java, package in Python, and Ruby, Groovy, C#, Perl, and PHP. The DSL has approximately 200 commands. PushToTest teaches a 3-day introductory class in Selenium. It takes 2-3 days for most students to become proficient in the Selenium DSL.

How well does the record/playback function work?

Selenium IDE is designed for Web 1.0 applications. In Web 1.0 environments Selenium's record/playback function works very well. For Rich Internet Applications that use Ajax techniques and widgets, not so much! Ajax is the Wild West of software development. Everything goes, there is no sheriff, and no standards body exists. Consequently Selenium does not know how to handle asynchronous applications where objects load at their own pace and there is no "wait until idle" function. Selenium misses Javascript driven clicks, Javascript created frames and pop-up windows. Each miss means the test author needs to manually add Selenium DSL commands.

Selenium IDE record/playback does not work well for tests that require looping and conditional branching. These are possible by manually coding Selenium DSL commands in a Java or other scripting language environment. The solution is to use Selenium IDE's record feature to create simple test use cases, transform them into a Java class, and treat each use case as a component incorporated into a test suite. There are emerging frameworks to help implement a component approach to Selenium test authoring.

Is there a coverage tool that plugs into Selenium?

ThoughtWorks Twist is a tool to facilitate a continuous test and integration methodology built on Selenium tests in a team-oriented development environment. There may be others. There is a lot of excitement and energy in the Selenium space so PushToTest wouldn't be surprised to find coverage tools.

Are there any missing features in Selenium? Is there a new release due soon?
A key drawback is Selenium's requirement to playback tests in a real browser. When the browser needs to interact with a user the Selenium test stops running. For example, when a browser moves from an HTTPS secured page to an HTTP page many browser's security policy kicks in and the browser asks the user to confirm the action. The Selenium project is working on a next generation technology called WebDriver. WebDriver uses each browser's native API to operate the test commands.

Selenium lacks a data production library system to provide operational test data, lacks a way to repurpose tests into load tests, lacks a platform to manage Selenium tests as test assets, and lacks a way to derive the root cause to functional and performance issues. Thankfully these are provided by many side projects in the Selenium biosphere, including PushToTest, BrowserMob, SauceLabs, and others.

The words "coming soon" and Selenium are mutually exclusive. It took the Selenium project 18 months to move from "beta" to "release candidate." PushToTest is optimistic but not hopeful that Selenium 2.0 with WebDriver technology will appear in 2010.

The Future 

PushToTest is building a new Ajax-based record/playback utility that will output Selenese tests. This is aimed at providing an open source alternative to HP QTP. The design is available at http://downloads.pushtotest.com/201001/TM_IDE_Design.pdf

Resources

Consider the following resources to help you:

PushToTest Test OnDemand Service

TestMaker Cloud Testing User Support Forum



 

 
alt

Additional documentation, product downloads and updates are at www.PushToTest.com. While the PushToTest testMaker software is distributed under an open-source license, the documenation remains (c) 2010 PushToTest. All rights reserved. PushToTest is a trademark of the PushToTest company.