Archived PushToTest site


The Selenium TestMaker Tutorial

Build A Test of a Rich Internet Application
In 30 Minutes Or Less.



Please note: This tutorial has been replaced with an updated tutorial.


This article shows how to build a test of a Rich Internet Application (RIA, using Ajax) in PushToTest TestMaker and Selenium in minutes. This article then shows the tremendous power to repurpose the test as a load and performance test and a business service monitor with no extra coding.

Download TestMaker and Run Selenium Tests  Watch the Tutorial Screencast

Note: Find all of the code and data described in this article in the TestMaker 6.1 or higher installation under tutorials/Selenium_WebTest_Tutorial

Enterprises are challenged to surface performance and load issues, regression and functional issues, and to accomplish business service monitoring in Rich Internet Application (RIA) environments that use Ajax technology. Successful Internet applications - Google Docs, Flikr, Facebook, and Google Maps - are driving users and managers to expect Ajax functions in an organization's Web applications. For instance, the following Calendar application has the look-and-feel of a desktop application, but it is running entirely in a Web browser.


Calendar Ajax example application

This tutorial builds a test of the Calendar application. The tutorial will show how to use the test for regression and smoke testing. Then the tutorial will repurpose the test as a load and performance test and business service monitor.

The tutorial will make use of Selenium support in PushToTest TestMaker. Selenium is a popular and widely used open-source Web test suite development tool that works well with Ajax applications. Selenium enables testers to build functional tests of Ajax applications in a Firefox-based graphical test development environment. Playback these tests in Firefox, Internet Explorer, Opera, Safari and other browsers. Then repurpose the same tests as functional tests, load and performance tests, and business service monitors in the TestMaker distributed test environment.

Tasks and Scenario

Our challenge in this tutorial is to build a test of the Calendar application. The tutorial will show how to use the test for regression and smoke testing. Then the tutorial will repurpose the test as a load and performance test and business service monitor.
  1. Build and run a functional test of the Calendar application.
  2. Run the test in a TestNode using the HTMLUnit browser
  3. Run the test in a TestNode using Selenium RC and a browser.
  4. Reuse the test as a data-driven test with a Data Production Library that reads data from a Comma-Separated-Value (CSV) file.
  5. Reuse the test as a load and performance test.
  6. Reuse the test as a business service monitor.
  7. Transform the test into a Jython script and Java test class.
  8. Debugging Ajax Applications: Tips and Techniques

Test Use Case

A software quality tester needs to test an Calendar system. The tester follows these steps:
  1. Schedule an event using the Calendar application.
  2. Confirm that the event appears in the Calendar in a list of events.

Implementation

We have already built the test and provide the test in a set of files available in the TestMaker_home/example_agents/AjaxRIASeleniumTest directory. See the following files for the implementation:
dat

Preparation To Use This Tutorial

This tutorial uses the Calendar application that ships with TestMaker 5.2 and later. TestMaker Installer automatically installs the Calendar tutorial service. In the event you chose not to install the Calendar tutorial service, follow these steps:
  1. Copy the file
    TestMaker_home/example_agents/Miscellaneous/ExampleServices/calendar.war
    to
    TestMaker_home/TestNetwork/TestNode/webapps

  2. Exit and start TestMaker again. Calendar will cause TestMaker to take a few more seconds than normal to start-up. You can see Calendar's start-up messages in the Local TestNode Output tab of the Output panel in TestMaker.

  3. Access Calendar in a browser at http://localhost:8080/calendar

Calendar is an example application built with Appcelerator, an excellent Ajax development environment.

This tutorial uses the following file name conventions:
Test Scenario files end in .scenario

Task 1: Record and Playback A Web Application Functional Test

In this task we will create a test of the Ajax application using the Selenium record/playback utility.
  1. The PushToTest installer adds Selenium IDE to your Firefox browser. Open Firefox.

  2. From the Tools drop-down menu choose Selenium IDE. A special Selenium IDE window opens above the main Firefox window.

    Selenium IDE main window1



  3. Click back to the main Firefox window. Point Firefox to http://localhost:8080/calendar. You will see the Calendar user interface.

    Calendar Ajax example application


  4. By default Selenium IDE opens in its recording mode. Selenium IDE is ready to record your use of the Calendar application.

  5. Click the New Event menu item. Note that Selenium IDE records two steps: open the Calendar and click on app_6 (the New Event element ID.)

    Creating a new event in Calendar



  6. Type Work On Homework in the Event text box. Enter today's date in the mm/dd/yy format (or click the Calendar icon and click today's date. Then click the Save Event button. You will see the event in the Today's Events list.

    After creating the new Event



  7. As a final step, right click on the Work On Homework entry in Today's Event's. Selenium IDE will show commands to assert the event inserted correctly into the Today's Events list.

    Pop-up menu to verify the event


  8. Click the Record icon in the upper right of the Selenium IDE window to stop recording the test.

  9. Notice the Selenium IDE window displays the steps for the test.

    Selenium IDE with the recorded test


  10. Next we will see the test operate in play-back mode. Click and drag the horizontal slider in Selenium IDE to the mid-point between Fast and Slow. Click the Play Current Test Case icon in the Selenium IDE window. It can be useful (and entertaining) to see the play-back.

    In the above steps you operated the test of an Ajax application using a Web browser. The Selenium IDE reads the Selenese test instructions and operates the test in the Firefox Web browser. This works well when building and debugging tests.
    How it works

    In the next part of this tutorial you will operate the recorded test in the PushToTest TestMaker test automation platform. TestMaker automates operation of the Selenium test, data-enables the Selenium test, produces reports for the operation of the Selenium test, enables the Selenium test to be a step within a larger use case, and repurposes the Selenium to be a load and performance test and business service monitor from the same single Selenium test.


Task 2: Run The Test In A TestNode


PushToTest distributed test environment


Operating a test using the internal HTMLUnit browser

The PushToTest console identifies the operating parameters of a functional test, load and performance test, and business service monitor in a TestScenario. Use the TestMaker Editor to define a TestScenario. Find a TestScenario document already created for you in AjaxRIASeleniumTest/Calendar_Functional_Test.scenario.
  1. Start TestMaker using TestMaker_home/TestMaker.bat (or .sh for Unix and Mac environment.)

  2. Choose File -> Open and select the file TestMaker_home/example_agents/AjaxRIASeleniumTest/Calendar_Functional_Test.scenario

  3. TestMaker opens a new controller panel for the TestScenario. The controller panel includes icons to run, pause, and stop the test.

    Controller panel shows the TestScenario
  4. Click the Edit icon in the controller panel to view the source of the TestScenario document. This action opens the Editor.

    Editor for building TestScenarios


  5. Editor displays the TestScenario. Use the tabs to view the parts of the TestScenario. For example, the TestScenario defines use of the Selenium test in the Use Cases tab.

    Editor use case tab

    The TestNodes tab tells TestMaker to send the Selenium recorded test to the TestNode(s) that will operate the test.

    The UseCases tab tells TestMaker to run a use case that runs the Selenium test in CalendarTest.selenium.

    The General tab tells TestMaker that this is a functional test and to run the use case once.

  6. Click the Run button and watch the results.

    Results of running the functional test


    Functional tests run a test as a single user. TestMaker displays check-marks next to each step of the Selenium test that operate successfully. TestMaker shows the amount of time to operate the use case and each step in the Selenium test.

    TestMaker displays an X mark for a step that fails. Use the Windows menu -> Output command to view the logs to learn the cause of the failure. Click the Local TestNode Error and Local TestNode Output tabs.

    See tips and techniques to solve failures.
Please note: Selenium IDE's recording of the Calendar test lacks a few important changes to run in TestMaker. We made the following changes to the CalendarTest.selenium:
  1. Add a setSpeed command as the first line. Right click on the first line of the test, choose Insert New Command. Set the command as setSpeed and enter 3000 in the value field. This value is in milliseconds (1000 milliseconds = 1 second.)

  2. Change the base URL to include the domain name: http://localhost:8080/

  3. Add a waitForElementPresent command. This makes sure the test waits until the JavaScript finishes updating the page to show the New Event form.
  4. Add an Open command. This makes sure the assertion tests the main page content, and not just the New Event form content.

Optional Task: Run A Test In A TestNode Using Selenium RC

Task 2 shows how to operate a test in a TestNode using the internal HTMLUnit browser. PushToTest supports operating a test in a TestNode using Selenium Remote Control (RC) and a browser of your choice for smoke tests and regression tests. This enables test automation of functional testing across multiple browsers and operating environments. PushToTest TestMaker operates tests in a grid of browsers.

Using Selenium RC and a browser to run a test

  1. In the TestMaker window use the File -> Open TestScenario command. Use the file selector to choose AjaxRIASeleniumTest/Calendar_Functional_Test_RC.scenario

  2. TestMaker opens a new controller panel for the TestScenario. The controller panel includes icons to run, pause, and stop the test.

  3. Click the Edit icon in the controller panel. We only made 1 change from the TestScenario that operates the Selenium test with the internal HTMLUnit browser:

    Editing a Selenium RC test

  4. That's all! With Firefox installed, click the PushToTest Run button and watch the test operate.

    TestMaker running a test in the Firefox browser

By the way, TestMaker supports Selenium RC with other browsers too. For example the method attribute may be any of the following installed browsers: chrome, konqueror, piiexplore, iehta, mock, opera, pifirefox, safari, iexplore and custom. The * is important.

Some operating environments require the method attribute to identify the path to the browser. Use the Editor's XML view to set the path to the browser. Click the Form/XML view icon at the bottom of the Editor. Find the statement and add the path to the method attribute. For example, the following example works for Firefox on Ubuntu 7:

method="*firefox /usr/lib/firefox-3.0.3/firefox" langtype="seleniumrc">


Task 3: Make A Data-Driven Functional Test

Task 3 enhances the functional test of the Calendar application from Task 1 and 2 to use a Data Production Library (DPL.) The DPL reads data from a comma separated value (CSV) flat file. The values provide input to the test (id, password, product number for search) and validation data (the Assert-Text-Exists) value.
  1. Create a Comma-Separated-Value file. Use your favorite text editor or spreadsheet program. Name the file data.csv. The contents must be in the following form.

    Comma separate value (CSV) file

    A quick explanation: The first row of the data file contains column names. These will be used to map values into the Selenium test.


  2. Change the Selenium test to refer to the column of data. In Selenium IDE change test step 4 from:

    type, event, Work On Homework

    to

    type, event, EventName


    Edited Selenium test


    A quick explanation
    : PushToTest maps the data from the named column in the CSV data file to the Selenium test data using the name you just entered.

  3. Connect the Data Production Library (DPL) to the Selenium test. Open the AjaxRIASeleniumTest/Calendar_Functional_DataDriven.scenario in the Editor. Click the Use Cases tab.


  4. Click the Add DPL link. Set the DPL type to HashDPL. HashDPL reads data from a comma separated value (CSV) file and provides it at test run time to the ScriptRunner. Set the Action to Get Next Row of Data.

    Setting a DPL in the use case


  5. By default TestMaker runs the use case in a Functional Test once. Click the General tab and set the Repeat value to a higher value to repeat the use case for the additional rows of data in the data.csv file.


  6. That's all! Click the PushToTest Run button and watch the results.

    Showing the results of a data driven test

A quick explanation: The TestScenario operates a functional test by running the Selenium test once. The data file we created contains 5 rows of data. Change the repeat value to 5 to have TestMaker repeat the test for each row of data.


Task 4: Repurpose A Functional Test As A Load And Performance Tests

The test we built in Task 3 operated a functional test by a single user. Load and performance testing identifies the scalability index of a target host application by operating a test with multiple simultaneous concurrently running users. PushToTest reuses functional tests as load and performance tests without requiring any change to the test!
  1. In the TestMaker window use the File -> Open TestScenario command. Use the file selector to choose AjaxRIASeleniumTest/Calendar_Load_Test.scenario.

  2. Click the Edit icon in the TestScenario Controller panel.

  3. Set the test to be a load test in the General tab.

    Editing a load test


    The settings tells TestMaker to operate a load and performance test at three levels of concurrently running simulated users (crlevel.) The test operates 1 user, then 2 users, and then 4.


  4. That's all! Click the PushToTest Run button and watch the results.

    Test results from a load test

    PushToTest operates the test in 1 concurrently running simulated users, then 2 users, then 4 users. The Real Time Scalability Index contrasts the Transaction Per Second (TPS) for each of the three user levels. A perfectly scalable system will increase TPS in linear proportion to the increase in users. For example, at 2 users doing 0.03 TPS a 4 user level should be twice the TPS, or 0.06 or higher. The above chart shows an application with linear scalability. The Step Times chart shows the amount of time each step took to operate as the test proceeded.



Task 5: Repurpose A Functional Test As A Business Service Monitor

Business Service Monitor (BSM) testing enforces and proves a Service Level Agreement (SLA) by operating a test periodically. For example, a monitor runs a test every 30 minutes. TestMaker reuses functional tests as a monitor with no changes to the Selenium test and one change .
  1. In the TestMaker window use the File -> Open TestScenario command. Use the file selector to choose AjaxRIASeleniumTest/Calendar_Monitor.scenario.

  2. Click the Edit icon in the TestScenario Controller panel.

    Editing a business service monitor


    This TestScenario runs the test use case as a business service monitor. The monitor operates the test use case every 10 minutes. The monitor keeps running until it encounters an exception.

  3. That's all! Click the PushToTest Run button and watch the results.

    Business Service Monitor (BSM)

    The monitor controller panel is a dashboard to the status of the service being tested.



  1. In the TestMaker window use the File -> Open TestScenario command. Use the file selector to choose AjaxRIASeleniumTest/Calendar_ScriptDriven.scenario.

  2. Click the Edit icon in the TestScenario Controller panel.




    A quick explanation: We ran the Transformer utility on the Selenium recording in Task 3. Access the Transformer in the Tools drop-down menu in TestMaker. The Transformer created the TestScenario you just opened.
  3. That's all! Click the PushToTest Run button and watch the results.




Add your own custom logging to a test script using the PTTStepListener. Add the following commands to a Jython script:

from com.pushtotest.tool import PTTStepListener
PTTStepListener.startStep( "setUp" );
PTTStepListener.endStep();

8. Debugging Ajax Applications: Tips and Techniques

Ajax applications provide rich user experiences. In an Ajax application components may update themselves asynchronously. Ajax applications communicate to information sources external to the browser - like the backend server - or even with other components running on the same page. This flexibility challenges the tests we write. Our Ajax tests must work within the asynchronous nature of the Ajax application. And, we often need to make manual changes to the test because of the lack of a standard for Selenium IDE to follow to record an Ajax application.

You may find the following tips and techniques useful:

a) When you are not certain of the state of your test and the application, use a special savePage and saveSource commands. The target value is a file. saveSource saves the current HTML.

b) Set HtmlUnit into "debug" mode. In the Editor use the Options tab to set "Save received responses to temp directory" and SeleniumHtmlUnit log level.


Download TestMaker and Run Selenium Tests  Watch the Tutorial Screencast