Archived PushToTest site


The Top 3 Issues In Website Load Testing

Website Load Testing for Web Applications and You

I talk with hundreds of CIOs, CTOs, Testers, and Developers every month. These are the top3 issues they ask me about:

How Many Users Can My Website Handle?

Performance testing identifies your Website's ability to achieve linear scalability. In a linear scaling system every user gets the same performance - no matter the number of users concurrently on the Website. If 100 people use the site concurrently they get the same speed as when 100,000 people use the same site.

Stress testing identifies how your Website will fail when too many users are on the site. How it fails is valuable for resource planning and disaster planning.

The best solution combines Website load testing technology with performance monitoring. An integrated solution correlates the load generated against a site by virtual users to the monitored resource statistics of the backend (CPU, network, and memory needs.) These performance solutions show how many users your Web site can handle before it slows and fails. Repurposing technology uses the same test to show how your Website will act as it fails. Look for a Linear Scalability Index chart is the fastest and easiest measurement of your Website's ability to serve 100 users to 5 million users.

Do I Have To Use Test Scripting Languages To Write Effect Tests?

The alternative to writing tests in a scripting language is Record/Playback tools. These watch your use of a browser to create a test script for you. The downside is when it comes to test script maintenance. In the long-term a good object oriented approach to test development will reduce maintenance costs. For Selenium tests that means PageObjects. For Java that means jUnit.

But that does not solve the issue for testers that have little scripting experience and knowledge. For these cases I recommend a Record/Playback tool that exports test scripts in Jav, PHP, Ruby, Python, and other scripting languages. These make the need to tip into test scripting a little further off.

I recommend you search for a test script design tool that watches you operate a Website in Internet Explorer (IE,) Firefox, Chrome, Safari, and Opera. The tool should automatically handle wait/pause functions in the Ajax, Flex, Flash, Applets, and Oracle Forms in your Web pages. Your goal is to create reusable test objects for functional testing, load and performance testing, and production monitoring. Stay away from tools with Beginner/Expert modes because "expert" normally means complicated with hand-coding, and the scripting languages are often proprietary.

How Do I Manage Thousands of Agile Stories and Tests Effectively?

Writing a few test scripts is easy. Supporting a dozen testers and 10 or more test suites means managing 100 or more tests. The hand-written back of an envelope and/or an Excel spreadsheet is not enough to scale-up your management of the test.

Agile test management services like Rally, Zephyr, QMetry, and InfoLink work around Agile stories, use cases, and tests. They make it easy to scale from a dozen tests to thousands. Connect test scripts to Agile stories to immediatly learn which tests have run (pass and fail,) which tests ran with slower than acceptable performance, and how the performance changes from sprint to sprint. Look for test management that integrates Website load testing from soapUI, Selenium, WebDriver, Sahi, jUnit, and TestNG.

Now that you know the top 3, consider learning more by reading my advanced report on Website load testing.

Get The Advanced Report on Website Load Testing
Get the advanced report on Website load testing.


Website Load Testing for Web Applications Blog

Website Load Testing for Web Applications and You

"You know the PushToTest site - with its hundreds of articles on testing - does not actually say anything about Website performance testing," a friend told me a few days ago. A quick check of the site showed that he is correct! :-)

PushToTest TestMaker is the leading open source solution to load test your Web site, social mobile integrated marketing, and Web applications. PushToTest offers load testing and reporting as a performance testing tool to e-commerce and business-to-business (B2B) sites all over the world.

PushToTest TestMaker for Website Load Testing, the standard at:
Best Buy manages large scale Agile tests with PushToTest TestMaker Deloitte monitors inter-office connectivity using PushToTest TestMaker PepsiCo tests with integrated marketing campaign Websites with PushToTest TestMaker

We get the following 3 questions most:

How Many Users Can Your Website Handle?

Scalability Index shows how close the application is to linear scalability

Advanced load testing technology combines with performance monitoring in PushToTest TestMaker to forecast your server CPU, network, and memory needs. TestMaker shows you how many users your Web site can handle before it slows and fails. TestMaker repurposes the same test to show how your Website will act as it fails. TestMaker's Linear Scalability Index chart is the fastest and easiest measurement of your Website's ability to serve 100 users to 5 million users.

No Scripting Languages To Learn

TestMaker Object Designer for script-less record/playback

Cut your load testing time in half with TestMaker Object Designer - the most advanced and easy-to-use test authoring tool. Designer watches a tester operate a Website in Internet Explorer (IE,) Firefox, Chrome, Safari, and Opera. Designer automatically handles the Ajax, Flex, Flash, Applets, and Oracle Forms in your Web pages. Designer creates reusable test objects that are perfect for functional testing, load and performance testing, and production monitoring. Designer's easy record and configuration process removes the need for complicated "expert" modes and hand-coding as there are no scripting languages to learn.

Manage Thousands of Agile Stories and Tests Effectively

Manage thousands of tests with TestMaker and Rally

TestMaker Enterprise integrates with Rally Test Management to make it easy to scale from a dozen tests to thousands. Connect test scripts to Agile stories to immediate learn what tests have run (pass and fail,) which tests ran with slower than acceptable performance, and how the performance changes from sprint to sprint. TestMaker orchestrates test suites from soapUI, Selenium, WebDriver, Sahi, jUnit, and TestNG.

Want to see all of this in action? Attend the next Open Source Test Workshop. It's free!

Download free open source TestMaker too Learn with other tests in our free Workshop
Learn with our free videos

We hope you will:
Since 2001, PushToTest helps PepsiCo, Best Buy, Deloitte and the world’s most well-known brands bullet-proof their Websites and social mobile Web applications against large volumes of traffic. Using our software and global services, PushToTest helps optimize your site for any amount of visitors. How can we help you grow?

Social Mobile Web Application vs. Enterprise Application Testing

Performance Infrastructure

I was surprised to see a big disagreement on the value of social mobile Web applications at the Red Herring conference.

PushToTest is selected as a Finalist for Red Herring's Top 100 Americas award. The award recognizes the year's most promising private technology ventures in North America. Red Herring selected PushToTest for developing performance infrastructure necessary to deliver social mobile Web applications reliably. I presented PushToTest at the Red Herring conference last week among panels of venture capitalists, merger managers at large businesses, and entrepreneurs. The discord around social mobile Web applications and enterprise software surprises me.

Red Herring Top 100 Americas Award

For example, Ray Rothrock of Venrock Capital said "Could the world live without enterprise software... no. Could it live without Facebook... yes." Wow. It's not a competition. Social mobile applications build an audience and marketplace of interest in brands. Enterprise software is the means to deliver a great Web user experience. Social mobile is the next generation in marketing tactics.

Social mobile Web applications intensify a divide between the IT team and business team at an organization. PushToTest delivers performance infrastructure to PepsiCo, BestBuy, and Deloitte. While the IT team wants to defend the enterprise systems from crowds of users - and the performance spikes that happen when the Web site gets slammed - the business team couldn't be happier when the site goes viral.

Most recently PushToTest saw this divide a few weeks ago when PepsiCo updated the Web site. The new site moves PepsiCo closer to the heart of its target market, the popular culture. Their new home page incorporates elements of Facebook, Twitter, and live information feeds for their concerts, participation in the Summer Olympics, and other promotions. It is an outstanding exciting integrated site.

Competing needs from the IT team and business team

The challenge for all organizations today is the IT team has all the responsibility to provide a site that works in a social, viral, and mobile marketplace. How a CIO gets any sleep is a mystery to me! These sites need to be load balanced, have great reliability during performance spikes, be elastic to scale up and down, and protect user and business data from unapproved access.

For business managers, getting to the heart of the target market means attracting as many people as possible to use a social mobile Web application. The business team wants the site to be social to attract the attention of the heart of the market, be viral to create awareness on its own, feature a great user experience (UX,) and be accessible from mobile devices.

At a minimum the performance infrastructure needs to accomplish the following:
  1. Test the actual scalability of edge caching networks like Akamai
  2. Surface functional problems in the User Experience at high levels of load
  3. Monitor the existing plumbing between Twitter, Facebook, voting services and other integration functions in the site
  4. Stress the origin servers to identify how the application breaks when it fails
  5. Enable globally outsourced developers and testers to run performance tests at any time
In our experience PepsiCo raised the bar for all business managers in its social mobile Web initiatives. For IT managers that means integrated marketing campaign testing - like PepsiCo - requires performance infrastructure. Learn how PushToTest delivers a solution that works for the IT team and business team by attending one of our free live Webinar workshops.

Free Live Webinar Workshops


P.S. - If you would like to learn more about PepsiCo's performance test efforts, ask me for a briefing.

Summer Of Love, Training, Education

PushToTest Summer of Testing Love Workshop Series 2012

PushToTest is necessary infrastructure for organizations to successfully connect with the heart of their target markets.  We help PepsiCo, Best Buy, Deloitte, and 20 other organizations load and performance test their social, Web, and mobile applications. As an open source company we believe in education and training from our front-line experiences in the battle to achieve reliable IT.

PushToTest announced 17 free Workshops, Webinars, and training classes to help improve your knowledge in the following topics:

Open Source Test Workshop for CIOs, CTOs, Business Managers
Learn the Open Source Test way to solve testing applications and business processes that use Rich Internet Application (RIA, using Ajax, Flex, Flash, Oracle Forms, Applets,) and SOAP and REST Web Service interfaces.
June 6, 2012
July 25, 2012

Open Source Test Workshop for Testers, QA, Developers, IT Ops
Solve performance bottlenecks and function problems in your web applications. Learn the Open Source Test way to accomplish functional tests, load and performance tests, and service monitoring of Web applications, Rich Internet Applications (RIA, using Ajax, Flex, Flash, Oracle Forms, Applets,) and SOAP and REST Web services.
June 7, 2012
July 26, 2012

Agile Performance Test Workshop for CIOs, CTOs, Business Managers
Your organization may have adopted Agile Software Development Methodology and forgot about load and performance testing! Learn the Open Source Testing way to load and performance test your Web applications, Rich Internet Applications (RIA, using Ajax, Flex, Flash, Oracle Forms, Applets,) and SOAP and REST Web services. This free Webinar delivers a testing methodology, tools, and best/worst practices.
June 13, 2012
August 1, 2012

Agile Performance Test Workshop for for Testers, QA, Developers, IT Ops
Learn the Agile Open Source Testing way to load and performance test your Web applications, Rich Internet Applications (RIA, using Ajax, Flex, Flash, Oracle Forms, Applets,) and SOAP and REST Web services. This free Webinar delivers a testing methodology, tools, and best/worst practices.
June 14, 2012
August 2, 2012

What Is New In TestMaker 6.5 Briefing
Learning the major new features in TestMaker 6.5
June 20, 2012 9 am to 10 am Pacific time

TestMaker Object Designer Basics Workshop
TestMaker Object Designer is for fast and easy data-driven test authoring of Ajax and Flex Applications. Designer is free open source test (OST) tool published under the GPL license from PushToTest. Designer is an alternative to HP QuickTest Pro (QTP) and Selenium IDE.
June 27, 2012
August 29, 2012

Advanced TestMaker Object Designer Workshop
Take your Designer skills to the next level.
June 28, 2012
August 30, 2012

soapUI Basics Training Workshop
soapUI is a fun and easy tool for authoring test suites of SOAP and REST based Web services. soapUI is a free open source test (OST) tool published by SmartBears. PushToTest TestMaker is test infrastructure to repuprose and operate soapUI test suites to be functional tests, load and performance tests, and production service monitors.
July 11, 2012
August 21, 2012

Advanced soapUI Training Workshop
Take your soapUI skills to the next level.
July 12, 2012
August 23, 2012

All of these training Webinars are free and limited to a small size. Register now to secure your seat.


5 Essential Elements To The Emerging Cloud Platforms

In 5 Years Every Application Will Run In The Cloud

I love the new options that are available to host application software. In my role at PushToTest I have to choose where the company hosts Java application servers, Web servers, RDBMS servers, and media image servers. Plus we are a small organization and need easy ways to manage the whole system on a budget. The more management I can outsource the better off we will operate - both on price and up-time.

A viable cloud platform emerges when it provides these features:
  • Elastic utility computing - hosting the server equipment on-demand
  • Hybrid cloud - run the Cloud on the public Internet or in your private datacenter
  • Management - API to start/stop instances, Web based application control panel
  • Managed Data - managed RDBMS or NoSQL data store
  • Managed Storage - managed storage and warehousing
The ones that qualify emerge as:
  • Amazon AWS with Eucalyptus to run the cloud internally and RightScale for Web management
  • Rackspace Cloud with OpenStack to manage the cloud (almost in Beta)
  • Microsoft Azure with the Microsoft management services and Software Development Kit (SDK)
Others to consider: GoGrid, Savvis (Not really sure if they see themselves as a cloud provider,) VMWare (Except they don't provide an elastic hosting,) and Google App Engine (depending on Google seeing itself as a platform provider.)

There are still challenges for your success on a Cloud platform, including effectiveness of management, scalability, broken functions, up-time, and oversubscription. In the real world, these systems need to identify the end-user experience as input to an automated cloud scaling system.

Learn what you need to be ready for the Cloud:

Watch the Cloud Testing workshop screencast

Watch The Cloud Testing Workshop Screencast


Testing in The Wild West

Lack of Standards Causes a Greater Need for Test Expert Services

I set the company name to PushToTest to show playfullness in the face of a daunting task. To be a tester today requires expertise in software development, software service interfaces and APIs, test scripting tools and languages, and operational test data creation and use. All of these keep changing - sorry innovating. The average CIO, CTO, V.P. Engineering, and Test Manager expects the same thing:
  • They want to see TestMaker work flawlessly
  • They want to see TestMaker work with their applications
  • They want to see their testers and developers use TestMaker productively
  • They want to see TestMaker integrate with their development systems seemlessly
TestMaker is not designed to test the entire Internet - only big portions of it! For example:
  • For Web services (SOAP, REST, JMS, ESB) that have a well defined and managed standard interface, TestMaker works very well with customer applications. The W3C standards body governs the SOAP standards and Oracle's JCP governs the JMS standard.
  • For Rich Internet Applications (Ajax, Flash, Flex, SOA) that have no standards body, this is the wild west of test scripting. TestMaker is an organization's best choice because of its rich support of Open Source Test tools. For example, if Selenium won't solve an Ajax problem then TestMaker also supports Sahi for Ajax testing, and Java for Ajax testing too.
For those times when TestMaker encounters compatibility issues with applications under test, PushToTest Global Services provides a paid Test Architect and Test Subject Matter Expert (SME) to solve compatibility issues.

At PushToTest we call this "well defined standards" vs "wild west" and it really cuts to the core issue. There is a lack of standards that causes a greater need for expert services. This is not that any particular test tool is difficult to use, it is that there is a "wild west" variety of possible implementations.

Todd Bradfute of PushToTest takes the metaphore to an extreme: So you can choose to live back east (SOAP/JMS) where the legislature passes laws and people obey them. But the taxes are higher (contracts and validation.) So some choose to live on the frontier (REST) ... taxes are lower, people can do more of their own thing ... but you have to rely on neighbors more for help (forums). Or you could take your wagon out West (AJAX, Flex, Flash, etc.) ... there's gold in them there hills (beautiful RIA presentations) people are ingenious (custom controls for textboxes) but sometimes they bend the laws (miss keystrokes on password entry, fail to respond to keyboard input.) Thus the town needs to hire a sheriff (QA Architect) to make the town work.

Learn What PushToTest Global Service Experts Do For Your Organization

Learn What PushToTest Global Service Experts Do For Your Organization

Get along little doggie.


Web Security Test Solutions with TestMaker and soapUI

9 Top Web Service Security Attacks

Web Service security testing requires a thorough and orchestrated approach to mitigate business risk effectively. Web Services using SOAP or REST interfaces are vulnerable to many types of attacks:
  1. Boundary Scans exploit bad handling of values that are outside of defined ranges
  2. Cross Site Scripting enables functional changes in services and bypasses access controls
  3. Parameter Fuzzy Values use unexpected values in service calls to access restricted functions
  4. Malformed XML exploits bad handling of invalid XML in your service
  5. Malicious Attachments exploit bad handling of attached files
  6. SQL Injection exploits bad database integration coding
  7. XML Bomb exploits bad handling of malicious XML requests
  8. XPath Injection exploits bad XML processing inside your target service
  9. Invalid Types tests handling of invalid input data
Each of these attacks can happen from multiple networks, data centers, and geographic regions. And, each attack can happen at any point in time. Continuous Web Service security testing is mandatory to reduce business risk. Read the soapUI tutorial to learn how.

PushToTest TestMaker with soapUI Pro provides a continuous end-to-end Web Service security testing platform for your SOAP and REST based services.
Deploying security tests to multiple locations
TestMaker integrates with Continuous Integration environments like Hudson, Jenkins, and Bamboo. These CI tools schedule security test operation to coincide with new software releases and timed schedules. TestMaker deploys the soapUI tests to one or more TestNodes operating around your network and the globe. They pummel the service under test with cross-site scripting attacks, malicious attachments, XPath injection attacks and all the rest.

soapUI Pro provides automatic generation of security suites. White-box testers write security tests and TestMaker deploys the tests to be data-driven and grid and cloud deployed.

Running security tests in parallel with stress and performance testing uncovers cases that neither stress testing nor security testing would reveal on their own. This frees the testers to think about what constitutes a failure and a mitigation.

Running a soapUI security test in a TestMaker Production Monitor enables continuous security testing. TestMaker logs the results to a historic results repository for on-going compliance with Service Level Agreements (SLA) and Security Agreements.

Click to read the soapUI tutorial


Selenium Tutorials for Beginners and Experts

Hi Seleniumistas!

I updated the Selenium Tutorial Screencasts for Beginners and Experts at:

These are free Webinar screencasts that cover the following topics:
  • Selenium Basics
  • Adam Goucher's Selenium, You Are Doing It Wrong
  • Creating Robust Selenium Tests with PageObjects
  • Advanced Selenium Testing
  • David Burns on Selenium Testing Basics
Also, the Selenium Tutorial for Beginners starts at:



TestMaker Object Designer vs HP QTP

TestMaker Object Designer vs. HP QuickTest Professional (QTP)

Last week PushToTest completed a Proof Of Concept (POC) project for a medium-size IT organization. The experience showed us the strengths and weaknesses of the TestMaker Object Designer when evaluated by a bunch of QA testers brought up on HP's QuickTest Professional (HP QTP.) Open Source Testing (OST) is a much better solution:
  1. While QTP did a good job at getting testers productive, it has not done the QA tester community any favors. QTP has not kept up with the innovative ways software developers and IT Ops managers build and operate Rich Internet Applications (RIA, using Ajax, Flex, Flash, Applets, Silverlight, Oracle Forms.) QTP does not recognize the thousands of new Web objects in Ajax applications using GWT, JQuery, Yui, and Extjs.

  2. QTP is too expensive for Agile where testers team with developers.

  3. Only PushToTest TestMaker offers repurposing capability where a single test script runs as a functional test, load and performance test, and production service monitor. QTP scripts are incompatible with Load Runner.
Still, QTP is a familiar face. The testers we worked with had a lonesome, longing face when they considered moving from QTP to PushToTest TestMaker. The testers talk about QTP like it is some old college friend who got them into a fraternity. To those that resist change, we put together a screencast showing how to make the transition easy and what to expect.

Watch A Screencast on TestMaker Object Designer

-Frank Cohen, PushToTest

New Company Name Announced

PushToTest Changes Company Name

Announcement date April 1, 2012

I've been doing a fair amount of thinking about the future of PushToTest. I'm glad to share some of this thinking today and a big announcement.

The company name has helped us over the past 10 years to identify our testing solutions as powerful, effective, and easy. But, perhaps it is time for a change. I've considered changing the company name to simply TestMaker. That makes it more understandable what we do. I did some research and found these new company names work too:




I say if you're going to expand you should go big. So, I've decided to rename the company to:


Please let me know which of the above you like.


CEO and Founder

New PushToTest Logo

Publish Test Results From OpenOffice Directly

Using OpenOffice To Query TestMaker Results

Michael Acosta-Madiedo, an engineer on the PushToTest TestMaker project, developed a technique to publish performance test results data using OpenOffice. His technique uses the TestMaker architecture where logged results data is available in a Results Repository - normally a copy of MySQL or Oracle.

This is the kind of work PushToTest Global Services does for customers everyday. Learn more by clicking here.

Special note: Click here to view a description of the TestMaker Results Repository schema.

The Problem

While TestMaker produces hundreds of results reports there are several reasons why business managers, testers, and IT Ops managers would query results outside of TestMaker. The query and reports are able to deliver custom reports and workaround limitations in the built-in reports. For example, one organization needed weighted average response times from test execution.

The Solution

OpenOffice provides access methods to work with data in a database. We will show how to get to TestMaker data in an OpenOffice spreadsheet. In this case we show how this works with MySQL.
  1. Create a Mysql data source
  2. Add the mysql-connector.jar to the OpenOffice classpath
  3. Open an OpenOffice Spreadsheet

    Windows operating environment:
    a) Start
    b) Use Tools menu - Options - General Java.
    c) Click on the button Class Path..., then choose Add Archive... and navigate to where the mysql-connector-x.x.x-java-bin.jar file is located.
    d) Click on Open, and the file should then appear in the list of Assigned folders and Archives.
    e) You can then click on OK and close completely.

    Note: You may receive a message indicating that you have to do that anyway in order for to be able recognize your changes correctly. If the quickstarter is running, you should close that application as well.

    Mac OS X operating environment:
    a) Use Tools menu -> Options -> -> General -> Java is -> Preferences -> -> Java
    b) In either case, you can use the mysql-connector.jar in the TestMaker/TestNode/bundles/mysql/5.0 directory.
    c) If a connection file has already been created … Load it in the Registered

  4.>Preferences-> Base->Databases
  5. Click New
  6. Select the odf file (example svn/testmaker-ent/ETM/cloud/<client>/<client>_db.odb)
  7. Click OK
  8. If a connection file hasn’t already been created … Use the Database Wizard to connect to your repository File->New->Database
  9. Select “connect to to an existing database”, and “MySQL”, click Next
  10. Use “Connect using JDBC”, click Next
  11. Database name = “testmaker”, Server=server name where you have mysqld running, Port=3306, driver=com.mysql.jdbc.Driver
  12. Click “Test class”, click Next
  13. User name = testmaker, check Password required, click Test connection, enter password, click Next
  14. Select “Yes, register the database for me”, click Finish
  15. Save to a filename (example svn/testmaker-ent/ETM/cloud/<client>/<client>_db.odb)
  16. Add a query for the data that interests you
  17. Open the ‘Data Sources’ Window

    Windows: F4
    Mac: View->Data Sources

  18. Right-Click on Queries and select “Edit Database File”
  19. Click on Queries
  20. Now use one of the tasks to create the query The easiest way is to have a SQL call that you want to use and select “Creat Query in SQL View” This opens an empty design window for you to paste your .sql into
  21. Let’s start with one of the queries used by one of the TestMaker reports (see svn/testmaker/TM/tags/tm601/modules/charting/src/com/pushtotest/tm/charting/reports/jasper/StepDuration.jrxml … specifically the data in the <queryString> area. Also, click here for the repository schema.)

    stepdefinitions.NAME AS stepdefinitions_Name,
    steps.DURATION AS steps_Duration,
    TRANSACTIONS transactions INNER JOIN STEPS steps ON transactions.TransactionID = steps.TransactionID
    INNER JOIN SEQUENCES sequences ON transactions.SequenceID = sequences.SequenceID
    INNER JOIN ITERATIONS iterations ON sequences.IterationID = iterations.IterationID
    INNER JOIN EXECUTIONS executions ON iterations.ExecutionID = executions.ExecutionID
    INNER JOIN STEPDEFINITIONS stepdefinitions ON steps.StepDefinitionID = stepdefinitions.StepDefinitionID
    TRANSACTIONS.SUCCESS = 1 and executions.ExecutionID = :Enter_ExecutionID

  22. Save with some memorable name (e.g. StepDuration)
  23. Make sure to save your changes to the .odb file before exiting
  24. Pull results into a spreadsheet
  25. With the ‘Data Sources’ View visible, open the connection and Queries tree
  26. Drag the Query you want down into your spreadsheet
  27. If it takes parameters enter them now
  28. The data will take a few moments but will then populate into the spreadsheet view
If you need to narrow the data you can either modify the query (adding AND to the WHERE clause) or you can use spreadsheet magic (e.g. =IF(A2==”Geo_LoadTestbyAddress”;B2;””) I tend to do this to get statistical information (e.g. =AVERAGE(D2:D1776))

Need Help With This Solution? Ask PushToTest Global Services

Page 1 of 10

  • «
  •  Start 
  •  Prev 
  •  1 
  •  2 
  •  3 
  •  4 
  •  5 
  •  6 
  •  7 
  •  8 
  •  9 
  •  10 
  •  Next 
  •  End 
  • »