Archived PushToTest site


What Is The Cloud Tipping Point?


Every organization reaches a tipping point where new systems and processes work better than the existing ones. We are at a cloud tipping point. This will impact every organization;s ability to effectively deliver IT systems and services.


Cloud technology impacts our business operations, software development plans, and customer services. At a business operation level my company, PushToTest, is representative of organizations that reached the cloud tipping point. We used to host 3 servers in a co-location facility.Today we host no servers of our own. In the past year we found cloud service providers for most of our needs: Google for company mail and Web site analytics, Fonality for telephone PBX operations, Epicom for hosting customer relationship management (CRM) services, and for Joomla Web site hosting. The Cloud has been wonderfully beneficial to our small company. Our costs are half, we can scale up as our business grows, and our services delivery reliability improved.


We achieved the cloud tipping point in our product development plans in 2008. TestMaker 5.4 introduced Cloud Testing and TestMaker added Grid Testing. We partnered with Amazon EC2, Collabnet, GoGrid, and Rackspace. Our costs are tiny, we deploy up to millions of virtual test users, and we deploy tests to a variety of geographic locations around the world. We now have Ant build scripts in TestMaker to a build an Amazon Machine Image(AMI) file for the TestNode automatically. We even use a hosted svn and Bamboo continuous integration environment from Atlassian. Next we will build automatic nightly builds, deployment, and tests.


I’m wondering what other organizations are doing with the Cloud? Please comment about the following:

  • Are you building any cloud-based applications? How are you using external cloud-based resources as building blocks for internally-written applications.



PushToTest is a small business serving a large user community and many global enterprises. We are often challenged to spend our cash on capital investments. The faster you achieve a cloud tipping point the closer you are to realizing the cloud's benefits.


I would love to learn your thoughts on these trends. A new general consensus appears to be coalescing.



TestMaker 5.6 Briefing: Refactoring The TestNode


PushToTest is refactoring the TestNode system to enable faster test operation, increased test scalability, and improved test stability. Read the details here. PushToTest intends the work for release in TestMaker 5.6 in late 2010.


We are hosting a special live Webinar briefing. This live Webinar will provide an early look at the refactored TestNode running functional tests, load and performance tests, and production monitoring. You will hear from the engineers the current status, see a demonstration, and learn the next steps. The Briefing will be open to your questions, feedback, and criticisms. We look forward to your participation!


This is a free event. Registration required:





Record/Playback For The Future Web


PushToTest is working on a next generation Record/Playback tool for modern Web applications. We are building this tool to be a compelling open source alternative for HP QTP and VUGen Users.


TestMaker Object Designer Briefing

Thursday, September 9, 2010
9 pm to 10 pm Pacific Daylight Time (PDT, GMT-7)


This live Webinar will provide an early look at the new TestMaker Object Designer (TOD) utility.


You will hear from the engineers the current status, see a demonstration, and learn the next steps. The Briefing will be open to your questions, feedback, and criticisms.


We look forward to your participation!


Sign-up for the Briefing at:



Web Application Test Tools


A recent discussion on LinkedIn began with a question: “Trying to find a tool to test websites. I am evaluating QTP but have problems identifying objects on any website. I also tried Selenium IDE, but I am worried about it being just on Firefox. Any ideas?”


The discussion that followed revealed what developers, testers, and entrepreneurs are considering for modern test tools. Here are the products recommended:


  • QTP is a good option but very costly, TestComplete is another good option.


  • If you’re looking to test with Selenium in an environment where Eclipse is used as well, have a look at CubicTest too. It provides a really nice way to setup and manage your tests.



  • You might want to also look at badboy ( I have not used it in anger but I believe it is free if you are using less than 5 copies within the same company.


  • I’d recommend WatiN – while it’s developement seems to be stalled a bit, it’s still very capable and pleasant to work with; it is focused on IE, but works with FF as well, and (unlike WatiJ/WebSpec ) is quite fast. Should work with any .NET language of your choice and SharpDevelop makes nice replacement for VisualStudio. If your project is Java based you yourself have understanding of language and tools you may wish to try new kid on the block – that is Tellurium. Offers very nice approach to test driven testing. AutoIt – which is already mentioned – is very nice as long as your automation requires not only interacting with browser itself (in this case, IE only) and some other applications (copy from Excel -> commit to DB -> some action in browser -> launch another app ). Works only on Windows, IE only (through COM, other apps – click and type scenarios only, not very reliable). Also, function list provided by AutoIt library is significantly different form other tools. Unless Canoo Web Test has been rebuild recently, don’t bother with it – it’s test description and reporting are great, but fails miserably on JavaScript. Anyway, I think it all depends on which tool will be less incapable of handling your application… I found it useful to have “test page” with all hacks used througout application for use as “proving ground” for tools.


  • Lets start off with defining what your needs really are. First, do you only want to use a automation tool for “browser based testing” ? Are you expecting to automate any application based applications? Second, what type of shop are you in? Are you primarily a open source/java/php/ruby type shop? are you a .net shop? MAC shop? Does your company mind paying licensing fees? Third, what type of staff budgets to you have or do you expect to do this with existing staff? if existing staff… what kind of skills do they currently have? If you just want to support Browser based automation.. and not application based testing. What browsers are you expecting to support through your automation tool? There’s a big difference between open source tools and licensed software, and there’s a lot that goes into tool selection. Take your time. don’t rush it. You could be stuck with the results for a long time.


  • My guess is that you have a very basic beginer lack of knowladge in QTP and those QTP doesn’t identify your web application. 
    QTP web support is great and if you need any help i would recommand you to spend few min to go over the documenttation and few of the forums and you will find what was the root casue of the problem.


  • In my view Selenium needs lots of programming skill. But overall its a very powerful tool to automate the web application. Apart from that I also suggest you to check TestComplete its also a very good tool for both webbased and desktop application.


  • Absolutely Selenium needs a lot of programming skill. I would be somewhat suspicious of any test automation tool that claimed it DIDN’T require programming chops.


  • I used to agree regarding a tool requiring programming chops. I have used most of the popular tools on the market: WinRunner, QTP, Silk, Robot, Autotester, TPNS, etc. etc. I evaluated Certify, truly a tool that requires no programming. It is parameter driven and I teach SMEs and BAs to be successful with it on a regular basis at some of the most prestige’s companies. I enjoy the other tools but can’t compete with the productivity of Certify (by Worksoft of Dallas. It’s better than 60% faster for development and maintenance. I have been using it for over 10 years now at major companies for IBM mainframe, html Web, .Net, VB, Java, SAP and SAP Portal transactions.


  • I think each tool have it’s own pros and cons and open source tools are coming close to proprietary tools and Selenium is one of them. It all depends what skill you have if you want to pick one and off course Dont forget about licenses.


  • After using Certify for 10 years I guarantee there is no programming. It is all parameter driven and much easier and quicker to learn and use, hence the higher price tag. Better than 60% faster development and time to market justifies the price tag. Most managers are tied into supporting their past and current choices and staff and I agree are hesitant to make a change but if they do the result is considerable savings all around and in technical resources and time to market. As far as cost, for the basics of MF, html, .Net, VB, Java, etc. it’s comparable; however, as far as SAP there is a considerable advantage that comes with the tool along with the support by a crack team at corporate headquarters in Dallas. Having that advantage, support, team and no programming approach is what an SAP test initiative does not have to gamble on. The team in Dallas is made of professionals at your service that you would be hard pressed to duplicate at any cost. Again, I’m not a sales person, just a long time experienced engineer and user of software and test automation.


  • You can try ITKO LISA which essentially works on HTTP GET and POST methods and doesnt requires object repository and can do Functional / Performance testing. We have successfully used it in ecom website and its excellent since ecom websites contents keeps changing including pricing info. on a daily basis. But a level of programming in JavaScript/Java is required. Hope this helps.


  • My last quote (although it was 2 years ago) for Certify was well into 6 digits. It was Certifyablely.. not comparable by any sense in cost. I’m not trying to “blast” the tool. It’s approach is certainly different than any of the other automation tools out there.


  • I know the SAP interface is priced seperately by Worksoft because it comes with some awesome benefits developed and supported in Dallas. I haven’t priced other tools lately, and of course in the software sales business pricing is by usage, but, I know that a one seat license about 2 years ago without the SAP Interface was about $7K. That gave you MF, html Web. VB, .Net, Java, etc all for the same price.That would be the most expensive way to purchase as there are of course discounts by volumn. I don’t think that is out of range with the others especially for a no programming approach where you can train capable SMEs to automate and maintain testing? Sorry I sound like a sales person. It’s just that I have used to tool for a long time and the only complaint I have heard on pricing is for the SAP interface which I don’t know the cost of but having taught and used it I think its worth a good deal.


  • Okay, so really, you can generate eggPlant scripts with zero coding knowledge — any SME can be creating scripts within a few minutes of sitting down with the tool. Aside from knowing how to use the software you’ll be testing, all you really have to be able to do is drag out a selection rectangle and click a button. eggPlant tests the app the way the user uses the app; the users don’t care what the objects in your app think they are doing and neither does eggPlant. What matters is what gets displayed to the user, that the user can interact with the interface and that the output is what is expected. * single license of eggPlant* running on Windows, Linux, or Mac OS will test any application running on any operating system — Windows, Linux, Mac, Unix. It will test against *any* browser. It will test Flash, Javascript, HTML5, and Silverlight. Using that same license, you can test applications on smart phones, including Android and iPhone (currently a feature of the Linux version, but coming soon to the Mac and Windows versions). You can switch between applications and even machines from within a script — test a chat client running on Windows, Mac, and iPhone from within a single script. And again, you don’t have to do any programming to do this (but when you’re ready, a rich, full-featured, yet syntactically simple programming language is available to you.) Really — it slices; it dices; it juliennes.


  • You can try SAHI, I found it much versatile and is true cross browser.


  • I also suggests SAHI, which I found very useful for testing web applications. The scripting using SAHI is pretty simple while comparing with many other open source tools


  • I think try out CODEUITest Builder.. it’s a new component by Microsoft in VSTS 2010.. It gives you good blend of recording and coding for Automation.


  • You could also use Oracle’s Openscript (a new technology that came from Empirix, which Oracle aqquired recently). The scripting language is Java, which is cool, and it supports web applications very well (I have have tested many web apps), including ADF, Oracle Forms, Visual Web objects etc . The great thing with this tool is if you can’t test something out-of-the-box, but you know some Java, you can write the test yourself.


  • I plan to Use Selenium IDE and Selenium RC with .NET application and IE7. But I believe RC would require Java to be installed. Does this lead to any conflict in Using Selenium to test .Net applications?


  • QTP is a good option but very costly, TestComplete is another good option.


  • If you’re looking to test with Selenium in an environment where Eclipse is used as well, have a look at CubicTest too ( ). It provides a really nice way to setup and manage your tests.



I chimed-in with my own ideas:


I am a firm believer in open source test tools. Most open source test tools show signs of maturity previously only held by the proprietary tools. There is not much difference between HP QTP and Selenium at this point.


I would invite you to join us at the Open Source Test Workshop. This is a free live webinar meet-up to see what can be accomplished using the open source test tools. It’s free and normally a couple of hundred fellow testers, developers, and architects get together to share ideas. Details are at


The cool thing about the open source tools like Selenium is the amazing things that can be done by combining and integrating them. For example, we are sponsoring a new open source record/playback tool that combines Selenium, Sahi, and TestMaker. The new tool will record and playback in Firefox, IE, Chrome, Safari, and Opera. It also adds conditionals, looping, and branching. All the scripts are data driven. And these scripts easy repurpose to be functional tests, load and performance tests, and production monitors. We intend to release an ‘alpha’ next month and have a final release in Fall. Watch a screencast here.



Web App, RIA, Mobile, Desktop Testing


I speak with CIOs and CTOs weekly to understand their challenges and needs. Testing, monitoring, and sustainability are top on their list. The problem for many of them is how to achieve reliable delivery of complicated systems across many deployment platforms. For example, several organizations in the PushToTest community are service enabling their applications for deployment to mobile platforms, desktop platforms, and for integration with other services.


I founded PushToTest with the core idea that tests should be viewed as repurposable components. For example, repurpose a functional test for smoke testing to later be a load and performance test. TestMaker is very good at repurposing. TestMaker is also very good in the following areas:



Over the next few weeks you will find case studies, cost benefit analysis (CBA) models, and guides to testing on these pages. Please let me know your feedback.



New Screencast on TestMaker Object Designer (TOD)



 PushToTest funded a new open source project to deliver a modern Web application test record/playback system. We named the project TestMaker Object Designer (TOD.) Our hope is to move Selenium IDE and other record/playback test tools into the era of modern Web and Rich Internet Applications (RIA, using Ajax, Flex, Flash.) The work on this new tool started in March 2010.

 Last week we hosted a special briefing meeting with the PushToTest community to show our progress on the project. We recorded the briefing.

 Watch the TestMaker Object Designer (TOD) June 2010 Briefing (45 minutes)

 We intend to release a “beta” version of the Designer in August 2010. We will have another briefing shortly before the release.



Managing SOA Sustainment




I speak weekly with CIOs and CTOs over the past 2 years to understand the extent to which modern IT systems are built on Service Oriented Architecture (SOA,) Rich Internet Application (RIA, using Ajax, Flex, Flash) and Business Process Management (BPM) technologies and methodology. These technologies are wide-spread and rarely understood well. Particularly difficult is long term sustainment and maintenance of these systems.


The Air Force IT Conference is coming up at the end of this month. PushToTest will be participating through our partner Dynamic Research Corporation (DRC.) DRC is in a great situation to deliver state of the art development, testing, monitoring, and sustainment of information services that use SOA, RIA, and BPM technology. DRC’s choice of technology and service partners is a good illustration for CIOs and CTOs effort to select and source technology, solutions, and vendors. DRC breaks down its choices:


Vocabulary Tools Understand new technologies and implement new methodologies. For example, brining SOA, Cloud and BPM technology into your development, test, and operations teams. Agile-Path, DRC, Tri-Cor
Test Integration testing, production monitoring, regression testing, load and performance testing, stress testing, and SLA compliance PushToTest, Managed Methods
Modeling Tools for semantic data integration, including metadata and data ETL functions, business rule reuse, and project lifecycle management. Expressor, Denodo, Opnet, 3DRC
Cyber Information assurance/cyber security services includes management support, operational support and technical support. Organization’s managers need to understand the command and control architecture to be effective. Lunar Line
SOA Architecture, construction, maintenance Adnet, Appistry, FGMAysnchrony Solutions, SSAI, OST
System Sustainment Maintenance on system and critical components Trident Technologies, Indus, Infinite Technologies, LDSS, Kadix Systems, Enterprise Solutions Group, Dayton Aerospace
Project Lifecycle Management (PLM) Manage resources, tasks, and schedules. ARAS


DRC’s selection is a good template for your own organization.


Fill out the request for information form and I will be happy to introduce you to DRC and its partners.



TestMaker 5.5 Ships!




PushToTest officially releases TestMaker 5.5 today! TestMaker 5.5 is a major feature enhancement and bug fix release.


Software engineering met Continuous Integration (CI) years ago. Tools like Hudson, Bamboo, Collabnet TeamForge, and Cruise Control automate the build and deploy cycle. Now, PushToTest drives an innovative way to achieve Continuous Testing (CT) in your organization.


New capabilities in TestMaker 5.5 include:


  • Grid Testing – TestMaker runs your tests on a Grid of your test equipment, in a Cloud Computing environment, and both.


  • New Script Runners to operate tests written in .NET, Visual Basic (VB) and from the command line


  • Scalability Index Plus ™ – TestMaker shows graphically where the application under test should be performing with linear scalability


  • Reports now show error messages and codes for easy drill-down


  • Relational Database logging


  • Macintosh native application packaging


Download the new software at


Read the Release Notes for details.


PushToTest tests TestMaker 5.5 on Windows XP SP2, Windows Vista SP1, Windows 7, Mac OS X 10.5.8, and Ubuntu 9.10 (Karmic Koala).


TestMaker 5.5 and later requires a special Microsoft patch to Windows XP for the Editor to operate correctly.

New record/playback tool briefing on Tuesday


We have a functional prototype to show everyone. Please join us for a briefing to see it work and for us to get your feedback, ideas, and criticisms. All are invited.


TestMaker Object Designer Briefing
Join us for the briefing on Tuesday, June 29, 2010 at 8 am Pacific time


Meeting details at:

PushToTest is working on a next generation Record/Playback tool for modern Web applications. We are building this tool to be a compelling open source alternative for HP QTP and VUGen Users. This live Webinar will provide an early look at the new TestMaker Object Designer (TOD) utility. You will hear from the engineers the current status, see a demonstration, and learn the next steps. The Briefing will be open to your questions, feedback, and criticisms. We look forward to your participation!


Title:    TestMaker Object Designer Briefing
Date:   Tuesday, June 29, 2010
Time:   8:00 AM – 9:00 AM PDT


After registering you will receive a confirmation email containing information about joining the Webinar.


System Requirements
PC-based attendees
Required: Windows® 7, Vista, XP, 2003 Server or 2000
Macintosh®-based attendees
Required: Mac OS® X 10.4.11 (Tiger®) or newer



How do you secure customer data in the cloud?


Last year the European Commission invited me to speak at their meeting of IT managers. During the questions and answers panel the IT managers asked many questions about cloud computing technology. I heard a common theme in the questions: data security and how to handle failures were keeping the IT manager community from moving applications into the cloud.


A friend is the IT manager for a large digital advertising agency. He is tasked with preparing the agency with enough equipment, bandwidth, and test automation capability to serve the agency customers. I asked him the same questions I heard from the EC IT managers: How do you secure customer data in the cloud?


My friend pointed to SAS 70, Cloud Security Alliance, Amazon and Rackspace.


SAS 70 is the Statement on Auditing Standards from the American Institute of Certified Public Accountants (AICPA). SAS 70 is different from the ISO 9000 standard certification in that SAS 70 is not a pre-determined set of standards that a service organization must meet to “pass”. Instead an independent auditor evalutes and audits a service organization’s security controls. You sit down with an auditor to conduct a security review. The auditor asks you how you comply with standard. And it only costs you $25,000 USD to go through the audit.


The problems begin when you are on the receiving end of a SAS 70. For example, a customer of Amazon or Rackspace may see SAS 70 certification on the Web site and believe the company to be risk free for security controls. SAS 70 is a reporting tool for auditors. While the Web site may herald SAS 70 certification, the actual audit may determine that the company is only complying with a few security controls. For example, an SAS 70 certification may have found the security controls are inadequate and the staff not trained. Making sense of a SAS 70 requires the reader to be well versed in IT control and compliance. My friend says some are pretty clear, others obscure.


As an IT industry we need the following:


1) A standards body to define cloud security best practices and controls. The Cloud Security Alliance does a good job at fulfilling this need.


2) A standard way to report an organization’s compliance with the best practices and controls. SAS 70 publishes the data in an “auditor”-focused lingo. It is not entirely obscured.


3) A way to transform the certification and audit publication into a set of APIs that are easy for software developers to adopt in their cloud-hosted applications. Platform-level security management is an opportunity for the platform providers: Oracle for Java, Microsoft for .NET, Zend for PHP.


Combining security policy management with scalable platform support will help cloud computing take off. And hopefully our jobs will multiply to support the infrastructure. Every IT and development organization leader will eventually need to answer the changing requirements for data security and when to build a better hosting platform.



TestMaker 5.5 RC3 Now Available!


We are very happy to make TestMaker 5.5 Release Candidate 3 available for immediate download. TestMaker 5.5 is a major feature enhancement and bug fix release. Download the new software from these links:


TestMaker 5.5 Release Candidate 3 for Windows

TestMaker 5.5 Release Candidate 3 for Macintosh


TestMaker 5.5 Release Candidate 3 for Linux

Release Candidates are functionally complete, are in-use by our customers, and may contain bugs. We had expected Release Candidate 2 was the final release; However, there turned out to be a significant SeleniumHtmlunit issue we needed to solve. We are issuing Release Candidate 3 expecting this is the final release and that we will declare this the final version on Monday, June 21, 2010. Please discuss issues here.


Release Candidate 3 has these changes:


  • Added MaxRampDown timer to force the end of a test ramp down. Set in the Options tab of the Editor.


  • Added Htmlunit headless Web browser version selector to Options tab of the Editor. TestMaker 5.5 ships with Htmlunit 2.7 and 2.8.1705. In our experience Htmlunit 2.7 is better for performance and 2.8 is better for Ajax compatibility.


  • Added Grid support for multiple TestNodes running on one machine. The Grid Configurator utility (in the Tools drop-down menu) accepts a full URL, including port number and path.


PushToTest tests TestMaker 5.5 on Windows XP SP2, Windows Vista SP1, Windows 7, Mac OS X 10.5.8, and Ubuntu 9.10 (Karmic Koala).



Page 6 of 10