We were recently speaking with a prospective customer who asked us:

“HP is offering a license for Quick Test Professional (QTP) at $7,500 per seat. PushToTest TestMaker Enterprise cost is much higher. Why should I buy TestMaker Enterprise?”

The prospective customer is doing software development of a Web application. The application incorporates Rich Internet Application (RIA, using Ajax and Flex/Flash,) SOAP-based Web services, and Service Oriented Architecture (SOA.) They have a single QA tester now. They need to do functional and regression testing. They have a single tester now and their needs are expanding.

Here is what we told the prospective customer:

Thank you for learning about the PushToTest solution. From our experience we see that you have a number of options:

1. Buy the QTP license at $7,500 per seat, plus $1,500 per seat per year in maintenance. We hear rumors of the Ajax protocol handler for QTP costs an extra $12,000. That will get you functional testing for Web 1.0 applications. We wonder if you will succeed with QTP in an Ajax environment from what our customers tell us about their experiences.
2. Use Selenium IDE and Selenium RC in TestMaker Community at $0 cost. TestMaker Community and Selenium do a great job at functional testing for Web 1.0 and RIA applications. It has a record/playback tool, simple test language support, and plays tests in IE, Firefox, Chrome and others. We typically get testers up and running on Selenium with a 2 day training course. Download TestMaker Community (with Selenium integrated) at http://www.pushtotest.com. Read our comparison of Selenium to QTP at  http://www.pushtotest.com/docs/thecohenblog/web-testing-tools-comparison
3. Buy a TestMaker Enterprise license. The price starts at $30,000 per year, depending on options. This will get you the test environment you need now for functional testing of RIA and SOA applications plus you will be ready to repurpose those tests to be load and performance tests and business service production monitors at no additional cost or effort.

Our suspicion is that HP’s pricing really looks like this:

This estimates the costs to run functional test of an Ajax application and load and performance tests at 1,000 virtual users with 2 tester seats. The proprietary alternative delivers separate tools for functional testing and load testing. Lack repurposing of tests between these environments adds test recoding and re-qualification costs.

Most of our customers report savings of 87% to 310% in their first year of using PushToTest solutions.

And don’t forget that your company benefits from PushToTest in the following ways:

• Increased visibility of functional issues and performance bottlenecks in customer facing Web services.
• Minimize service interruptions by predicting service delivery from current application health status and load testing delivered from the PushToTest solution.
• Increase production monitoring test coverage to interoperate with your application’s new Ajax-based user interface.
• Produce more actionable knowledge for your organization’s management by using the latest results analysis charting capability in the new TestMaker 5.4 product.
• Continue and expand your technical team’s knowledge and expertise in its continued use of the PushToTest solution in 2010.

PushToTest is the best solution for your business needs.

-Frank

Jason Huggins, one of the founders of Selenium, spoke at the Agile Developer user group meeting recently. Matt Raible of DZone gives a good summary of Jason’s comments.

Among them:

• Selenium started at Thoughtworks. They were challenged to fix an Ajax bug in their expense reporting system. JWebUnit, HtmlUnit, JsUnit, Driftwood, and FIT did not meet their needs. They invented Selenese as a notation for tests first.
• Selenium Core, as a JavaScript embedded test playing robot, came next. Then Selenium RC and Grid.
• Selenium test playback is slow. Parallelization can solve some of the slowness problems.
• JavaScript sandbox, Flash, Applets, Silverlight, and HTML 5’s Canvas all present problems in Selenium.

Read the article here.

Here are my thoughts:

Thanks for the good write-up. Too bad about your battery. I would like to hear more.

PushToTest integrated Selenium into TestMaker a couple of years ago. Selenium works very well for testing Ajax applications. And, TestMaker runs Selenium tests as functional tests, load and performance tests, and business service monitors. TestMaker runs these tests in your QA lab, in the Cloud, or both. See http://www.pushtotest.com/products/cloudtesting

Jason’s comments about FlexMonkey and GPL are an interesting take.  FlexMonkey is a GPL implementation of the new Adobe Automation API. The Automation API defines a Selenese-like testing language. So it’s the API that needs to be implemented, not adoption of FlexMonkey. PushToTest is working on a contribution to Selenium IDE that implements the Automation API and test language. That will provide a standard way to test Flex applications within the Selenium (and consequently TestMaker) environments. http://www.adobe.com/devnet/flex/samples/custom_automated/

We are also building a new Ajax-based record/playback utility that will output Selenese tests. This is aimed at providing an open source alternative to QTP. The design is available at http://downloads.pushtotest.com/201001/TM_IDE_Design.pdf

Lastly, I’ve written up a comparison of QTP to Selenium at http://www.pushtotest.com/docs/thecohenblog/web-testing-tools-comparisonThanks!

-Frank

CMC Media produced a set of interviews at the STARWest testing conference. They asked a dozen thought leaders in the testing space:

“Is agile affecting testing?”

Watch the video interviews at:

http://www.youtube.com/watch?v=zhITAFaITCA

I am part of the interviews. Here is my opinion:

“Seems like 2008 was the year that a lot of people came to the realization that using waterfall techniques to build software incrementally – where you went from requirements into development and then into testing – is pretty much an easy way to kill your developers and testers. It is just to hard to in an Ajax environment or an SOA environment, to use waterfall.

Agile, on the other hand, says “Let’s test first! Let’s build the interfaces to the application first.” That’s great. Agile also brings peer development and a bunch of other methodologies.

I am seeing testing move up to leverage Agile techniques as well. When you are building new applications and services using some of the new frameworks there is no reason you can’t build the test at the same time you build the application. That is beyond what Agile itself originally said. Agile wants you to do test first. I say build and test at the same time using Agile techniques.”

-Frank

PushToTest introduced Cloud Testing in TestMaker last year. The results have been outstanding. In 2009 we saved our customers $650,000 in fees that they would have otherwise spent on building out their own QA labs. With this enabling TestMaker technology, customers run their automated tests using continuous integration servers like Bamboo and Hudson.

Cloud computing environments used to be the pervue of businesses like Amazon,
Collabnet, and Rackspace. An open source software stack emerges to build and manage your own cloud:

• Ubuntu Enterprise Cloud combines Ubuntu Linux with Eucalyptus
  and other cloud management tools. Eucalyptus joined the world of commercial open source with the launch of Eucalyptus Systems in 2009. Eucalyptus incorporates the Apache Axis2 Web services engine, Mule enterprise service bus, Rampart security, and Libvirt virtualization. Eucalyptus comes with its own implementation of the Amazon API.
• The Red Hat Deltacloud Portal manages all of the cloud deployments that may
  exist within an environment, providing a common integration platform. Deltacloud helps ease the integration between public and private clouds. Deltacloud creates a REST-based API to map to Amazon EC2 and private clouds that use VMware or Red Hat Enterprise Linux.
• Nimbus is an open source toolkit that turns clusters into a cloud environment. Nimbus has a suprisingly complete set of APIs and management systems to provision customized compute nodes.
• PHP developers have Simple Cloud API from Zend Technologies to manage cloud services from GoGrid, IBM, Microsoft, Nirvanix Storage Delivery Network, and Rackspace Files. Java developers have Typica to access Amazon’s SQS, EC2, SimpleDB and DevPay LS web services. We use Typica in TestMaker with great results.
• PushToTest TestMaker repurposes Selenium, soapUI, and Java, Ruby, Perl, PHP unit tests as functional tests, load and performance tests, and business service monitors. TestMaker operates tests from a private cloud, public cloud, or both.

Thanks to Michael Biddick’s article in Information Week to motivate me to write this blog entry.

Frank

Download the Press Release (Adobe Acrobat PDF format, 81 Kbytes)

CAMPBELL, Calif. and BRISBANE, Calif., Jan. 21, 2010 —  PushToTest® and CollabNet® announced today an expansion of their business relationship to deliver test capability in the cloud – backed by the latest releases of CollabNet TeamForge® and PushToTest TestMaker™ for enterprise clients, government organizations, and open communities worldwide.

TestMaker, the open source alternative for test automation, has been added to the CollabNet collabXchange™ marketplace.  CollabNet, the leader in distributed application lifecycle management (ALM) solutions, and PushToTest have integrated TestMaker with the CollabNet platform; organizations can reduce unplanned service interruptions, increase developer and tester productivity, and improve the ability to achieve service level agreements (SLA).

“We designed PushToTest TestMaker to leverage Cloud Computing. CollabNet TeamForge enables us to deliver huge price and feature advantages to our customers,” said Frank Cohen, chief executive officer, PushToTest, and world-renowned Test Geek. “With TeamForge Lab Management support built into TestMaker 5.3, users click the start button, watch TestMaker create load injector machine instances in their own QA lab or a cloud provider such as Amazon Web Services (EC2), run the test, and then take the machines down. This saves customers huge amounts of budget, relieves them of running their own QA lab, enables tests with up to millions of simulated virtual users, and runs tests from multiple geographic locations – and all on-demand.”

For organizations needing to surface and solve performance bottlenecks and functional problems, TestMaker provides a single tool to verify advanced Rich Internet Applications (RIA), Service Oriented Architecture (SOA,) and Business Process Management (BPM) applications.

PushToTest supplies the complete test automation platform to deliver functional testing, load testing, and business service monitoring by repurposing the same single test script. CollabNet’s TeamForge Lab Management and CollabNet Subversion reduce time spent on configuring build and test environments and securely managing component reuse libraries or test assets in a centralized repository.

“CollabNet has built the collabXchange to best serve our customer base with options to extend and expand their CollabNet platform,” said Jim Ensell, vice president, business development, CollabNet. “Together, TestMaker and CollabNet TeamForge Lab Management provide a compelling solution for test automation in the cloud.”

About PushToTest

PushToTest is the open source application performance management and test automation solutions provider. PushToTest is paving the open source test (OST) solutions migration into the mainstream by making high-quality, low-cost technology accessible to medium and large size global organizations and by providing world class professional services. PushToTest provides a software testing orchestration platform along with cloud testing capabilities and outsourced testing solutions, as well as support, training, and consulting services to customers worldwide and through top-tier partnerships. PushToTest’s open source strategy offers customers a long-term plan for building testing infrastructures that are based on and leverage open source technologies with a focus on rapid adoption of the latest technologies, reuse of test asset investments, and ease of management. For more information, visit www.pushtotest.com or send email info@pushtotest.com

About CollabNet

CollabNet is the leader in application lifecycle management (ALM) platforms for distributed software development teams.  CollabNet TeamForge is the industry’s most open ALM platform, supporting every environment, methodology, and technology.  With an integrated suite of easy-to-use tools that share a centralized repository, it is the only ALM platform that enables a culture of collaboration, improving productivity 10-50% and reducing the cost of software development by up to 80%.  As the corporate sponsor of the open source Subversion project, the best version control and software configuration management (SCM ) solution for distributed teams, collaborative development is in CollabNet’s DNA.  Millions of users at more than 900 organizations, including Applied Biosystems, Capgemini, Deutsche Bank, Reuters, and the U.S. Department of Defense, have transformed the way they develop software with CollabNet.  For more information, visit www.collab.net or call (888) 778-9793.

Media Contact:

Paige Schoknecht

Prequent, Inc. (for CollabNet)

+1 (408) 275-1419 office

+1 (650) 223-4085 mobile

paige@prequent.com

Frank Cohen

PushToTest

+1 (408) 871-0122

info@pushtotest.com

Try out the new TestMaker 5.4 and let us know your experience.

TestMaker 5.4 is a major feature enhancement release. The new software includes major feature improvements in the following areas:

* Easier and More Productive User Interface – The new TestMaker Editor is a fun and easy graphical tool to build and operate tests.

* Continuous Integration Features – Compatibility with Hudson, Bamboo, Collabnet TeamForge, Ant, JUnit, and other continuous integration environments

* Updated Tutorials – 30-minute tutorials on building tests of Ajax applications using Selenium, building tests of SOAP-based services using Selenium, and building tests of Flex-based services using TestMaker AMF protocols.

* Improvements to Selenium IDE – We began to make changes to Selenium IDE to improve its support for Ajax-based Web applications.

* Bug Fixes – Dozens of small fixes and improvements.

The project includes contributions from dozens of engineers in the TestMaker open-source community and PushToTest funded engineering effort by Andrew Zuercher, Ali Faiz, William Martinez Pomares, Luis Carlos Lara Lopez, and Peter Schumacher.

We are planning to release TestMaker 5.4 next week. We would appreciate your feedback now. Please try one of the download links for TestMaker Enterprise.

For Macintosh users:
http://tm53rc4.s3.amazonaws.com/PushToTest_TestMaker_Install_Mac.zip

For Windows users:
http://tm53rc4.s3.amazonaws.com/PushToTest_TestMaker_Install_Windows.zip

For Linux/Unix users:
http://tm53rc4.s3.amazonaws.com/PushToTest_TestMaker_Install_Linux.tar.gz

Barring no show-stoppers, this is the final release candidate and the final 5.4 software.

Please let me know any issues you run into by posting a message to the support forums.

Thank you.

-Frank

Open Source Test Workshop dates announced for 2010

The Open Source Test Workshop is a free meet-up for people and organizations needing to test information systems for functionality, load and performance, and service delivery. PushToTest hosts the meet-ups. One recent student to the Workshop said this afterwards:

“Thank you so much for inviting and presenting a great useful webinar. The information present there was excellent, I got lots of information. Just after the webinar I google the importance and advantage of selenium over the QTP as tools available in market. I underestimated this webinar and didnot prepare myself for it. But I gained alot. Thank you so mcuh. Your webinar has sparked me alot for learning selenium. I would appreciate if you let me know some more good resources for that.”

PushToTest recently announced the dates for the Workshop in January and February via live Webinar.

PushToTest hosts two forms of the Open Source Test Workshop:
Workshop for CIOs, CTOs, and Senior Managers

Workshop for QA Testers, Developers, and IT Managers

We hope to see you at the Workshop!

-Frank

Test Management with PushToTest

PushToTest Global Services provides our customers with test management and business performance management solutions that include our TestMaker solution. We do so by integrating TestMaker into a test management stack that includes the following technology:

• Test case\suite management – PushToTest partners with Collabnet TeamForge and other open source packages to provide test management, bug tracking, and revision control for tests

 

• Test data management, (parsing , sanitisation & storage) – TestMaker Data Production Library (DPL) technology data enables tests from RDBMS solutions like MySQL. We train your team to build tests using DBUnit for data generation and replication.

 

• Security test case management – TestMaker supports security testing of X509, SSL, SAML, and application level security in test suites.

 

• Automated regression with command interpreter – TestMaker repurposes unit tests to be automated regression tests, load and performance tests, and business service monitors from one test.

 

• Integrated defect management – TestMaker provides integrated defect management through Collabnet TeamForge, Trac, and other ticket system.

 

• Test result storage and reporting – Fully automation storage and results reporting.

 

• Key Performance Indicator (KPI) reporting – Provided as part of PushToTest’s reporting capability.

 

PushToTest delivers the above solution, provide training, and on-going support.

We would be glad to discuss this you in more depth. Please contact us

Frank