The Cohen Blog

CMC Media produced a set of interviews at the STARWest testing conference. They asked a dozen thought leaders in the testing space:

"Is agile affecting testing?"

Watch the video interviews at: 

I am part of the interviews. Here is my opinion:

"Seems like 2008 was the year that a lot of people came to the realization that using waterfall techniques to build software incrementally - where you went from requirements into development and then into testing - is pretty much an easy way to kill your developers and testers. It is just to hard to in an Ajax environment or an SOA environment, to use waterfall. 

Agile, on the other hand, says "Let's test first! Let's build the interfaces to the application first." That's great. Agile also brings peer development and a bunch of other methodologies.

I am seeing testing move up to leverage Agile techniques as well. When you are building new applications and services using some of the new frameworks there is no reason you can't build the test at the same time you build the application. That is beyond what Agile itself originally said. Agile wants you to do test first. I say build and test at the same time using Agile techniques."

-Frank

PushToTest introduced Cloud Testing in TestMaker last year. The results have been outstanding. In 2009 we saved our customers $650,000 in fees that they would have otherwise spent on building out their own QA labs. With this enabling TestMaker technology, customers run their automated tests using continuous integration servers like Bamboo and Hudson.

• Ubuntu Enterprise Cloud combines Ubuntu Linux with Eucalyptus and other cloud management tools. Eucalyptus joined the world of commercial open source with the launch of Eucalyptus Systems in 2009. Eucalyptus incorporates the Apache Axis2 Web services engine, Mule enterprise service bus, Rampart security, and Libvirt virtualization. Eucalyptus comes with its own implementation of the Amazon API.
  
• The Red Hat Deltacloud Portal manages all of the cloud deployments that may exist within an environment, providing a common integration platform. Deltacloud helps ease the integration between public and private clouds. Deltacloud creates a REST-based API to map to Amazon EC2 and private clouds that use VMware or Red Hat Enterprise Linux.
  
• Nimbus is an open source toolkit that turns clusters into a cloud environment. Nimbus has a suprisingly complete set of APIs and management systems to provision customized compute nodes.
  
• PHP developers have Simple Cloud API from Zend Technologies to manage cloud services from GoGrid, IBM, Microsoft, Nirvanix Storage Delivery Network, and Rackspace Files. Java developers have Typica to access Amazon's SQS, EC2, SimpleDB and DevPay LS web services. We use Typica in TestMaker with great results.
  
• PushToTest TestMaker repurposes Selenium, soapUI, and Java, Ruby, Perl, PHP unit tests as functional tests, load and performance tests, and business service monitors. TestMaker operates tests from a private cloud, public cloud, or both.

Frank

Download the Press Release (Adobe Acrobat PDF format, 81 Kbytes)

CAMPBELL, Calif. and BRISBANE, Calif., Jan. 21, 2010 —  PushToTest® and CollabNet® announced today an expansion of their business relationship to deliver test capability in the cloud – backed by the latest releases of CollabNet TeamForge® and PushToTest TestMaker™ for enterprise clients, government organizations, and open communities worldwide.

TestMaker, the open source alternative for test automation, has been added to the CollabNet collabXchange™ marketplace.  CollabNet, the leader in distributed application lifecycle management (ALM) solutions, and PushToTest have integrated TestMaker with the CollabNet platform; organizations can reduce unplanned service interruptions, increase developer and tester productivity, and improve the ability to achieve service level agreements (SLA). 

“We designed PushToTest TestMaker to leverage Cloud Computing. CollabNet TeamForge enables us to deliver huge price and feature advantages to our customers,” said Frank Cohen, chief executive officer, PushToTest, and world-renowned Test Geek. “With TeamForge Lab Management support built into TestMaker 5.3, users click the start button, watch TestMaker create load injector machine instances in their own QA lab or a cloud provider such as Amazon Web Services (EC2), run the test, and then take the machines down. This saves customers huge amounts of budget, relieves them of running their own QA lab, enables tests with up to millions of simulated virtual users, and runs tests from multiple geographic locations – and all on-demand.”

For organizations needing to surface and solve performance bottlenecks and functional problems, TestMaker provides a single tool to verify advanced Rich Internet Applications (RIA), Service Oriented Architecture (SOA,) and Business Process Management (BPM) applications.  

PushToTest supplies the complete test automation platform to deliver functional testing, load testing, and business service monitoring by repurposing the same single test script. CollabNet’s TeamForge Lab Management and CollabNet Subversion reduce time spent on configuring build and test environments and securely managing component reuse libraries or test assets in a centralized repository.

“CollabNet has built the collabXchange to best serve our customer base with options to extend and expand their CollabNet platform,” said Jim Ensell, vice president, business development, CollabNet. “Together, TestMaker and CollabNet TeamForge Lab Management provide a compelling solution for test automation in the cloud.”

About PushToTest

PushToTest is the open source application performance management and test automation solutions provider. PushToTest is paving the open source test (OST) solutions migration into the mainstream by making high-quality, low-cost technology accessible to medium and large size global organizations and by providing world class professional services. PushToTest provides a software testing orchestration platform along with cloud testing capabilities and outsourced testing solutions, as well as support, training, and consulting services to customers worldwide and through top-tier partnerships. PushToTest's open source strategy offers customers a long-term plan for building testing infrastructures that are based on and leverage open source technologies with a focus on rapid adoption of the latest technologies, reuse of test asset investments, and ease of management. For more information, visit www.pushtotest.com or send email info@pushtotest.com

About CollabNet

CollabNet is the leader in application lifecycle management (ALM) platforms for distributed software development teams.  CollabNet TeamForge is the industry’s most open ALM platform, supporting every environment, methodology, and technology.  With an integrated suite of easy-to-use tools that share a centralized repository, it is the only ALM platform that enables a culture of collaboration, improving productivity 10-50% and reducing the cost of software development by up to 80%.  As the corporate sponsor of the open source Subversion project, the best version control and software configuration management (SCM ) solution for distributed teams, collaborative development is in CollabNet’s DNA.  Millions of users at more than 900 organizations, including Applied Biosystems, Capgemini, Deutsche Bank, Reuters, and the U.S. Department of Defense, have transformed the way they develop software with CollabNet.  For more information, visit www.collab.net or call (888) 778-9793.

Media Contact:

Paige Schoknecht

Prequent, Inc. (for CollabNet)

+1 (408) 275-1419 office

+1 (650) 223-4085 mobile

paige@prequent.com

Frank Cohen

PushToTest

+1 (408) 871-0122

info@pushtotest.com

TestMaker 5.4 is a major feature enhancement release. The new software includes major feature improvements in the following areas: 

• Easier and More Productive User Interface - The new TestMaker Editor is a fun and easy graphical tool to build and operate tests.
• Continuous Integration Features - Compatibility with Hudson, Bamboo, Collabnet TeamForge, Ant, JUnit, and other continuous integration environments
• Updated Tutorials - 30-minute tutorials on building tests of Ajax applications using Selenium, building tests of SOAP-based services using Selenium, and building tests of Flex-based services using TestMaker AMF protocols.
• Improvements to Selenium IDE - We began to make changes to Selenium IDE to improve its support for Ajax-based Web applications.
• Bug Fixes - Dozens of small fixes and improvements.

The project includes contributions from dozens of engineers in the TestMaker open-source community and PushToTest funded engineering effort by Andrew Zuercher, Ali Faiz, William Martinez Pomares, Luis Carlos Lara Lopez, and Peter Schumacher.

We are planning to release TestMaker 5.4 next week. We would appreciate your feedback now. Please try one of the download links for TestMaker Enterprise. 

For Macintosh users:

http://tm53rc4.s3.amazonaws.com/PushToTest_TestMaker_Install_Mac.zip

For Windows users:

http://tm53rc4.s3.amazonaws.com/PushToTest_TestMaker_Install_Windows.zip

For Linux/Unix users:

http://tm53rc4.s3.amazonaws.com/PushToTest_TestMaker_Install_Linux.tar.gz

Barring no show-stoppers, this is the final release candidate and the final 5.4 software.

Please let me know any issues you run into by posting a message to the support forums. 

Thank you.

-Frank

The Open Source Test Workshop is a free meet-up for people and organizations needing to test information systems for functionality, load and performance, and service delivery. PushToTest hosts the meet-ups. One recent student to the Workshop said this afterwards:

PushToTest recently announced the dates for the Workshop in January and February via live Webinar.

PushToTest hosts two forms of the Open Source Test Workshop:

We hope to see you at the Workshop!

-Frank

PushToTest Global Services provides our customers with test management and business performance management solutions that include our TestMaker solution. We do so by integrating TestMaker into a test management stack that includes the following technology:

• Test case\suite management - PushToTest partners with Collabnet TeamForge and other open source packages to provide test management, bug tracking, and revision control for tests
• Test data management, (parsing , sanitisation & storage) - TestMaker Data Production Library (DPL) technology data enables tests from RDBMS solutions like MySQL. We train your team to build tests using DBUnit for data generation and replication.
• Security test case management - TestMaker supports security testing of X509, SSL, SAML, and application level security in test suites.
• Automated regression with command interpreter - TestMaker repurposes unit tests to be automated regression tests, load and performance tests, and business service monitors from one test.
• Integrated defect management - TestMaker provides integrated defect management through Collabnet TeamForge, Trac, and other ticket system.
• Test result storage and reporting - Fully automation storage and results reporting.
• Key Performance Indicator (KPI) reporting - Provided as part of PushToTest's reporting capability.

PushToTest delivers the above solution, provide training, and on-going support.

Frank

Many testers, developers and IT managers ask PushToTest to compare HP QuickTest Professional (QTP) to PushToTest TestMaker with Selenium IDE. PushToTest provides open source test automation solutions for organizations needing to improve the reliability and reduce service interruptions in their Web applications, Rich Internet Applications (RIA, using Ajax, Flex, Flash,) Web services, and Business Process Management services. PushToTest is the Open Source Test (OST) expert for dozens of tools. This comparison document helps organizations looking for an inexpensive open source solution that works with their latest applications.

We are asked "How's your business running?" frequently. PushToTest is a software publishing company. Most of our business is a set of applications that run in our datacenter. They should ask, "How are my application services running?"

Answering that question is no simple task. We have applications that use Web services, Service Oriented Architecture (SOA,) Rich Internet Applications (RIA, using Ajax, Flex, Flash,) and Business Process Management technology. Most of the applications are stateful. For example, to know if an application is up and running we need to log-in, navigate a few pages, and enter values. The resulting test use case requires operational data (the log-in, for example, is a real log-in with real id and password.) And validating the results requires us to assert the correct values are present. Plus, we want to see the results in a set of actionable charts that update over time.

When we think of what PushToTest goes through to ensure its business is running, it makes us wonder how the average medium to large organization handles these requirements? This is one of the main drivers for organizations to benefit from PushToTest services. We deliver solutions to:

• Reduce the risk of service interruptions.
• Mitigate the expense of running a QA lab.
• Advise with our expert insight on the latest initiatives, including Cloud Computing, Application Monitoring, Application Development Using Ajax, and Continuous Integration.

IN THE BRAIN OF OLE LENSMAR, AGILE TESTING OF REST SERVICES WITH SOAPUI

Perhaps the top most requested new feature from TestMaker users is an easier way to create and manage TestScenarios. A TestScenario is the orchestration instruction to run tests in the TestMaker distributed test environment. TestScenarios are XML formatted documents. They contain details on which TestNodes will run the test and define use cases by identifying which tests (Selenium, Windmill, soapUI, unit tests) to run. Unfortunately, TestMaker users hand-edit the XML file. This is at times cumberson and error prone.

We are building a new Forms Editor for TestMaker 5.4 This enhancement adds a GUI-based forms editor for TestMaker TestScenarios. We encourage you to preview and comment on the new design.

The resulting project is available for download at:

We implemented the Editor using JQuery, Titanium, Extjs and custom JavaScript code. The source code is at:

A screencast on how we built the Editor is at:

We ran several usability labs for the Editor already. View a PDF for the design of the changes at:

We hope you enjoy the new features! 

Some organizations are choosing a very risky strategy. They are putting load and performance testing of their mission critical applications on hold for now because of their budget. They risk service delivery outages, service crashes, and slowly performing service functions. PushToTest has a better idea.

Put PushToTest services and solutions to work for you in 2009 and pay for TestMaker in your 2010 budget. 

license good for the rest of 2009.

OR 

B) Sign-up for a PushToTest Turnkey Outsourced Load Test and receive a TestMaker Enterprise license good for the rest of 2009.

The TestMaker Enterprise solution comes with the following: 

• Load Testing Solution (up to 1000 virtual users)
• Run The Tests On PushToTest OnDemand Servers (up to 15 hours)
• Receive Training and Support (Instructor led live Webinar)
• Our Experts Build and Operate Your First Test (up to 40 hours)

Continue To Benefit From PushToTest TestMaker and Support in 2010 for a guaranteed 2009 price. 

Contact PushToTest today:

http://www.pushtotest.com/about/requestforinformation 

The offer expires on October 15, 2009.

Details on TestMaker Enterprise at:

The Open Source Test Workshop is a great place to meet your peers and learn the best practices to test your application. The Workshop is the ideal meet-up for QA testers, software developers, architects, and IT managers. We are all being asked to do more, faster, with less resources. Is there a better time to get together to ask questions and learn the best way to bring together Open Source Test tools and methodology?

The Workshop is the one place where you can learn how to use tools like Selenium, soapUI, HTMLUnit, Windmill, Mozmill, JUnit, and TestMaker. Join us at the next Workshop events: 

and 

Comments from past Workshop attendees:

A little while ago a TestMaker user in our open source community wondered when it makes sense to move up to a paid subscription to TestMaker Enterprise.

TestMaker is ideal for load testing Rich Internet Applications (RIA) using Ajax, DWR, Spring, Hibernate, OCI/OCJ, and Oracle database technology. Our SeleniumHtmlUnit technology is unsurpassed at repurposing Selenium test scripts to be load and performance tests and business service monitors. 

We are a professional open source software company. Your organization greatly reduces business risk, avoids unscheduled down-time, and increases customer satisfaction when using our solutions in the wild RIA environment. We participate in many of the RIA open source projects and can authoritatively identify best practices for using our solutions.

When our customers encounter a technology problem we immediate apply our engineers to deliver a solution. Most of our support organization is made up of engineers working on the core of TestMaker. That is the nature of our business and that is why we offer the TestMaker Enterprise product, with its professional support, training, and consulting services.

For example, one of our customers needed a SeleniumHtmlUnit method implemented. We delivered the implemented method in 2 days.

While we encourage everyone's participation in the open source test (OST) community, I am wondering if you have budget and an interest in buying TestMaker Enterprise? If so, what is your evaluation process to arrive at a decision?

Let us know your answers and we will propose a solution for your organization.

-Frank

PushToTest produced a new DZone RefCard "Getting Started With Selenium" is now available for free download. The RefCard will save you hours of time learning Selenium and show you how to save your organization time and risk through test automation.

PushToTest achieved a milestone today: We ran a test from 30,000 feet!

PushToTest did a load test for a customer. The customer was running a fidelity program where teenagers played a Flash-based game. Every 15 minutes the service awards gifts to people playing the game. The Flash component checks the backend system every 15 minutes for the new gifts. The customer implemented no time-out mechanism.

Know what happens next?

The teenagers leave the Web page open, even when they are sleeping. The Flash components turn into a distributed denial of service attach engine. The site operations managers notice that every 15 minutes the server load increases, and it never reduces.

It was really interesting and fun to surface this problem and offer a mitigation.

PushToTest is a media sponsor to QCon, we wanted you to know about the third annual San Francisco conference (November 18-21, 2009) for enterprise software development people. QConSF features 80 technical sessions (Nov 18-20) and 2 days of tutorials (Nov 16-17). 

We have arranged a special discount for PushToTest subscribers of an additional $50 off when you use the discount code "pushtotes_qconsf09" when you register for the conference.

$600 off to PushToTest users discount code: pushtotes_qconsf09 before August 28, 2009

While most of the other software development conferences have been cancelled, QConSF attendance is 200% higher than it was at the same time last year (which was even before the economic crisis was announced). The venue is the same small and cozy Westin on Market Street, placing limits on the number of attendees. 

The last ten years have seen an intense amount of energy devoted to how programs and data relate. This keynote at QConSF in November by WCF Architect & MS Distinguished Engineer Don Box will talk about the current state of the practice and how we arrived here, as well as identifying the next set of challenges ahead.

The track themes for QCon San Francisco & track hosts are as follows: 

• Technical Skills for Agile Development
• The Cloud: Platform or Utility
• Architectures you've always wondered about
• Agility as a Craft
• Architecture for the Architect
• Browser as a Platform
• Cool Stuff with Java
• DSL in Practice
• Emerging Languages
• Kanban/ Lean Software Development
• Soa: Hits and Misses
• The Enterprise-IT Ecology

For a taste of the types of sessions at QCon, check out the following talks:

Software Architecture for Cloud Applications

While it is possible to transplant a traditionally-structured application onto a cloud platform, that isn't the best way to go. Release it! author Michael Nygard will discuss ways that running on a cloud computing infrastructure should affect your software architecture.

Clean Code: Craftsmanship and Ethics

Uncle Bob Martin of Agile, OO, UML fame outlines the practices used by software craftsmen to maintain their professional ethics. He resolves the dilemma of speed vs. quality, and mess vs schedule. He provides a set of principles and simple Dos and Donts for teams who want to be counted as professional craftsmen.

Open Source at Unibet.com - 10x Scalability at Half the Cost

With over four million registered customers in more than 100 countries, Unibet is one of Europe's largest online gaming operators. In this session you will learn about Unibet's continuously ongoing rearchitecture initiatives at a deeply technical level.

Lessons Learned from Architecture Reviews

Rebecca Wirfs-Brock of OO and Agile fame presents lessons learned from preparing for, presenting, and conducting architecture reviews. The software architecture review is one tool that helps reveal architectural risks and strengths as well as uncover unidentified issues that need addressing.