Lack of Standards Causes a Greater Need for Test Expert Services

I set the company name to PushToTest to show playfullness in the face of a daunting task. To be a tester today requires expertise in software development, software service interfaces and APIs, test scripting tools and languages, and operational test data creation and use. All of these keep changing - sorry innovating. The average CIO, CTO, V.P. Engineering, and Test Manager expects the same thing:

• They want to see TestMaker work flawlessly
• They want to see TestMaker work with their applications
• They want to see their testers and developers use TestMaker productively
• They want to see TestMaker integrate with their development systems seemlessly

TestMaker is not designed to test the entire Internet - only big portions of it! For example:

• For Web services (SOAP, REST, JMS, ESB) that have a well defined and managed standard interface, TestMaker works very well with customer applications. The W3C standards body governs the SOAP standards and Oracle's JCP governs the JMS standard.
  
• For Rich Internet Applications (Ajax, Flash, Flex, SOA) that have no standards body, this is the wild west of test scripting. TestMaker is an organization's best choice because of its rich support of Open Source Test tools. For example, if Selenium won't solve an Ajax problem then TestMaker also supports Sahi for Ajax testing, and Java for Ajax testing too.

For those times when TestMaker encounters compatibility issues with applications under test, PushToTest Global Services provides a paid Test Architect and Test Subject Matter Expert (SME) to solve compatibility issues.

At PushToTest we call this "well defined standards" vs "wild west" and it really cuts to the core issue. There is a lack of standards that causes a greater need for expert services. This is not that any particular test tool is difficult to use, it is that there is a "wild west" variety of possible implementations.

Todd Bradfute of PushToTest takes the metaphore to an extreme: So you can choose to live back east (SOAP/JMS) where the legislature passes laws and people obey them. But the taxes are higher (contracts and validation.) So some choose to live on the frontier (REST) ... taxes are lower, people can do more of their own thing ... but you have to rely on neighbors more for help (forums). Or you could take your wagon out West (AJAX, Flex, Flash, etc.) ... there's gold in them there hills (beautiful RIA presentations) people are ingenious (custom controls for textboxes) but sometimes they bend the laws (miss keystrokes on password entry, fail to respond to keyboard input.) Thus the town needs to hire a sheriff (QA Architect) to make the town work.



Learn What PushToTest Global Service Experts Do For Your Organization

Get along little doggie.

Frank

9 Top Web Service Security Attacks

Web Service security testing requires a thorough and orchestrated approach to mitigate business risk effectively. Web Services using SOAP or REST interfaces are vulnerable to many types of attacks:

1. Boundary Scans exploit bad handling of values that are outside of defined ranges
2. Cross Site Scripting enables functional changes in services and bypasses access controls
   
3. Parameter Fuzzy Values use unexpected values in service calls to access restricted functions
   
4. Malformed XML exploits bad handling of invalid XML in your service
5. Malicious Attachments exploit bad handling of attached files
6. SQL Injection exploits bad database integration coding
7. XML Bomb exploits bad handling of malicious XML requests
   
8. XPath Injection exploits bad XML processing inside your target service
9. Invalid Types tests handling of invalid input data

Each of these attacks can happen from multiple networks, data centers, and geographic regions. And, each attack can happen at any point in time. Continuous Web Service security testing is mandatory to reduce business risk. Read the soapUI tutorial to learn how.

PushToTest TestMaker with soapUI Pro provides a continuous end-to-end Web Service security testing platform for your SOAP and REST based services.
TestMaker integrates with Continuous Integration environments like Hudson, Jenkins, and Bamboo. These CI tools schedule security test operation to coincide with new software releases and timed schedules. TestMaker deploys the soapUI tests to one or more TestNodes operating around your network and the globe. They pummel the service under test with cross-site scripting attacks, malicious attachments, XPath injection attacks and all the rest.

soapUI Pro provides automatic generation of security suites. White-box testers write security tests and TestMaker deploys the tests to be data-driven and grid and cloud deployed.

Running security tests in parallel with stress and performance testing uncovers cases that neither stress testing nor security testing would reveal on their own. This frees the testers to think about what constitutes a failure and a mitigation.

Running a soapUI security test in a TestMaker Production Monitor enables continuous security testing. TestMaker logs the results to a historic results repository for on-going compliance with Service Level Agreements (SLA) and Security Agreements.




-Frank

Hi Seleniumistas!

I updated the Selenium Tutorial Screencasts for Beginners and Experts at:

http://www.pushtotest.com/selenium-webinar-series

These are free Webinar screencasts that cover the following topics:

• Selenium Basics
• Adam Goucher's Selenium, You Are Doing It Wrong
• Creating Robust Selenium Tests with PageObjects
• Advanced Selenium Testing
• David Burns on Selenium Testing Basics

Also, the Selenium Tutorial for Beginners starts at:
http://www.pushtotest.com/selenium-tutorial-for-beginners-toc

Thanks!

-Frank

TestMaker Object Designer vs. HP QuickTest Professional (QTP)

Last week PushToTest completed a Proof Of Concept (POC) project for a medium-size IT organization. The experience showed us the strengths and weaknesses of the TestMaker Object Designer when evaluated by a bunch of QA testers brought up on HP's QuickTest Professional (HP QTP.) Open Source Testing (OST) is a much better solution:

1. While QTP did a good job at getting testers productive, it has not done the QA tester community any favors. QTP has not kept up with the innovative ways software developers and IT Ops managers build and operate Rich Internet Applications (RIA, using Ajax, Flex, Flash, Applets, Silverlight, Oracle Forms.) QTP does not recognize the thousands of new Web objects in Ajax applications using GWT, JQuery, Yui, and Extjs.
   
   
2. QTP is too expensive for Agile where testers team with developers.
   
   
3. Only PushToTest TestMaker offers repurposing capability where a single test script runs as a functional test, load and performance test, and production service monitor. QTP scripts are incompatible with Load Runner.
   

Still, QTP is a familiar face. The testers we worked with had a lonesome, longing face when they considered moving from QTP to PushToTest TestMaker. The testers talk about QTP like it is some old college friend who got them into a fraternity. To those that resist change, we put together a screencast showing how to make the transition easy and what to expect.


-Frank Cohen, PushToTest

PushToTest Changes Company Name

Announcement date April 1, 2012

I've been doing a fair amount of thinking about the future of PushToTest. I'm glad to share some of this thinking today and a big announcement.

The company name has helped us over the past 10 years to identify our testing solutions as powerful, effective, and easy. But, perhaps it is time for a change. I've considered changing the company name to simply TestMaker. That makes it more understandable what we do. I did some research and found these new company names work too:

PushToTestTestMaker

PushToTestTestMakerMakesTestingEasier

PushToTestTestMakerMakesTestingEasierForRIAWebBPMSolutions

I say if you're going to expand you should go big. So, I've decided to rename the company to:

PushToTestAndMakeTestingEasierAndWhyWontHPBuyUsAsWeKeepTakingBusinessAway-FromThemOhYeaItsBecauseOpenSourceTestingRocks

Please let me know which of the above you like.

Thanks!

Frank
CEO and Founder
PushToTestAndMakeTestingEasierAndWhyWontHPBuyUsAsWeKeepTakingBusinessAway-FromThemOhYeaItsBecauseOpenSourceTestingRocks.com

Using OpenOffice To Query TestMaker Results

Michael Acosta-Madiedo, an engineer on the PushToTest TestMaker project, developed a technique to publish performance test results data using OpenOffice. His technique uses the TestMaker architecture where logged results data is available in a Results Repository - normally a copy of MySQL or Oracle.

This is the kind of work PushToTest Global Services does for customers everyday. Learn more by clicking here.

Special note: Click here to view a description of the TestMaker Results Repository schema.

The Problem

While TestMaker produces hundreds of results reports there are several reasons why business managers, testers, and IT Ops managers would query results outside of TestMaker. The query and reports are able to deliver custom reports and workaround limitations in the built-in reports. For example, one organization needed weighted average response times from test execution.

The Solution

OpenOffice provides access methods to work with data in a database. We will show how to get to TestMaker data in an OpenOffice spreadsheet. In this case we show how this works with MySQL.

1. Create a Mysql data source
2. Add the mysql-connector.jar to the OpenOffice classpath
3. Open an OpenOffice Spreadsheet
   
   Windows operating environment:
   a) Start OpenOffice.org
   b) Use Tools menu - Options - OpenOffice.org General Java.
   c) Click on the button Class Path..., then choose Add Archive... and navigate to where the mysql-connector-x.x.x-java-bin.jar file is located.
   d) Click on Open, and the file should then appear in the list of Assigned folders and Archives.
   e) You can then click on OK and close OpenOffice.org completely.
   
   Note: You may receive a message indicating that you have to do that anyway in order for OpenOffice.org to be able recognize your changes correctly. If the OpenOffice.org quickstarter is running, you should close that application as well.
   
   Mac OS X operating environment:
   a) Use Tools menu -> Options -> OpenOffice.org -> General -> Java is OpenOffice.org -> Preferences -> OpenOffice.org -> Java
   b) In either case, you can use the mysql-connector.jar in the TestMaker/TestNode/bundles/mysql/5.0 directory.
   c) If a connection file has already been created … Load it in the Registered
   
   
4. DatabaseOpenOffice.org->Preferences->OpenOffice.org Base->Databases
5. Click New
6. Select the odf file (example svn/testmaker-ent/ETM/cloud/<client>/<client>_db.odb)
7. Click OK
8. If a connection file hasn’t already been created … Use the Database Wizard to connect to your repository File->New->Database
9. Select “connect to to an existing database”, and “MySQL”, click Next
10. Use “Connect using JDBC”, click Next
11. Database name = “testmaker”, Server=server name where you have mysqld running, Port=3306, driver=com.mysql.jdbc.Driver
12. Click “Test class”, click Next
13. User name = testmaker, check Password required, click Test connection, enter password, click Next
14. Select “Yes, register the database for me”, click Finish
15. Save to a filename (example svn/testmaker-ent/ETM/cloud/<client>/<client>_db.odb)
16. Add a query for the data that interests you
17. Open the ‘Data Sources’ Window
    
    Windows: F4
    Mac: View->Data Sources
    
    
18. Right-Click on Queries and select “Edit Database File”
19. Click on Queries
20. Now use one of the tasks to create the query The easiest way is to have a SQL call that you want to use and select “Creat Query in SQL View” This opens an empty design window for you to paste your .sql into
21. Let’s start with one of the queries used by one of the TestMaker reports (see svn/testmaker/TM/tags/tm601/modules/charting/src/com/pushtotest/tm/charting/reports/jasper/StepDuration.jrxml … specifically the data in the <queryString> area. Also, click here for the repository schema.)
    
    SELECT
    stepdefinitions.NAME AS stepdefinitions_Name,
    steps.DURATION AS steps_Duration,
    CONTROLLERS.CONTROLLERNAME AS CONTROLLERS_CONTROLLERNAME
    FROM
    TRANSACTIONS transactions INNER JOIN STEPS steps ON transactions.TransactionID = steps.TransactionID
    INNER JOIN SEQUENCES sequences ON transactions.SequenceID = sequences.SequenceID
    INNER JOIN ITERATIONS iterations ON sequences.IterationID = iterations.IterationID
    INNER JOIN EXECUTIONS executions ON iterations.ExecutionID = executions.ExecutionID
    INNER JOIN TESTMAKER.CONTROLLERS CONTROLLERS ON iterations.CONTROLLERID = CONTROLLERS.CONTROLLERID
    INNER JOIN STEPDEFINITIONS stepdefinitions ON steps.StepDefinitionID = stepdefinitions.StepDefinitionID
    WHERE
    TRANSACTIONS.SUCCESS = 1 and executions.ExecutionID = :Enter_ExecutionID
    
    
22. Save with some memorable name (e.g. StepDuration)
23. Make sure to save your changes to the .odb file before exiting
24. Pull results into a spreadsheet
25. With the ‘Data Sources’ View visible, open the connection and Queries tree
26. Drag the Query you want down into your spreadsheet
27. If it takes parameters enter them now
28. The data will take a few moments but will then populate into the spreadsheet view

If you need to narrow the data you can either modify the query (adding AND to the WHERE clause) or you can use spreadsheet magic (e.g. =IF(A2==”Geo_LoadTestbyAddress”;B2;””) I tend to do this to get statistical information (e.g. =AVERAGE(D2:D1776))

Need Help With This Solution? Ask PushToTest Global Services

Orlando Free Workshop for Testers, Developers, IT Ops, Managers

Solve performance bottlenecks and function problems in your web applications. Learn the Open Source Test way to accomplish functional tests, load and performance tests, and service monitoring of Web applications, Rich Internet Applications (RIA, using Ajax, Flex, Flash, Oracle Forms, Applets,) and SOAP and REST Web services.

Orlando, Florida
Thursday, March 8th, 2012
3:00 pm to 4:30 pm

Register Now And We Will Send You Location Details

First Name *
Last Name *
Email *
Company *
Phone *
Job Title *

What We Have For You

• A free technical Workshop hosted by Frank Cohen, CEO and Founder at PushToTest, presented in a live meet-up format at a hotel conference center in Orlando, Florida, including answers to your questions
• A systematic, simple, Agile way to understand and manage effective tests of your application
• How To Record Selenium Tests In MS Internet Explorer and other Browsers
• Run Selenium, soapUI, Sahi, JUnit Tests As Load And Performance Tests
• Open Source Test software to build tests and deploy to desktop, grid, and cloud environments
• At a minimum you will see Selenium, Sahi, TestMaker, soapUI, HtmlUnit, JUnit, Flex, Oracle Forms testing demonstrated
• How To Run Tests from Continuous Integration (CI) and Test Management Environments
• A clear tutorial approach to the PushToTest methodology of building and repurposing tests to understand the correct functioning, performance, and scalability of your application
• An organized reference guide to the "best of the best" tips, techniques, patterns and antipatterns from PushToTest over the years, and how it all fits together

Agenda

• The Why, What, and How of Open-Source Test Tools vs. Proprietary Tools
• An Architecture To Use Open Source Test Tools Together
• Repurposing A Single Selenium or soapUI Test For Functional Tests, Load and Performance Tests, and Production Monitors
• Building Tests With A Component Architecture For Easy Maintenance
• Demonstrations of Open Source Test Tools and How They Hang Together
• Results Analysis To Surface The Root Cause of Performance Bottlenecks and Functional Issues
• Questions and Answers

See you in Orlando!

-Frank

Free Workshops, Webinars, Screencasts on Open Source Testing

Need to learn Selenium, soapUI or any of a dozen other Open Source Test (OST) tools? Join us for a free Webinar Workshop on OST. We just updated the calendar to include the following Workshops:

• Free Webinar: Solve Performance Bottlenecks and Function Problems In Your Web Applications
  February 22, 2012
• Open Source Test Workshop for Developers, Testers, IT Ops - Learn how the Open Source Test Tools Make Test Development and Operation Easy
  February 23, 2012 
• Open Source Test Workshop for CIOs, CTOs, Business Managers - Learn how to bring Open Source Test tools and methodology into your organization
  March 21, 2012
• Selenium, soapUI, Sahi, TestMaker Workshop for Testers, Developers, IT Ops
  March 22, 2012
• Agile Open Source Performance Test Workshop for CIOs, CTOs, Business Managers - Load and performance testing without hassle and cost
  March 28, 2012
• Agile Open Source Performance Test Workshop for Developers, Testers, IT Managers - The PushToTest Calibration Test Methodology explained and trained
  March 29, 2012
• Use Selenium, soapUI, Sahi, TestMaker Performance Testing In Your Organization
  April 17, 2012
• Agile Open Source Performance Test Workshop for Developers, Testers, IT Managers
  April 18, 2012
• Open Source Test Workshop for CIOs, CTOs, Business Managers
  May 2, 2012
• Selenium, soapUI, Sahi, TestMaker Workshop for Testers, Developers, IT Ops
  May 3, 2012

And If you are not available for the above Workshops, try watching a screencast recording.

Selenium Tutorial for Beginners

Selenium is an open source technology for automating browser-based applications. Selenium is easy to get started with for simple functional testing of a Web application. I can usually take a beginner with some light testing experience and teach them Selenium in a 2 day course. A few years ago I wrote a fast and easy tutorial Building Selenium Tests For Web Applications tutorial for beginners.


The Selenium Tutorial for Beginners has the following chapters:

• Selenium Tutorial 1: Write Your First Functional Selenium Test
• Selenium Tutorial 2: Write Your First Functional Selenium Test of an Ajax application
• Selenium Tutorial 3: Choosing between Selenium 1 and Selenium 2
• Selenium Tutorial 4: Install and Configure Selenium RC, Grid
• Selenium Tutorial 5: Use Record/Playback Tools Instead of Writing Test Code
• Selenium Tutorial 6: Repurpose Selenium Tests To Be Load and Performance Tests
• Selenium Tutorial 7: Repurpose Selenium Tests To Be Production Service Monitors
• Selenium Tutorial 8: Analyze the Selenium Test Logged Results To Identify Functional Issues and Performance Bottlenecks
• Selenium Tutorial 9: Debugging Selenium Tests
• Selenium Tutorial 10: Testing Flex/Flash Applications Using Selenium
• Selenium Tutorial 11: Using Selenium In Agile Software Development Methodology
• Selenium Tutorial 12: Run Selenium tests from HP Quality Center, HP Test Director, Hudson, Jenkins, Bamboo
• Selenium Tutorial 13: Alternative To Selenium

A community of supporting open source projects - including my own PushToTest TestMaker - enables you to apply your Selenium tests as functional tests for smoke testing, regression testing, and integration tests, load and performance tests, and production service monitors. These techniques and tools make it easy to run Selenium tests from test management platforms, including HP Quality Center, HP Test Director, Zephyr, TestLink, QMetry, from automated Continuous Integration (CI) tests, including Hudson, Jenkins, Cruise Control, and Bamboo.

I wrote a Selenium tutorial for beginners to make it easy to get started and take advantage of the advanced topics. Download TestMaker Community to get the Selenium tutorial for beginners and immediately build and run your first Selenium tests. It is entirely open source and free!

SOA Testing with Open Source Test Tools

PushToTest helps organizations with large scale Service Oriented Architecture (SOA) applications achieve high performance and functional service delivery. But, it does not happen at the end of SOA application development. Success with SOA at Best Buy requires an Agile approach to software development and testing, on-site coaching, test management, and great SOA oriented test tools.

Distributing the work of performance testing through an Agile epoc, story, and sprints reduces the testing effort overall and informs the organization's business managers on the service's performance. The biggest problem I see is keeping the testing transparent so that anyone - tester, developer, IT Ops, business manager, architect - follows a requirement down to the actual test results.

With the right tools, methodology, and coaching an organization gets the following:

• Process identification and re-engineering for Test Driven Development (TDD)
  
• Installation and configuration of a best-in-class SOA Test Orchestration Platform to enable rapid test development of re-usable test assets for functional testing, load and performance testing and production monitoring
• Integration with the organization's systems, including test management (for example, Rally and HP QC) and service asset management (for example, HP Systinet)
• Construction of the organization's end-to-end tests with a team of PushToTest Global Professional Services, using this system and training of the existing organization's testers, Subject Matter Experts, and Developers to build and operate tests
• On-going technical support

On-Site Coaching Leads To Certification

The key to high quality and reliable SOA service delivery is to practice an always-on management style. That requires on-site coaching. In a typical organization the coaches accomplish the following:

• Test architects and test developers work with the existing Testing Team members. They bring expert knowledge of the test tools. Most important is their knowledge of how to go from concept to test coding/scripting
  
• Technical coaching on test automation to ensure that team members follow defined management processes
  

Cumulatively this effort is referred to as "Certification". When the development team produces quality product as demonstrated by simple functional tests, then the partner QA teams take these projects and employ "best practice" test automation techniques. The resulting automated tests integrate with the requirements system (for example, Rally), the continuous integration system, and the governance systems (for example, HP Systinet.)

Agile, Test Management, and Roles in SOA

Agile software development process normally focuses first on functional testing - smoke tests, regression test, and integration tests. Agile applied to SOA service development deliverables support the overall vision and business model for the new software. At a minimum we should expect:

1. Product Owner defines User Stories
2. Test Developer defines Test Cases
3. Product team translates Test Cases into soapUI, TestMaker Designer, and Java project implementations
4. Test Developer wraps test cases into Test Scenarios and creates an easily accessible test record associated to the test management service
   
5. Any team member follows a User Story down into associated tests. From there they can view past results or execute tests again.
6. As tests execute the test management system creates "Test Execution Records" showing the test results
   

Learn how PushToTest improves your SOA software development life cycle. Click here to learn how.