Site MapAccessibilityContact Us

PushToTest - Open Source Test

 E-mail | Print |

7. Governing SOA Through Test Automation

PushToTest TestMaker is architected to surface performance, interoperability and functional issues in service environments. It should be natural to expect that PushToTest runs as a service itself.

Governing SOA Through Test Automation


Governing SOA Through Test Automation

PushToTest TestMaker is architected to surface performance, interoperability and functional issues in service environments. It should be natural to expect that PushToTest runs as a service itself.

Governing Services Using Registry / Repository And Test Automation

Software developers, QA technicians and IT managers are learning how SOA governance and test automation tools benefit their organization's ability to develop, manage and deliver services to determined quality levels. Examples of these governance platforms include SoftwareAG's X-Registry , BEA AquaLogic Enterprise Repository and IONA Repository . PushToTest publishes an open-source test automation tool that when combined with the power of an SOA repository delivers SOA governance benefits. PushToTest TestMaker delivers an inter-operability mechanism between the PushToTest test automation run-time and a repository platform:

  • • Delivers immediate useful test automation functions to governance platform users.
  • • Provides metadata for services being governed, including service up / down status, time-since-last-failure data, service deployment correct validation status and service reliability history.
  • • Provides a mechanism to do service monitoring, scalability and performance tests managed by a governance platform.
  • • Provides a widely-adopted test API, test runtime environment and test development toolkit to governance platform users.

An SOA governance platform provides controls and functions enabling enterprises to manage a set of interdependent information services. From the developer's perspective, a governance platform gives a developer a way to discover a service that implements a particular function which therefore wont need to be written. In an SOA context, governance adds a registry, enabling developers to search for services, a database of metadata about the services' qualities and a set of policies to control the service's use at development and deployment time. For example, a policy attached to a service creating new employee email accounts requiring a developer to also use service authorizing requests. Inter-operability between a governance and test automation platform, an example policy says, "You must run a test script successfully to verify this service is correctly configured before deploying this service to a production environment". This inter-operability, the governance tool manages policies, stores the functional tests scripts, issues a call for PushToTest to run a test and saves the results as metadata for the service entry in the repository. The inter-operability provides a way to accomplish "test by policy" methods.

A Practical Example

Using a governance platform, software developers and IT managers can see management policies prescribing developers to use a particular service or set of services in the application. The governance platform shows developers and IT managers service dependencies allowing them to determine a service's reliability and build redundancy into the service to handle instances when an upstream service goes down. Software developers can do this today, without SOA or governance platforms. For example, one developer may tell another about a great US Postal Zip code look-up service for use in a product registration Web page enabling users to type in their zip code and have the page automatically fill-in the city and state fields. The same developer may warn he noticed the service throws errors when used on the weekends. This developer-to-developer exchange conveys the following knowledge:

  • • The existence and location of the Zip code look-up service
  • • The service was not reliable on the weekends, but worked fine on weekdays
  • • The service functioned according to its published schema

In an ideal world, the SOA governance platform conveys the same knowledge. For instance, a search showing a list of services and their metadata, including their up / down status.

Integration with PushToTest runtime provides knowledge about services, reliability and quality. In an SOA environment, PushToTest runtime drives a composite application using its native protocols to determine its functionality, scalability, performance and up-time. PushToTest is popular with software developers because it turns their existing unit and functional tests into SOA function, scalability, performance and service monitor tests automatically. Additionally, PushToTest is a distributed test automation platform where tests run on remote TestNodes on multiple networks and machines. IT managers, software developers and architects like PushToTest TestMaker because its’ tests provide knowledge feeds to governance tools, like:

  1. 1. Up-Time Feed to determine Service Level Agreement (SLA) compliance
  2. 2. Functioning Feed to discover service regression bugs
  3. 3. Reliability Feed to quantify service reliability over time

Integrating SoftwareAG X-Registry and PushToTest TestMaker

PushToTest and SoftwareAG took the PushToTest As A Service concept into reality. A proof of concept (POC) project for adding governance information to SoftwareAG’s X-Registry . In the POC, TestMaker monitors a service registered in X-Registry and sends that service's governance results statistics information to the X-Registry governance’s registry system, including current up / down service status, time since last problem and average response times. Within X-Registry, the statistics appear as the registered service's metadata.

Introduction

TestMaker 5.x and later have the capacity to work as a soap-based Web service known as PushToTest As A Service (PAAS). SoftwareAG owns webMethods, a company providing business process integration to the world's largest corporations and government agencies. SoftwareAG publishes X-Registry (formerly WebMethods and Infravio X-Registry). X-Registry is a commercial software product providing registry, repository and governance essentials for successful company-wide adoption of service-oriented architecture (SOA). The integration of TestMaker and X-Registry delivers necessary functions to X-Registry users needing to govern services, by enabling X-Registry to realize the current state of a service and other governance information by adding PAAS to the governance platform. PAAS runs tests to the Web Services registered in X-Registry and stores governance information about the quality of those services. To accomplish this, several architectural and functional issues needed to be resolved:

  • • Who will write the test?
  • • How will PAAS information be accessed?
  • • Who will update the governance repository’s information?
  • • How often must governance information be updated?
  • • How will the security issues be managed?
  • • Does the monitor-test only check the system or should it also update the repository information?
Source Code, Resources, Files

All the necessary code, files and resources for this integration may be downloaded for free at:

http://downloads.pushtotest.com/tm5/SendSMS.zip

Specification of the TestMaker / X-Registry Integration

This figure illustrates the implementation of the Proof of Concept:

The following steps demonstrate the integration.

0. User adds SendSMS to X-Registry

  • • Start XRegistry
  • • Start SendSMS
  • • Compile SendSMS project

To compile SendSMS, use ant jar in the SendSMS folder.

  • • In file SendSMS/dist/TOMCAT/bin run startup.sh or startup.cmd according your OS.
  • • Add SendSMS to X-Registry
  • • Log in to X-Registry system as a provider manager:
  • • Click Add New Service :
  • • Add SendSMS service information . In the POC the service is named SendSMS and the WSDL
    is:

http://localhost:8686/SendSMS/ws/SMS?wsdl

where localhost is the name of the system hosting the SendSMS service

  1. 1. Connect to X-Registry
  2. • Start PAAS using the command startup.sh or startup.cmd according to your OS. The files are located in the directory TRUNK/tm5/testmaker_distribution/TestNetwork/TestNode/bin .
  3. • Start the PAAS test using the command PAASTest.bat or PAASTest.sh according to your OS. The files are located in the folder TRUNK/SendSMS .
  4. • The program will connect to X-Registry
  5. 2. Connect to PAAS
  6. • The program PAASTest will connect to both systems using web services:
  1. 3. PAAS check SendSMS Status
  2. • PAAS system will periodically check the Service SendSMS by a user-specified time; for POC this time is 2 minutes.
  3. • If the SendSMS test works, the system will state there were no errors.
  • • If the service is not running or is not working as expected, PAAS will state the test failed with the reason for the failure
  1. 4. Get GSRS information from PAAS
  2. • The PAAS test will automatically get historical service information from PAAS
  3. • The service information is XML data known as GSRS (Governance Statistics Result Set)
  4. • The GSRS is known as PushToTest-Governance and the document PushToTest_Governance.pdf describes its specification and design
  5. • This is a sample GSRS for a correctly running service:


Fri Jun 22 10:51:51 CST 2007

up


17

30082

0

30673

26


http://examples.pushtotest.com/axis/services/MessageService?wsdl

http://examples.pushtotest.com/axis/services/MessageService?wsdl

http://examples.pushtotest.com/axis/services/MessageService?wsdl


38ADKJF83JF838DJQIEUR3

fcohen

Accepts PGP and OpenSSL certificates

This is a sample GSRS for a service with a problem:


Fri Jun 22 10:56:01 CST 2007

exceptions found


0

30076

0

30673

30

error

Fri Jun 22 10:54:43 CST 2007

An error

java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

...

at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:138)

... 20 more


http://examples.pushtotest.com/axis/services/MessageService?wsdl

http://examples.pushtotest.com/axis/services/MessageService?wsdl

http://examples.pushtotest.com/axis/services/MessageService?wsdl


38ADKJF83JF838DJQIEUR3

fcohen

Accepts PGP and OpenSSL certificates


  1. 5. Update SendSMS information
  2. • PAAS will send the GSRS to X-Registry. Currently it is posted in the description place, but being restricted to 255 characters is a problem. The future plan is to have the GSRS file added as a separate file and a new field added to indicate the service status.
  3. • This figure shows a POC display of a service running as expected:
  • • This figure shows a POC display of a service not running as expected:

The preceding steps demonstrated how TestMaker monitors a service registered in X-Registry and sends service governance results statistics information, to X-Registry governance registry system, such as current up / down service status, time since last problem and average response times as well as how X-registry stores and displays registered service information as metadata.

Further information can be found at:

  • • X-Registry description (Adobe Acrobat PDF format)
  • • WebMethods

The PAAS Interface

TestMaker is an automation framework with a distributed test environment to test services for functionality, scalability and performance. The TestMaker Console sends required information to specialized and optimized TestNodes (located on several computers), runs the tests and sends results back to the Console. The Console processes the results to create statistics files, charts and reports. A simplified workflow of TestMaker

TestMaker 5 enables the Console to receive soap-based Web service requests to operate a TestScenario. TestMaker has been upgraded to allow any application using a connector class to connect to the Console and run a test, with PushToTest As A Service. The upgraded workflow of TestMaker;

Three significant changes to TestMaker enabled the upgrade:

Modify the TestNodes to Allow Them to Operate Multiple Tests Concurrently.

In previous versions of the TestMaker TestNetwork architecture, TestNodes could receive instructions from only one Console. The new TestMaker 5 architecture allows TestNodes to receive instructions from multiple applications (Consoles), which has been checked for availability, stability and confidentiality.

  • • Availability means a scenario can access all of its own resources.
    To accomplish this, each scenario is given an unique ID in the form ID_ plus an ID number. Using the ID as its name, a directory (the application's root directory) is created and contains alll the resources of each test.
  • • Stability means any process can not affect any other process.
    To accomplish this, each monitor is run as an isolated thread with its own variable and ClassLoader, used to load the new required jar resources for a JAVA test.
  • • Confidentiality means any scenario can not access any other scenario's data.
    To accomplish this, each scenario only knows its own root directory.
Develop a Class that Becomes a Service
  • • The goal was to create a new class and use Apache Axis to turn it into a Web Service. This involved two tasks, compiling the new class and configuring tomcat to accept the new service.
  • • The new class needed to be simple, yet functional and complete as described in Classes.
Classes

Function Name

Parameter

Explanation

RunTest

String

aFilenames

Run the test and get its session ID.

NOTE: This is required for the other functions.

DataHandler

aDHFiles

First argument is the scenario’s required filenames.

Second argument is the datahandlers containing the files’ information.

This argument is the test scenario’s filename.

Fourth and firth arguments are the username and password used for authentication.

String sScript

String user

String pswd

Returns an integer representing the session ID.

Error if zero; system was not able to create scenario.

TestStatus

int id

String user

String pswd

Get the test’s status.


First argument is the session ID (obtained using the Runtest function).

Second and third arguments are the username and password for authentication.

Returns a single value indicating test status:

0 - Test operating normally.

1 - Test running and encountered one or more exceptional conditions.

2 - Test paused.

3 - Test ended.

4 - Error; current errors failed authentication, or session ID is not valid (iD<=0, ID for non-existent test).

GetSnapshot

int id

String user

String pswd

Get test’s most recent result.

First argument is the session ID (obtained using RunTest function).

Second and third arguments are the username and password for authentication.

Returns a GSRS formatted XML document with the test’s most recent result.

Error if NULL (ID is not valid or authetication failed).

PauseTest

int id

String user

String pswd

Pause the test.

First argument is the session ID (obtained using the RunTest function).

Second and third arguments are the username and password for authentication.

Returns true when test paused successfully (even if already paused).

Error if false (authentication failed or ID is not valid).

ResumeTest

int id

String user

String pswd

Resume a paused test.

First argument is the session ID (obtained using the Runtest function).

Second and third arguments are the username and password for authentication.

Returns true when test resumes successfully (even if already running).

Error if false (authentication failed or ID is not valid).

StopTest

int id

String user

String pswd

Stop a test.

First argument is the session ID (obtained using RunTest function).

Second and third arguments are the username and password for authentication.

Returns true when test stops successfully.

Error if false (test already stopped, authentication failed, ID is not valid).

Configuring Axis to accept the new class as a new service was surprisingly easy; simply add a new service tag to the Axis configuration file (server-config.wsdd)



The new service is located on the server at:

http://ServerName:8080/TestNetwork/ws/PAAS

The requests in the service WSDL definition are:








The responses in the service’s WSDL are:








Develop a Class to Connect to PAAS

This is an interface between the application and PAAS. It is available as a standalone library for clients to connect their applications to the PAAS Farms. Class Method describes the class methods:

Class Method

Class

Description

PAASClient(String ServerURL, String Username, String Password)

The class constructor; creates a new object for connecting to PAAS.

The username and password are stored for future calls. The ServerURL is usually: http//ServerName:8080/TestNetwork/ws/PAAS

public int doRunTest (String[]filenames, StringsScript)

Calls PAAS RunTest and returns the test status. Requires an array of files required by the scenario and the filename to be executed (only the name, not the full path).

Generates a java.lang.Exception if the Scenario fails.

public int doTestStatus ()

Calls PAAS TestStatus to get the test status.

public boolean

doPauseTest ()

Calls PAAS PauseTest and returns the test status.

public boolean

doResumeTest ()

Calls PAAS ResumeTest and returns the test status.

public boolean

doStopTest ()

Calls PAAS StopTest and returns the test status.
Example implementation

This code snippet shows a simple communication connection between an application and PAAS:


PAASClient client= null; // First we create the client

String [] files = new String[2]; // We add the required files

files[0] = "./SMSMonitor.xml"; // The Scenario

files[1]= "SendSMS.jar"; // The resources needed

String GSRSFile = "GSRS.xml"; // The file to save the GSRS


try


{

/* We create a client using the localhost as the server

* with pushtotest as the username and paas as the password

*/

client = new PAASClient(

"http://localhost:8080/TestNetwork/ws/PAAS",

"pushtotest",

"paas");


// We launch the test using the SMSMonitor.xml as the

// running scenario

client.doRunTest(files,"SMSMonitor.xml");


// If nothing goes wrong we inform of the success

System.out.println("Connection to PAAS Done");

}

catch (Exception e)

{

// Something goes wrong, we inform the situation and quit

e.printStackTrace();

System.out.println("Connection to PAAS Failed");

System.exit(0);

}


/*

* Every amount of time we read the current GSRS and show it in

* the screen

*/

try

{

while (true)

{

// Get the Governance file

client.doGetSnapshot(GSRSFile);

BufferedReader buff = new BufferedReader(new java.io.FileReader(src));

String line = buff.readLine();

while (line != null)

//Prints lines of the of the Governance

System.out.println(line);

try { Thread.sleep(10000); } catch (Exception e){}

}

}

catch (Exception e)

{

//Errors obtaining the GSRS, we show it and quit

System.out.println("Error adding the GSRS");

System.exit(0);


We found these resources helpful while implementing PAAS:

Microsoft Developer Network: Software as a service

Microsoft Developer Network: Architecture Strategies for Catching the Long Tail

Anderson, Chris: The Long Tail

PAAS Response

Each PAAS request to TestMaker generates a PAAS response:

PAAS Request / TestMaker Response

PAAS Request

TestMaker Response

RunTest

Returns the session ID number.

Request body is a TestScenario XML document.

TestStatus

Requires session ID value from the RunTest operation.

Returns a single value indicating the test status:

0 - Test operating normally.

1 - Test running and encountered one or more exceptional conditions.

2 - Test paused.

3 - Test ended.

4 - Problems (test does not exist, authentication failed).

GetSnapshot

Requires session ID from the Runtest operation.

Returns a GSRS formatted XML document containing the test’s most recent result.

PauseTest

Requires session ID from the RunTest operation.

Puts test in "pause" condition.

ResumeTest

Requires session ID from the RunTest operation.

Changes test condition from "pause" to"run".

StopTest

Requires session ID from the RunTest operation.

Puts test in "stop" condition.

The corresponding Web service result values are:

Web Service Result Values

Function Name

Response Value

RunTest

0 - Error; something went wrong.

>0 - Session ID for the scenario.

TestStatus

Test status value.

GetSnapshot

GSRS formatted XML.

NULL - Error (no such id. authentication failed).

PauseTest

True - Test has been paused.

False - Unable to stop the test (no such id. authentication failed).

ResumeTest

Trus - Test has been resumed.

False - Unable to resume test (no such id. authentication failed).

StopTest

True - Test has been stopped.

False - Unable to stop the test (no such id. authentication failed).

Governance Statistics Result Set (GSRS)

PushToTest executes a test and returns a Governance Statistics Results Set (GSRS) response document. The XML Schema Definition (XSD) for the GSRS document is found in testmaker_home/example_agents/GSRS.xsd

A sample GSRS response:


Fri Jun 22 10:51:51 CST 2007

up


17

30082

0

30673

26


http://examples.pushtotest.com/axis/services/MessageService?wsdl

http://examples.pushtotest.com/axis/services/MessageService?wsdl

http://examples.pushtotest.com/axis/services/MessageService?wsdl


38ADKJF83JF838DJQIEUR3

fcohen

Accepts PGP and OpenSSL certificates

Additional documentation, product downloads and updates are at www.PushToTest.com. While the PushToTest testMaker software is distributed under an open-source license, the documenation remains (c) 2008 PushToTest. All rights reserved. PushToTest is a trademark of the PushToTest company.