Designing A Functional, Load, and Performance Testing Framework for Web Services

The lifecycle of any application typically involves four phases—requirements, design, development, and testing. Testing is the phase that guarantees value delivery to the customer. Testing, being an important process in the SDLC, is an area where better and efficient testing strategies meeting different requirements are sought after. In the new world of Web services, it becomes all the more important, as very few tools are available. Keeping the dynamic nature and overlap of technologies in which Web services work, we came up with an idea of a highly configurable, XML-based test harness for Web services. The XML test harness that we are proposing is a framework for functional and performance testing for a multitude of Web services.

XML Test Harness

The proposed test harness is a configurable XML-based framework; it can be used for the functional testing of Web services and to measure the performance of Web services at runtime. The test harness will be able to work with Web services that are jax-rpc or document exchange model (DEM)-based. The test harness will support performance testing, load testing, stress testing, and CHO (continuous hours of operation) testing for Web services.

The test harness is designed in such a way that it treats each ‘test’ as a task to be executed. Each task can accomplish different functions, depending on the configuration parameters it is supplied with. The configuration parameters for individual tasks are provided by the XML files. So, the test harness starts with the loading of configuration files, getting the tasks and its parameters out of configuration file, and finally running the task with the desired properties. The task in this process will invoke the Web service to be tested, get the results, and if the task is doing the performance testing, it will measure the execution time, also.

The harness will have options such as simulating multiple users using threads for loading a Web service. Simulation of multiple users will be useful for performance/stress testing to get real-time data. The harness also has an option of setting the number of iterations (and sleep time in between) for tests to average out the performance results and for performing CHO testing.

Depending on the type of Web service, either jax-rpc or DEM, the harness will use the appropriate XML file for configuration. The tester can provide the list of test groups (tests to be run in one group), or individual tests to be run for a particular Web service in another XML file. This tests list file will also have the pre- and post conditions to check the success or failure of the test. In either case, the test harness will report the successes and failures with a report manager. The report manager comes with an option of output to file, console, or both.

Architecture of the Web Services Test Harness Framework

As shown in Figure 1, the Web service Test harness has the following three subsystems:

• Configurator
• Runtime Engine
• Report generator

Click here for a larger image.

Figure 1.

Let’s understand the functionality of the individual components:

1. Configurator:
   This subsystem provides the much-wanted flexibility to the whole test harness. It processes the harness configuration XML file for the initial settings. Then, it loads the test list XML file and processes it to find out the test groups comprising individual test level information. The test level information contains the test name, style (jax-rpc, dem, or java invocation), input/output conditions, and number of threads for simulating multiple users.

   The Configurator also loads the style-specific (jax-rpc, dem, or java invocation) configurations, using their separate XML files. It also loads the configurations for the Report generation subsystem.

2. Runtime Engine:
   This subsystem forms the heart of the test harness. It basically acts as a controller that administers the other subsystems, starts the harness, initializes the whole system, manages the threads being spawned, and co-ordinates interactions necessary among all the subsystems.
3. Report Generator:
   This subsystem processes the test results, extracts the necessary information, and reports it to a comma-separated file (for loading as an MS Excel file) or console, or both, as per the configuration.

Future Directions

The test harness has some areas of improvements on which the future direction of the development will depend. We believe that the report manager should be enhanced for HTML-based reporting using XML outputs and applying style sheets to them. We are also thinking of enhancing the pre-conditions processing by use of JAXB in case of the SOAP-RPC style of Web services. Also, we are planning to extend the harness to be used for Java-based applications other than Web services.

Conclusion

This article gives a brief overview of the XML testing harness for Web services. The Framework we proposed and developed will be very useful for the projects that are working in Web services and finding it difficult to get a testing tool that can be suited for their comparatively new requirements. The current implementation is in its infancy and a lot needs to be done to make it available as a product. But, we are moving forward in the right direction and readers’ input would be greatly appreciated. As inventors of the above test harness, we are already in the process of filing a patent disclosure on the above invention.