Web Services Test Forum

The Web Services Test Forum (WSTF) provides a framework in which members of the Web Service community can develop interoperability scenarios and test implementations of those scenarios against other implementations. The WSTF does not charge dues and has no central governing authority (i.e. board). The WSTF was patterned by its initial creators (BEA Systems, Fujitsu, IBM, and Oracle) after the SoapBuilders mailing list/community. While its main focus is to test the various Web Service specifications, it also serves as a forum where the entire Web Service community can share ideas and concerns in an open fashion.

Principles

The WSTF is founded on the following basic principles:

Low barriers to participation

the WSTF seeks to be as inclusive as possible. Consequently it does not charge dues or any other form of membership fee. All that is required to join is have the individual or organization sign a Participation Agreement.

No centralized control

past experience has shown that, if a subset of members is allowed to control what is and isn't tested, they tend to try and direct testing efforts towards technologies and standards that they favor and discourage testing on technologies that, for whatever reason, they do not like. In the WSTF any member is free to propose a scenario, contribute to an existing scenario, or implement a scenario as they choose.

Interoperability by consensus

many interoperability issues arise either because the relevant specifications simply do not cover a particular area or there are conflicting, but equally valid interpretations of a specification. The WSTF seeks to resolve such cases using a common-sense, consensus approach.

Independent testing

any test that requires two or more people from separate organizations to participate at the same time is inherently more expensive and complex than a test which can be conducted by one person alone. The WSTF seeks to promote and support testing that does not require the active participation of all the parties involved in the test. For example, the WSTF site provides a directory of all the implementations of a given scenario. These implementations are expected to be long-lived and remain up and working for the benefit of other WSTF members.

Distribute testing costs

the WSTF provides a shared testbed where the cost of testing is spread out among the members by having each implementation host their own endpoints. This removes the need for each to duplicate this effort themselves in-house.

Scenarios

Unlike other interoperability organizations, the work of the WSTF does not center around individual specifications. Activities are organized around the concept of a "Test Scenario". Scenarios are made up of three parts:

1. A high-level use case that describes the problem to be solved and the constraints on that solution.
2. An architecture that describes the services technologies and standards that will be used to address the problem and how they will be used.
3. A set of test cases along with the artifacts (WSDL, XML Schema, etc.) necessary to implement those test cases.

Process

Once a scenario has been defined, members of the WSTF may implement it using their products or open source projects. They deploy these implementations onto publicly available systems and test interoperability with each other in a crosswise fashion. Problems and issues are discussed on the WSTF mailing lists. The scenario may need to be clarified or re-factored during this process. Once an implementation reaches a certain level of maturity, and the implementers choose to do so, the scenario and its implementations can be made visible outside the WSTF by publishing it. Whether published or not, the endpoints that provide the scenario implementations are expected to be maintained indefinitely. This allows other members of the WSTF to perform regression testing, test new implementations, verify behavior, etc. without requiring the active participation of the implementer.

The WSTF chose to keep most of their work private for a couple of reasons. First, in an entirely public forum the members may not feel as free to bring up sensitive topics. By signing the Participation Agreement, members agree to keep all discussions private to the group - thus allowing for a much more open and honest discussion. Second, the members of the WSTF wanted a "WSTF Published" scenario to mean that it had broad community support. Without a formal "Publish" step in the process it would be hard distinguish scenarios that had the community's support versus ones that were only implemented by one company.

A scenario can be published when it has five different implementations and at least two-thirds of those implementor choose to make it public. Only members of the WSTF who put up implementations/endpoints of the scenario are eligible to vote. This restriction was done to ensure that only those who "have skin in the game" are allowed to influence it. For more information on the process see the WSTF's Charter.

Regression testing of web service

Functional and non-functional web service testing is done with the help of WSDL parsing and regression testing is performed by identifying the changes made thereafter. Web service regression testing needs can be categories into three different ways, namely, changes in WSDL, changes in code, and selective re-testing of web service operations. To capture above three changes three intermediate forms of WSDL, namely, Difference WSDL (DWSDL), Unit WSDL (UWSDL), and Reduced WSDL (RWSDL), respectively can be used. These intermediate forms of WSDLs are then combined to form Combined WSDL (CWSDL) which is further used for regression testing of the web service. This will help in Automatic Web Service Change Management (AWSCM), by performing the selection of the relevant test cases to construct reduced test suite from the old test suite.

Results

The WSTF produces the following artifacts:

• The WSTF provides a directory of the endpoints that implement a scenario. The fact that these endpoints are long-lived means that you can re-check the interoperability results at any time. You also can create your own implementation of a scenario and check what you have done against any of the existing endpoints.
• Every scenario includes a set of "findings" for the scenario. These are notes that discuss what was discovered during testing including problems in the underlying specifications, possible work-arounds for problems, etc.
• A set of guidelines or best practices on how to address the business case using web services technologies.