Test automation

Test automation is the use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting functions. Commonly, test automation involves automating a manual process already in place that uses a formalized testing process.

Over the past few years, tools with graphical user interfaces (GUI) that help programmers quickly create applications have dramatically improved programmer productivity. This has increased the pressure on testers, who are often perceived as bottlenecks to the delivery of software products. Testers are being asked to test more and more code in less and less time. Test automation is one way to do this, as manual testing is time consuming. As different versions of software are released, the new features will have to be tested manually time and again. But, now there are tools available that help the testers in the automation of the GUI which reduce the test time as well as the cost; other test automation tools support execution of performance tests.

Many test automation tools provide record and playback features that allow users to record interactively user actions and replay it back any number of times, comparing actual results to those expected. However, reliance on these features poses major reliability and maintainability problems. Most successful automators use a software engineering approach, and as such most serious test automation is undertaken by people with development experience.

A growing trend in software development is to use testing frameworks such as the xUnit frameworks (for example, JUnit and NUnit) which allow the code to conduct unit tests to determine whether various sections of the code are acting as expected under various circumstances. Test cases describe tests that need to be run on the program to verify that the program runs as expected. All three aspects of testing can be automated.

Another important aspect of test automation is the idea of partial test automation, or automating parts but not all of the software testing process. If, for example, an oracle cannot reasonably be created, or if fully automated tests would be too difficult to maintain, then a software tools engineer can instead create testing tools to help human testers perform their jobs more efficiently. Testing tools can help automate tasks such as product installation, test data creation, GUI interaction, problem detection (consider parsing or polling agents equipped with oracles), defect logging, etc., without necessarily automating tests in an end-to-end fashion.

Test automation is expensive and it is an addition, not a replacement, to manual testing. It can be made cost-effective in the longer term though, especially in regression testing. One way to generate test cases automatically is model-based testing where a model of the system is used for test case generation, but research continues into a variety of methodologies for doing so.

Along with test automation, automation of the entire testing process is also very important. Therefore, one must keep following points when thinking of test automation:

• Platform and OS independence
• Data driven capability (Input Data, Output Data, Meta Data)
• Customizable Reporting (DB Access, crystal reports)
• Email Notifications (Automated notification on failure or threshold levels)
• Easy debugging and logging
• Version control friendly – minimum or zero binary files
• Extensible & Customizable (Open APIs to be able to integrate with other tools)
• Common Driver (Ant or Maven)
• Headless execution for unattended runs (For integration with build process or batch runs)
• Support distributed execution environment (distributed test bed)
• Distributed application support (distributed SUT)

Framework approach in automation

A framework is an integrated system that sets the rules of Automation of a specific product. This system integrates the function libraries, test data sources, object details and various reusable modules. These components act as small building blocks which need to be assembled in a regular fashion to represent a business process. Thus, framework provides the basis of test automation and hence simplifying the automation effort.

There are various types of frameworks. They are categorized on the basis of the automation component they leverage. These are:
1. Data-driven testing
2. Modularity-driven testing
3. Keyword-driven testing
4. Hybrid testing
5. Model-based testing

Test automation of regression testing and maintainability

Regression testing – i.e., checking to see if any modifications to the software code have broken legacy features – is one of the most popular subjects of test automation. For each new release, black box testers create new test cases, so the number of legacy test cases that check legacy features has a snowball effect. Thus, it's natural to find ways to automate the execution of those legacy test cases. The main challenge here is not to write code for test automation, but to maintain that automation once it is written.

In his tutorial, How to Become a Software Tester, Roman Savenkov calls this maintenance the "test automation killer."Savenkov, Roman (2008). How to Become a Software Tester. Roman Savenkov Consulting. p. 268. ISBN 978-0-615-23372-7.

The problem is that application, especially in a start-up environment, tends to change very often (and in many cases, dramatically). Because test automation interacts with application, that test automation should be changed every time after a linked piece of application changes. For example, if a developer changes a button name from "Log in" to "Log on" and the test automation script was programmed to press the “Log in” button, then the test automation must be changed as well or execution of automated test case will be blocked.

Therefore, in many cases the question is not how to automate, but rather, "Does it make sense to automate?"

See also

• Software testing
• Test automation framework
• Unit test

References

• Elfriede Dustin, et al: Automated Software Testing. Addison Wesley, 1999, ISBN 0-20143-287-0

External links

• Generating Test Cases Automatically
• Easy way to automate testing