Automated Testing Framework for Automation Success

By Steve Miller

This article discusses the key ingredients of an automated testing framework that can lead to test automation success.

Why Automate Test Cases?
Even though many companies run their regression test cases manually, it may make more sense to automate regression test cases. It makes sense to automate test cases when engineers can no longer run the regression test cases on each build created. For example, if developers are doing daily or weekly builds of the code for quality assurance and they cannot quickly run the regression test cases with each build, then it is time to consider automating them.

Benefits of Automating Test Cases

  • Quicker Releases
    By having regression test cases run automatically, software quality team can concentrate on testing new features of software and less time regressing existing features

  • Higher Quality Releases
    Software releases will have fewer bugs and require less customer support because they will be of higher quality

  • Happier Customers
    Customers will be happier and more willing to serve as testimonials for future prospects

What is an Automated Testing Framework?
Most automated tools require the test engineer to understand a scripting language (VB Script, Java Script, etc.) to write their automated test cases. These tools usually have the ability to create the scripts using record and playback, but this does not always write the most efficient scripting code and is not as re-usable and maintainable. An Automated Testing Framework is a set of assumptions, concepts, and practices that provide support for automated software testing, enable the re-use of automated test cases, reduce maintenance, and enhance maintainability.

Anatomy of a Successful Automated Test Framework
When designing Automated Testing Framework, consider these features, as they can dramatically improve success and reduce the chance of abandonment:

  • Keyword Driven Testing
    If test engineers are not experts at scripting languages, consider replacing automated scripts with keyword driven testing. This approach enables a tester (or even a subject matter expert) to create automated tests by describing each step of the automation. For example, if engineers are automating the login process of the application, the user will access the application, type in the user-id and password and press a button to login. Traditionally, testers would do this by writing VB Script that will navigate to the application, identify each object on the screen (user-id, password and login button), then write script to enter in the user-id, password and to press the login button. With keyword driven testing, the tester does not need to understand the scripting language to make this happen, they can simply describe the event (navigate to the application, enter in “abc” for the user-id, enter in “xxx” for the password, press the Login button when done). This is a much simpler approach to automated testing than scripting.

  • Allow Data Iterations
    It is good practice to allow testers to re-use automated test cases, but allow them to run them with different sets of data. For example, a company might use the same automated test case for logging into the software, but allow the test case to be run multiple times with different user-id and password combinations to test different scenarios. If the automated testing framework is flexible enough to enable testers to define different data iterations, it will reduce time spent creating automated test cases.

  • Allow Running on Multiple Hosts
    When designing the automated testing framework, consider multiple hosts. For example, as quality assurance demands grow, a company may need to setup a QA Lab with multiple servers. Once that is done, testers will need a way to launch the automated test cases on multiple servers, so consider building that into the automation framework.

  • Allow Scheduling of Automation Runs
    Once the automated test cases have been developed, companies will want to schedule them so that they can run unattended. When designing the automation framework, build in the capabilities to schedule automation runs at different intervals (days of the week), allow recurrence (run every day at 6 pm, etc.), and allow them to be scheduled on different hosts.

  • Reporting Run Results
    When developing the automation framework, consider how to report on the run activity of the automation effort. If engineers are using an existing test management solution for their manual test effort, consider integrating the automated test result data in with the reporting of the test management solution, that way companies can analyze the manual and automated test effort in a consolidated fashion.

Automated Testing Framework Resources

Steve Miller is the President of Pragmatic Software