Device Testing Slows Product Innovation

By Tom Ryan, CEO, The Fanfare Group

Constant product innovation has contributed to a thriving equipment-manufacturing industry over the last two decades. The enabler for innovation has been a steady stream of new tools and technologies that help software developers work faster and smarter. Developers have been empowered to be more productive, but QA groups have not been given the tools they need to keep up. No matter how quickly features are developed, products cannot be released before they are tested. As a result, testing is quickly becoming a barrier to innovation, threatening the pace of feature advancements.

Software developers have clearly benefited from productivity-enhancing tools and technologies. It is the testers who need 21st century tools to stay in parity. Sadly, the last notable advancements in testing were — Perl, Tcl, and automated GUI testing from that appeared in the ’80s and ’90s. Another strategy that emerged in the late ’90s was to off shore testing to organizations overseas. However, because labor-based solutions can only scale linearly, a wide gap has developed between what can be developed and tested. Most companies that outsource find an initial period of relief, and then discover that testing has once again surfaced as a barrier stalling the ability to innovate.

Today’s testers need new tools to empower them in several ways. Tools for earlier defect detection will help reduce costs and prevent product delays. Automating the testing process-without making it longer or more complex-will save time and increase coverage. Facilitating communication among developers, testers, and the automation team, even if they work on distributed global teams and speak different languages, will save even more time.

Earlier defect detection
Finding bugs when the equipment is already in the field is very costly, increasing the time needed to resolve the problem and eroding consumer confidence. In general, the later in the product release cycle that a bug is detected, the higher the remediation costs. If the QA group discovers a bug just one week before scheduled product release, for example, incremental costs can include overtime for testers and developers and lost revenues from delayed product introduction. The value of early defect detection is early defect correction. There are currently test automation tools available that enable QA to detect defects early on and easily reproduce the defect, and use that same mechanism to ensure the defect is fixed.

The value of automating certain aspects of tests is well accepted in the equipment industry, and many manufacturers have begun developing their own automation tools. However, tool development is not a core competency of QA groups, and therefore can result in low adoption rates. Home-grown solutions tend to be time-consuming, difficult to use and error-prone, which discourages testers from using them. In addition, executives express frustration at the time and effort the team spends building and maintaining in-house automation tools as compared to actually using them. Testers need automation tools that make their jobs easier and faster. Well-executed test automation tools are now available that will save testers hours and days of time-especially for repetitive, mundane tasks for which a human tester does not add value such as:

  • Configuring a testbed into a certain state
  • Repeating a test on a new software build
  • Repeating a test on different equipment or devices
  • Executing another test that is similar to one already performed-for example, repeating the same batch of tests for each port on a network device
  • Searching through output logs to identify root causes of failure

New breeds of automation tools are also helping to increase test coverage. Types of coverage that are feasible with automation, but not with manual testing include:

  • Testing the boundary conditions for a feature by testing over all possible combinations
  • Stressing a system by repeating a test or set of tests many times
  • Loading a system by performing numerous similar tests at the same time

But, at what point in the product development cycle is it best to add automation? The conventional wisdom has been to wait to perform any testing-manual or automated-until the rate of change slows. However, this no longer makes sense because product changes occur frequently throughout the development stage and can continue well into the QA cycle. Therefore, an organization that waits to introduce automation until the rate of change slows cannot start until the product is nearing release. By then, the benefits of automation are nominal.

More effective communication between testers, developers, and scripters
In the typical workflow, developers, testers, and customers communicate using multiple types of documents in different formats. Developers struggle to understand obscure bug reports submitted by testers, often finding that test cases do not properly document the state of the device and the steps required to surface the bug. Testers often don’t understand the vague feature definitions that developers provide. The automation team cannot understand or reproduce test plans written by feature testers-and sometimes cannot even understand each others’ scripts. Often, nobody can understand defect reports coming back from the field.

The problem is further exacerbated when global product team members speak different languages. Time spent interpreting vague communications and correcting misunderstandings will delay product introduction or result in quality problems. In no more time than required for manual testing, new test automation products can create documented and automated test case. This simple difference creates a breakthrough for testing by enabling QA groups to detect errors earlier in the product introduction lifecycle, increase coverage, and clearly communicate the bugs to developers so that they can replicate and fix the bugs more rapidly and validate it has been remedied. The technology behind Fanfare’s test automation solutions is the breakthrough in the capture-replay approach to automation which enables the test case document and the automated test script to be one and the same. Testers follow the same process that they already use for manual testing. The bonus: at the end of the process, they have an automated test. And, with no incremental effort testers can:

  • Use the devices interactively to prototype a test, just as if he or she were testing it manually
  • Execute the test iteratively until it behaves exactly as desired, including parameterization, looping, logic, and reporting
  • Receive a documented and automated script that other testers or the developer can use and replay
  • Optionally render the test into a Tcl script that can immediately be used in an existing automated regression testing system
  • Reuse and adapt the tests that have been captured-the same benefit that scripters have long enjoyed

New test automation approaches will also solve the communication challenges that have plagued testers, developers, and automation teams. To illustrate, consider the traditional scenario in which a tester in the United States writes a bug report and developers, who speak a different language, spend time trying to understand the report’s meaning and surface the bug. In contrast, today’s test automation solutions enable more effective and efficient communications by using the test case itself as the communications vehicle. The tester can provide a failed test case that developers can run themselves, eliminating ambiguity and thereby accelerating problem resolution.

In conclusion, to sustain today’s pace of innovation, equipment manufacturers need new ways to scale product testing to keep up with product advancements. Achieving this goal will require a collaborative effort from everyone involved in the testing process. Moreover, as other vendors join Fanfare in offering tools and technologies that accelerate and enhance testing, equipment manufacturers will be well positioned to break through the product innovation barrier.