We have all been guilty at one time or another. The classic scenario where one decides to cut costs by doing-it-yourself versus hiring a specialist and it turned out to cost you more in the long run. Taking on a test automation effort is no different. Technology has given us a lot of tools that allow the average user to develop test automation applications. However, there are some pitfalls that an automation specialist knows to look out for in order to truly make an automation solution a success.
Skimping on Hardware Costs
A test effort typically doesn’t involve just one test rack and therefore the automation equipment costs are multiplied across test racks. On top of that, extra labor costs are often associated with initial development of the test simulators and test scripts. Therefore, it is very common for test teams to cut corners when purchasing test automation equipment. Often times this is a major mistake. Our experience has shown that hardware costs are typically less than 15% of the total automation effort and choosing the wrong hardware can lead to expensive future maintenance costs.
Accounting for Future Design Changes
Products often evolve. If a minor design change in the future leads to a massive testing expense, we have seen where companies will abandon the automation effort altogether because the automation architecture was not laid out correctly and became difficult to maintain.You can never predict what changes are going to be made to a system in the future. You CAN minimize the impact.
At Mainline Test & Integration, our engineers take an architect approach to test automation from the very beginning of the product development life cycle. We understand that this approach often takes a lot of diligence as testing is typically being done in order to release a product and most projects have a schedule to uphold.
Knowing When Not to Automate
A proper automation architecture could increase your test coverage percentage of automation to 95%, but the last 5% may cost just as much as the first 95%. And don’t forget, some test sweeps may not be planned to be ran multiple times. One of the major reasons to automate is to obtain the costs and time savings associated with multiple test sweeps. Automating tests that require infrequent execution can be the mark of a misguided automation effort.
While we share an enthusiasm for automation, some things just do not make sense to automate. System integration testing with hardware-in-the-loop requires special attention. While simulation tools allow you to test the product in ways you can’t do manually, the last thing you want is to be in the field and be hooking up all the subsystems for the first time. Our engineers understand that there is a proper time and place to introduce test automation.