Software Test Automation Maturity Assessment and Improvement Based on Best Practices

Mature test automation is key for achieving high product quality at speed. However, many organizations still have immature test automation practices that impede them to reap expected benefits. Not all attempts to improve test automation practices are successful, usually caused by ineffective assessment and improvement.

This dissertation aims to develop the guidelines for test automation maturity assessment and improvement by synthesizing existing guidelines from various sources, demonstrating the adoption of developed guidelines, and evaluating the effectiveness of the developed guidelines with practitioners.

In this dissertation, we synthesized a taxonomy of test automation best practices from 18 test maturity models. We also reviewed 26 academic literature and 55 grey literature sources on the same topic, and found that the current literature includes the guidelines that are not covered in test maturity models, like new technical best practices and advice (e.g., improvement approaches, technical techniques, and experience-based heuristics) on conducting certain best practices. Using synthesized test automation best practices from test maturity models as the base, we developed a test automation maturity survey and distributed it in the current industry to explore the state of practice of test automation maturity. Based on 151 survey responses coming from more than 100 organizations in 25 countries, several observations were made to demonstrate the adoption of test automation best practices proposed by existing test maturity models, e.g., there is a lack of guidelines on designing and executing automated tests and the right metrics to measure test automation performance in general. We observed that high levels of test automation maturity (assessed by test automation best practices from test maturity models) can lead to high product quality at speed in modern software developments, using empirical evidence from an experience study (that examined the experience of a DevOps team who succeeded in test automation maturity improvement) and a quantitative study (that collected metric data from 37 open-source projects by running our test automation maturity survey and mined project repositories).

With the study results of this dissertation, future work in this research scope related to theory development works, empirical studies, and iterative studies is stimulated. Practitioners can benefit from the developed and evaluated guidelines in this dissertation to assess and improve test automation maturity.