Test Case Prioritization Using Build History and Test Distances: An Approach for Improving Automotive Regression Testing in Continuous Integration Environments
Thesis event information
Date and time of the thesis defence
Place of the thesis defence
University of Oulu, Linnanmaa IT115
Topic of the dissertation
Test Case Prioritization Using Build History and Test Distances: An Approach for Improving Automotive Regression Testing in Continuous Integration Environments
Doctoral candidate
M.Sc. Information Processing Science Alireza Haghighatkhah
Faculty and unit
University of Oulu Graduate School, Faculty of Information Technology and Electrical Engineering, Empirical Software Engineering in Software, Systems and Services (M3S)
Subject of study
Information Processing Science
Opponent
Professor Per Runeson, Lund University
Custos
Professor Markku Oivo, University of Oulu
Test Case Prioritization Using Build History and Test Distances: An Approach for Improving Automotive Regression Testing in Continuous Integration Environments
Agile software development has become a source of competitive advantage in many industrial sectors, including the automotive domain. Continuous integration (CI) is an agile practice and involves integrating software changes as often and as early as possible. CI in combination with regression testing is considered a good software development practice, because it ensures that recent changes have not negatively impacted the software’s functionality. The execution of a large test suite at each integration (also known as the build) cycle is not realistic. To detect regression faults earlier, one alternative is to permute the test cases within the regression test suite. This is known as test case prioritization (TCP).
This dissertation aims to shorten the build cycle in CI environments by detecting regression faults earlier, allowing software developers to integrate and verify their changes more frequently and continuously. The research was performed in three phases. In the 1st phase, the relevant literature was systematically surveyed to analyze the existing body of knowledge and identify research opportunities. In the 2nd phase, the TCP technique of history-based diversity was developed and subjected to several rounds of improvements and evaluations. In the 3rd phase, results from the previous two phases were synthesized in order to draw conclusions and outline implications. Our results bring to mind the well-known adage “don't put all your eggs in one basket”. For the initial stages, when no information about the system under test is available, one should spread the limited testing budget evenly across different parts of the system by running the most dissimilar test cases early in the process (i.e., maximizing diversity as early as possible). For the later stages, when historical build data are available, one should give higher priority to the test cases that have higher historical value (i.e., those that revealed the most failure in the most recent revisions) and that are most distant from those previously prioritized (i.e., cover different areas of the system).
One of the major contributions of this dissertation is the design, implementation, and experimental evaluation of a TCP technique using 12 open-source Java programs. Findings from the experiments provide empirical evidence in support of two previously proposed TCP heuristics ¬¬– failure recurrence and test diversity – and their combination in CI development environments. Furthermore, results from the systematic mapping studies contribute to the literature by providing a synthesized and structured body of knowledge and a basis for future studies. We conclude that TCP can be performed in CI environments with negligible investment using build history and test distances.
This dissertation aims to shorten the build cycle in CI environments by detecting regression faults earlier, allowing software developers to integrate and verify their changes more frequently and continuously. The research was performed in three phases. In the 1st phase, the relevant literature was systematically surveyed to analyze the existing body of knowledge and identify research opportunities. In the 2nd phase, the TCP technique of history-based diversity was developed and subjected to several rounds of improvements and evaluations. In the 3rd phase, results from the previous two phases were synthesized in order to draw conclusions and outline implications. Our results bring to mind the well-known adage “don't put all your eggs in one basket”. For the initial stages, when no information about the system under test is available, one should spread the limited testing budget evenly across different parts of the system by running the most dissimilar test cases early in the process (i.e., maximizing diversity as early as possible). For the later stages, when historical build data are available, one should give higher priority to the test cases that have higher historical value (i.e., those that revealed the most failure in the most recent revisions) and that are most distant from those previously prioritized (i.e., cover different areas of the system).
One of the major contributions of this dissertation is the design, implementation, and experimental evaluation of a TCP technique using 12 open-source Java programs. Findings from the experiments provide empirical evidence in support of two previously proposed TCP heuristics ¬¬– failure recurrence and test diversity – and their combination in CI development environments. Furthermore, results from the systematic mapping studies contribute to the literature by providing a synthesized and structured body of knowledge and a basis for future studies. We conclude that TCP can be performed in CI environments with negligible investment using build history and test distances.
Last updated: 1.3.2023