Reinforcement Learning for Automatic Test Case Prioritization and Selection in Continuous Integration

  • 2018-11-09 20:08:58
  • Helge Spieker, Arnaud Gotlieb, Dusica Marijan, Morten Mossige
  • 1

Abstract

Testing in Continuous Integration (CI) involves test case prioritization,selection, and execution at each cycle. Selecting the most promising test casesto detect bugs is hard if there are uncertainties on the impact of committedcode changes or, if traceability links between code and tests are notavailable. This paper introduces Retecs, a new method for automaticallylearning test case selection and prioritization in CI with the goal to minimizethe round-trip time between code commits and developer feedback on failed testcases. The Retecs method uses reinforcement learning to select and prioritizetest cases according to their duration, previous last execution and failurehistory. In a constantly changing environment, where new test cases are createdand obsolete test cases are deleted, the Retecs method learns to prioritizeerror-prone test cases higher under guidance of a reward function and byobserving previous CI cycles. By applying Retecs on data extracted from threeindustrial case studies, we show for the first time that reinforcement learningenables fruitful automatic adaptive test case selection and prioritization inCI and regression testing.

 

Quick Read (beta)

loading the full paper ...