Abstract
Search-based software testing (SBST) is a widely adopted technique fortesting complex systems with large input spaces, such as Deep Learning-enabled(DL-enabled) systems. Many SBST techniques focus on Pareto-based optimization,where multiple objectives are optimized in parallel to reveal failures.However, it is important to ensure that identified failures are spreadthroughout the entire failure-inducing area of a search domain and notclustered in a sub-region. This ensures that identified failures aresemantically diverse and reveal a wide range of underlying causes. In thispaper, we present a theoretical argument explaining why testing based on Paretooptimization is inadequate for covering failure-inducing areas within a searchdomain. We support our argument with empirical results obtained by applying twowidely used types of Pareto-based optimization techniques, namely NSGA-II (anevolutionary algorithm) and OMOPSO (a swarm-based Pareto-optimizationalgorithm), to two DL-enabled systems: an industrial Automated Valet Parking(AVP) system and a system for classifying handwritten digits. We measure thecoverage of failure-revealing test inputs in the input space using a metricthat we refer to as the Coverage Inverted Distance quality indicator. Ourresults show that NSGA-II-based search and OMOPSO are not more effective than ana\"ive random search baseline in covering test inputs that reveal failures.The replication package for this study is available in a GitHub repository.