Separating passing and failing test executions by clustering anomalies

Almaghairbe, Rafig and Roper, Marc (2016) Separating passing and failing test executions by clustering anomalies. Software Quality Journal. pp. 1-38. ISSN 1573-1367 (https://doi.org/10.1007/s11219-016-9339-1)

[thumbnail of Almaghairbe-Roper-SQJ-2016-Separating-passing-and-failing-test-executions]
Preview
 Text. Filename: Almaghairbe_Roper_SQJ_2016_Separating_passing_and_failing_test_executions.pdf
Final Published Version
License: Creative Commons Attribution 4.0 logo
Download (3MB)| Preview

Abstract

Developments in the automation of test data generation have greatly improved efficiency of the software testing process, but the so-called oracle problem (deciding the pass or fail outcome of a test execution) is still primarily an expensive and error-prone manual activity. We present an approach to automatically detect passing and failing executions using cluster-based anomaly detection on dynamic execution data based on firstly, just a system’s input/output pairs and secondly, amalgamations of input/output pairs and execution traces. The key hypothesis is that failures will group into small clusters, whereas passing executions will group into larger ones. Evaluation on three systems with a range of faults demonstrates this hypothesis to be valid—in many cases small clusters were composed of at least 60 % failures (and often more). Concentrating the failures in these small clusters substantially reduces the numbers of outputs that a developer would need to manually examine following a test run and illustrates that the approach has the potential to improve the effectiveness and efficiency of the testing process.

CORE (COnnecting REpositories)