Separating passing and failing test executions by clustering anomalies

Almaghairbe, Rafig and Roper, Marc (2016) Separating passing and failing test executions by clustering anomalies. Software Quality Journal. pp. 1-38. ISSN 1573-1367 (

[thumbnail of Almaghairbe-Roper-SQJ-2016-Separating-passing-and-failing-test-executions]
Text. Filename: Almaghairbe_Roper_SQJ_2016_Separating_passing_and_failing_test_executions.pdf
Final Published Version
License: Creative Commons Attribution 4.0 logo

Download (3MB)| Preview


Developments in the automation of test data generation have greatly improved efficiency of the software testing process, but the so-called oracle problem (deciding the pass or fail outcome of a test execution) is still primarily an expensive and error-prone manual activity. We present an approach to automatically detect passing and failing executions using cluster-based anomaly detection on dynamic execution data based on firstly, just a system’s input/output pairs and secondly, amalgamations of input/output pairs and execution traces. The key hypothesis is that failures will group into small clusters, whereas passing executions will group into larger ones. Evaluation on three systems with a range of faults demonstrates this hypothesis to be valid—in many cases small clusters were composed of at least 60 % failures (and often more). Concentrating the failures in these small clusters substantially reduces the numbers of outputs that a developer would need to manually examine following a test run and illustrates that the approach has the potential to improve the effectiveness and efficiency of the testing process.


Almaghairbe, Rafig ORCID logoORCID: and Roper, Marc ORCID logoORCID:;