a systematic literature review on benchmarks for evaluating debugging approaches

This systematic literature review provides an overview of benchmarks for evaluating debugging approaches. It supports researchers to choose the best suited benchmark(s) for evaluating their approaches.

Highlights •Identification of 73 benchmarks for evaluating debugging approaches.•Comparison of benchmarks w.r.t. size, language, and realization of FAIR criteria.•Discussion of problems in benchmar...

We compare the different benchmarks w.r.t. their size and the provided information such as bug reports, contained test cases, and other code metrics. This comparison is intended to help...

However, the reasons that inspire researchers to propose novel benchmarks are not fully understood. This article reports the investigation, identification, classification, and externalization of the state of the art about the proposition of benchmarks on software testing and debugging domains.

A systematic literature review on benchmarks for evaluating debugging approaches. Journal of Systems and Software 192 (2022), 111423. https://doi.org/10.1016/j.jss.2022.111423 Digital Library

A systematic literature review on benchmarks for evaluating debugging approaches Abstract Bug benchmarks are used in development and evaluation of debugging approaches, e.g. fault localization and automated repair.

For these reasons, we have performed a systematic literature review where we have identified 73 benchmarks that can be used to evaluate debugging approaches. We compare the different benchmarks with respect to their size and the provided information such as bug reports, contained test cases, and other code metrics.

A systematic literature review on benchmarks for evaluating debugging approaches. J. Syst. Softw. 192: 111423 (2022)

A novel bug benchmark for DLSW, Denchmark, is introduced, consisting of 4,577 bug reports from 193 popular D LSW projects, collected through a systematic dataset construction process, to provide an invaluable starting point for the automatic debugging techniques of DLSw.

This comparison is intended to help researchers to quickly identify all suitable benchmarks for evaluating their specific debugging approaches. More information can be found in the publication: Thomas Hirsch and Birgit Hofer: "A Systematic Literature Review on Benchmarks for Evaluating Debugging Approaches", submitted to the Journal of Systems ...