Not logged in.

Contribution Details

Type Conference or Workshop Paper
Scope Discipline-based scholarship
Published in Proceedings Yes
Title An Evaluation of Open-Source Software Microbenchmark Suites for Continuous Performance Assessment
Organization Unit
Authors
  • Christoph Laaber
  • Philipp Leitner
Presentation Type paper
Item Subtype Original Work
Refereed Yes
Status Published in final form
Language
  • English
Event Title MSR ’18: 15th International Conference on Mining Software Repositories
Event Type conference
Event Location Gothenburg, Sweden
Event Start Date May 28 - 2018
Event End Date May 29 - 2018
Place of Publication New York, NY, USA
Publisher ACM
Abstract Text Continuous integration (CI) emphasizes quick feedback to devel- opers. This is at odds with current practice of performance testing, which predominantely focuses on long-running tests against entire systems in production-like environments. Alternatively, software microbenchmarking attempts to establish a performance baseline for small code fragments in short time. This paper investigates the quality of microbenchmark suites with a focus on suitability to deliver quick performance feedback and CI integration. We study ten open-source libraries written in Java and Go with benchmark suite sizes ranging from 16 to 983 tests, and runtimes between 11 minutes and 8.75 hours. We show that our study subjects include benchmarks with result variability of 50% or higher, indicating that not all benchmarks are useful for reliable discovery of slow- downs. We further arti cially inject actual slowdowns into public API methods of the study subjects and test whether test suites are able to discover them. We introduce a performance-test quality metric called the API benchmarking score (ABS). ABS represents a benchmark suite’s ability to nd slowdowns among a set of de ned core API methods. Resulting benchmarking scores (i.e., fraction of discovered slowdowns) vary between 10% and 100% for the study subjects. This paper’s methodology and results can be used to (1) assess the quality of existing microbenchmark suites, (2) select a set of tests to be run as part of CI, and (3) suggest or generate benchmarks for currently untested parts of an API.
Free access at Related URL
Related URLs
Digital Object Identifier 10.1145/3196398.3196407
Other Identification Number merlin-id:16259
PDF File Download from ZORA
Export BibTeX
EP3 XML (ZORA)