Not logged in.

Quick Search - Contribution

Contribution Details

Type Journal Article
Scope Discipline-based scholarship
Title Does Reviewer Recommendation Help Developers?
Organization Unit
Authors
  • Vladimir Kovalenko
  • Nava Tintarev
  • Evgeny Pasynkov
  • Christian Bird
  • Alberto Bacchelli
Item Subtype Original Work
Refereed Yes
Status Published in final form
Language
  • English
Journal Title IEEE Transactions on Pattern Analysis and Machine Intelligence
Publisher Institute of Electrical and Electronics Engineers
Geographical Reach international
ISSN 0098-5589
Volume 46
Number 7
Page Range 710 - 731
Date 2020
Abstract Text Selecting reviewers for code changes is a critical step for an efficient code review process. Recent studies propose automated reviewer recommendation algorithms to support developers in this task. However, the evaluation of recommendation algorithms, when done apart from their target systems and users (i.e., code review tools and change authors), leaves out important aspects: perception of recommendations, influence of recommendations on human choices, and their effect on user experience. This study is the first to evaluate a reviewer recommender in vivo. We compare historical reviewers and recommendations for over 21,000 code reviews performed with a deployed recommender in a company environment and set out to measure the influence of recommendations on users' choices, along with other performance metrics. Having found no evidence of influence, we turn to the users of the recommender. Through interviews and a survey we find that, though perceived as relevant, reviewer recommendations rarely provide additional value for the respondents. We confirm this finding with a larger study at another company. The confirmation of this finding brings up a case for more user-centric approaches to designing and evaluating the recommenders. Finally, we investigate information needs of developers during reviewer selection and discuss promising directions for the next generation of reviewer recommendation tools. Preprint: https://doi.org/10.5281/zenodo.1404814.
Official URL https://ieeexplore.ieee.org/document/8453850
Digital Object Identifier 10.1109/TSE.2018.2868367
Other Identification Number merlin-id:20246
PDF File Download from ZORA
Export BibTeX
EP3 XML (ZORA)