Not logged in.

Contribution Details

Type Journal Article
Scope Discipline-based scholarship
Title On the User-centric Comparative Remote Evaluation of Interactive Video Search Systems
Organization Unit
  • Luca Rossetto
  • Ralph Gasser
  • Silvan Heller
  • Mahnaz Parian-Scherb
  • Loris Sauter
  • Florian Spiess
  • Heiko Schuldt
  • Ladislav Peska
  • Tomáš Souček
  • Miroslav Kratochvil
  • František Mejzlík
  • Patrik Veselý
  • Jakub Lokoč
Item Subtype Original Work
Refereed No
Status Published in final form
  • English
Journal Title IEEE MultiMedia
Publisher Institute of Electrical and Electronics Engineers
Geographical Reach international
ISSN 1070-986X
Volume 28
Number 4
Page Range 18 - 28
Date 2021
Abstract Text In the research of video retrieval systems, comparative assessments during dedicated retrieval competitions provide priceless insights into the performance of individual systems. The scope and depth of such evaluations are unfortunately hard to improve, due to the limitations by the set-up costs, logistics, and organization complexity of large events. We show that this easily impairs the statistical significance of the collected results, and the reproducibility of the competition outcomes. In this article, we present a methodology for remote comparative evaluations of content-based video retrieval systems and demonstrate that such evaluations scale-up to sizes that reliably produce statistically robust results, and propose additional measures that increase the replicability of the experiment. The proposed remote evaluation methodology forms a major contribution toward open science in interactive retrieval benchmarks. At the same time, the detailed evaluation reports form an interesting source of new observations about many subtle, previously inaccessible aspects of video retrieval.
Official URL
Digital Object Identifier 10.1109/mmul.2021.3066779
Other Identification Number merlin-id:20949
PDF File Download from ZORA
Export BibTeX