Not logged in.

Contribution Details

Type Conference or Workshop Paper
Scope Discipline-based scholarship
Published in Proceedings Yes
Title An Asynchronous Scheme for the Distributed Evaluation of Interactive Multimedia Retrieval
Organization Unit
  • Loris Sauter
  • Ralph Gasser
  • Abraham Bernstein
  • Heiko Schuldt
  • Luca Rossetto
Presentation Type paper
Item Subtype Original Work
Refereed Yes
Status Published in final form
  • English
ISBN 978-1-4503-9497-0
Page Range 33 - 39
Event Title MM '22: The 30th ACM International Conference on Multimedia
Event Type conference
Event Location Lisboa Portugal
Event Start Date November 10 - 2022
Event End Date November 14 - 2022
Place of Publication New York, NY, USA
Publisher ACM
Abstract Text Evaluation campaigns for interactive multimedia retrieval, such as the Video Browser Shodown (VBS) or the Lifelog Search Challenge (LSC), so far imposed constraints on both simultaneity and locality of all participants, requiring them to solve the same tasks in the same place, at the same time and under the same conditions. These constraints are in contrast to other evaluation campaigns that do not focus on interactivity, where participants can process the tasks in any place at any time. The recent travel restrictions necessitated the relaxation of the locality constraint of interactive campaigns, enabling participants to take place from an arbitrary location. Born out of necessity, this relaxation turned out to be a boon since it greatly simplified the evaluation process and enabled organisation of ad-hoc evaluations outside of the large campaigns. However, it also introduced an additional complication in cases where participants were spread over several time zones. In this paper, we introduce an evaluation scheme for interactive retrieval evaluation that relaxes both the simultaneity and locality constraints, enabling participation from any place at any time within a predefined time frame. This scheme, as implemented in the Distributed Retrieval Evaluation Server (DRES), enables novel ways of conducting interactive retrieval evaluation and bridged the gap between interactive campaigns and non-interactive ones.
Free access at Official URL
Official URL
Related URLs
Digital Object Identifier 10.1145/3552467.3554797
Other Identification Number merlin-id:22773
PDF File Download from ZORA
Export BibTeX