Not logged in.
Quick Search - Contribution
Contribution Details
| Type | Journal Article |
| Scope | Discipline-based scholarship |
| Title | On the User-centric Comparative Remote Evaluation of Interactive Video Search Systems |
| Organization Unit | |
| Authors |
|
| Item Subtype | Original Work |
| Refereed | No |
| Status | Published in final form |
| Language |
|
| Journal Title | IEEE MultiMedia |
| Publisher | Institute of Electrical and Electronics Engineers |
| Geographical Reach | international |
| ISSN | 1070-986X |
| Volume | 28 |
| Number | 4 |
| Page Range | 18 - 28 |
| Date | 2021 |
| Abstract Text | In the research of video retrieval systems, comparative assessments during dedicated retrieval competitions provide priceless insights into the performance of individual systems. The scope and depth of such evaluations are unfortunately hard to improve, due to the limitations by the set-up costs, logistics, and organization complexity of large events. We show that this easily impairs the statistical significance of the collected results, and the reproducibility of the competition outcomes. In this article, we present a methodology for remote comparative evaluations of content-based video retrieval systems and demonstrate that such evaluations scale-up to sizes that reliably produce statistically robust results, and propose additional measures that increase the replicability of the experiment. The proposed remote evaluation methodology forms a major contribution toward open science in interactive retrieval benchmarks. At the same time, the detailed evaluation reports form an interesting source of new observations about many subtle, previously inaccessible aspects of video retrieval. |
| Official URL | https://ieeexplore.ieee.org/document/9381648 |
| Digital Object Identifier | 10.1109/mmul.2021.3066779 |
| Other Identification Number | merlin-id:20949 |
| PDF File |
Download from ZORA
|
| Export |
BibTeX
EP3 XML (ZORA)
|