Portal de Periódicos da CAPES

On the User-Centric Comparative Remote Evaluation of Interactive Video Search Systems

2021; IEEE Computer Society; Volume: 28; Issue: 4 Linguagem: Inglês

10.1109/mmul.2021.3066779

1941-0166

Luca Rossetto, Ralph Gasser, Silvan Heller, Mahnaz Amiri Parian, Loris Sauter, Florian Spieß, Heiko Schuldt, Ladislav Peška, Tomáš Soućek, Miroslav Kratochvíl, František Mejzlík, Patrik Veselý, Jakub Lokoč,

Video Analysis and Summarization

In the research of video retrieval systems, comparative assessments during dedicated retrieval competitions provide priceless insights into the performance of individual systems. The scope and depth of such evaluations are unfortunately hard to improve, due to the limitations by the set-up costs, logistics, and organization complexity of large events. We show that this easily impairs the statistical significance of the collected results, and the reproducibility of the competition outcomes. In this article, we present a methodology for remote comparative evaluations of content-based video retrieval systems and demonstrate that such evaluations scale-up to sizes that reliably produce statistically robust results, and propose additional measures that increase the replicability of the experiment. The proposed remote evaluation methodology forms a major contribution toward open science in interactive retrieval benchmarks. At the same time, the detailed evaluation reports form an interesting source of new observations about many subtle, previously inaccessible aspects of video retrieval.