Video Browser Showdown: Interactive Video Benchmark

• Video Browser Showdown is an annual, live interactive video retrieval competition that tests rapid human-in-the-loop search on large video collections under time constraints.
• The benchmark has evolved to include visual and textual known-item search, ad-hoc video search, question answering, and conversational search tasks.
• Systems integrate multimodal retrieval, deep feature embedding, and optimized interaction designs to address segmentation challenges and domain-specific nuances.

Searching arXiv for recent and foundational Video Browser Showdown papers to ground the article. Video Browser Showdown (VBS) is an annual, live, interactive video retrieval competition and evaluation platform that tests how effectively systems support rapid human-in-the-loop search over large video collections under time pressure. Across the literature, VBS is described as a competitive benchmark or evaluation campaign for ad-hoc and known-item video search, with both expert and novice users operating systems on shared datasets and submitting results to a competition server (Schoeffmann et al., 2018). Its task portfolio has expanded over time from visual and textual Known-Item Search (KIS) and Ad-hoc Video Search (AVS) to include Question Answering (QAS) in 2024 and conversational known-item search (KISC) in 2025, reflecting a shift from segment retrieval alone toward broader multimodal and mixed-initiative interaction (Rossetto et al., 2024, Rossetto et al., 15 Sep 2025).

1. Historical development and institutional role

VBS has been characterized as a recurring, live competition co-located with the International Conference on MultiMedia Modeling (MMM), where research teams bring interactive systems into a shared evaluation setting (Rossetto et al., 2024, Rossetto et al., 15 Sep 2025). Earlier system papers frame it as a benchmark for time-constrained interactive video retrieval on a shared corpus, with public operation in front of an audience and strong emphasis on iterative querying, browsing, and submission efficiency (Rossetto et al., 2018, Nguyen et al., 2018).

The event has evolved substantially in scale and scope. In 2018, system papers centered on tasks over collections aligned with TRECVID resources such as IACC.3 and master-shot keyframes (Schoeffmann et al., 2018, Rossetto et al., 2018). By 2019, VBS used the V3C1 dataset, reported as 7,475 web videos totaling about 1,000 hours and segmented into 1,082,657 non-overlapping shots with associated keyframes and thumbnails (Amato et al., 2020). In 2024, the competition used two shards of V3C comprising 2,300 hours from 17,235 Vimeo videos, together with the Marine Video Kit (MVK) and a gynecological laparoscopies dataset (Rossetto et al., 2024). In 2025, the scale increased further to the complete V3C collection, 3,800 hours from 28,450 videos, alongside MVK version 2 and LapGyn, yielding nearly 4,000 hours overall (Rossetto et al., 15 Sep 2025).

This trajectory suggests that VBS functions not only as a system comparison exercise but also as a longitudinal stress test for interaction design under increasing dataset heterogeneity, domain shift, and temporal pressure. The introduction of domain-focused marine and medical collections appears to have intensified the need for domain-adapted indexing and operator expertise (Truong et al., 2022, Rossetto et al., 2024).

2. Task taxonomy and evaluation setting

VBS task design is centered on time-constrained interactive retrieval. In 2018, system papers consistently describe three primary task types: visual KIS, textual KIS, and AVS (Schoeffmann et al., 2018, Nguyen et al., 2018, Watcharapinchai et al., 2018). Visual KIS typically required locating an exact segment given a video clip or visual exemplar, while textual KIS required locating a segment from a textual description. AVS differed by asking for as many relevant segments as possible for a general description, and multiple correct submissions could be possible, with immediate confirmation from the server in some editions (Schoeffmann et al., 2018).

Later reports formalize the task durations more explicitly. For VBS 2018, one system paper states that the benchmark allocates 5 minutes per query (Nguyen et al., 2018). In VBS 2024, expert tasks were organized as KISV with 5 minutes, KIST with 7 minutes, AVS with 5 minutes, and QAS with 5 minutes (Rossetto et al., 2024). In VBS 2025, KISV had 5 minutes, KIST 7 minutes, KISC 7 minutes, AVS 5 minutes, and QAS 5 minutes (Rossetto et al., 15 Sep 2025).

A central structural feature is the distinction between expert and novice sessions. Multiple papers note that experts are system developers or experienced operators, whereas novices are first-time or non-developer users, making VBS simultaneously an evaluation of retrieval effectiveness and usability (Rossetto et al., 2019, Rossetto et al., 2024). The novice/expert split has repeatedly exposed differences in preferred modalities and interface tolerance. For example, text-based querying was often favored by both groups because it is faster to formulate under time pressure, whereas more complex sketch or concept-combination interfaces were harder for novices to exploit effectively (Rossetto et al., 2019, Nguyen et al., 2018, Watcharapinchai et al., 2018).

The evaluation infrastructure in recent editions is built around the Distributed Retrieval Evaluation Server (DRES), which manages task orchestration, submissions, and scoring (Rossetto et al., 2024, Rossetto et al., 15 Sep 2025). However, the reports do not publish a formal LaTeX scoring formula. In tasks such as AVS and QAS, live judging has been necessary because full pre-annotation is impractical and correct answers may have multiple valid forms (Rossetto et al., 2024, Rossetto et al., 15 Sep 2025).

3. Core retrieval paradigms and system architectures

VBS has served as a venue for a wide range of interactive video retrieval architectures, but several recurring design patterns are visible.

A first pattern is multimodal retrieval with explicit query composition. The VIREO 2018 system integrated three main modalities—query-by-color-sketch, query-by-text, and query-by-concept—with relevance feedback and later added an object detector built on R-FCN using additional training data from multiple sources (Nguyen et al., 2018). The Sloth Search System supported text-based and sketch-based search, combining them with a weighted formulation:

$$sim_{all}=w_{t} \cdot sim_{t}+w_{c} \cdot (1 - dist_{c})+ w_{o} \cdot (1-dist_{o})$$

where $sim_t$ is the text score and $dist_c$, $dist_o$ are cosine distances for color and object sketch matching, with user-defined weights (Watcharapinchai et al., 2018).

A second pattern is the heavy use of concept detection and textual surrogates. vitrivr’s 2019 deployment compensated for missing precomputed ASR in V3C by generating speech transcripts using the Google Cloud Speech-to-Text API and adding action labels with I3D pretrained on Kinetics-600, thereby making text queries effective even for visually specified tasks (Rossetto et al., 2019). Similarly, diveXplore systems across years relied on deep concepts, objects, places, events, OCR, and speech-to-text to broaden textual access paths into visual content (Leibetseder et al., 28 Aug 2025, Schoeffmann et al., 28 Aug 2025).

A third pattern is vector-space retrieval and cross-modal embedding search. The 2024 diveXplore redesign integrated OpenCLIP trained on LAION-2B with the ViT-H/14 backbone, storing 1024-dimensional image and text embeddings in FAISS for free-text and visual similarity search, with cosine similarity

$s(x, y) = \frac{x \cdot y}{\|x\| \|y\|}$

used for ranking (Schoeffmann et al., 28 Aug 2025). This reflects the broader shift toward joint embedding models in later VBS systems, especially for heterogeneous datasets and free-text interaction (Rossetto et al., 2024).

A fourth pattern is retrieval unification through text indexing. VISIONE encodes scene tags, object and color locations, and deep visual features as text in a single Apache Lucene index (Amato et al., 2020). Its multimodal architecture uses fielded textual encodings for scene tags, object/color classes, location-coded tokens, and surrogate text representations of R-MAC descriptors, thereby allowing retrieval and rescoring within a standard text search engine (Amato et al., 2020).

These architectural tendencies indicate that VBS has progressively favored systems that minimize friction between modalities, support fast pivots between ranked retrieval and browsing, and expose enough internal structure to let users reformulate queries quickly.

4. Interaction design and browsing strategies

The competition’s time-constrained setting has made interaction design as important as underlying retrieval models. Many companion papers devote more space to interface behavior than to back-end formulas, and they repeatedly identify browsing efficiency, navigation granularity, and submission speed as decisive factors (Schoeffmann et al., 2018, Rossetto et al., 2019).

One influential interface family is map-based browsing. diveXplore historically organized keyframes in self-organizing maps, arranged by similarity features such as color or semantic concepts, later extending to pre-filtered concept maps and textual search over more than 1200 feature maps (Schoeffmann et al., 2018). In version 5.0, the system deliberately reduced dependence on a single global featuremap and replaced it with smaller concept-based featuremaps and a video-based similarity filter to reduce complexity and improve responsiveness (Leibetseder et al., 28 Aug 2025). In version 6.0, diveXplore introduced a dual shot-search and map-search paradigm, where shot search returns ranked keyframes and map search uses video summary images navigated horizontally by result ranking and vertically by inter-video similarity (Leibetseder et al., 28 Aug 2025).

Another recurring strategy is high-density result presentation with rapid local context inspection. vitrivr 2018 introduced a similarity-ordered global thumbnail view and a per-video temporal thumbnail view, together with color annotations, higher playback speeds, and direct submission from the current playback position (Rossetto et al., 2018). vitrivr 2019 further improved throughput through client-side history caching, lazy loading, result filtering by score threshold and dominant shot color, and a Shift-click submission shortcut (Rossetto et al., 2019).

Systems also differ in how they represent temporal context. VIREO used dynamic-image browsing that automatically showed candidate shots from a video on mouse hover, while also allowing switching to a traditional frame-based interface (Nguyen et al., 2018). diveXplore 2024 used hover-scrubbing over result thumbnails, where horizontal mouse position dynamically scrubbed through keyframes from the same video, and provided immediate access to video summaries, full video playback, metadata, visual similarity search, and DRES submission (Schoeffmann et al., 28 Aug 2025).

Outside VBS proper but directly relevant to its design goals, the Content Flow Bar (CFB) proposes transcript-driven semantic timeline snippets as a navigation aid for rapid fragment identification (Perez-Ortiz et al., 2022). In a user study, CFB increased time spent exploring in the results screen by +139.160 s ($p=0.0004$), increased deepest rank of video explored by +1.873 ($p=0.0336$), and reduced the number of videos played before first selection by −0.533 as a trend (Perez-Ortiz et al., 2022). This suggests that semantically enriched navigation tools may address a persistent VBS bottleneck: rapid within-video verification once a plausible result has been surfaced.

5. Datasets, segmentation, and domain effects

A persistent theme across VBS literature is that dataset characteristics strongly shape system effectiveness. In 2018, diveXplore explicitly described IACC.3 as having “low visual quality,” which led to disabling the web-example search component and replacing it with a color-sketch feature (Schoeffmann et al., 2018). vitrivr likewise found that IACC.3 master-shot references were often too coarse, with targets embedded inside longer segments and therefore hard to isolate (Rossetto et al., 2018).

Segmentation has therefore been a recurrent technical issue. diveXplore 6.0 identified its earlier uniform 10-second segmentation as a “major bottleneck for KIS,” since short targets could be missed inside long windows, and changed to 1-second sampling for VBS 2022 to improve recall and timestamp precision (Leibetseder et al., 28 Aug 2025). The 2021 diveXplore rebuild already split V3C1 into custom shots and uniformly sampled frames using a one-second interval (Leibetseder et al., 28 Aug 2025). Sloth Search likewise noted that searching individual keyframes rather than shot compositions limited visual KIS performance (Watcharapinchai et al., 2018).

The expansion to domain-specific datasets has introduced additional challenges. The Marine Video Kit paper states that MVK was used in a special session during VBS 2023 and reports that general-purpose CLIP-based text-image retrieval is limited in underwater content (Truong et al., 2022). In a 4,000-pair experiment using ClipCap-generated queries, about 30% of targets were retrieved within the top 2,000 items and only 6.6% within the top 100, underscoring the difficulty of homogeneous, noisy marine footage (Truong et al., 2022). The same paper found that a VBS-experienced annotator outperformed novice users and ClipCap-generated captions in query formulation, indicating that user expertise becomes even more consequential in narrow domains (Truong et al., 2022).

Medical video collections introduced in VBS 2024 and extended in 2025 add a different kind of domain shift, with repetitive visual patterns and specialized terminology (Rossetto et al., 2024, Rossetto et al., 15 Sep 2025). diveXplore 2024 addressed this with domain-specific segmentation for surgical videos based on ORB keypoint tracking, plus Mask R-CNN for instruments and anatomy and a Bi-LSTM action recognizer for surgical actions (Schoeffmann et al., 28 Aug 2025). This suggests that VBS increasingly rewards systems that can specialize feature extraction and indexing to the operational domain rather than relying on general-purpose web-video priors alone.

6. Quantitative outcomes, system trade-offs, and recurrent failure modes

Although many VBS companion papers focus on engineering and qualitative lessons, several provide concrete performance outcomes. diveXplore reports “2nd place at VBS2017 and VBS2018” (Schoeffmann et al., 2018). VIREO finished 6th overall out of 9 teams in VBS 2018 and was the winner for visual known-item search, with users able to complete a color-sketch query within 0.5 second (Nguyen et al., 2018). Sloth Search reports that its team was ranked first in AVS in VBS 2018 (Watcharapinchai et al., 2018). vitrivr’s 2018 paper reports 2nd place in competition textual KIS (55/100 points), 4th place in visual expert KIS (79/100 points), 5th place in expert AVS (64/100 points), 7th place in novice AVS (35/100 points), no placement in novice visual KIS (0/100 points), and 7th place overall (47/100 points) (Rossetto et al., 2018). vitrivr then ranked first overall in VBS 2019 with 91/100, including 1st place in visual expert KIS, 1st place in novice visual KIS, 2nd place in expert AVS, and 1st place in novice AVS (Rossetto et al., 2019).

More recent official result reports show the increasing breadth of the field. In 2024, the final ranking placed VISIONE first, followed by Vibro and PraK, with diveXplore fifth, VIREO seventh, and vitrivr ninth (Rossetto et al., 2024). In 2025, seventeen teams participated, and the final ranking placed NII-UIT first, PraK Tool V3 second, diveXplore third, VIREO fifth, and vitrivr thirteenth (Rossetto et al., 15 Sep 2025).

Several recurrent trade-offs emerge from these reports:

Trade-off	Typical benefit	Typical limitation
Text-first interaction	Fast query formulation under time pressure	Fails when labels, ASR, or OCR are missing or mismatched
Rich multi-feature interfaces	Broad coverage across task types	Can become complex, error-prone, and hard for novices
Coarse segmentation	Lower indexing cost	Misses short KIS targets and reduces timestamp precision
Concept-heavy retrieval	Strong AVS and textual access	Ambiguity and granularity issues in concept selection

The “less is more” redesign of diveXplore 5.0 explicitly linked declining performance from VBS2017–VBS2020 to accumulating complexity around a central global featuremap, which had become slow and error-prone (Leibetseder et al., 28 Aug 2025). This is a rare direct articulation of a broader VBS lesson: adding retrieval options does not necessarily improve competitive effectiveness if the resulting interaction costs exceed the gains.

Failure modes are also strikingly consistent across systems. Coarse shot segmentation repeatedly caused missed KIS targets (Rossetto et al., 2018, Leibetseder et al., 28 Aug 2025). Browsing bottlenecks caused false negatives even when the correct item was retrieved; vitrivr reports that in the 9th visual KIS task in 2019, the correct item had been retrieved but was overlooked while browsing (Rossetto et al., 2019). Large concept banks could cause ambiguity and excessive reformulation, particularly for novices (Nguyen et al., 2018). Sketch-based querying was often valuable for some visual KIS tasks but insufficiently selective for large-scale collections (Rossetto et al., 2019, Watcharapinchai et al., 2018).

7. Emerging directions: conversational, question-answering, and domain-robust VBS

Recent VBS editions indicate a broadening of the benchmark’s definition of interactive retrieval. The 2024 introduction of QAS added tasks where a scene in loop plus a textual question required contextual reasoning within a known video, rather than simple scene matching (Rossetto et al., 2024). The 2025 introduction of KISC went further by formalizing conversational search: teams began with a short textual summary and could ask questions to an oracle panel, with answers shared to all teams simultaneously (Rossetto et al., 15 Sep 2025).

This shift suggests that VBS is increasingly probing mixed-initiative workflows rather than only retrieval speed from fixed queries. A plausible implication is that future high-performing systems may need to unify classic interactive retrieval with conversational state tracking, dynamic query reformulation, and richer cross-modal grounding.

Another clear direction is the integration of LLMs and generative components. The 2024 results report identifies TalkSee, VIREO, and Exquisitor as examples of systems emphasizing LLMs, generative models, or conversational search (Rossetto et al., 2024). However, the reports do not yet establish a definitive correlation between LLM integration and final ranking. The evidence instead remains stronger for interface quality, responsive infrastructure, and multimodal indexing as the common traits of successful systems (Rossetto et al., 2024, Rossetto et al., 15 Sep 2025).

Finally, domain robustness has become a central concern. The use of MVK and medical videos demonstrates that relatively small but homogeneous or specialized collections can be as challenging as much larger open-domain web video corpora (Truong et al., 2022, Rossetto et al., 2024). This has implications for both system design and evaluation philosophy. General-purpose embedding models may offer strong initial performance, but VBS increasingly exposes their limitations under domain shift, noisy visual conditions, and task-specific temporal reasoning.

Taken together, the literature portrays VBS as a benchmark that has moved from evaluating interactive segment retrieval on generic web video toward evaluating full-stack human-in-the-loop search systems under conditions of scale, modality diversity, domain specialization, and conversational uncertainty. Its continuing value lies in making those interacting constraints visible in a live setting where retrieval models, user interfaces, operator skill, and evaluation infrastructure all materially affect outcome.