---
title: 'Video Browser Showdown: Interactive Video Benchmark'
url: https://www.emergentmind.com/topics/video-browser-showdown
type: topic
---

# Video Browser Showdown: Interactive Video Benchmark

Searching arXiv for recent and foundational Video Browser Showdown papers to ground the article.
Video Browser Showdown (VBS) is an annual, live, interactive video retrieval competition and evaluation platform that tests how effectively systems support rapid human-in-the-loop search over large video collections under time pressure. Across the literature, VBS is described as a competitive benchmark or evaluation campaign for ad-hoc and known-item video search, with both expert and novice users operating systems on shared datasets and submitting results to a competition server [1804.01863]. Its task portfolio has expanded over time from visual and textual Known-Item Search (KIS) and Ad-hoc Video Search (AVS) to include Question Answering (QAS) in 2024 and conversational known-item search (KISC) in 2025, reflecting a shift from segment retrieval alone toward broader multimodal and mixed-initiative interaction [2502.15683] [2509.12000].

## 1. Historical development and institutional role

VBS has been characterized as a recurring, live competition co-located with the International Conference on MultiMedia Modeling (MMM), where research teams bring interactive systems into a shared evaluation setting [2502.15683] [2509.12000]. Earlier system papers frame it as a benchmark for time-constrained interactive video retrieval on a shared corpus, with public operation in front of an audience and strong emphasis on iterative querying, browsing, and submission efficiency [1805.02371] [1805.11814].

The event has evolved substantially in scale and scope. In 2018, system papers centered on tasks over collections aligned with TRECVID resources such as IACC.3 and master-shot keyframes [1804.01863] [1805.02371]. By 2019, VBS used the V3C1 dataset, reported as 7,475 web videos totaling about 1,000 hours and segmented into 1,082,657 non-overlapping shots with associated keyframes and thumbnails [2008.02749]. In 2024, the competition used two shards of V3C comprising 2,300 hours from 17,235 Vimeo videos, together with the Marine Video Kit (MVK) and a gynecological laparoscopies dataset [2502.15683]. In 2025, the scale increased further to the complete V3C collection, 3,800 hours from 28,450 videos, alongside MVK version 2 and LapGyn, yielding nearly 4,000 hours overall [2509.12000].

This trajectory suggests that VBS functions not only as a system comparison exercise but also as a longitudinal stress test for interaction design under increasing dataset heterogeneity, domain shift, and temporal pressure. The introduction of domain-focused marine and medical collections appears to have intensified the need for domain-adapted indexing and operator expertise [2209.11518] [2502.15683].

## 2. Task taxonomy and evaluation setting

VBS task design is centered on time-constrained interactive retrieval. In 2018, system papers consistently describe three primary task types: visual KIS, textual KIS, and AVS [1804.01863] [1805.11814] [1806.01081]. Visual KIS typically required locating an exact segment given a video clip or visual exemplar, while textual KIS required locating a segment from a textual description. AVS differed by asking for as many relevant segments as possible for a general description, and multiple correct submissions could be possible, with immediate confirmation from the server in some editions [1804.01863].

Later reports formalize the task durations more explicitly. For VBS 2018, one system paper states that the benchmark allocates 5 minutes per query [1805.11814]. In VBS 2024, expert tasks were organized as KISV with 5 minutes, KIST with 7 minutes, AVS with 5 minutes, and QAS with 5 minutes [2502.15683]. In VBS 2025, KISV had 5 minutes, KIST 7 minutes, KISC 7 minutes, AVS 5 minutes, and QAS 5 minutes [2509.12000].

A central structural feature is the distinction between expert and novice sessions. Multiple papers note that experts are system developers or experienced operators, whereas novices are first-time or non-developer users, making VBS simultaneously an evaluation of retrieval effectiveness and usability [1902.10647] [2502.15683]. The novice/expert split has repeatedly exposed differences in preferred modalities and interface tolerance. For example, text-based querying was often favored by both groups because it is faster to formulate under time pressure, whereas more complex sketch or concept-combination interfaces were harder for novices to exploit effectively [1902.10647] [1805.11814] [1806.01081].

The evaluation infrastructure in recent editions is built around the Distributed Retrieval Evaluation Server (DRES), which manages task orchestration, submissions, and scoring [2502.15683] [2509.12000]. However, the reports do not publish a formal LaTeX scoring formula. In tasks such as AVS and QAS, live judging has been necessary because full pre-annotation is impractical and correct answers may have multiple valid forms [2502.15683] [2509.12000].

## 3. Core retrieval paradigms and system architectures

VBS has served as a venue for a wide range of interactive video retrieval architectures, but several recurring design patterns are visible.

A first pattern is multimodal retrieval with explicit query composition. The VIREO 2018 system integrated three main modalities—query-by-color-sketch, query-by-text, and query-by-concept—with relevance feedback and later added an object detector built on R-FCN using additional training data from multiple sources [1805.11814]. The Sloth Search System supported text-based and sketch-based search, combining them with a weighted formulation:
$$
sim_{all}=w_{t} \cdot sim_{t}+w_{c} \cdot (1 - dist_{c})+ w_{o} \cdot (1-dist_{o})
$$
where $sim_t$ is the text score and $dist_c$, $dist_o$ are cosine distances for color and object sketch matching, with user-defined weights [1806.01081].

A second pattern is the heavy use of concept detection and textual surrogates. vitrivr’s 2019 deployment compensated for missing precomputed ASR in V3C by generating speech transcripts using the Google Cloud Speech-to-Text API and adding action labels with I3D pretrained on Kinetics-600, thereby making text queries effective even for visually specified tasks [1902.10647]. Similarly, diveXplore systems across years relied on deep concepts, objects, places, events, OCR, and speech-to-text to broaden textual access paths into visual content [2508.20687] [2508.20560].

A third pattern is vector-space retrieval and cross-modal embedding search. The 2024 diveXplore redesign integrated OpenCLIP trained on LAION-2B with the ViT-H/14 backbone, storing 1024-dimensional image and text embeddings in FAISS for free-text and visual similarity search, with cosine similarity
$$
s(x, y) = \frac{x \cdot y}{\|x\| \|y\|}
$$
used for ranking [2508.20560]. This reflects the broader shift toward joint embedding models in later VBS systems, especially for heterogeneous datasets and free-text interaction [2502.15683].

A fourth pattern is retrieval unification through text indexing. VISIONE encodes scene tags, object and color locations, and deep visual features as text in a single Apache Lucene index [2008.02749]. Its multimodal architecture uses fielded textual encodings for scene tags, object/color classes, location-coded tokens, and surrogate text representations of R-MAC descriptors, thereby allowing retrieval and rescoring within a standard text search engine [2008.02749].

These architectural tendencies indicate that VBS has progressively favored systems that minimize friction between modalities, support fast pivots between ranked retrieval and browsing, and expose enough internal structure to let users reformulate queries quickly.

## 4. Interaction design and browsing strategies

The competition’s time-constrained setting has made interaction design as important as underlying retrieval models. Many companion papers devote more space to interface behavior than to back-end formulas, and they repeatedly identify browsing efficiency, navigation granularity, and submission speed as decisive factors [1804.01863] [1902.10647].

One influential interface family is map-based browsing. diveXplore historically organized keyframes in self-organizing maps, arranged by similarity features such as color or semantic concepts, later extending to pre-filtered concept maps and textual search over more than 1200 feature maps [1804.01863]. In version 5.0, the system deliberately reduced dependence on a single global featuremap and replaced it with smaller concept-based featuremaps and a video-based similarity filter to reduce complexity and improve responsiveness [2508.20569]. In version 6.0, diveXplore introduced a dual shot-search and map-search paradigm, where shot search returns ranked keyframes and map search uses video summary images navigated horizontally by result ranking and vertically by inter-video similarity [2508.20687].

Another recurring strategy is high-density result presentation with rapid local context inspection. vitrivr 2018 introduced a similarity-ordered global thumbnail view and a per-video temporal thumbnail view, together with color annotations, higher playback speeds, and direct submission from the current playback position [1805.02371]. vitrivr 2019 further improved throughput through client-side history caching, lazy loading, result filtering by score threshold and dominant shot color, and a Shift-click submission shortcut [1902.10647].

Systems also differ in how they represent temporal context. VIREO used dynamic-image browsing that automatically showed candidate shots from a video on mouse hover, while also allowing switching to a traditional frame-based interface [1805.11814]. diveXplore 2024 used hover-scrubbing over result thumbnails, where horizontal mouse position dynamically scrubbed through keyframes from the same video, and provided immediate access to video summaries, full video playback, metadata, visual similarity search, and DRES submission [2508.20560].

Outside VBS proper but directly relevant to its design goals, the Content Flow Bar (CFB) proposes transcript-driven semantic timeline snippets as a navigation aid for rapid fragment identification [2201.03408]. In a user study, CFB increased time spent exploring in the results screen by +139.160 s ($p=0.0004$), increased deepest rank of video explored by +1.873 ($p=0.0336$), and reduced the number of videos played before first selection by −0.533 as a trend [2201.03408]. This suggests that semantically enriched navigation tools may address a persistent VBS bottleneck: rapid within-video verification once a plausible result has been surfaced.

## 5. Datasets, segmentation, and domain effects

A persistent theme across VBS literature is that dataset characteristics strongly shape system effectiveness. In 2018, diveXplore explicitly described IACC.3 as having “low visual quality,” which led to disabling the web-example search component and replacing it with a color-sketch feature [1804.01863]. vitrivr likewise found that IACC.3 master-shot references were often too coarse, with targets embedded inside longer segments and therefore hard to isolate [1805.02371].

Segmentation has therefore been a recurrent technical issue. diveXplore 6.0 identified its earlier uniform 10-second segmentation as a “major bottleneck for KIS,” since short targets could be missed inside long windows, and changed to 1-second sampling for VBS 2022 to improve recall and timestamp precision [2508.20687]. The 2021 diveXplore rebuild already split V3C1 into custom shots and uniformly sampled frames using a one-second interval [2508.20569]. Sloth Search likewise noted that searching individual keyframes rather than shot compositions limited visual KIS performance [1806.01081].

The expansion to domain-specific datasets has introduced additional challenges. The Marine Video Kit paper states that MVK was used in a special session during VBS 2023 and reports that general-purpose CLIP-based text-image retrieval is limited in underwater content [2209.11518]. In a 4,000-pair experiment using ClipCap-generated queries, about 30% of targets were retrieved within the top 2,000 items and only 6.6% within the top 100, underscoring the difficulty of homogeneous, noisy marine footage [2209.11518]. The same paper found that a VBS-experienced annotator outperformed novice users and ClipCap-generated captions in query formulation, indicating that user expertise becomes even more consequential in narrow domains [2209.11518].

Medical video collections introduced in VBS 2024 and extended in 2025 add a different kind of domain shift, with repetitive visual patterns and specialized terminology [2502.15683] [2509.12000]. diveXplore 2024 addressed this with domain-specific segmentation for surgical videos based on ORB keypoint tracking, plus Mask R-CNN for instruments and anatomy and a Bi-LSTM action recognizer for surgical actions [2508.20560]. This suggests that VBS increasingly rewards systems that can specialize feature extraction and indexing to the operational domain rather than relying on general-purpose web-video priors alone.

## 6. Quantitative outcomes, system trade-offs, and recurrent failure modes

Although many VBS companion papers focus on engineering and qualitative lessons, several provide concrete performance outcomes. diveXplore reports “2nd place at VBS2017 and VBS2018” [1804.01863]. VIREO finished 6th overall out of 9 teams in VBS 2018 and was the winner for visual known-item search, with users able to complete a color-sketch query within 0.5 second [1805.11814]. Sloth Search reports that its team was ranked first in AVS in VBS 2018 [1806.01081]. vitrivr’s 2018 paper reports 2nd place in competition textual KIS (55/100 points), 4th place in visual expert KIS (79/100 points), 5th place in expert AVS (64/100 points), 7th place in novice AVS (35/100 points), no placement in novice visual KIS (0/100 points), and 7th place overall (47/100 points) [1805.02371]. vitrivr then ranked first overall in VBS 2019 with 91/100, including 1st place in visual expert KIS, 1st place in novice visual KIS, 2nd place in expert AVS, and 1st place in novice AVS [1902.10647].

More recent official result reports show the increasing breadth of the field. In 2024, the final ranking placed VISIONE first, followed by Vibro and PraK, with diveXplore fifth, VIREO seventh, and vitrivr ninth [2502.15683]. In 2025, seventeen teams participated, and the final ranking placed NII-UIT first, PraK Tool V3 second, diveXplore third, VIREO fifth, and vitrivr thirteenth [2509.12000].

Several recurrent trade-offs emerge from these reports:

| Trade-off | Typical benefit | Typical limitation |
|---|---|---|
| **Text-first interaction** | Fast query formulation under time pressure | Fails when labels, ASR, or OCR are missing or mismatched |
| **Rich multi-feature interfaces** | Broad coverage across task types | Can become complex, error-prone, and hard for novices |
| **Coarse segmentation** | Lower indexing cost | Misses short KIS targets and reduces timestamp precision |
| **Concept-heavy retrieval** | Strong AVS and textual access | Ambiguity and granularity issues in concept selection |

The “less is more” redesign of diveXplore 5.0 explicitly linked declining performance from VBS2017–VBS2020 to accumulating complexity around a central global featuremap, which had become slow and error-prone [2508.20569]. This is a rare direct articulation of a broader VBS lesson: adding retrieval options does not necessarily improve competitive effectiveness if the resulting interaction costs exceed the gains.

Failure modes are also strikingly consistent across systems. Coarse shot segmentation repeatedly caused missed KIS targets [1805.02371] [2508.20687]. Browsing bottlenecks caused false negatives even when the correct item was retrieved; vitrivr reports that in the 9th visual KIS task in 2019, the correct item had been retrieved but was overlooked while browsing [1902.10647]. Large concept banks could cause ambiguity and excessive reformulation, particularly for novices [1805.11814]. Sketch-based querying was often valuable for some visual KIS tasks but insufficiently selective for large-scale collections [1902.10647] [1806.01081].

## 7. Emerging directions: conversational, question-answering, and domain-robust VBS

Recent VBS editions indicate a broadening of the benchmark’s definition of interactive retrieval. The 2024 introduction of QAS added tasks where a scene in loop plus a textual question required contextual reasoning within a known video, rather than simple scene matching [2502.15683]. The 2025 introduction of KISC went further by formalizing conversational search: teams began with a short textual summary and could ask questions to an oracle panel, with answers shared to all teams simultaneously [2509.12000].

This shift suggests that VBS is increasingly probing mixed-initiative workflows rather than only retrieval speed from fixed queries. A plausible implication is that future high-performing systems may need to unify classic interactive retrieval with conversational state tracking, dynamic query reformulation, and richer cross-modal grounding.

Another clear direction is the integration of large language models and generative components. The 2024 results report identifies TalkSee, VIREO, and Exquisitor as examples of systems emphasizing LLMs, generative models, or conversational search [2502.15683]. However, the reports do not yet establish a definitive correlation between LLM integration and final ranking. The evidence instead remains stronger for interface quality, responsive infrastructure, and multimodal indexing as the common traits of successful systems [2502.15683] [2509.12000].

Finally, domain robustness has become a central concern. The use of MVK and medical videos demonstrates that relatively small but homogeneous or specialized collections can be as challenging as much larger open-domain web video corpora [2209.11518] [2502.15683]. This has implications for both system design and evaluation philosophy. General-purpose embedding models may offer strong initial performance, but VBS increasingly exposes their limitations under domain shift, noisy visual conditions, and task-specific temporal reasoning.

Taken together, the literature portrays VBS as a benchmark that has moved from evaluating interactive segment retrieval on generic web video toward evaluating full-stack human-in-the-loop search systems under conditions of scale, modality diversity, domain specialization, and conversational uncertainty. Its continuing value lies in making those interacting constraints visible in a live setting where retrieval models, user interfaces, operator skill, and evaluation infrastructure all materially affect outcome.

Source: https://www.emergentmind.com/topics/video-browser-showdown