diveXplore: Deep Interactive Video Exploration
- diveXplore is a distributed, interactive video exploration system that integrates deep learning, self-organizing maps, and modular concept-based search for ad-hoc video browsing.
- The system employs advanced retrieval methods including shot segmentation, temporal context search, and OpenCLIP-based free-text and visual similarity search to enhance query effectiveness.
- Key design evolutions focus on reducing complexity while improving interaction efficiency, making diveXplore a robust tool for competitive environments like the Video Browser Showdown.
diveXplore is a distributed, interactive video exploration system, and more specifically a feature-rich Deep Interactive Video Exploration system, developed by the Institute of Information Technology (ITEC) at Klagenfurt University for flexible, ad-hoc video search and browsing and for competitive interactive retrieval settings such as the Video Browser Showdown (VBS). Across the versions documented for VBS2018, VBS2021, VBS2022, and VBS2024, the system evolves from a broad set of retrieval and collaboration features centered on self-organizing feature maps toward a leaner, modular, web-based architecture that combines concept search, map search, temporal context search, and, in the 2024 revision, OpenCLIP-based free-text and visual similarity search (Schoeffmann et al., 2018, Leibetseder et al., 28 Aug 2025, Leibetseder et al., 28 Aug 2025, Schoeffmann et al., 28 Aug 2025).
1. Historical trajectory and competitive setting
diveXplore was formerly known as CoViSS and was used by team ITEC1 for the Video Browser Showdown 2018. It was described as a longstanding participating system in the annual Video Browser Showdown from VBS2017 to VBS2020 and in two iterations of the Lifelog Search Challenge, LSC2018 and LSC2019. The system’s development is tightly coupled to competitive interactive retrieval, and the papers consistently frame changes in terms of search effectiveness, browsing speed, and usability under time pressure (Schoeffmann et al., 2018, Leibetseder et al., 28 Aug 2025).
The documented version history shows a clear sequence of design emphases.
| Version context | Reported system state | Reported emphasis |
|---|---|---|
| VBS2018 | diveXplore (formerly CoViSS) | flexible, ad-hoc video search and browsing |
| VBS2021 / version 5.0 | completely rebuilt version 5.0, implemented from scratch | greatly reducing the system's complexity and keeping proven useful features in a modular manner |
| VBS2022 / version 6.0 | less feature rich, yet, more modern, leaner and faster than the original system | shot segmentation, map search, concept search, temporal context search |
| VBS2024 | largely revised system | OpenCLIP, query server, fast browsing UI, exploration view |
The 2018 report states that the system secured second place at both VBS 2017 and VBS 2018. Later reports characterize version 5.0 as a response to declining performance after progressive feature accumulation, while version 6.0 is presented as a continuation of the leaner architecture introduced in 5.0. This suggests that the project’s central technical problem was not only retrieval quality, but also the interaction cost imposed by coupling many functions to a single core visualization (Schoeffmann et al., 2018, Leibetseder et al., 28 Aug 2025, Leibetseder et al., 28 Aug 2025).
2. Early design: self-organizing maps, multimodal retrieval, and collaboration
The 2018 system centered on feature map browsing. Users could browse maps of the entire dataset, arranged either by similar color or by similar high-level features, defined as semantic classification results from CNNs. The maps were laid out using a self-organizing map algorithm guided by either color or semantic similarity. Pre-filtered feature maps highlighted only keyframes with specific characteristics such as faces, text, or other semantic concepts, and a textual search feature supported navigation across the large number of maps (Schoeffmann et al., 2018).
Several additional retrieval modes were already present. The system included a storyboard or shot view for quickly browsing each video, a color filter, textual concept search, similarity search by selecting a keyframe, and a color-sketch feature. Compared to earlier versions that used uniformly sampled thumbnails, the shot view allowed users to jump directly to relevant points and supported variable playback with different speeds. The interface also maintained a history of search results and a dedicated similarity tab to track outcomes of recent similarity-based searches, supporting iterative and back-and-forth search workflows (Schoeffmann et al., 2018).
A notable feature of the early system was explicit support for multi-user search. In collaborative settings, users could see each other's positions on feature maps and send hints about relevant shots. A special SpectatorView summarized the status of all collaborators. The 2018 analysis further distinguishes two user groups, experts and novices. Experts tended to utilize almost the full spectrum of features, combining concept search, map search, map browsing, similarity search, shot filtering, and video inspection, whereas novices relied primarily on textual concept search and only sometimes used map search and browsing for AVS tasks. This suggests that diveXplore was designed as a single environment that could expose high interaction bandwidth to expert users while preserving a simpler concept-driven path for less experienced operators (Schoeffmann et al., 2018).
3. Version 5.0: complete rebuild and modularization
The 5.0 paper identifies a structural limitation in earlier versions. After initial successful employment as a competitive tool, performance declined as new features were introduced increasing overall complexity. The reported explanation is that many additions to the system needed to revolve around the system's core element, an interactive self-organizing browseable featuremap, which, as an integral component, did not accommodate the addition of new features well. The consequences were described as sluggish performance, error-proneness, and difficulty in maintenance and extension (Leibetseder et al., 28 Aug 2025).
Version 5.0 therefore constitutes a completely rebuilt version, implemented from scratch with the aim of greatly reducing the system's complexity as well as keeping proven useful features in a modular manner. The redesign decouples core system components and removes unwieldy dependencies around the featuremap interface. Backend processes such as preprocessing and feature extraction are separated from the interactive frontend, and the modular codebase is described as plug-and-play, with features easier to maintain, extend, and optimize (Leibetseder et al., 28 Aug 2025).
The architectural stack for 5.0 is explicitly web-based. Preprocessing splits the V3C1 dataset into custom shots and uniformly-sampled frames at 1 second intervals, followed by feature extraction using hand-crafted and deep learning based descriptors for concepts, colors, textures, and motions. The backend uses two NodeJS servers, one for serving keyframes, thumbnails, and featuremaps, and one as a RESTful middleware for client-server communication, with MongoDB as the database. The frontend is an AngularJS client application for interactive search and exploration (Leibetseder et al., 28 Aug 2025).
A central redesign concerns the featuremap itself. Instead of a single, massive, core featuremap, 5.0 moves to concept-based smaller featuremaps. Users search individual deep concepts such as “car” and receive a set of smaller, concept-focused maps. Each map can be organized by similarity, including concept, color, texture, and motion, by confidence, or by video affiliation. The system also introduces multiple, independently-updateable views for storyboards and shot navigation, concept search over deep neural network outputs, and shot-based and frame-based granularity in result ordering. These views do not require reloading or synchronizing the featuremap on each interaction. In addition, a video-based similarity filter allows users to filter and group videos by metadata such as creation period, by frequency or confidence of detected concepts, and by combinations of multiple filters (Leibetseder et al., 28 Aug 2025).
4. Version 6.0: finer segmentation and temporal-context retrieval
Version 6.0 is presented as an extension of the refactored 5.0 architecture. The paper states that since version 5.0 the system is less feature rich, yet, more modern, leaner and faster than the original system, and that this proved to be a sensible decision because the new system showed increasing performance in VBS2021 when compared to the most recent former competitions. The explicit goals for 6.0 are reconsidering shot segmentation and map search and introducing new features for improving concept as well as temporal context search (Leibetseder et al., 28 Aug 2025).
The architecture remains modular and web-based, divided into preprocessing, back end, and front end. Preprocessing extracts metadata and deep features from video keyframes and ten-second video segments. Shot, map, and video information is hosted on a Node.js server, while feature collections are stored in MongoDB and exposed through middleware as a JSON-based REST API. The web UI comprises two distinct interfaces for shot and map search and communicates with the backend via POST requests, enabling combinable queries and complex logic such as feature selection and thresholds (Leibetseder et al., 28 Aug 2025).
The feature inventory reported for 6.0 is broader and more explicitly model-based. Concepts are extracted with Inception v3 and correspond to 1000 ImageNet categories; objects are extracted with YOLO v4 and correspond to 80 MS COCO categories; events use Moments in Time with 304 action events; places use Places365 with 365 categories; and similarity at the map level is calculated from Inception v3’s last fully connected layer. Concepts, objects, and places are extracted from keyframes, whereas events require temporal context from full ten-second segments (Leibetseder et al., 28 Aug 2025).
The most emphasized 6.0 change is shot segmentation. Previous versions used uniform 10 second segmentation for all videos; version 6.0 moves to one-second segmentation intervals. The stated motivation is that 10 second segments made it nearly impossible to retrieve targets in very short segments, especially for Known Item Search tasks. The paper also describes improved map search through alternative retrieval strategies beyond simple frequency counts, and an enhanced concept interface with smart, bash-like autocomplete and multi-feature combination queries. Additional search modalities include OCR and STT, leveraging Tesseract and Kaldi. Interface enhancements include search history, context menus, and a video preview overlay with video metadata and top-scoring objects, concepts, events, and places (Leibetseder et al., 28 Aug 2025).
5. Version 2024: OpenCLIP retrieval, federated querying, and cluster exploration
For VBS2024, diveXplore is described as having undergone significant revisions based on experience from VBS2023 and feedback from the IVR4B special session at CBMI2023. The updated system integrates OpenCLIP trained on the LAION-2B dataset for image and text embeddings, and these embeddings are used for free-text search and visual similarity search. The summary specifies the OpenCLIP ViT-H/14 variant and reports 1024-dimensional image/text embeddings (Schoeffmann et al., 28 Aug 2025).
The retrieval backend is also revised. A new query server implemented in Node.js can distribute queries to multiple backends, such as FAISS for embeddings or MongoDB for metadata, and merge the results using various strategies including rank fusion and filtering. The architecture supports parallel and federated querying, for example combining free-text retrieval with metadata or object filters. Frontend-to-server communication is via WebSocket, and the papers emphasize interactive and fast updates (Schoeffmann et al., 28 Aug 2025).
The 2024 interface is optimized for fast browsing. Reported features include keyboard shortcuts for result browsing, summary inspection, and video playback; on-hover interaction for quick context exploration, where horizontal mouse movement previews temporally adjacent keyframes; and an “expert mode” in the search bar for advanced, mixed-modality queries using prefixes such as -c, -o, -e, and -t. The system also adds an exploration view, or “explore” mode, for large clusters of visually similar videos, including examples such as weddings, paraglider events, and snow and ice scenery. The stated purpose is to support browsing when the search need is exploratory and to aid tasks requiring diversity or discovery of repeated content patterns (Schoeffmann et al., 28 Aug 2025).
In this revision, keyframes are embedded and indexed in FAISS for efficient nearest-neighbor search. The data summary further states that videos are segmented into shots using TransNetv2 for the V3C dataset and custom methods for surgery and marine collections. This suggests a gradual shift from a system organized primarily around concept maps toward one that can unify embedding search, metadata filtering, temporal constraints, and exploratory cluster navigation within a federated query layer (Schoeffmann et al., 28 Aug 2025).
6. Usage patterns, design philosophy, and technical significance
Across the documented versions, diveXplore repeatedly confronts the trade-off between feature breadth and interaction efficiency. The early system explicitly endorsed a “flexible system with many different retrieval options,” and the 2018 user analysis found that experts benefited from combining multiple features while novices relied mainly on concept search. Later papers report the opposite pressure: additional features increased code and UI coupling, especially when every new view or result needed to realign with the core map, and this harmed responsiveness and maintainability (Schoeffmann et al., 2018, Leibetseder et al., 28 Aug 2025).
The 5.0 and 6.0 papers crystallize this shift into a design philosophy summarized by the phrase “Less is more.” In 5.0, retention of only proven useful features, loose coupling of features, and independent views are presented as remedies for sluggish performance and error-proneness. In 6.0, the system remains less feature rich than earlier generations but adds back features that are framed as directly improving search success, such as one-second segmentation, alternative map retrieval, and temporal-context search. The 2024 revision extends this logic by introducing a query server that can distribute and merge different query types while keeping the interface optimized for fast browsing (Leibetseder et al., 28 Aug 2025, Leibetseder et al., 28 Aug 2025, Schoeffmann et al., 28 Aug 2025).
A common misconception is that diveXplore is defined solely by its self-organizing featuremap. The historical record is more precise. The self-organizing featuremap was historically the centerpiece of the system, but later versions convert it into modular, concept-specific maps and eventually place it alongside shot search, map search, OCR, STT, OpenCLIP-based free-text retrieval, visual similarity search, and an exploration view for similar-video clusters. Another misconception is that the system followed a monotonic expansion in functionality. The papers instead document a cycle of expansion, performance decline, rebuilding, and selective reintroduction of capabilities, with competitive results used as an external validation signal (Leibetseder et al., 28 Aug 2025, Leibetseder et al., 28 Aug 2025, Schoeffmann et al., 28 Aug 2025).
From an interactive retrieval perspective, diveXplore is best understood as a family of systems rather than a fixed interface. Its stable themes are web-based deployment, multi-view interaction, competitive VBS usage, and a recurrent effort to balance expert-oriented combinability with low-friction browsing. Its variable elements are the retrieval substrate: self-organizing maps in the early system, modular concept-focused maps and video-based filters in 5.0, finer segmentation and temporal-event retrieval in 6.0, and OpenCLIP-plus-FAISS with federated query execution in the 2024 revision (Schoeffmann et al., 2018, Leibetseder et al., 28 Aug 2025, Leibetseder et al., 28 Aug 2025, Schoeffmann et al., 28 Aug 2025).