OmniDCBench: Dense Audio-Visual Captioning

• OmniDCBench is a benchmark for omni dense captioning, requiring continuous temporal segmentation and structured, scene-by-scene narratives.
• It employs a six-dimensional annotation schema that covers audio-visual events, background, camera state, shot editing, dialogue, and acoustics to capture rich multimedia details.
• The benchmark integrates human annotations with dynamic programming evaluation (SodaM) to assess both scene segmentation accuracy and caption quality in dense video content.

Searching arXiv for OmniDCBench and closely related benchmark papers to ground the article. arXiv search query: "OmniDCBench TimeChat-Captioner benchmark" OmniDCBench is a human-annotated benchmark for Omni Dense Captioning, a video–language task in which a model must generate a continuous, scene-by-scene, time-stamped, and structured audio-visual narrative for an input video (Yao et al., 9 Feb 2026). In the task formulation, the input video contains both visual frames and audio signals, $V = (F, A) = \{f_1,\dots,f_T\}, \{a_1,\dots,a_T\}$, and the target output is a sequence $C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\}$, where each $t_i$ is a timestamp interval and each $c_i$ is a per-scene structured caption (Yao et al., 9 Feb 2026). The benchmark is designed to evaluate joint time-aware segmentation and fine-grained audio-visual description, with captions organized along six explicit semantic dimensions rather than as unconstrained free-form summaries (Yao et al., 9 Feb 2026).

1. Definition and task formulation

OmniDCBench is built specifically to evaluate Omni Dense Captioning, which the underlying work defines as a task requiring two simultaneous forms of density: temporally dense coverage of the whole video and description dense characterization of each scene (Yao et al., 9 Feb 2026). A video is segmented into a sequence of continuous scenes, each scene is assigned an explicit timestamp in MM:SS format, and each scene receives a multi-sentence structured caption (Yao et al., 9 Feb 2026).

Formally, for a video $V$, the ground-truth script is

$$C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$

where each timestamp is

$t_i = [t_{i,s}, t_{i,e}]$

and each caption is represented as

$$c_i = (\text{Events}_i, \text{Background}_i, \text{Camera}_i, \text{ShotEdit}_i, \text{Dialogue}_i, \text{Acoustic}_i).$$

The predicted output is correspondingly

$$P = \{(\hat{t}_1, \hat{c}_1), \ldots, (\hat{t}_M, \hat{c}_M)\},$$

with $M$ potentially different from $C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\}$0 (Yao et al., 9 Feb 2026).

The task objective is not merely event spotting or summary captioning. Rather, the goal is that a reader can mentally reconstruct the video “scene by scene” as if reading a film script, including both visual and acoustic content (Yao et al., 9 Feb 2026). This emphasis on continuous scene coverage differentiates OmniDCBench from earlier dense-captioning setups that often focus on sparse salient events.

2. Structural schema and annotation targets

Each scene in OmniDCBench is annotated using a six-dimensional schema intended to produce “script-like” captions (Yao et al., 9 Feb 2026). The six dimensions are:

1. Overall Audio-Visual Events (Events / segment_detail_caption): actions, interactions, reactions, temporal relations, and integrated video-audio narration.
2. Background and Environment (Background / video_background): location, atmosphere, lighting, time of day, objects, and layout.
3. Camera State (Camera / camera_state): camera angles, movements, and framing.
4. Multi-shot Editing Style (ShotEdit / shooting_style): cut types, montage, flashback, long take, dissolves, and shot organization.
5. Dialogue Content (Dialogue / speech_content): speaker attribution, key utterances or summaries, and speech acts.
6. Acoustic Cues (Acoustic / acoustics_content): background sounds, sound effects, background music, and speech tone or prosody (Yao et al., 9 Feb 2026).

This schema covers both content and form. The event and background fields capture what is happening and where it occurs, while the camera and shot-editing fields encode cinematic construction. Dialogue and acoustic cues explicitly require audio-aware understanding, making sound a first-class component of evaluation rather than an auxiliary signal (Yao et al., 9 Feb 2026).

The benchmark enforces continuous coverage, explicit timestamps, multi-sentence narratives per scene, and six-dimension fields per scene (Yao et al., 9 Feb 2026). The annotation format is implemented as JSON in prompts and APIs, and model outputs are expected to mirror that structure with fields such as "timestamp", "segment_detail_caption", "video_background", "camera_state", "shooting_style", "speech_content", and "acoustics_content" (Yao et al., 9 Feb 2026).

3. Dataset construction and annotation protocol

OmniDCBench contains 1,122 videos selected to emphasize multi-scene, audio-visual richness (Yao et al., 9 Feb 2026). The sources are Movie101v2, which provides high-resolution movie clips with rich cinematography and narrative complexity, and YT-Temporal-1B, which contributes general YouTube videos with diverse topics and clear audio (Yao et al., 9 Feb 2026). Selection criteria include clear visuals, meaningful audio, multiple scenes, and diversity in content and difficulty (Yao et al., 9 Feb 2026).

The annotation pipeline is a three-stage pipeline (Yao et al., 9 Feb 2026). First, videos undergo video screening & difficulty labeling. Candidate clips are watched by crowd annotators, low-quality or unsuitable content is filtered out, and difficulty tags are assigned for later balancing (Yao et al., 9 Feb 2026). Second, annotators perform scene segmentation with timestamps, segmenting each video into scenes based on continuity of time, location, narrative, changes in visual setting, audio context, or plot progression, and assigning MM:SS start–end intervals (Yao et al., 9 Feb 2026). Third, annotators produce six-dimensional captioning for every scene, with some dimensions potentially handled by different annotators, especially those requiring cinematographic expertise such as camera state and shot-editing style (Yao et al., 9 Feb 2026).

Quality control is explicit. Both timestamps and captions are double-checked by independent annotators, and the paper states: “To further ensure data integrity, both the timestamp and caption annotations are double-checked by independent annotators” (Yao et al., 9 Feb 2026). The annotation interface is described as providing clear instructions and real-time feedback (Yao et al., 9 Feb 2026).

The benchmark’s annotation richness is substantial: the average caption length is ≈995 words per video (Yao et al., 9 Feb 2026). This suggests that the benchmark is designed to assess sustained long-form structured generation rather than brief descriptive snippets. A plausible implication is that performance on OmniDCBench depends not only on local perceptual recognition but also on long-context discourse coherence across scenes.

4. Evaluation protocol and SodaM

OmniDCBench evaluates an integrated task combining multi-scene segmentation and time-aware dense captioning quality (Yao et al., 9 Feb 2026). For segmentation, the reported metrics are F1 and mean IoU over temporal overlaps between predicted and ground-truth scenes (Yao et al., 9 Feb 2026). For caption quality, the benchmark uses per-dimension checklist scores and the unified metric SodaM (Yao et al., 9 Feb 2026).

Timestamp overlap is computed using

$C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\}$1

After alignment, the benchmark reports F1 across thresholds $C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\}$2 and mIoU over matched pairs (Yao et al., 9 Feb 2026).

For caption quality, each ground-truth dimension is decomposed into atomic elements

$C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\}$3

and a judge model, Gemini-2.5-Flash, determines whether each atomic element is covered by the predicted caption (Yao et al., 9 Feb 2026). The CheckList score is

$C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\}$4

This is described as a semantically grounded recall measure rather than an n-gram-overlap metric (Yao et al., 9 Feb 2026).

SodaM extends SODA$C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\}$5 through two mechanisms: dynamic-programming temporal alignment and many-to-one prediction merging (Yao et al., 9 Feb 2026). The dynamic programming table is

$C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\}$6

and is used to align predicted and reference scenes purely from timestamp IoU (Yao et al., 9 Feb 2026). If multiple predicted scenes align to one ground-truth scene, predictions are merged by taking the union of their timestamps and concatenating their captions, while ground-truth scenes are not merged (Yao et al., 9 Feb 2026).

The stated advantages of SodaM are twofold. First, text evaluation complexity is reduced from $C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\}$7 to $C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\}$8, because textual scoring is applied only after temporal alignment (Yao et al., 9 Feb 2026). Second, scene boundary ambiguity is mitigated: a model that splits a human-annotated scene into multiple finer segments is not penalized heavily if the merged prediction covers the same content (Yao et al., 9 Feb 2026). This addresses a common issue in scene-level video annotation, where plausible boundaries may differ without altering semantic fidelity.

5. Baselines, benchmark difficulty, and reported performance

OmniDCBench is the primary evaluation benchmark for TimeChat-Captioner-7B, which is trained for Omni Dense Captioning using TimeChatCap-42K and a two-stage SFT + GRPO regime (Yao et al., 9 Feb 2026). The model uses Qwen2.5-Omni-7B as backbone, combines Qwen2.5-VL and Qwen2-Audio, interleaves audio and visual tokens in time, and uses Multi-modal RoPE for temporal position encoding (Yao et al., 9 Feb 2026). During GRPO, the reward includes format reward $C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\}$9, length reward $t_i$0, timestamp reward $t_i$1, and time-aware caption reward $t_i$2, with final reward

$t_i$3

and weights $t_i$4 (Yao et al., 9 Feb 2026).

The benchmark compares proprietary and open-source baselines, including Gemini-2.5-Pro, Gemini-2.5-Flash, LongVALE, Qwen2.5-Omni, MiniCPM-o-2.6, OmniVinci, Qwen3-Omni Instruct, and the two TimeChat-Captioner checkpoints (Yao et al., 9 Feb 2026). Reported scene segmentation (F1 / mIoU) scores include 68.5 / 74.9 for Gemini-2.5-Pro, 62.4 / 70.8 for TimeChat-Captioner-7B-SFT, and 61.2 / 69.6 for TimeChat-Captioner-7B-GRPO (Yao et al., 9 Feb 2026). Reported SodaM scores are 33.7 for Gemini-2.5-Pro, 32.6 for TimeChat-Captioner-7B-SFT, and 35.0 for TimeChat-Captioner-7B-GRPO (Yao et al., 9 Feb 2026).

The paper attributes the difficulty of OmniDCBench to several requirements: temporal consistency and segmentation, fine-grained, long-form descriptions, multi-scene structure, audio-visual fusion, and explicit treatment of cinematic aspects such as camera state and editing style (Yao et al., 9 Feb 2026). Qualitative examples indicate that general-purpose multimodal models may hallucinate character identity or focus on peripheral details while missing the central event (Yao et al., 9 Feb 2026). This suggests that the benchmark is not merely testing generic caption fluency but cross-scene grounding, identity consistency, and alignment between narrative salience and audiovisual evidence.

6. Relation to prior benchmarks and the benchmark-design lineage

OmniDCBench occupies a distinct position relative to prior video-language benchmarks. The paper contrasts it with representative earlier tasks such as ActivityNet Captions, Vid2Seq, LongVALE, ARC-Chapter, YouCook, and TVQA (Yao et al., 9 Feb 2026). The principal differences are continuous omni-modal coverage, a structured six-dimension schema, time-aware script-style annotations, audio as a first-class citizen, and unified evaluation of segmentation plus caption content (Yao et al., 9 Feb 2026).

The naming of OmniDCBench also invites comparison with an earlier benchmark tradition in systems research. DCBench was introduced as a benchmark suite for datacenter computing with both data analysis workloads and service workloads, motivated by the claim that “only one application is not enough” to represent the diversity of real datacenter behavior (Jia et al., 2013). BigDataBench extended that design logic by emphasizing diversity across the “4V”s of big data, broad application domains, multiple application types, scalable realistic data, and multiple software stacks (Wang et al., 2014). Those earlier works target architecture, systems, and data-management evaluation rather than video-language modeling, but they share a common benchmark philosophy: representativeness, diversity, realistic data, extensibility, and explicit methodology (Jia et al., 2013, Wang et al., 2014).

This historical connection should not be overstated. OmniDCBench is not a datacenter benchmark in the sense of DCBench or BigDataBench; it is a benchmark for time-aware, structured audio-visual captioning (Yao et al., 9 Feb 2026). However, a plausible implication is that the choice of the “Bench” suffix and the emphasis on joint evaluation reflect a broader research norm inherited from benchmark-suite design: a benchmark is treated as a structured research instrument rather than merely a static dataset.

7. Practical use, transfer effects, and limitations

OmniDCBench is used in the source work as a pure evaluation benchmark, while the training set TimeChatCap-42K is described as disjoint and synthetic (Yao et al., 9 Feb 2026). The announced implementation details include 2 FPS frame sampling, 32K tokens context length, JSON-style structured outputs, and an evaluation pipeline with dynamic-programming temporal alignment and checklist evaluation using Gemini-2.5-Flash as a judge model (Yao et al., 9 Feb 2026). Datasets, models, and code are announced as publicly available through the project repository (Yao et al., 9 Feb 2026).

A notable result is that optimization for OmniDCBench correlates with downstream improvements. In the paper’s pipeline for audio-visual reasoning, dense captions generated by caption models are read by Gemini-2.5-Pro to answer questions in Daily-Omni and WorldSense, and the captions from TimeChat-Captioner-7B-GRPO yield 52.8 on Daily-Omni and 22.6 on WorldSense (Yao et al., 9 Feb 2026). For Charades-STA, the model achieves [email protected]: 68.7, [email protected]: 48.3, and mIoU: 58.8 after fine-tuning for temporal grounding (Yao et al., 9 Feb 2026). The authors argue that training on Omni Dense Captioning improves temporal understanding and transfer to other audiovisual tasks (Yao et al., 9 Feb 2026).

The paper also records ethical and practical caveats. The videos come from public academic datasets rather than scraped private material, and the release is described as operating under responsible-use licenses, with risks including inherited biases and misuse of detailed video descriptions for surveillance or misinformation (Yao et al., 9 Feb 2026). At the same time, the work highlights potential benefits for accessibility and education (Yao et al., 9 Feb 2026).

A common misconception would be to view OmniDCBench as only a denser version of conventional dense video captioning. The benchmark’s core novelty lies not simply in longer captions, but in the conjunction of continuous scene segmentation, explicit timestamps, six-field structured semantics, and audio-visual-cinematic coverage (Yao et al., 9 Feb 2026). Another misconception would be to interpret its results as pure segmentation performance; in fact, the benchmark is explicitly designed so that segmentation quality and semantic coverage are evaluated jointly through SodaM (Yao et al., 9 Feb 2026).