---
title: 'OmniDCBench: Dense Audio-Visual Captioning'
url: https://www.emergentmind.com/topics/omnidcbench
type: topic
---

# OmniDCBench: Dense Audio-Visual Captioning

Searching arXiv for OmniDCBench and closely related benchmark papers to ground the article.
arXiv search query: "OmniDCBench TimeChat-Captioner benchmark"
OmniDCBench is a human-annotated benchmark for **Omni Dense Captioning**, a video–language task in which a model must generate a continuous, scene-by-scene, time-stamped, and structured audio-visual narrative for an input video [2602.08711]. In the task formulation, the input video contains both visual frames and audio signals, \(V = (F, A) = \{f_1,\dots,f_T\}, \{a_1,\dots,a_T\}\), and the target output is a sequence \(C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\}\), where each \(t_i\) is a timestamp interval and each \(c_i\) is a per-scene structured caption [2602.08711]. The benchmark is designed to evaluate joint **time-aware segmentation** and **fine-grained audio-visual description**, with captions organized along six explicit semantic dimensions rather than as unconstrained free-form summaries [2602.08711].

## 1. Definition and task formulation

OmniDCBench is built specifically to evaluate **Omni Dense Captioning**, which the underlying work defines as a task requiring two simultaneous forms of density: **temporally dense** coverage of the whole video and **description dense** characterization of each scene [2602.08711]. A video is segmented into a sequence of continuous scenes, each scene is assigned an explicit timestamp in MM:SS format, and each scene receives a multi-sentence structured caption [2602.08711].

Formally, for a video \(V\), the ground-truth script is
\[
C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},
\]
where each timestamp is
\[
t_i = [t_{i,s}, t_{i,e}]
\]
and each caption is represented as
\[
c_i = (\text{Events}_i, \text{Background}_i, \text{Camera}_i, \text{ShotEdit}_i, \text{Dialogue}_i, \text{Acoustic}_i).
\]
The predicted output is correspondingly
\[
P = \{(\hat{t}_1, \hat{c}_1), \ldots, (\hat{t}_M, \hat{c}_M)\},
\]
with \(M\) potentially different from \(N\) [2602.08711].

The task objective is not merely event spotting or summary captioning. Rather, the goal is that a reader can mentally reconstruct the video “scene by scene” as if reading a film script, including both visual and acoustic content [2602.08711]. This emphasis on continuous scene coverage differentiates OmniDCBench from earlier dense-captioning setups that often focus on sparse salient events.

## 2. Structural schema and annotation targets

Each scene in OmniDCBench is annotated using a **six-dimensional schema** intended to produce “script-like” captions [2602.08711]. The six dimensions are:

1. **Overall Audio-Visual Events (Events / segment_detail_caption)**: actions, interactions, reactions, temporal relations, and integrated video-audio narration.
2. **Background and Environment (Background / video_background)**: location, atmosphere, lighting, time of day, objects, and layout.
3. **Camera State (Camera / camera_state)**: camera angles, movements, and framing.
4. **Multi-shot Editing Style (ShotEdit / shooting_style)**: cut types, montage, flashback, long take, dissolves, and shot organization.
5. **Dialogue Content (Dialogue / speech_content)**: speaker attribution, key utterances or summaries, and speech acts.
6. **Acoustic Cues (Acoustic / acoustics_content)**: background sounds, sound effects, background music, and speech tone or prosody [2602.08711].

This schema covers both **content** and **form**. The event and background fields capture what is happening and where it occurs, while the camera and shot-editing fields encode cinematic construction. Dialogue and acoustic cues explicitly require audio-aware understanding, making sound a first-class component of evaluation rather than an auxiliary signal [2602.08711].

The benchmark enforces **continuous coverage**, **explicit timestamps**, **multi-sentence narratives per scene**, and **six-dimension fields per scene** [2602.08711]. The annotation format is implemented as JSON in prompts and APIs, and model outputs are expected to mirror that structure with fields such as `"timestamp"`, `"segment_detail_caption"`, `"video_background"`, `"camera_state"`, `"shooting_style"`, `"speech_content"`, and `"acoustics_content"` [2602.08711].

## 3. Dataset construction and annotation protocol

OmniDCBench contains **1,122 videos** selected to emphasize **multi-scene, audio-visual richness** [2602.08711]. The sources are **Movie101v2**, which provides high-resolution movie clips with rich cinematography and narrative complexity, and **YT-Temporal-1B**, which contributes general YouTube videos with diverse topics and clear audio [2602.08711]. Selection criteria include clear visuals, meaningful audio, multiple scenes, and diversity in content and difficulty [2602.08711].

The annotation pipeline is a **three-stage pipeline** [2602.08711]. First, videos undergo **video screening & difficulty labeling**. Candidate clips are watched by crowd annotators, low-quality or unsuitable content is filtered out, and difficulty tags are assigned for later balancing [2602.08711]. Second, annotators perform **scene segmentation with timestamps**, segmenting each video into scenes based on continuity of time, location, narrative, changes in visual setting, audio context, or plot progression, and assigning MM:SS start–end intervals [2602.08711]. Third, annotators produce **six-dimensional captioning** for every scene, with some dimensions potentially handled by different annotators, especially those requiring cinematographic expertise such as camera state and shot-editing style [2602.08711].

Quality control is explicit. Both timestamps and captions are **double-checked** by independent annotators, and the paper states: “To further ensure data integrity, both the timestamp and caption annotations are double-checked by independent annotators” [2602.08711]. The annotation interface is described as providing clear instructions and real-time feedback [2602.08711].

The benchmark’s annotation richness is substantial: the average caption length is **≈995 words per video** [2602.08711]. This suggests that the benchmark is designed to assess sustained long-form structured generation rather than brief descriptive snippets. A plausible implication is that performance on OmniDCBench depends not only on local perceptual recognition but also on long-context discourse coherence across scenes.

## 4. Evaluation protocol and SodaM

OmniDCBench evaluates an integrated task combining **multi-scene segmentation** and **time-aware dense captioning quality** [2602.08711]. For segmentation, the reported metrics are **F1** and **mean IoU** over temporal overlaps between predicted and ground-truth scenes [2602.08711]. For caption quality, the benchmark uses **per-dimension checklist scores** and the unified metric **SodaM** [2602.08711].

Timestamp overlap is computed using
\[
\text{IoU}(\hat{t}, t) = \frac{|\hat{t} \cap t|}{|\hat{t} \cup t|}.
\]
After alignment, the benchmark reports **F1** across thresholds \(\{0.3, 0.5, 0.7, 0.9\}\) and **mIoU** over matched pairs [2602.08711].

For caption quality, each ground-truth dimension is decomposed into **atomic elements**
\[
\mathcal{E}_d = \{e_1, e_2, \ldots, e_{|\mathcal{E}_d|}\},
\]
and a judge model, **Gemini-2.5-Flash**, determines whether each atomic element is covered by the predicted caption [2602.08711]. The CheckList score is
\[
\text{CheckList}(\hat{c}, c) = \frac{\sum_{d \in \mathcal{D}} \sum_{i=1}^{|\mathcal{E}_d|} \text{Judge}(\hat{c}, e_i)}{\sum_{d \in \mathcal{D}} |\mathcal{E}_d|}.
\]
This is described as a semantically grounded recall measure rather than an n-gram-overlap metric [2602.08711].

**SodaM** extends SODA\(_c\) through two mechanisms: **dynamic-programming temporal alignment** and **many-to-one prediction merging** [2602.08711]. The dynamic programming table is
\[
S[i][j] = \max
\begin{cases}
S[i-1][j] \\
S[i][j-1] \\
S[i-1][j-1] + \text{IoU}(t_i, \hat{t}_j)
\end{cases}
\]
and is used to align predicted and reference scenes purely from timestamp IoU [2602.08711]. If multiple predicted scenes align to one ground-truth scene, predictions are merged by taking the union of their timestamps and concatenating their captions, while ground-truth scenes are not merged [2602.08711].

The stated advantages of SodaM are twofold. First, text evaluation complexity is reduced from \(O(MN)\) to \(O(K)\), because textual scoring is applied only after temporal alignment [2602.08711]. Second, **scene boundary ambiguity** is mitigated: a model that splits a human-annotated scene into multiple finer segments is not penalized heavily if the merged prediction covers the same content [2602.08711]. This addresses a common issue in scene-level video annotation, where plausible boundaries may differ without altering semantic fidelity.

## 5. Baselines, benchmark difficulty, and reported performance

OmniDCBench is the primary evaluation benchmark for **TimeChat-Captioner-7B**, which is trained for Omni Dense Captioning using **TimeChatCap-42K** and a two-stage **SFT + GRPO** regime [2602.08711]. The model uses **Qwen2.5-Omni-7B** as backbone, combines **Qwen2.5-VL** and **Qwen2-Audio**, interleaves audio and visual tokens in time, and uses **Multi-modal RoPE** for temporal position encoding [2602.08711]. During GRPO, the reward includes format reward \(\mathcal{R}_F\), length reward \(\mathcal{R}_L\), timestamp reward \(\mathcal{R}_T\), and time-aware caption reward \(\mathcal{R}_C\), with final reward
\[
\mathcal{R} = \alpha_f R_F + \alpha_l R_L + \alpha_t R_T + \alpha_c R_C
\]
and weights \(0.5, 0.5, 1.0, 1.0\) [2602.08711].

The benchmark compares proprietary and open-source baselines, including **Gemini-2.5-Pro**, **Gemini-2.5-Flash**, **LongVALE**, **Qwen2.5-Omni**, **MiniCPM-o-2.6**, **OmniVinci**, **Qwen3-Omni Instruct**, and the two TimeChat-Captioner checkpoints [2602.08711]. Reported **scene segmentation (F1 / mIoU)** scores include **68.5 / 74.9** for Gemini-2.5-Pro, **62.4 / 70.8** for TimeChat-Captioner-7B-SFT, and **61.2 / 69.6** for TimeChat-Captioner-7B-GRPO [2602.08711]. Reported **SodaM** scores are **33.7** for Gemini-2.5-Pro, **32.6** for TimeChat-Captioner-7B-SFT, and **35.0** for TimeChat-Captioner-7B-GRPO [2602.08711].

The paper attributes the difficulty of OmniDCBench to several requirements: **temporal consistency and segmentation**, **fine-grained, long-form descriptions**, **multi-scene structure**, **audio-visual fusion**, and explicit treatment of **cinematic aspects** such as camera state and editing style [2602.08711]. Qualitative examples indicate that general-purpose multimodal models may hallucinate character identity or focus on peripheral details while missing the central event [2602.08711]. This suggests that the benchmark is not merely testing generic caption fluency but cross-scene grounding, identity consistency, and alignment between narrative salience and audiovisual evidence.

## 6. Relation to prior benchmarks and the benchmark-design lineage

OmniDCBench occupies a distinct position relative to prior video-language benchmarks. The paper contrasts it with representative earlier tasks such as **ActivityNet Captions**, **Vid2Seq**, **LongVALE**, **ARC-Chapter**, **YouCook**, and **TVQA** [2602.08711]. The principal differences are **continuous omni-modal coverage**, a **structured six-dimension schema**, **time-aware script-style annotations**, **audio as a first-class citizen**, and **unified evaluation of segmentation plus caption content** [2602.08711].

The naming of OmniDCBench also invites comparison with an earlier benchmark tradition in systems research. **DCBench** was introduced as a benchmark suite for datacenter computing with both **data analysis workloads** and **service workloads**, motivated by the claim that “only one application is not enough” to represent the diversity of real datacenter behavior [1307.8013]. **BigDataBench** extended that design logic by emphasizing diversity across the “4V”s of big data, broad application domains, multiple application types, scalable realistic data, and multiple software stacks [1401.1406]. Those earlier works target architecture, systems, and data-management evaluation rather than video-language modeling, but they share a common benchmark philosophy: representativeness, diversity, realistic data, extensibility, and explicit methodology [1307.8013] [1401.1406].

This historical connection should not be overstated. OmniDCBench is not a datacenter benchmark in the sense of DCBench or BigDataBench; it is a benchmark for **time-aware, structured audio-visual captioning** [2602.08711]. However, a plausible implication is that the choice of the “Bench” suffix and the emphasis on joint evaluation reflect a broader research norm inherited from benchmark-suite design: a benchmark is treated as a structured research instrument rather than merely a static dataset.

## 7. Practical use, transfer effects, and limitations

OmniDCBench is used in the source work as a **pure evaluation benchmark**, while the training set **TimeChatCap-42K** is described as disjoint and synthetic [2602.08711]. The announced implementation details include **2 FPS** frame sampling, **32K tokens** context length, JSON-style structured outputs, and an evaluation pipeline with dynamic-programming temporal alignment and checklist evaluation using **Gemini-2.5-Flash** as a judge model [2602.08711]. Datasets, models, and code are announced as publicly available through the project repository [2602.08711].

A notable result is that optimization for OmniDCBench correlates with downstream improvements. In the paper’s pipeline for audio-visual reasoning, dense captions generated by caption models are read by **Gemini-2.5-Pro** to answer questions in **Daily-Omni** and **WorldSense**, and the captions from **TimeChat-Captioner-7B-GRPO** yield **52.8** on Daily-Omni and **22.6** on WorldSense [2602.08711]. For **Charades-STA**, the model achieves **R1@0.5: 68.7**, **R1@0.7: 48.3**, and **mIoU: 58.8** after fine-tuning for temporal grounding [2602.08711]. The authors argue that training on Omni Dense Captioning improves temporal understanding and transfer to other audiovisual tasks [2602.08711].

The paper also records ethical and practical caveats. The videos come from **public academic datasets** rather than scraped private material, and the release is described as operating under **responsible-use licenses**, with risks including inherited biases and misuse of detailed video descriptions for surveillance or misinformation [2602.08711]. At the same time, the work highlights potential benefits for accessibility and education [2602.08711].

A common misconception would be to view OmniDCBench as only a denser version of conventional dense video captioning. The benchmark’s core novelty lies not simply in longer captions, but in the conjunction of **continuous scene segmentation**, **explicit timestamps**, **six-field structured semantics**, and **audio-visual-cinematic coverage** [2602.08711]. Another misconception would be to interpret its results as pure segmentation performance; in fact, the benchmark is explicitly designed so that segmentation quality and semantic coverage are evaluated jointly through SodaM [2602.08711].

Source: https://www.emergentmind.com/topics/omnidcbench