SodaM: Metric for Omni Dense Captioning

• SodaM is a unified metric evaluating time-aware, structured, multi-scene audio-visual narrations through temporal alignment and checklist-based caption quality scoring.
• It combines IoU-based temporal segmentation with per-dimension evaluation across Events, Background, Camera, ShotEdit, Dialogue, and Acoustic cues.
• SodaM serves as both a benchmark metric and a reinforcement learning reward to improve captioning accuracy in multi-scene video narration.

SodaM is a unified metric proposed for the Omni Dense Captioning task in "TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions" (Yao et al., 9 Feb 2026). It is designed to evaluate time-aware detailed descriptions while mitigating scene boundary ambiguity. In this setting, the output is a continuous, fine-grained, and structured audio-visual narrative with explicit timestamps, and SodaM jointly evaluates temporal alignment and semantic coverage for multi-scene, script-like captions.

1. Definition and task setting

SodaM is introduced in the context of Omni Dense Captioning, where a model must segment a video into continuous scenes and generate a detailed paragraph for each scene with explicit timestamps (Yao et al., 9 Feb 2026). The task is formulated over a video $V$ with frames and audio,

$$V \rightarrow F = \{f_1, \ldots, f_T\},\ A = \{a_1, \ldots, a_T\},$$

and an output script

$$C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$

where $t_i$ is a timestamp range and $c_i$ is a multi-dimensional scene caption.

Each scene description is structured along six dimensions:

1. Overall Audio-Visual Events (Events)
2. Background and Environment (Background)
3. Camera State (Camera)
4. Multi-shot Editing Style (ShotEdit)
5. Dialogue Content (Dialogue)
6. Acoustic Cues (Acoustic)

The benchmark associated with this task, OmniDCBench, contains 1,122 videos with continuous multi-scene timestamps and full 6-D captions, with average $\sim 995$ words per video (Yao et al., 9 Feb 2026). Within this benchmark, SodaM is the primary metric reflecting the quality of temporally-aligned, multi-dimensional captions.

2. Rationale and design objective

SodaM was introduced because Omni Dense Captioning is substantially more complex than conventional dense video captioning (Yao et al., 9 Feb 2026). The outputs are script-like, long, time-stamped, and explicitly structured. Standard text metrics such as BLEU, METEOR, CIDEr, and SPICE are described as ill-suited for paragraph-length, multi-dimensional descriptions, and they are not time-aware. The metric is therefore designed around two requirements.

First, it must evaluate time-aware detailed descriptions. This means rewarding outputs that both segment the video into scenes with accurate timestamps and provide detailed multi-dimensional text per segment.

Second, it must mitigate scene boundary ambiguity. Different annotators or models may use different numbers of segments and place boundaries slightly differently even when describing essentially the same event. SodaM addresses this by aligning predicted and ground-truth scenes flexibly, allowing many-to-one or one-to-many relationships, using temporal overlap to match segments, and then evaluating semantic coverage within those aligned segments (Yao et al., 9 Feb 2026).

The paper explicitly characterizes SodaM as a two-stage metric combining temporal IoU-based alignment with a CheckList-based caption quality score. This design is intended to make the metric both efficient and robust under ambiguous scene boundaries.

3. Formal definition and computation

Let the prediction be

$$P = \{(\hat{t}_1, \hat{c}_1), \ldots, (\hat{t}_M, \hat{c}_M)\},$$

and the ground truth be

$G = \{(t_1, c_1), \ldots, (t_N, c_N)\},$

where $M$ and $N$ may differ (Yao et al., 9 Feb 2026). SodaM is computed through per-pair scoring, temporal alignment, prediction merging, and aggregation.

Per-pair timestamp accuracy

For a predicted timestamp $$V \rightarrow F = \{f_1, \ldots, f_T\},\ A = \{a_1, \ldots, a_T\},$$0 and ground-truth timestamp $$V \rightarrow F = \{f_1, \ldots, f_T\},\ A = \{a_1, \ldots, a_T\},$$1, temporal Intersection over Union is defined as

$$V \rightarrow F = \{f_1, \ldots, f_T\},\ A = \{a_1, \ldots, a_T\},$$2

Per-pair caption quality

For each dimension $$V \rightarrow F = \{f_1, \ldots, f_T\},\ A = \{a_1, \ldots, a_T\},$$3, the ground-truth scene caption $$V \rightarrow F = \{f_1, \ldots, f_T\},\ A = \{a_1, \ldots, a_T\},$$4 is decomposed into atomic elements

$$V \rightarrow F = \{f_1, \ldots, f_T\},\ A = \{a_1, \ldots, a_T\},$$5

Given a predicted caption $$V \rightarrow F = \{f_1, \ldots, f_T\},\ A = \{a_1, \ldots, a_T\},$$6, the CheckList score is defined as

$$V \rightarrow F = \{f_1, \ldots, f_T\},\ A = \{a_1, \ldots, a_T\},$$7

where $$V \rightarrow F = \{f_1, \ldots, f_T\},\ A = \{a_1, \ldots, a_T\},$$8 indicates whether the predicted caption covers element $$V \rightarrow F = \{f_1, \ldots, f_T\},\ A = \{a_1, \ldots, a_T\},$$9 (Yao et al., 9 Feb 2026). The judge model is Gemini-2.5-Flash.

Temporal alignment by dynamic programming

Since $$C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$0 in general, SodaM first aligns predicted scenes to ground-truth scenes using dynamic programming with temporal IoU as the matching score:

$$C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$1

This alignment stage uses only temporal IoU, not text, which makes it inexpensive relative to LLM-based judging (Yao et al., 9 Feb 2026).

Many-to-one merging

After alignment, multiple predicted scenes may align to the same ground-truth scene. SodaM handles this by merging those predictions:

$$C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$2

$$C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$3

The paper states that only predictions are merged while keeping the ground truth unchanged to ensure evaluation fairness (Yao et al., 9 Feb 2026).

After merging, one obtains a set of aligned pairs

$$C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$4

Aggregation into SodaM

For each matched pair, SodaM evaluates caption quality with the CheckList procedure and then computes an F1 score for all pairs following $$C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$5 (Yao et al., 9 Feb 2026). The paper describes this as a story-level, time-aware F1 over ground-truth keypoints aggregated across scenes and dimensions after temporal alignment.

The same alignment is also used to report timestamp metrics: F1 at IoU thresholds $$C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$6 and mean IoU. These are reported separately as Multi-Scene Seg.: F1 & mIoU rather than being folded into the SodaM scalar (Yao et al., 9 Feb 2026).

4. Temporal robustness and boundary ambiguity

A central property of SodaM is that it is explicitly temporal in two senses (Yao et al., 9 Feb 2026). First, alignment is based on temporal IoU, so only segments that overlap in time can accumulate high alignment scores. Second, the benchmark also reports timestamp accuracy through F1 at multiple IoU thresholds and mean IoU, using the same alignment path.

Its treatment of boundary ambiguity has three components. Soft matching via IoU avoids requiring exact boundary coincidence. Dynamic programming allows skipping segments in either sequence and selects a global path that maximizes total IoU rather than imposing a rigid one-to-one mapping. Many-to-one merging then consolidates several finer-grained predicted scenes into a single aligned prediction when they correspond to the same human-annotated scene.

This design matters because scene granularity is not stable across annotators or models. Human annotations may treat a temporally extended event as one scene, while a model may split it into two or three smaller segments. SodaM is constructed to evaluate such cases holistically rather than penalizing them as categorical mismatches.

The paper also contrasts SodaM with a direct application of $$C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$7. Compared to $$C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$8, SodaM reduces judge-model cost from $$C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$9 to $t_i$0 where $t_i$1 by decoupling IoU matching from text evaluation, and it gracefully handles many-to-one alignments through merging (Yao et al., 9 Feb 2026).

5. Role in training and benchmark evaluation

SodaM is not only an offline metric; it is also used as a reward component in GRPO training for TimeChat-Captioner-7B (Yao et al., 9 Feb 2026). In that setup, the reward consists of four parts:

• format reward $t_i$2
• length reward $t_i$3
• timestamp reward $t_i$4
• time-aware caption reward $t_i$5

The paper states that $t_i$6 adopts the unified SodaM metric as the reward to encourage comprehensive and temporally-aligned structural captions. The full reward is

$t_i$7

with default weights $t_i$8, $t_i$9, $c_i$0, and

$c_i$1

(Yao et al., 9 Feb 2026).

This coupling between evaluation and reinforcement learning gives SodaM a dual role. As a benchmark metric, it measures temporally aligned semantic coverage. As a reward, it biases the model toward captions that jointly satisfy timestamp accuracy and six-dimensional content coverage.

The ablation reported in the paper shows that removing the SodaM caption reward degrades benchmark performance: GRPO without $c_i$2 yields OmniDCBench SodaM $c_i$3, while GRPO with $c_i$4 yields OmniDCBench SodaM $c_i$5 (Yao et al., 9 Feb 2026). The same paper also states that TimeChat-Captioner-7B achieves state-of-the-art performance, surpassing Gemini-2.5-Pro.

6. Properties, limitations, and disambiguation

SodaM is presented as better aligned with Omni Dense Captioning than conventional n-gram metrics because it combines temporal matching with checklist-based semantic recall across Events, Background, Camera, ShotEdit, Dialogue, and Acoustic (Yao et al., 9 Feb 2026). Its strengths, as explicitly described, include dense semantic coverage, robustness to temporal ambiguity, and improved efficiency relative to $c_i$6.

Several limitations are stated or can be marked as plausible implications. An explicit dependency is that CheckList and thus SodaM rely on Gemini-2.5-Flash as judge model. A plausible limitation is sensitivity to judge-model biases or paraphrase failures, because the metric depends on model-based recognition of atomic keypoints. A plausible limitation is that SodaM, being recall-oriented, may favor very long captions that mention many keypoints at once. A plausible limitation is dependence on the quality of the decomposition of ground-truth captions into atomic keypoints per dimension (Yao et al., 9 Feb 2026).

A common source of confusion is the name itself. In contemporary arXiv usage, SodaM is not interchangeable with SODA. The exact string “SodaM” does not appear in "Spectrum-Aware Parameter Efficient Fine-Tuning for Diffusion Models" (Zhang et al., 2024), where the method is SODA for spectral adaptation of diffusion models; it does not appear in "Optimistic Dual Averaging Unifies Modern Optimizers" (Pethick et al., 11 May 2026), where the named methods are SODA, MODA, and SODA wrappers; and it does not appear in "SODA: Semantic-Oriented Distributional Alignment for Generative Recommendation" (Xue et al., 28 Feb 2026), where SODA denotes a recommendation objective. In the present context, SodaM specifically names the metric for time-aware, structurally rich, multi-scene audio-visual caption evaluation introduced in TimeChat-Captioner (Yao et al., 9 Feb 2026).

SodaM therefore occupies a distinct position: it is a benchmark and reward metric for temporally grounded, multi-dimensional video narration rather than a fine-tuning method, an optimizer framework, or a recommendation objective.