TimeChatCap-42K: Dense Captioning Dataset

• TimeChatCap-42K is a large-scale, time-aware audio-visual captioning dataset that produces continuous, fine-grained, script-like narratives with explicit timestamps.
• It uses a two-stage pipeline with Gemini-2.5-Pro to generate coarse scene segmentation followed by detailed six-dimensional descriptions covering events, background, camera, editing, dialogue, and acoustics.
• The dataset underpins TimeChat-Captioner-7B training and integrates SodaM for evaluating timestamp accuracy and descriptive coverage, setting new benchmarks in dense captioning.

TimeChatCap-42K is a large-scale, time-aware, structured audio-visual captioning dataset created to train and study Omni Dense Captioning, defined as generating continuous, fine-grained, “script-like” narratives for multi-scene videos with explicit timestamps (Yao et al., 9 Feb 2026). Its design targets two forms of density simultaneously: temporally dense annotation through continuous scene segmentation with explicit start–end times, and description-dense annotation through six complementary dimensions per scene across audio and visual semantics. Within the same framework, TimeChatCap-42K functions as the training corpus for TimeChat-Captioner-7B, while OmniDCBench provides fully human-annotated evaluation and SodaM provides a unified metric for timestamp accuracy and time-aware descriptive coverage.

1. Task setting and conceptual scope

TimeChatCap-42K was built for Omni Dense Captioning, a task centered on producing continuous, fine-grained, and structured audio-visual narratives with explicit timestamps for multi-scene videos (Yao et al., 9 Feb 2026). The task is motivated by the need for captions that do more than summarize salient events: they describe videos scene by scene in a manner analogous to a cinematographic screenplay.

The dataset’s notion of “dense” has two explicit facets. The first is temporal density: scenes partition the video continuously, with absolute start–end times and no overlaps. The second is descriptive density: each scene is described through six dimensions spanning events, background, camera, editing, dialogue, and acoustic cues. This structure is intended to provide dense supervision for time-aware audio-visual alignment rather than sparse event-level captioning.

TimeChatCap-42K sits alongside two other components in the same research program. OmniDCBench is a 1,122-video, fully human-annotated benchmark used for fair evaluation and designed to expose challenges such as scene-boundary ambiguity. SodaM is the evaluation metric that jointly assesses timestamp accuracy and time-aware descriptive coverage while mitigating boundary ambiguities through alignment and merging. TimeChat-Captioner-7B is the 7B baseline trained on TimeChatCap-42K via supervised fine-tuning and Group Relative Policy Optimization.

2. Dataset composition and annotation schema

The “42K” in TimeChatCap-42K denotes 42,000 synthetic video–caption pairs, each consisting of a short video clip with audio and a multi-scene, six-dimensional caption set with explicit timestamps (Yao et al., 9 Feb 2026). The videos are multimodal, including both visual frames and audio tracks, and the captions are synthesized with Gemini-2.5-Pro through a two-stage prompting pipeline and then filtered. Source videos are sampled from two public academic datasets: MMTrail-2M, described as diverse trailer videos, and Movie101, described as movie commentary videos.

From the appendix statistics reported for the dataset, 73.9% of clips are 50–60 seconds long, the average segment length is 10.04 seconds, and captions average 877 words per video across the six dimensions. The pipeline initially segments raw videos into approximately 3-minute chunks and then constructs multi-scene annotations; after filtering, the released clips are mostly around one minute, with dense segments that reflect semantic changes.

TimeChatCap-42K enforces a fixed six-dimensional structural schema per scene:

Dimension	Operational field name	Description
Overall Audio-Visual Events	segment_detail_caption	Detailed narration of the scene’s actions and semantics
Background and Environment	video_background	Setting, location, objects, and atmosphere
Camera State	camera_state	Camera movements, angles, framing
Multi-shot Editing Style	shooting_style	Editing techniques and shot organization
Dialogue Content	speech_content	Spoken content with speaker attribution
Acoustic Cues	acoustics_content	Background sounds, music, voice tones

Timestamps are represented as absolute start–end times in minutes:seconds, for example 00:01-00:10, and scenes partition the video continuously without overlaps. Operationally, each scene is stored as a JSON-like object with the fields timestamp, segment_detail_caption, video_background, camera_state, shooting_style, speech_content, and acoustics_content.

A concrete scene example reported for the dataset uses the following structure:

$\{r_1, \ldots, r_G\}$8

The captions are produced in English, as implied by the English prompting and schema. The joint attention to video and audio is explicit: dialogue and acoustics are treated as first-class components rather than auxiliary metadata.

3. Construction pipeline and quality control

The annotation process is automated through a two-stage, coarse-to-fine prompting pipeline with Gemini-2.5-Pro (Yao et al., 9 Feb 2026). In Stage 1, the model proposes a dense segmentation with brief captions, exemplified by a form such as “0:00–0:15: a boy singing…”. In Stage 2, each segment is expanded into detailed six-dimensional text directly from the video’s audio and visuals using the fixed schema.

Filtering is an integral part of the dataset construction. The reported criteria are: removing videos with fewer than two segments, removing videos without audio tracks, discarding samples with JSON formatting errors or missing fields, and filtering out segments below a minimum duration. These constraints are intended to preserve structural consistency and multimodal completeness.

The paper distinguishes sharply between synthetic training supervision and human evaluation. TimeChatCap-42K is synthetic and automatically filtered, whereas OmniDCBench is fully human-annotated, with multi-annotator checks and double verification of timestamps and captions, and is used only for evaluation rather than training. This separation is important because scene boundaries are described as inherently semantic and therefore ambiguous. TimeChatCap-42K uses clear, non-overlapping timestamp strings, while the evaluation protocol is designed to tolerate reasonable prediction–reference mismatches.

A plausible implication is that the dataset prioritizes annotation scale and structural regularity over direct human authorship. The paper explicitly notes that this synthetic supervision can introduce stylistic or content biases and may propagate upstream model errors.

4. Role in training TimeChat-Captioner-7B

TimeChatCap-42K is the training dataset for TimeChat-Captioner-7B, and the reported training process has two stages: supervised fine-tuning and Group Relative Policy Optimization (Yao et al., 9 Feb 2026). The supervised phase uses 40K samples for 2 epochs with learning rate 5e-5, batch size 128, 32K-token context, and 2 FPS video sampling. The objective is next-token prediction over the structured target, comprising timestamps together with the six descriptive dimensions.

The second stage, GRPO, is motivated by two reported issues: timestamps are only a small fraction of tokens, approximately 0.7%, so cross-entropy under-optimizes temporal accuracy; and models may overfit to typical scene counts. GRPO therefore directly optimizes time-aware quality. Using sampled responses $\{o_1, \ldots, o_G\}$ per input $q$ from the current policy $\pi_{\theta_{\text{old}}}$ and rewards $\{r_1, \ldots, r_G\}$, the group-relative advantages are defined as:

$$A_i = \frac{r_i - \operatorname{mean}(\{r_k\})}{\operatorname{std}(\{r_k\})}$$

The GRPO objective is given as:

$$J_{\text{GRPO}}(\theta) = \mathbb{E} \left[ \frac{1}{G} \sum_{i=1}^{G} \min\!\left( \frac{\pi_\theta(o_i|q)}{\pi_{\theta_{\text{old}}}(o_i|q)} A_i,\; \operatorname{clip}\!\left(\frac{\pi_\theta}{\pi_{\theta_{\text{old}}}}, 1-\epsilon, 1+\epsilon\right) A_i \right) - \beta D_{KL}(\pi_\theta \| \pi_{\text{ref}}) \right]$$

The reward is a weighted sum:

$$R = \alpha_f R_F + \alpha_l R_L + \alpha_t R_T + \alpha_c R_C$$

The components are specified as follows. The format reward $R_F$ equals 1 if the output is a valid, parseable JSON list of segments and 0 otherwise. The length reward $R_L$ penalizes overlong outputs as a regularizer against repetitive or stalled generations. The timestamp reward $R_T$ is average F1 over IoU thresholds $q$0 using the optimal alignment path. The time-aware caption reward $q$1 is the SodaM score computed on aligned pairs to encourage temporally grounded, multi-dimensional detail. Default weights during GRPO are $q$2, $q$3, $q$4, and $q$5.

The GRPO setup uses 2K training samples, 1 epoch, learning rate 1e-5, batch size 64, rollout size G=8, 2 FPS, and 32K-token context. This arrangement makes TimeChatCap-42K not merely an annotation resource but a training substrate whose structure is directly reflected in both the supervised targets and the reinforcement learning reward design.

5. Coupling with SodaM and evaluation methodology

TimeChatCap-42K is tightly coupled to SodaM, the time-aware evaluation metric developed for Omni Dense Captioning (Yao et al., 9 Feb 2026). Let predictions be $q$6 and ground truth be $q$7. Each timestamp $q$8 uses absolute times in seconds, though presented as mm:ss, and the caption dimensions $q$9 are the six schema fields.

The per-pair temporal overlap is measured by timestamp IoU:

$\pi_{\theta_{\text{old}}}$0

Alignment is then obtained by dynamic programming over the $\pi_{\theta_{\text{old}}}$1 grid using only IoU scores and allowing many-to-one and one-to-many alignments. The recurrence is:

$\pi_{\theta_{\text{old}}}$2

If multiple predicted segments $\pi_{\theta_{\text{old}}}$3 align to a single ground-truth scene $\pi_{\theta_{\text{old}}}$4, the predictions are merged for caption evaluation. The merged timestamp is

$\pi_{\theta_{\text{old}}}$5

and the merged caption is

$\pi_{\theta_{\text{old}}}$6

This procedure preserves ground-truth scenes while consolidating over-segmentation on the prediction side. It is explicitly intended to mitigate boundary ambiguity.

Caption quality is assessed dimension by dimension using checklist-based scoring. For each dimension $\pi_{\theta_{\text{old}}}$7, the ground-truth caption $\pi_{\theta_{\text{old}}}$8 is decomposed into atomic keypoints $\pi_{\theta_{\text{old}}}$9. A judge model returns a binary decision $\{r_1, \ldots, r_G\}$0 indicating whether the predicted caption covers the keypoint. The Checklist score for dimension $\{r_1, \ldots, r_G\}$1 is:

$\{r_1, \ldots, r_G\}$2

SodaM reports temporal accuracy through F1 over IoU thresholds $\{r_1, \ldots, r_G\}$3 and mean IoU after alignment. Caption quality is reported through per-dimension F1, following SODA, for Camera, Events, Background, Acoustics, ShotEdit, and Dialogue. The main SodaM (Avg.) score is the average of these six per-dimension F1s. The paper further states that SodaM reduces judge-model cost from $\{r_1, \ldots, r_G\}$4 to $\{r_1, \ldots, r_G\}$5 after alignment, where $\{r_1, \ldots, r_G\}$6 matched pairs.

In practical usage, the reported protocol is to run a model to produce JSON outputs, perform dynamic-programming alignment using timestamp IoU, merge many-to-one predicted scenes, compute timestamp F1 and mean IoU plus the six Checklist-based F1 scores, and average the six dimension scores to obtain SodaM (Avg.). The paper also recommends strict JSON validation and, for RL training, the inclusion of format and length rewards to stabilize long generations.

6. Empirical performance, differentiation from prior datasets, and reported limitations

Training on TimeChatCap-42K yields a specialist model that the paper describes as state of the art among open-source systems for time-aware dense captioning and as surpassing Gemini-2.5-Pro on the primary SodaM metric (Yao et al., 9 Feb 2026). On OmniDCBench, TimeChat-Captioner-7B-GRPO achieves SodaM (Avg.) 35.0, compared with 33.7 for Gemini-2.5-Pro. For scene segmentation quality, TimeChat-Captioner-7B-GRPO records F1 61.2 and mIoU 69.6, while Gemini-2.5-Pro remains higher on segmentation F1 and mean IoU but lower on caption quality.

The reported downstream transfer results are also tied to captions generated from this training regime. For VideoQA transfer, where captions are used by a QA system, the model achieves 52.8 on Daily-Omni and 22.6 on WorldSense. For temporal grounding on Charades-STA fine-tuning, the reported metrics are [email protected] = 68.7, [email protected] = 48.3, and mIoU = 58.8, described as consistently stronger than Qwen2.5-Omni-7B and several expert models. Ablation studies further report that scaling SFT data from 20K to 40K improves OmniDCBench from 31.3 to 32.6, and that adding the SodaM reward $\{r_1, \ldots, r_G\}$7 during GRPO improves OmniDCBench from 32.5 to 35.0 and Daily-Omni from 50.4 to 52.8.

The paper identifies several characteristics that distinguish TimeChatCap-42K from prior datasets. These are: temporal continuity with explicit timestamps for every scene rather than only sparse event-level captions; comprehensive, multi-scene coverage rather than salient-only summaries; a structured six-dimensional schema spanning visuals, audio, camera, and editing; and a joint audio-visual focus in which acoustic cues and dialogue are first-class citizens.

The reported limitations are equally specific. Because the supervision is synthetic and generated by a proprietary model with automatic filtering, it can introduce stylistic or content biases and may propagate upstream model errors. There is also a duration bias: most clips are approximately one minute and the average segment is approximately ten seconds, so generalization to hour-long or highly irregular videos remains challenging. Training used a 32K-token context window, and the paper notes that scaling to longer videos will require larger context or token compression. Future directions stated in the paper are to collect more diverse, longer-form videos to improve duration generalization and timestamp segmentation accuracy, incorporate efficient token compression to reduce sequence lengths and training costs, especially for RL, and continue refining time-aware rewards and judge models to further improve caption faithfulness.

The dataset, models, and code are stated to be released at https://github.com/yaolinli/TimeChat-Captioner. The resources are sourced from public academic datasets and are to be released under responsible-use terms, with specific licenses to be provided at release.