TimeChat-Captioner-7B Overview

• The paper introduces TimeChat-Captioner-7B, a multimodal LLM that produces detailed, timestamped, multi-scene scripts using a structured six-dimensional schema.
• It leverages a Qwen2.5-Omni backbone with interleaved audio-visual tokens and time-aware positional encoding to improve scene segmentation and temporal alignment.
• Training combines supervised fine-tuning and GRPO reinforcement learning with specialized rewards to optimize format adherence, length control, and caption quality.

TimeChat-Captioner-7B is a specialized multimodal LLM introduced for Omni Dense Captioning, a task that requires continuous, fine-grained, and structured audio-visual narratives with explicit timestamps. In the 2026 formulation, the model is built on a 7B Qwen2.5-Omni backbone with interleaved audio-visual tokens and time-aware positional encoding, and it is trained through supervised fine-tuning and GRPO with task-specific rewards. Its target output is a multi-scene, “script-like” description organized across six dimensions, rather than a single coarse paragraph, and its reported performance on OmniDCBench reaches a SodaM average of 35.0, surpassing Gemini-2.5-Pro at 33.7 while also improving downstream audio-visual reasoning and temporal grounding (Yao et al., 9 Feb 2026).

1. Task definition and nomenclature

Omni Dense Captioning takes a video with synchronized audio and outputs a multi-scene script that is dense in both time and description. The task comprises temporal segmentation into successive scenes with explicit timestamps and the generation of six-dimensional structured captions per scene. A scene is defined as a semantically coherent segment, not necessarily a single shot, unified by continuity in time, location, or narrative context; scene transitions are indicated by changes in visual setting, audio context, or narrative progression (Yao et al., 9 Feb 2026).

The task is formalized with input frames and audio over $T$ time steps,

$$F = \{f_1, f_2, \ldots, f_T\}, \quad A = \{a_1, a_2, \ldots, a_T\},$$

and output

$$S = C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$

where each $t_i$ is a start–end range in MM:SS and each $c_i$ is a structured multi-dimensional caption. Internally, the model uses a JSON-like schema with strict time ranges in the form “m:ss - m:ss” and non-overlapping boundaries (Yao et al., 9 Feb 2026).

A recurrent source of confusion is the model name itself. An earlier paper used “TimeChat-Captioner-7B” to denote the 7B-parameter TimeChat configuration for long-video dense captioning, built around a timestamp-aware frame encoder and a sliding Video Q-Former on top of LLaMA-2 7B (Ren et al., 2023). In the 2026 literature, the same label denotes the Qwen2.5-Omni-based baseline for Omni Dense Captioning (Yao et al., 9 Feb 2026).

2. Six-dimensional script schema

The defining representational feature of TimeChat-Captioner-7B is its six-dimensional structural schema. Each scene is described through a fixed set of fields intended to provide holistic audio-visual coverage, camera and editing context, and dialogue and acoustic cues.

Dimension	Function	Internal field
Events	Detailed narration of actions and salient events	segment_detail_caption
Background	Setting, location, atmosphere, salient environmental elements	video_background
Camera	Camera movement, angle, framing, zoom, pan, tilt, shot transitions	camera_state
ShotEdit	Editing techniques and multi-shot organization	shooting_style
Dialogue	Transcription and summary of spoken content with speaker attribution	speech_content
Acoustic	Background sounds, music, speech tones, sound effects, ambiance	acoustics_content

The script is emitted as a sequence of scenes, each annotated with a strict time range “MM:SS - MM:SS” and a six-field caption. Supervised fine-tuning and reinforcement learning both use this aligned schema, and inference follows the learned output template to produce parseable JSON (Yao et al., 9 Feb 2026).

The qualitative contrast supplied in the paper clarifies the motivation for this structure. A representative successful output contains two temporally separated scenes, each with explicit Events, Background, Camera, ShotEdit, Dialogue, and Acoustic fields. By contrast, a baseline failure collapses the same material into a single coarse paragraph, omits timestamps, misses speaker attribution and acoustic cues, and misidentifies the passenger’s gender. This suggests that the six-dimensional design is intended not merely to increase verbosity, but to regularize completeness and factual grounding under a screenplay-like format (Yao et al., 9 Feb 2026).

3. Architecture and temporal modeling

TimeChat-Captioner-7B uses Qwen2.5-Omni as its base LLM, within the Qwen2.x Omni multimodal family. The architecture employs Qwen2.5’s “Thinker” module to integrate multi-modality, with Qwen2.5-VL as the vision encoder and Qwen2-Audio as the audio encoder. The model is therefore not a simple video-only captioner; it is an audio-visual system that decodes jointly fused representations into timestamps and six-dimensional textual outputs (Yao et al., 9 Feb 2026).

Its temporal modeling centers on two mechanisms. First, audio and visual tokens are arranged in a temporally interleaved sequence so that tokens at similar times are adjacent, enabling cross-attention to fuse audio cues with concurrent visual frames. Second, Multimodal Rotary Position Embeddings, or M-RoPE, encode absolute temporal positions across modalities, enhancing scene boundary localization and continuous timestamp prediction. The Thinker module consumes the interleaved audio-visual tokens and applies cross-attention over the joint embedding space, while M-RoPE provides temporal indices (Yao et al., 9 Feb 2026).

Generation is also explicitly format-aware. The model outputs a sequence of scene objects, each containing a timestamp plus the six schema fields. Time-awareness is injected through both positional encodings and explicit timestamp tokens, and format adherence is further reinforced during reinforcement learning through a Format Reward. During supervised fine-tuning, instruction prompts are split into Stage-1 and Stage-2: Stage-1 enforces segmentation logic, while Stage-2 expands each segment into six-dimensional detail. At inference time, the model produces segments directly; many-to-one merging is part of evaluation alignment rather than generation (Yao et al., 9 Feb 2026).

4. Training data and optimization

The training dataset, TimeChatCap-42K, is synthesized from MMTrail-2M and Movie101 through a two-step procedure using Gemini-2.5-Pro. Stage 1 performs boundary segmentation with brief captions such as “0:00–0:15: a boy singing…”, and Stage 2 enriches each segment with six-dimensional captions using the structured schema. Quality filtering removes clips with fewer than two segments, missing audio, JSON errors, missing fields, and overly short segments. The resulting corpus contains 42K pairs; most video clips are around 50–60s; the average caption length is 877 words per video across the six dimensions; and the average segment length is approximately 10.04s (Yao et al., 9 Feb 2026).

Supervised fine-tuning uses next-token prediction on the structured outputs:

$$L_{\mathrm{SFT}} = -\sum_t \log p_\theta(y_t \mid x, y_{<t}),$$

where $x$ denotes interleaved audio-visual tokens and $y$ the JSON-formatted multi-scene caption. The reported schedule is 40K samples, 2 epochs, learning rate $5 \times 10^{-5}$, batch size 128, 2 FPS video sampling, and a maximum sequence length of 32K tokens, trained with 32 × 80G GPUs under DeepSpeed ZeRO-2. The maximum pixels per frame are 297,920 and per video 20,070,400 (Yao et al., 9 Feb 2026).

Reinforcement learning is performed with GRPO because timestamps are sparse tokens, approximately 0.7% of the output, and scene counts vary, so SFT alone underweights time accuracy and format adherence. For an input $q$, the model samples $$F = \{f_1, f_2, \ldots, f_T\}, \quad A = \{a_1, a_2, \ldots, a_T\},$$0 candidate outputs from the current policy, computes rewards, and normalizes them into advantages:

$$F = \{f_1, f_2, \ldots, f_T\}, \quad A = \{a_1, a_2, \ldots, a_T\},$$1

The GRPO objective is

$$F = \{f_1, f_2, \ldots, f_T\}, \quad A = \{a_1, a_2, \ldots, a_T\},$$2

The reward design combines four terms: a Format Reward $$F = \{f_1, f_2, \ldots, f_T\}, \quad A = \{a_1, a_2, \ldots, a_T\},$$3, a Length Reward $$F = \{f_1, f_2, \ldots, f_T\}, \quad A = \{a_1, a_2, \ldots, a_T\},$$4 following AVoCaDO, a Timestamp Reward $$F = \{f_1, f_2, \ldots, f_T\}, \quad A = \{a_1, a_2, \ldots, a_T\},$$5 based on F1 over IoU thresholds $$F = \{f_1, f_2, \ldots, f_T\}, \quad A = \{a_1, a_2, \ldots, a_T\},$$6, and a Time-aware Caption Reward $$F = \{f_1, f_2, \ldots, f_T\}, \quad A = \{a_1, a_2, \ldots, a_T\},$$7 based on SodaM. The final reward is

$$F = \{f_1, f_2, \ldots, f_T\}, \quad A = \{a_1, a_2, \ldots, a_T\},$$8

with $$F = \{f_1, f_2, \ldots, f_T\}, \quad A = \{a_1, a_2, \ldots, a_T\},$$9. The GRPO stage uses 2K samples, 1 epoch, learning rate $$S = C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$0, batch size 64, rollout $$S = C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$1, and $$S = C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$2 (Yao et al., 9 Feb 2026).

5. Benchmark design, metrics, and empirical profile

Evaluation is carried out on OmniDCBench, a human-annotated benchmark of 1,122 videos drawn from Movie101 and YT-Temporal-1B. Annotation follows a three-stage pipeline consisting of candidate filtering and difficulty tagging, holistic scene segmentation with continuous timestamps, and six-dimensional scene descriptions, with camera and editing annotations provided by specialists and then double-checked by independent annotators. The benchmark is characterized as high-resolution with clear audio, and its descriptive richness averages 995 words per video (Yao et al., 9 Feb 2026).

Timestamp accuracy is measured by IoU over time ranges together with F1 at thresholds $$S = C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$3 and mean IoU. Caption quality uses CheckList, which decomposes each ground-truth caption dimension into atomic elements and scores coverage by a judge model. The unified metric, SodaM, aligns predicted and ground-truth scenes by dynamic programming:

$$S = C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$4

When multiple predicted segments align to one ground-truth scene, the evaluation merges them by taking the min start time, max end time, and concatenated captions. SodaM is then computed as the final temporally aligned caption quality score, macro-averaged across the six dimensions. The paper emphasizes two consequences: boundary ambiguity is mitigated, and judge-model cost is reduced from $$S = C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$5 to $$S = C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$6 (Yao et al., 9 Feb 2026).

A concise view of the main OmniDCBench results is given below.

Model	Timestamp F1 / mIoU	SodaM Avg
TimeChat-Captioner-7B-SFT	62.4 / 70.8	32.6
TimeChat-Captioner-7B-GRPO	61.2 / 69.6	35.0
Gemini-2.5-Pro	68.5 / 74.9	33.7

The empirical profile is therefore mixed in a technically significant way. TimeChat-Captioner-7B-GRPO is state of the art on time-aware dense captioning, with per-dimension SodaM scores of Camera 12.4, Events 39.6, Background 49.2, Acoustics 38.2, ShotEdit 43.5, and Dialogue 54.3, but its multi-scene segmentation F1 and mIoU are second among all models, and the SFT variant is slightly better than GRPO on the raw timestamp segmentation metrics. This is not a contradiction: GRPO is optimized for format validity, length control, timestamp accuracy, and temporally aligned caption quality, and the reported gains concentrate on SodaM rather than uniformly improving every submetric (Yao et al., 9 Feb 2026).

The baseline gap is large. Selected SodaM averages are 4.6 for Qwen2.5-Omni(7B), 5.4 for MiniCPM-o-2.6(8B), 6.9 for OmniVinci(9B), 14.3 for Qwen3-Omni(30B-A3B), and 1.8 for LongVALE(7B). The ablations further indicate that larger SFT data helps, with SFT(20K) scoring 31.3 and SFT(40K) scoring 32.6; adding the SodaM reward increases performance from 32.5 to 35.0; and moving from one to two SFT epochs raises SodaM from 30.7 to 32.6. Small changes to $$S = C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$7 have marginal effects of at most 0.5% (Yao et al., 9 Feb 2026).

6. Downstream utility, limitations, and related directions

The paper reports that the generated dense descriptions improve downstream audio-visual reasoning and temporal grounding when used as structured intermediate supervision. On Daily-Omni, TimeChat-Captioner-7B-GRPO reaches 52.8, compared with 29.9 for video-SALMONN-2 and 13.4 for Qwen2.5-Omni, while Gemini-2.5-Pro reaches 60.2. On WorldSense, the reported scores are 22.6 for the model, 18.2 for video-SALMONN-2, 8.6 for Qwen2.5-Omni, and 33.8 for Gemini-2.5-Pro. On Charades-STA, the model reports [email protected] 79.8, [email protected] 68.7, [email protected] 48.3, and mIoU 58.8, surpassing Qwen2.5-Omni-7B at 65.9 on [email protected] and TimeSuite at 67.1 on the same metric. The stated mechanism is that rich, temporally grounded scripts provide dense audio-visual supervision signals together with factual dialogue and acoustic cues, improving temporal alignment and cross-modal reasoning in QA and grounding (Yao et al., 9 Feb 2026).

The limitations are equally explicit. The 32K-token context window constrains long-input and long-output training, so hour-long videos require segment-then-caption strategies such as approximately 1-minute chunks. Performance degrades on very long content with numerous scenes. Sparse supervision of time tokens can still produce minor boundary drifts. Synthetic training annotations depend on Gemini-2.5-Pro and may inherit its biases, while the human benchmark differs in style and granularity. RL with judge models remains costly, although the SodaM alignment reduces evaluation complexity to $$S = C = \{(t_1, c_1), (t_2, c_2), \ldots, (t_N, c_N)\},$$8 (Yao et al., 9 Feb 2026).

Within the broader literature, several adjacent systems illuminate what is specific about TimeChat-Captioner-7B. “TimeChat” emphasizes long-video understanding with a timestamp-aware frame encoder and a sliding Video Q-Former, using a 7B LLaMA-2 backbone and instruction tuning across six time-sensitive tasks (Ren et al., 2023). “TCA-Captioner” targets temporal and cross-modal alignment for audiovisual captioning through the Observer-Checker-Corrector framework and TCA-Bench, and its paper explicitly states that it does not mention a model named “TimeChat-Captioner-7B” (Zhao et al., 2 Jul 2026). “TAC” and TAC-V focus on timestamped audio captioning and an audio-visual “semantic bridge” for downstream reasoning (Kumar et al., 17 Feb 2026). “Video ChatCaptioner” instead uses a controller-driven question-answer loop over uniformly sampled frames to obtain enriched spatiotemporal descriptions (Chen et al., 2023). A plausible implication is that TimeChat-Captioner-7B occupies a distinct point in this design space: it couples explicit scene timestamps, audio-visual synchrony, and a fixed six-field screenplay schema inside a single structured generation framework.