video-SALMONN 2: Audio-Visual Captioning LLM

Updated 7 July 2026

The paper introduces caption-focused preference learning by decomposing captions into atomic events and using directed DPO for improved caption accuracy.
It employs frozen audio-visual encoders with trainable modality aligners and LoRA adapters, ensuring efficient fusion of audio and visual modalities.
Quantitative results show a 28% error reduction on a custom caption benchmark, demonstrating superior performance over competing models.

video-SALMONN 2 is an audio-visual LLM for detailed video captioning with paired audio. In its published formulation, it is a 7B-parameter system built upon an internally trained visual LLM and further fine-tuned on LLaVA-OneVision-7B, with frozen audio and visual encoders, trainable modality aligners, and LoRA adaptation on the backbone LLM. Its defining contribution is caption-focused preference learning: ground-truth captions are decomposed into atomic events, completeness and hallucination are measured at the event level, and those signals are used to construct directed preference pairs for DPO. Training is further stabilized by multi-round DPO (MrDPO), which periodically refreshes the reference model and merges and re-initializes LoRA as a proxy for backbone updates (Tang et al., 18 Jun 2025).

1. Position within the video-SALMONN family

video-SALMONN 2 is the caption-enhanced descendant of the original video-SALMONN, which introduced a speech-enhanced audio-visual LLM built around a Multi-Resolution Causal Q-Former for joint speech, non-speech audio, and vision (Sun et al., 2024). The earlier system emphasized end-to-end speech-aware video understanding, whereas video-SALMONN 2 reorients the line toward detailed audio-visual captioning and toward caption-quality optimization through event-based evaluation, directed DPO, and MrDPO (Tang et al., 18 Jun 2025).

Within the broader family, later named variants are not simple renamings of video-SALMONN 2. video-SALMONN-R $^3$ is described as a distinct variant centered on ReWatch, ReAsk, and ReAnswer for efficient video QA, and not as the “second version” of the original model (Li et al., 23 Jun 2026). video-SALMONN S is the streaming sibling, designed for multi-hour video under a fixed memory budget rather than for short-form caption enhancement (Sun et al., 13 Oct 2025). This family structure matters because the designation “video-SALMONN 2” refers specifically to the captioning-oriented branch, not to the later re-watch or streaming branches.

The model also differs from the prior video-SALMONN variant at the level of training emphasis. The prior variant used a multi-resolution causal Q-Former to process audio-visual streams and did not center training on caption-quality optimization via preference learning; video-SALMONN 2 instead introduces targeted captioning improvements via a new event-based evaluation pipeline, directed DPO, and multi-round DPO with LoRA proxy merging and reinitialization, plus GPT-based automated feedback on atomic events (Tang et al., 18 Jun 2025).

2. Architecture and multimodal sequence construction

The architecture consists of separate audio and visual branches producing modality-specific tokens, which are synchronously interleaved and fused with the text prompt tokens before entering the LLM. Encoders are kept frozen; modality aligners and LoRA adapters are trained in subsequent stages (Tang et al., 18 Jun 2025).

On the visual side, input frames are downsampled to a fixed frame rate $\phi$ , with default $\phi = 1$ fps. For a video of duration $T$ seconds, $n=\phi T$ frames are initially sampled. If $n$ exceeds a hardware or latency budget $m$ , where $m=110$ frames max, the frame rate is reduced to $\phi'=\lfloor m/T \rfloor$ , making $n=\phi' T \le m$ . Each frame $\phi$ 0 is mapped to visual tokens via a frozen visual encoder followed by a visual modality aligner:

$\phi$ 1

On the audio side, the encoder is Whisper-Large-v3, also frozen. To handle long audio, the waveform $\phi$ 2 is split into

$\phi$ 3

segments, with $\phi$ 4 seconds per segment:

$\phi$ 5

Segment-level positional embeddings $\phi$ 6 are added before the audio aligner:

$\phi$ 7

followed by concatenation and alignment:

$\phi$ 8

The audio aligner is a window-level Q-Former with 0.5 s window; for a 30 s input it produces 60 audio tokens.

Fusion is chronological rather than late-stage. Let

$\phi$ 9

be audio segment indices aligned to frame $\phi = 1$ 0. The fused token block at time step $\phi = 1$ 1 is

$\phi = 1$ 2

and the overall multimodal sequence is

$\phi = 1$ 3

The text prompt $\phi = 1$ 4 is concatenated with $\phi = 1$ 5 to form the LLM input, and the model generates caption $\phi = 1$ 6 by maximizing $\phi = 1$ 7 (Tang et al., 18 Jun 2025).

LoRA adapters are attached to the backbone LLM to enable efficient training while keeping encoders frozen. In the published version, LoRA uses rank $\phi = 1$ 8 and scaling $\phi = 1$ 9. During audio-visual SFT and MrDPO, LoRA on the LLM is trained; encoders remain frozen. The paper denotes generic backbone weights $T$ 0 as the LoRA targets but does not enumerate specific layers beyond “weights to adapt” (Tang et al., 18 Jun 2025).

3. Training procedure: supervised alignment, directed DPO, and MrDPO

Training proceeds in stages. Before preference optimization, the model undergoes audio modality alignment and then audio-visual supervised fine-tuning. Audio alignment trains the audio aligner on LibriSpeech-960h and AudioCaps via cross-entropy on reference transcripts or captions:

$T$ 1

Audio-visual SFT then trains on paired video data with synchronized audio to learn interleaved audio-visual token processing, again with cross-entropy against ground-truth captions or questions:

$T$ 2

LoRA is introduced during audio-visual SFT and then reused and updated in MrDPO (Tang et al., 18 Jun 2025).

The standard DPO objective is defined on paired preferences $T$ 3 with policy $T$ 4 and reference $T$ 5:

$T$ 6

video-SALMONN 2 makes this directed for captioning by constructing preferences from atomic-event metrics. Ground-truth captions are decomposed into atomic events by a powerful LLM, and generated captions are evaluated for missing and hallucination events; in each pair, the caption with the lower total error becomes $T$ 7 and the other becomes $T$ 8. Pairs with small metric differences are filtered out to reduce evaluation noise and improve training efficiency (Tang et al., 18 Jun 2025).

MrDPO addresses the problem that single-round offline DPO can become ineffective as the reference model becomes stale relative to the updated policy. Each round comprises three steps. First, the previous LoRA is merged into the backbone:

$T$ 9

where $n=\phi T$ 0 and $n=\phi T$ 1. Second, a fresh LoRA proxy $n=\phi T$ 2 is initialized and attached to the merged backbone $n=\phi T$ 3; the merged backbone is frozen as the round-specific reference $n=\phi T$ 4, and only the new proxy is updated. Third, training uses guided DPO (gDPO), which adds a stabilizing cross-entropy term with ground-truth captions:

$n=\phi T$ 5

Here $n=\phi T$ 6, $n=\phi T$ 7, and $n=\phi T$ 8 weights the SFT term (Tang et al., 18 Jun 2025).

The round schedule is approximately 1,000 updates per round for Rounds 1–5, plus a sixth round of 841 updates. Caption-pair thresholds become stricter across rounds, and repetition-rate filtering is also applied. In ablation, $n=\phi T$ 9 is selected as the best stability-performance tradeoff; larger values greater than 1 hinder DPO optimization, while very small values below 0.01 reduce stability in later rounds. The LoRA learning rate is $n$ 0 for Rounds 1–2, $n$ 1 for Round 3, and $n$ 2 thereafter. Batch size per GPU is 1. Audio alignment used 32 H800 GPUs for 30k steps at approximately 3 h; SFT used 32 H800 for 15,475 steps at approximately 14 h; MrDPO used 8 H800 for Rounds 1–5 at approximately 2 h per round and 32 H800 for Round 6 at approximately 1 h. The optimizer is not explicitly reported (Tang et al., 18 Jun 2025).

4. Atomic-event evaluation and the caption benchmark

A central component of video-SALMONN 2 is its atomic-event-based evaluation pipeline. Let $n$ 3 be the set of ground-truth atomic events for a video, with $n$ 4. For a model-generated caption, an evaluation LLM outputs three categories: missing events $n$ 5, incorrect events $n$ 6, and hallucination events $n$ 7. The paper defines

$n$ 8

From these, completeness and accuracy are derived as

$n$ 9

and one can also define overall caption “accuracy” as $m$ 0 (Tang et al., 18 Jun 2025).

Ground-truth captions are decomposed into atomic events by GPT-4o, with human refinement for the test set. For each generated caption, missing and hallucination events are identified by an evaluation LLM: GPT-3.5 in the paper, or Qwen3-4B for training efficiency. The same event-based signals direct preference construction during MrDPO and support evaluation at test time. Human ELO ranking is reported as corroborating the improvements measured by these metrics (Tang et al., 18 Jun 2025).

The custom caption benchmark contains 483 videos of 30–60 s duration, with average 51 s, human-annotated audio-visual captions, and refined atomic events. It contains an average of 34.2 atomic events per video, including 6.1 audio-related events, with 4.6 speech-related and 1.5 non-speech events. The test set is open-sourced at HuggingFace. Beyond the caption benchmark, evaluation also includes Video-MME, NExT-QA, MLVU, and VDC, with the VDC detailed caption subset used for correlation (Tang et al., 18 Jun 2025).

This benchmark design places unusual emphasis on omissions and hallucinations at event granularity rather than on overlap-based caption metrics. A plausible implication is that the benchmark is better aligned with dense audio-visual description than with purely stylistic fluency judgments, particularly for dialogue attribution, sound events, and chronology.

5. Quantitative performance and ablation findings

On the custom caption benchmark, video-SALMONN 2 reports Miss 10.0%, Hall 12.9%, and Total 22.9%. The corresponding numbers for Ours-Visual Base are Miss 23.3%, Hall 27.4%, and Total 50.7%. GPT-4o Visual reports 17.0 / 14.2 / 31.2, and Gemini-1.5-Pro reports 21.8 / 16.5 / 38.3. The published abstract states that MrDPO reduces the captioning error rates by 28%, and the detailed benchmark establishes that the final 7B model surpasses GPT-4o Visual and Gemini-1.5-Pro on this task (Tang et al., 18 Jun 2025).

System	Caption benchmark (Miss / Hall / Total)	Video-MME
GPT-4o Visual	17.0 / 14.2 / 31.2	—
Gemini-1.5-Pro (A+V)	21.8 / 16.5 / 38.3	—
Ours-Visual Base (V)	23.3 / 27.4 / 50.7	62.9
video-SALMONN 2 (A+V)	10.0 / 12.9 / 22.9	67.4

On QA-related benchmarks, video-SALMONN 2 remains highly competitive among models of similar size. It reports 46.1 accuracy and 2.5 score on VDC Detailed, 67.4 on Video-MME, 83.0 on NExT-QA, and 68.0 on MLVU. These numbers are not uniformly superior to its own visual base on every benchmark: Ours-Visual Base reports 83.6 on NExT-QA and 69.2 on MLVU, whereas video-SALMONN 2 reports 83.0 and 68.0. This directly supports the paper’s stated limitation that caption-focused RL does not consistently improve QA performance (Tang et al., 18 Jun 2025).

A further proxy test evaluates caption utility by using only captions to answer Video-MME through GPT-4o. In that setting, GPT-4o captions yield 64.3% QA accuracy, while video-SALMONN 2 captions yield 65.9%, the best among the tested systems. The paper interprets this as evidence of caption completeness and usefulness (Tang et al., 18 Jun 2025).

The ablation results isolate the training recipe. gDPO outperforms classical DPO after Round 2, improving stability and avoiding local optima. Periodic LoRA merging and fresh LoRA initialization outperform reusing the same LoRA, especially in early rounds, by integrating accumulated improvements into the backbone and enabling exploration of new low-rank subspaces. The audio modality contributes materially: Whisper-Large-v3-based audio tokens with segment-level positional encoding reduce omission of speech and sound events and improve caption granularity relative to purely visual models (Tang et al., 18 Jun 2025).

6. Limitations, later reuse, and broader significance

The paper explicitly notes that video QA performance does not consistently improve with caption-focused RL, because MrDPO is trained only on captioning and captioning and QA may engage different capabilities. Reported failure modes include temporal ordering errors, weak cross-modal alignment, hallucinations for subtle audio events or visually ambiguous scenes, and coverage difficulties on long videos beyond the sampling limit of 110 frames and 30 s audio segments. Out-of-distribution content and noisy audio can degrade performance, and potential biases from training data and automated evaluators warrant scrutiny and mitigation (Tang et al., 18 Jun 2025).

The model’s inference and deployment profile reflects its short-form captioning emphasis. Visual inputs are capped at 110 frames at 1 fps; audio is processed in 30 s encoder passes and aggregated through the audio aligner; tokens are interleaved chronologically with the prompt before entering the LLM. Standard instruction prompts can be used for user-facing captioning, but the paper does not exhaustively report deployment decoding settings. The code is available at https://github.com/bytedance/video-SALMONN-2, and the caption benchmark is released at https://huggingface.co/datasets/videoSALMONN2/video-SALMONN_2_testset (Tang et al., 18 Jun 2025).

Later work has reused the video-SALMONN-2 test set as a dense captioning benchmark rather than as a model architecture. “Script-a-Video” applies its Multi-Stream Scene Script representation to the video-SALMONN-2 test set and reports an average reduction of roughly 25% in Total Error when replacing monolithic captions with factorized scripts under a Gemini-2.5-Pro judge (Team, 13 Apr 2026). This suggests that video-SALMONN 2 has influenced not only captioning models but also the design of caption interfaces and evaluation protocols.

Within the family, subsequent variants clarify the scope of the original system. video-SALMONN-R $m$ 1 targets efficient QA through re-watch localization learned by RL rather than caption-enhanced instruction tuning (Li et al., 23 Jun 2026). video-SALMONN S addresses streaming long-video understanding with test-time-training memory rather than short-form caption optimization (Sun et al., 13 Oct 2025). Those developments underscore the specific significance of video-SALMONN 2: it established an event-centric framework for audio-visual captioning in which missing content and hallucinated content are first-class optimization targets, and it showed that a 7B model could surpass GPT-4o Visual and Gemini-1.5-Pro on a human-annotated benchmark for detailed audio-visual description (Tang et al., 18 Jun 2025).