Video Captioning Models

• Video captioning models are algorithms that automatically generate descriptive text for videos by integrating visual, audio, and temporal cues.
• They employ advanced architectures such as CNNs, LSTMs, transformers, and memory-augmented networks to extract, fuse, and decode multimodal information.
• Recent trends focus on improving temporal localization, reducing hallucination, and enhancing performance via reinforcement learning and modular, open-source pipelines.

Video captioning models are algorithms and architectures designed to generate natural language descriptions that accurately characterize the content of videos. These systems are fundamentally multimodal, requiring the visual parsing of time-varying pixel streams and the generation (or ranking) of associated text sequences. Current research spans classical sequence-to-sequence approaches, memory-augmented neural networks, large multimodal model (LMM) fusion, and techniques exploiting both audio and visual signal integration. Benchmark performance is often measured on datasets such as MSVD, MSR-VTT, ActivityNet, YouCook2, and domain-specific corpora, with metrics including BLEU, METEOR, ROUGE-L, CIDEr, and increasingly, custom metrics evaluating grounding and hallucination.

1. Model Architectures and Fusion Strategies

Video captioning systems can be categorized by their feature extraction, fusion, and decoding paradigms. Traditional models employ a two-stage pipeline: frame/clip feature extraction via CNNs (e.g., VGG, ResNet, I3D), followed by a recurrent decoder (LSTM, GRU) that produces word sequences conditioned on aggregate visual representations (Adewale et al., 2023, Wang et al., 2016). Memory-augmented architectures like Multimodal Memory Models (M³) (Wang et al., 2016) and Memory-Attended Recurrent Networks (MARN) (Pei et al., 2019) extend this with external memory slots, capturing longer visual-text dependencies and enabling cross-video context.

Recent advances utilize transformer or LLM-based decoders. QCaption (Wang et al., 10 Jan 2026) implements a modular, late-fusion pipeline combining: (1) frame sampling (e.g., Katna clustering, regular or random sampling), (2) per-frame captioning with a frozen LMM (CLIP/LLaVA backbone), and (3) summary-level aggregation via an LLM (Vicuna-based), using natural language concatenation as the fusion mechanism. This modularity enables state-of-the-art captioning and video QA, outperforming end-to-end video-LMMs like Video-LLaVA or VideoChatGPT.

Other contemporary frameworks, such as WAFTM (V et al., 2021), employ memory-augmented multimodal transformers and learn weighted-additive fusion of appearance, motion, and object tokens at the cross-attention sublayers, supporting flexible input modalities and mitigating long-range context loss. Streaming variants (e.g., (Zhou et al., 2024)) introduce token clustering memories for bounded-complexity processing over arbitrarily long videos.

2. Feature Representation and Multimodal Integration

Early captioners relied on static per-frame (2D CNN) and short-clip (3D CNN) features, optionally composed with optical flow or handcrafted descriptors (Wang et al., 2016, Adewale et al., 2023). Subsequent architectures, such as dual-graph models (Jin et al., 2023), explicitly aggregate multi-scale appearance, motion, and object features with graph attention networks and relational graphs, followed by gated fusion blocks to control per-word cross-modality integration.

Recent large multimodal models use pretrained image-text encoders (BLIP-2, CLIP, Q-Former) and concatenate visual tokens from sampled frames, implicitly encoding temporality (Zhang et al., 19 Feb 2025). QCaption (Wang et al., 10 Jan 2026) demonstrates that combining strong per-frame LMMs with large LLM aggregators yields superior results—even without explicit temporal reasoning modules or end-to-end finetuning. For audio-visual captioning, models such as video-SALMONN 2 (Tang et al., 18 Jun 2025) and UGC-VideoCaptioner (Wu et al., 15 Jul 2025) implement parallel audio and visual branches, jointly encoding information as interleaved modality-specific token sequences that are fused in the transformer backbone.

3. Training Objectives, Optimization, and Regularization

The canonical supervised objective is sequence-level cross-entropy over the predicted caption, occasionally modulated by coverage penalties or multi-task losses (Rimle et al., 2020). To better align generation with human-centric metrics (e.g., CIDEr), self-critical sequence training (SCST) and other RL-based objectives are adopted (V et al., 2021, Zhang et al., 19 Feb 2025). In reinforcement learning settings, rewards correspond to non-differentiable scores or atomic-event metrics—quantifying content completeness and hallucination, as exemplified by video-SALMONN 2's DPO/MrDPO framework (Tang et al., 18 Jun 2025).

Exposure bias and overfitting are mitigated via scheduled sampling (Chen et al., 2019), variational dropout (Chen et al., 2020), layer normalization, memory regularization, and length-aware losses. WAFTM leverages WordPiece tokenization and REINFORCE to address OOVs and directly optimize CIDEr (V et al., 2021). Longer-range memory and global feature priors help maintain relevant context for action and object recognition over long sequences (Wang et al., 2016, Zhou et al., 2024).

For zero-shot or data-sparse settings, prompt-tuning and retrieval-enhanced test-time adaptation (RETTA) learn soft tokens to bridge frozen retrieval, vision, and text models, enabling strong out-of-the-box video captioning (Ma et al., 2024).

4. Temporal Localization and Dense Captioning

Dense video captioning demands temporally localized event detection and description. State-space models (SSMs) with transfer state propagation support online event segmentation, scaling linearly with video length and reducing computational cost by up to 7× compared to traditional attention-based methods (Piergiovanni et al., 3 Sep 2025). Streaming models maintain fixed-size clustering-based memory and emit intermediate outputs at multiple decoding points, avoiding loss of recent or earlier events (Zhou et al., 2024).

TA-Prompting (Cheng et al., 6 Jan 2026) extends LLM-based pipelines by interpolating predicted temporal anchors (via detector-style Transformer modules) and conditioning generation on dedicated time-token embeddings. Event selection is refined via event-coherent sampling, balancing autoregressive coherence with cross-modal CLIP similarity to select the most faithful captions.

5. Evaluation Metrics, Benchmarks, and Hallucination Analysis

Standard evaluation uses n-gram matching (BLEU, METEOR, ROUGE-L, CIDEr), but these often fail to reflect semantic grounding or hallucination. Datasets include MSVD, MSR-VTT, ActivityNet, YouCook2, ViTT, UGC-VideoCap, and domain-specific corpora (e.g., EgoSchema, NewsVideo).

To address hallucination—where models mention objects or actions detached from source video—custom metrics have been introduced. COAHA (Caption Object and Action Hallucination Assessment) (Ullah et al., 2022) penalizes object and action errors weighted by semantic distance. Recent work, particularly video-SALMONN 2 (Tang et al., 18 Jun 2025), operationalizes atomic-event extraction to quantify missing events and hallucinations, using these as preference signals for preference-optimized RL (DPO/gDPO/MrDPO). Empirical results indicate substantial relative reduction in error rates compared to leading LLMs (e.g., 27.8% reduction in total error relative to a Vicuna-based baseline, surpassing GPT-4o and Gemini-1.5-Pro in atomic completeness/accuracy).

Ablation studies across architectures consistently demonstrate the importance of learned fusion (vs. concatenation), memory-augmented encoders, multimodal tokenization, and robust sampling schedules (V et al., 2021, Wang et al., 10 Jan 2026, Ullah et al., 2022). Both per-frame LMM fusion and transformer-based temporal aggregation independently support substantial metric gains (e.g., up to +44.2%/48.9% on YouCook2/ActivityNet-QA for QCaption (Wang et al., 10 Jan 2026)).

6. Practical Deployment and Data Efficiency

Modular and late-fusion pipelines offer flexibility for on-premise, security-sensitive deployments by using exclusively open-source components (e.g., LLaVA, Vicuna) and avoiding reliance on external APIs (Wang et al., 10 Jan 2026). QCaption's design permits rapid substitution and upgrade of any keyframe extractor, per-frame LMM, or aggregator LLM. With frozen backbones and no need for expensive video-transformer pretraining, these methods achieve low overhead for finetuning and inference.

For resource-constrained scenarios, research demonstrates that adapting a pretrained image-text model (BLIP-2) via video-frame concatenation and limited RL finetuning (6,000 pairs) can achieve top-3 performance on major benchmarks, outperforming far larger, fully supervised video-text models (Zhang et al., 19 Feb 2025). Group Relative Policy Optimization (GRPO) and other RL variants further boost data efficiency and caption granularity in limited settings (e.g., TikTok UGC), delivering competitive audio-visual and cross-modal performance with smaller numbers of human annotations (Wu et al., 15 Jul 2025).

7. Recent Trends and Future Directions

The field is witnessing a convergence of approaches: fusion of strong per-frame LMMs, on-the-fly memory and streaming mechanisms, explicit detection and control of hallucination, and preference-based reinforcement learning. Recent frameworks such as TA-Prompting (Cheng et al., 6 Jan 2026) and video-SALMONN 2 (Tang et al., 18 Jun 2025) demonstrate modular stacking of temporal localization, LoRA-augmented LLMs, and atomic-event evaluation, enabling detailed, temporally-faithful, and low-hallucination captions across long videos. Fine-tuned models with hybrid control tokens integrate both action and static scene descriptions for superior downstream Q&A over long-form, multi-scene content (Sasse et al., 22 Jul 2025).

Open research problems include robust adaptation to unseen domains, fully online and hierarchical long-video processing, multi-domain semi-supervised learning, and tighter integration of audio, text, and high-resolution streaming video features. Quantifying and optimizing real-world effectiveness—especially with human-in-the-loop validation and targeted event/error decomposition—remains a priority as video captioning systems are incorporated into broader video analytics and QA pipelines for surveillance, multimedia retrieval, and human-computer interaction.