CapRiCorn-1K: A Comprehensive Benchmark for Video Captioning and Subject Referential Consistency Across Temporal Scales

Abstract: Accurate and comprehensive video captions with consistent subject references are critical for downstream understanding and generation tasks. However, few existing benchmarks can objectively and comprehensively evaluate these properties across diverse durations and scenarios, thereby hindering the advancement of video captioning models. To bridge this gap, we propose CapRiCorn-1K, a comprehensive benchmark designed to evaluate both video captioning quality and subject referential consistency across long temporal horizons and diverse video domains. To accommodate varied evaluation needs, our benchmark supports both audiovisual and visual-only settings. Extensive experiments on CapRiCorn-1K reveal that current models generally struggle to generate accurate and comprehensive captions while maintaining consistent subject references. Moreover, as video duration increases, both the overall caption quality and subject referential consistency decline. Notably, our evaluation metrics exhibit strong correlations with the performance of downstream understanding and generation tasks conditioned on the generated captions, further validating their effectiveness. The project is available at https://github.com/xlchen0205/CapRiCorn-1K .

• The paper introduces CapRiCorn-1K, a benchmark that rigorously evaluates both caption quality and subject referential consistency in long-video contexts.
• It employs a human-in-the-loop annotation pipeline with dual metrics—accuracy/coverage and referential consistency—to capture scene transitions and subject events.
• Empirical results link referential stability to downstream reasoning and video generation, highlighting architectural insights for future multimodal models.

CapRiCorn-1K: Benchmarking Video Captioning and Subject Referential Consistency Across Temporal Scales

Motivation and Limitations of Existing Benchmarks

Comprehensive and coherent video captioning is a key enabler for multimodal LLMs, with notable impacts on both downstream perception and generation tasks. CapRiCorn-1K addresses extensive gaps in prior benchmarks, which are typically limited by short video durations, homogeneity of content, and the lack of scene and subject-level referential tracking evaluation. Critically, conventional benchmarks rarely probe the subject referential consistency challenge that is exacerbated in long video sequences with multiple scene transitions and subject state changes.

Ambiguous and inconsistent references, as illustrated in the paper, propagate reasoning failures in agentic LLM settings and induce subject collapse in video generation contexts, thus undermining actual model utility despite favorable benchmark scores.

Figure 1: The effect of inconsistent subject references on downstream reasoning and video generation tasks, highlighting the need for referential consistency evaluation.

CapRiCorn-1K Data Construction and Annotation Pipeline

CapRiCorn-1K consists of 1,000 manually collected and annotated videos, each selected for dynamic scene transitions and a wide variety of content spanning eight categories and 36 subcategories. The temporal span ranges from 15 seconds to 10 minutes, with a mean duration of 252s, capturing a much wider action and context space than previous benchmarks.

The annotation protocol involves fine-grained subject identification (average 4.4 subjects/video) and the dense marking of keypoints for salient subject-related events (21.5/video) and other context anchors (14.9/video). Annotation is strictly human-in-the-loop, with multi-step cross-validation to ensure accuracy of both subject coreference and key event identification. Separate visual-only and audiovisual keypoint sets allow for modality-agnostic evaluation, supporting both AV and V-only models.

Figure 2: CapRiCorn-1K features broad topical diversity and a balanced duration distribution, with each video containing multiple scene transitions.

Evaluation Metrics: Caption Quality and Referential Consistency

CapRiCorn-1K introduces dual evaluation axes:

• Overall Caption Quality ($\text{Acc}$, $\text{Cov}$): Using a strong judge LLM (GPT-4.1 by default), each keypoint is assessed for presence and accuracy in the generated caption; "Accuracy" counts only fully correct mentions, while "Coverage" combines correct and partial mentions.
• Subject Referential Consistency ($\text{Ref}$): For each unique subject, all local subject descriptions (across all keypoints where that subject appears in the caption) are clustered using context-informed coreference decisions. The referential consistency metric is based on a Rand Index-inspired calculation that penalizes both over-fragmentation and under-description, quantifying the stability of coreference chains at subject level.

Critically, the protocol penalizes captions that superficially boost consistency scores via excessive brevity or omission (i.e., by undergenerating descriptions).

Figure 3: CapRiCorn-1K's score computation—keypoint-level mention checking for accuracy/coverage and clustering of subject mentions for referential consistency assessment.

Empirical Results and Comparative Analysis

Comprehensive model evaluation on CapRiCorn-1K reveals several trends:

• Performance Gradient: Closed-source Gemini-3 models outperform all open models, maintaining only marginal performance degradation as video length increases. In contrast, open-source systems (e.g., AVoCaDO and DiaDem) exhibit pronounced declines in both accuracy and referential consistency, especially on videos longer than two minutes.
• Shortcomings in Previous Benchmarks: Several open-source models achieve high accuracy on short segments but fail to maintain referential stability—demonstrating prior benchmarks' inefficacy at surfacing real-world weaknesses.
• Architectural Factors: Model scaling improves performance within families, but parameter count alone is not predictive—AVoCaDO (7B) outperforms video-SALMONN-2+ (72B) due to architectural and data curation differences.

Metric Validation: Downstream Task Correlation

To verify the practical significance of the new metrics, CapRiCorn-1K captions are used as "memory" input for LLM agent reasoning (M3-Bench-web) and as intermediate representations for video re-generation (LTX-2.3). Results confirm:

• $\text{Acc}$ and $\text{Cov}$ scores correlate strongly with agentic multimodal reasoning success.
• $\text{Ref}$ is even more tightly linked to performance on downstream person understanding and subject consistency in video generation, with Pearson coefficients above 0.98 in both cases.

  Figure 4: Strong correlation between CapRiCorn-1K metrics and downstream agentic reasoning and video generation outcomes, establishing metric reliability.

Error Analysis in Challenging Scenarios

Qualitative error inspection isolates three persistent model weaknesses:

• Clothing Changes: Models frequently lose subject identity across uniform changes in the same individual, yielding fragmented or ambiguous references.

  Figure 5: Typical failure in tracking a subject through multiple clothing changes; captions fail to maintain a stable referential chain.

• Multiple Subjects: Increasing subject count (especially with visually similar individuals) results in coreference breakdown and subject swap errors.

  Figure 6: Referential confusion among multiple visually similar subjects, leading to merging or fragmentation of subject tracks.

• Multi-Scene Transitions: Frequent and non-trivial scene changes severely challenge the maintenance of subject continuity in captions.

  Figure 7: Scene transitions disrupt subject tracking, resulting in ambiguous or incorrect coreference in captions.

Practical Implications and Future Directions

CapRiCorn-1K provides a highly diagnostic setting for future model improvement, compelling architectural, data, and training innovations targeted specifically at referential consistency and long-horizon temporal modeling. Practically, robust referentially coherent captioning is necessary for long-term agentic reasoning, memory-based video analysis, and controllable video generation pipelines.

Key future research directions include:

• Extension toward >10 minute videos and more complex interleaving of human and non-human entities.
• Development of architectures and training objectives explicitly targeting stable subject coreference chains over scene and state changes.
• Use of CapRiCorn-1K not only as an evaluation resource but as reinforcement/plug-in supervision for generation of real-world usable multimodal captions.

Conclusion

CapRiCorn-1K fills a significant void in video captioning research, introducing a challenging and fine-grained benchmark that measures both caption informational richness and the stability of subject referential chains throughout diverse, long-form videos. The benchmark, annotation protocol, and metrics are shown to be highly predictive of downstream success, setting a new experimental standard for progress in video captioning and multimodal reasoning (2606.21949).