LVSum: A Benchmark for Timestamp-Aware Long Video Summarization

Abstract: Long video summarization presents significant challenges for current multimodal LLMs (MLLMs), particularly in maintaining temporal fidelity over extended durations and producing summaries that are both semantically and temporally grounded. In this work, we present LVSum, a human-annotated benchmark designed specifically for evaluating long video summarization with fine-grained temporal alignment. LVSum comprises diverse long-form videos across 13 domains, each paired with human-generated summaries containing precise temporal references. We conduct a comprehensive evaluation of both proprietary and open-source MLLMs on LVSum, assessing performance using newly introduced LLM-based metrics for content relevance and modality coherence, alongside standard evaluation metrics. Our experiments reveal systematic gaps in temporal understanding among existing MLLMs and offer insights that establish a new foundation for advancing temporal reasoning in long video summarization.

• The paper introduces LVSum, a benchmark offering multi-reference, human-annotated, interval-level supervision for long video summarization.
• It proposes novel evaluation metrics, Content Relevance and Modality Coherence, to measure temporal alignment and cross-modal grounding.
• The results expose challenges like over-coverage, temporal misalignment, and modality interference in current multimodal large language models.

Timestamp-Aware Long Video Summarization: The LVSum Benchmark

Introduction

Long video summarization, which requires extracting concise, temporally-localized and semantically rich representations from extended video content, remains a core unsolved problem for multimodal LLMs (MLLMs). Existing benchmarks focus on short or narrowly scoped videos and fail to provide the necessary granularity and diversity to rigorously assess current MLLMs' capabilities in temporally grounded summarization. "LVSum: A Benchmark for Timestamp-Aware Long Video Summarization" (2604.10024) systematically addresses this gap by introducing a high-quality, human-annotated dataset for long-form video summarization, accompanied by novel evaluation metrics that probe both temporal and cross-modal grounding.

Dataset Construction and Benchmark Design

LVSum consists of 72 videos with durations ranging from 10 to 55 minutes (average: 16 min) drawn from 13 diverse categories, including lectures, vlogs, documentaries, podcasts, news, and more. Each video is annotated by up to 10 independent annotators who identify and precisely timestamp salient segments, assign importance scores, and provide concise textual descriptions, all under a strict 15% total summary-duration constraint. This fine-grained, multi-reference protocol ensures dense perceptual supervision, supports robust human agreement measurement, and offers strong subjective evaluation coverage.

In contrast to existing benchmarks—such as SumMe [gygli2014creating], TVSum [Song2015TVSumSW], VideoXum [lin2023videoxum], and Instruct-V2Xum [hua2025v2xum]—LVSum is uniquely positioned for long-form, interval-anchored, multi-annotator evaluation. Importantly, no prior dataset jointly offers (i) multi-reference long videos, (ii) interval-level importance supervision, and (iii) explicit timestamped descriptions at scale, making LVSum a critical resource for systematic benchmarking of MLLMs in real-world, temporally-aware summarization settings.

Methodology: Model Evaluation and Metrics

Evaluation is conducted on prominent proprietary and open-source MLLMs, including Opus-4.5, Gemini-2.5-Pro, and Qwen3-VL-235B-A22B-Instruct, using both video frames and aligned transcripts as input modalities. The models are prompted to produce temporally anchored, length-constrained summaries consisting of ranked segments with start/end timestamps, importance labels, and descriptions.

Metrics

To address the inadequacy of traditional metrics in capturing temporal and semantic fidelity, two new LLM-based automatic metrics are proposed:

• Content Relevance (CR): Directly assesses semantic alignment between predicted and reference summaries, penalizing omission of key events and inclusion of irrelevant information.
• Modality Coherence (MC): Quantifies cross-modal grounding by scoring whether the predicted descriptions are supported by the corresponding visual/audio content (i.e., penalizing hallucinations, temporal drift, and cross-modal inconsistencies).

These are used alongside rank-based metrics (Kendall’s $\tau$, Spearman’s $\rho$) computed at second-level granularity and enforced summary length budgets, reflecting practical summarization requirements in long-form settings.

Results and Failure Modes

Benchmarking results highlight three systematic failure modes in current models:

1. Over-coverage: Models such as Opus-4.5 and Qwen3-VL-235B generate summaries with durations that overshoot the imposed 15% budget, revealing inability to perform effective temporal compression and selection.
2. Temporal misalignment: Models often capture semantically important content but with imprecise timestamp localization, leading to degraded temporal fidelity and narrative flow.
3. Cross-modal mismatch: Descriptions assigned to timestamped intervals hallucinate actions/objects or contain internal contradictions with respect to the actual video segment, a failure robustly detected by the MC metric.

While rank correlations (e.g., $\tau \sim 0.09-0.10$ for top models vs. 0.13-0.14 human upper bound) indicate that state-of-the-art models approach 75% of human-level performance, these numbers obscure significant deficits. Notably, Gemini-2.5-Pro adheres most closely to budget constraints and exhibits the highest MC score (4.32/5), whereas Qwen3-VL-235B is highly verbose and shows negative multimodal synergy: temporal ranking accuracy drops when both text and visual inputs are present, indicating modality interference rather than fusion.

These findings are illustrated by the qualitative comparison of summary outputs for a sample video: human summaries are event-centric and selective, Gemini-2.5-Pro yields compact, coherent summaries, Opus-4.5 produces detailed but lengthy segments, and Qwen3-VL-235B outputs lengthy transcript-like paraphrases with weak prioritization.

(Figure 1)

Figure 1: Failure cases illustrating distinct evaluation modes. (a) Low Content Relevance (CR): summary omits salient events. (b) Low Modality Coherence (MC): textual descriptions contradict visual evidence within the predicted interval.

Ablation Studies

Modality and Frame Sampling

Ablation on input modalities reveals that transcript-only input consistently outperforms video-only, suggesting that linguistic cues (dialogue/narration) are the primary driver for salience ranking in current MLLMs. However, incorporating both video and transcript confers some gain, except for Qwen3-VL-235B, where multimodal input degrades performance and inflates summary length—further validating the modality interference phenomena.

Increasing video frame sampling density from 24 to 128 frames improves ranking accuracy for both Gemini-2.5-Pro and Qwen3-VL-235B, with performance gains plateauing at higher values, indicating diminishing returns and model sensitivity to sampling density, especially for models not optimized for dense long-context fusion.

Category-Level and Constraint Compliance

Category-wise evaluation reveals content-dependent model behavior: Gemini-2.5-Pro is robust across structured and narrative-rich domains, Opus-4.5 excels for lectures but is prone to over-coverage, and Qwen3-VL-235B exhibits severe drift and verbosity in event-dense domains. Additional ablations show that prompt structure (i.e., re-positioning of length constraint in Qwen3-VL-235B) partially mitigates, but does not eliminate, instruction drift and overrun effects.

Qualitative Analysis

Comparisons across models and diverse video types confirm that Gemini-2.5-Pro excels in semantic selection and temporal constraint adherence, Opus-4.5 delivers semantically accurate but lengthy summaries, and Qwen3-VL-235B serializes events with little compression. Failure case visualization (see Figure 1) demonstrates that classical rank-based metrics alone fail to penalize missing content or hallucinations, validating the necessity of concurrent use of CR and MC.

Theoretical and Practical Implications

LVSum establishes a rigorous empirical foundation for timestamp-aware, budget-constrained long-video summarization and exposes fundamental challenges—temporal compression, multimodal grounding, and constraint-aware selection—that are largely unmet by current MLLMs. The newly introduced MC and CR metrics provide critical diagnostic tools for progress on these axes, and their adoption will likely become standard for temporally-grounded video understanding evaluation.

On the practical front, the systematic over-coverage and misalignment issues observed mean that deployment of current MLLMs for end-user-facing summarization, search, or retrieval applications should be accompanied by careful constraint-enforcement or post-processing. Theoretically, the modality interference findings and abysmal multimodal fusion in some models suggest that future architectures must more deeply integrate temporal reasoning mechanisms, cross-modal alignment modules, and explicit compression controllers.

Future directions include the design of models with built-in temporal salience filters, better multimodal fusion representations, dynamic context windowing, and hierarchical summarization heads. As LVSum exposes clear gaps to human-level temporal abstraction and cross-modal grounding, it will be a driving force in incentivizing granular multitask pretraining, improved temporal localization objectives, and refinement of summary budgeting mechanisms.

Conclusion

LVSum provides the first comprehensive benchmark for timestamp-aware summarization of long videos with multi-reference, interval-aligned human labels. Empirical evidence demonstrates that while leading MLLMs are semantically capable, they lack robust mechanisms for temporal compression, precise grounding, and multimodal integration. Through novel metric design and detailed error analysis, this work charts essential directions for the next generation of temporally and cross-modally grounded video summarization systems.