LongShOTBench: Omni-Modal Video Benchmark

• LongShOTBench is a comprehensive benchmark for long-form video understanding, integrating vision, audio, and speech to evaluate multimodal large language models.
• It employs a five-stage pipeline—including multimodal captioning, scenario design, and human-validated rubrics—to ensure precise, interpretable scoring.
• Extensive experiments reveal challenges in temporal coherence and tool use, pushing advancements in omni-modal video analysis and reasoning.

LongShOTBench is a diagnostic benchmark and evaluation framework for long-form, omni-modal video understanding with tasks spanning vision, audio, and speech, integrating open-ended Q&A and tool-augmented agentic reasoning. It was introduced to address the limitations of prior benchmarks that either limit scope to vision-only or short clips, or rely on single-score metrics that obscure failure modes. With hour-scale videos and a rigorous, rubric-based, human-validated annotation pipeline, LongShOTBench advances the assessment of multimodal LLMs (MLLMs) along axes of temporal coherence, multi-modality, interpretability, and tool-use ability (Kurpath et al., 18 Dec 2025).

1. Origin, Motivation, and Goals

LongShOTBench was created from the observation that as MLLMs and multimodal agents evolve, existing video-understanding benchmarks fail to stress models’ abilities in truly long-horizon, multi-modal, and intent-driven contexts. Legacy datasets typically focus on either the temporal axis (long but uni-modal), or on multi-modality in brief segments, and they commonly reduce model evaluation to single exact-match metrics or ambiguous end-to-end scores. Such evaluation obscures failures such as losing track of events, missing off-screen audio cues, or failing to choose and orchestrate external tools.

LongShOTBench was designed to:

• Encompass extended video content (mean 44.7 min per video, total 117.5 hours across 157 samples).
• Require coherent reasoning over aligned vision, raw audio, and speech transcript streams.
• Support open-ended, scenario-driven questions, both single-turn and multi-turn, mapped to real or plausible user intents.
• Encourage interpretable and traceable scoring via fine-grained, criterion-weighted rubrics, rather than undifferentiated single numbers.
• Explicitly test agentic tool use by requiring models to invoke search, OCR, calculators, summarization, extraction modules, etc., within natural dialogues.

2. Benchmark Construction and Dataset Properties

LongShOTBench’s construction follows a five-stage pipeline:

1. Video Collection: Videos are selected from 92 public Video-MME corpus samples plus 65 manually curated YouTube sources, totaling 157 videos with average duration of 44.7 minutes and median 45.6 minutes.
2. Multimodal Caption Generation:
   • Segment-level splitting uses speech activity detection.
   • Speech is transcribed via Whisper-large-v3.
   • Visual dense captions are generated with Qwen-2.5-VL-32B.
   • Audio-event detection for music, applause, and ambient environment employs Audio-Flamingo-3.
   • Cross-modal summarization with Qwen3-30B fuses all modalities into coherent, time-aligned narratives.
3. Question and Scenario Design:
   • Up to five scenario-driven “viewing contexts” are attached to each video, strictly grounded in metadata.
   • Each is mapped to one of 32 defined tasks within five top-level categories.
   • Automatic question generation yields diverse, multi-step dialogues spanning difficulties (Level 1–5, with ∼30% at Level 3, ∼35% at Level 4, ∼25% at Level 5).
4. Answer and Rubric Creation:
   • Reference answers are produced by Qwen3-30B, with strict adherence to grounding (no unsupported inferences).
   • Each QA instance is paired with a rubric: 3–5 hierarchical criteria per answer, each assigned weighted scores (high: 7–10, medium: 4–6, low: 1–3, or negative for penalties).
   • The QA score is computed as

   $\text{score}_{QA} = \sum_{k=1}^K w_k s_k$

   where $w_k$ is criterion weight and $s_k \in \{+1, -1\}$ indicates satisfaction.

5. Human Validation: Ten expert annotators iteratively refine Q&A and rubrics, discard ambiguous or corrupted samples, and enforce strict metadata adherence.

The final dataset contains 2,083 video segments and 3,092 QA items, spanning heterogeneous narratives, conversational styles, and degrees of multimodal dependency.

3. Task Taxonomy and Agentic Tool Usage

LongShOTBench’s design introduces a hierarchical taxonomy of five main task categories, each with specialized sub-tasks:

Category	# Subtasks	Example Subtasks
Core Perception	4	Entity recognition, event understanding
Information	4	Retrieval, summarization, sentiment analysis
Multimodal	4	Cross-modal verification, motion analysis
Reasoning	4	Causal, quantitative, compositional reasoning
Agentic	16	Tool calls: web search, OCR, calculator, etc.

Within “agentic” tasks, questions may require the model to chain multiple tool invocations (e.g., transcribe_speech(…), extract_scene_text(…), web_search(…)), all orchestrated within potentially multi-turn dialogues. In total, 193 multi-turn conversational scenarios involve 891 explicit tool invocations.

The tool suite includes, among others: transcribe_speech, extract_scene_text, object_detection, calculator, web_search, detect_audio_events, summarize_segment, memory_tool, translate_text, and execute_code.

4. Multimodal Fusion, Model Ingestion, and Scoring Protocol

LongShOTBench mandates multimodal fusion at both preprocessing and task-execution levels:

• Vision is processed via frame sampling guided by scene detection and SigLIP embedding.
• Speech uses Whisper-small and Whisper-large for preprocessing/transcription.
• Audio is analyzed via Audio-Flamingo-3 for non-speech event recognition.
• OCR uses PaddleOCR for on-screen textual content.
• All features are consolidated in a multimodal vector database for abstraction and retrieval.

Model evaluation follows a native-video-only protocol: each model is required to ingest the full video using its own documented settings (not a universal sampling strategy), thereby avoiding frame-selection biases or mismatches with the model's native temporal handling.

Answer evaluation is two-stage:

1. Generation: Candidate model produces free-form answers (possible tool calls included) for each QA or multi-turn dialogue, with access to all modalities.
2. Scoring: Qwen3-14B, an open-source judge, examines satisfaction of each rubric criterion, yielding a QA score by weighted sum. For multi-turn tasks, an ideal trajectory protocol means each response is evaluated against the perfect dialog history, preventing error accumulation.

5. Experimental Results and Comparative Strengths

Zero-shot baseline results reveal substantial headroom for improvement:

Model	Overall Score (%)
Gemini-2.5-Flash	52.95
Qwen3-VL-8B	29.12
Qwen2.5-Omni	20.49
InternVL3.5-8B	18.39
LongShOTAgent (ours)	44.66

Category-level breakdown demonstrates that Gemini-2.5-Flash leads across Core Perception (41.04%), Reasoning (61.65%), Information (54.65%), and Multimodal (54.49%), with open-source systems rarely exceeding 30% in any category. The agentic pipeline achieves 38.25% on agentic tasks and 44.66% overall, closing much of the gap to closed-source baselines.

Ablation studies by video length indicate that all models degrade on longer samples: Gemini drops from 55.6% (≤30 min) to 47.2% (>60 min), underscoring the challenge of context retention and reasoning over hour-scale temporal spans.

Relative to prior datasets:

• LongShOTBench's average duration (44.7 min) surpasses most multimodal benchmarks (LongVALE ≈4 min, TriSense 15 min).
• It is among the first to align raw audio, speech, and vision directly (not via subtitles), allowing detailed analysis of cross-modal dependencies.
• Open-ended, scenario-driven queries and tool-use requirements push beyond simple captioning or retrieval.
• Diagnostic, rubric-based scoring allows partial credit and error attribution, instead of black-box accuracy.

6. Limitations and Prospective Extensions

While LongShOTBench sets new standards in long-form, multimodal, and agentic benchmark methodology, several limitations remain:

• Resource Consumption: Each complete round of evaluation requires approximately 30 GPU-hours per model with 4× A6000, limiting exploration of broad model pools.
• Scenario Constraints: Strict metadata grounding in Q&A avoids language drift but may under-represent highly creative or speculative user goals.
• Tool Scope: The current set of 16 tool types could be extended to include scene graphs, knowledge graphs, or live open-world retrieval.
• Protocol Evolution: As MLLM architectures expand in context window and fine-grained temporal ingestion, the native-video-only evaluation protocol may require adaptation.

A plausible implication is that advances in memory management, online feature adaptation, and hybrid symbolic-neural pipelines could yield significantly stronger results in agentic, hour-long multimodal tasks.

7. Relationship to Prior Long-Term Evaluation Frameworks

LongShOTBench is a semantic and technical successor to earlier efforts in long-term evaluation, exemplified by the TLP ("Track Long and Prosper") benchmark (Moudgil et al., 2017). Whereas TLP's focus was on pixel-level, single-object visual tracking over 8+ minute HD video (an order of magnitude longer than OTB), LongShOTBench generalizes the challenge to reasoning, question answering, and agentic tool use over hour-scale, multimodal content. Both frameworks underline that performance on short sequences or unimodal data does not extrapolate to long-duration, distributed, or compositional settings, and both advocate for benchmarks that diagnose failure modes and drive new methodologies specifically adapted for real-world, long-range tasks.

---

References:

• (Kurpath et al., 18 Dec 2025) A Benchmark and Agentic Framework for Omni-Modal Reasoning and Tool Use in Long Videos
• (Moudgil et al., 2017) Long-Term Visual Object Tracking Benchmark