SceneBench: Scene-Aware Video Benchmark
- SceneBench is a long-video benchmark that evaluates scene-level comprehension by focusing on coherent, multi-minute segments rather than isolated clips.
- It encompasses diverse tasks—such as Title Pred, Scene QA, and SceneQA-Audio—to diagnose failures in global context retention in vision-language models.
- The introduction of Scene-RAG, which dynamically retrieves scene context, demonstrates improvements (up to +4.3% average gain) in long-context reasoning tasks.
SceneBench is a long-video understanding benchmark for vision-LLMs (VLMs) that foregrounds scene-level comprehension rather than only fine-grained perception or coarse summarization. The paper defines a scene as “a coherent segment of a video in which both visual and semantic contexts remain consistent, aligning with human perception,” and frames the central question as whether current VLMs can reason effectively over long, scene-level contexts. In that formulation, SceneBench is intended to reveal failures of long-range contextual retention in long-video reasoning and to motivate more robust, human-like video comprehension (Chen et al., 28 Mar 2026).
1. Conceptual framing and scene definition
Long video understanding is presented as “a core challenge in multimodal learning,” and SceneBench is introduced against the observation that existing benchmarks mainly emphasize either fine-grained perception or coarse summarization. The benchmark therefore shifts emphasis to scene-aware long-context reasoning, with the scene used as the basic semantic-temporal unit rather than the isolated clip or the whole-video summary (Chen et al., 28 Mar 2026).
A central methodological claim in the available material is that long-video evaluation cannot be reduced to a trivial scaling of short-video evaluation. The rebuttal fragment states:
“A direct same-compute comparison is fundamentally flawed for long videos, since feeding equivalent frames directly causes OOM while processing frames sequentially forces the baseline to treat the long video as disjointed short clips, destroying the global context required for reasoning.”
This framing makes SceneBench a benchmark not simply for perception over extended duration, but for global context preservation across scenes. A plausible implication is that the benchmark is designed to expose a specific failure mode: models may successfully process local content while still failing to maintain scene continuity over multi-minute videos.
2. Available benchmark structure and task suite
The available material identifies SceneBench as a scene-aware long-video benchmark with “scene-level, multi-minute data” and “task diversity,” but it does not provide the full dataset specification. What can be recovered reliably is that the evaluation suite includes the following task categories: Title Pred, Comm. Pred, Clip QA, Scene QA, SceneQA-Audio, and I-VQA. The fragment therefore supports describing SceneBench as a multi-task benchmark rather than a single-task evaluation set (Chen et al., 28 Mar 2026).
The task names indicate a distinction between clip-level and scene-level reasoning, and the inclusion of SceneQA-Audio indicates that audio is part of the intended benchmark scope. However, the full task definitions are not specified in the available material. The same limitation applies to dataset size, number of videos, video sources, segmentation procedure, scene annotation protocol, and train/validation/test partitioning.
One concrete implementation detail is available for audio-related evaluation. The text states that, for open-source models, SceneQA-Audio was evaluated in a visual-only setting because “current omini models (e.g., Qwen2.5-Omni, InteractiveOmni) process audio with whisper-based model, only support audio under 30 seconds.” This establishes an important practical constraint: the benchmark includes audio-sensitive tasks, but current long-duration audio support in evaluated open-source omni models is limited.
3. Scene Retrieval-Augmented Generation
To validate the benchmark’s diagnosis of long-context failure, the paper introduces Scene Retrieval-Augmented Generation (Scene-RAG). Its stated purpose is to “construct[] a dynamic scene memory by retrieving and integrating relevant context across scenes,” thereby giving the model access to scene-level evidence without forcing monolithic long-context encoding (Chen et al., 28 Mar 2026).
The available latency table makes the modular structure partly visible. It names InternVideo2 as the visual encoder, QwenAudio2 as the audio encoder, and Qwen3 as the reasoning model. On a 2,767-second video, the table reports:
- Offline Preprocess: Visual Enc. 273.52, Audio 3.20, Total 276.72
- Online Inference: Visual Enc. 1.04, Audio 0.19, LLM 61.06, Total 62.29
This suggests a two-stage pipeline. A plausible implication is that Scene-RAG first builds a scene-level multimodal memory offline, then retrieves relevant scene evidence at question time and passes the retrieved context to the LLM for answer generation. The precise retrieval strategy, scoring function, memory representation, and fusion mechanism are not specified in the available material.
4. Quantitative evidence of scene-level difficulty
The abstract states that SceneBench reveals “a sharp drop in accuracy when VLMs attempt to answer scene-level questions,” and reports that Scene-RAG improves VLM performance by +2.50%. The fragmentary results table further provides a concrete task breakdown for MPLUG-OWL3-V (7B), including a baseline and a Scene-RAG variant (Chen et al., 28 Mar 2026).
| Task | MPLUG-OWL3-V | + Scene-RAG |
|---|---|---|
| Title Pred | 86.7 | 87.7 |
| Comm. Pred | 65.1 | 71.3 |
| Clip QA | 73.1 | 78.1 |
| Scene QA | 53.8 | 57.6 |
| SceneQA-Audio | 50.4 | 56.3 |
| I-VQA | 70.8 | 74.6 |
| Avg. | 66.6 | 70.9 |
The table reports an average gain of +4.3 for MPLUG-OWL3-V with Scene-RAG. It also shows that Clip QA = 73.1, while Scene QA = 53.8 and SceneQA-Audio = 50.4 in the baseline setting. The supplied analysis explicitly identifies this as a 19.3-point drop from Clip QA to Scene QA and a 22.7-point drop from Clip QA to SceneQA-Audio. This is consistent with the paper’s argument that scene-level, long-context reasoning is substantially more difficult than local clip understanding.
The same fragment reports task-wise improvements with Scene-RAG across all listed categories: +1.0 on Title Pred, +6.2 on Comm. Pred, +5.0 on Clip QA, +3.8 on Scene QA, +5.9 on SceneQA-Audio, and +3.8 on I-VQA. These numbers support the claim that retrieval over scene memory improves performance broadly rather than only on a single subtask.
5. Benchmark significance within long-video evaluation
SceneBench is positioned as addressing “gaps in long-video benchmarks through scene-level, multi-minute data.” On the evidence available, its distinctive contribution is not merely longer temporal windows, but a re-centering of evaluation around the scene as the unit of temporal-semantic coherence. This matters because the benchmark is intended to test whether a model can retain and use context across multiple scenes, rather than only extract local details or produce coarse summaries (Chen et al., 28 Mar 2026).
The available material also identifies Scene-RAG as a complementary methodological argument: if sequential clip processing “destroy[s] the global context required for reasoning,” then improved retrieval over scene memory becomes indirect evidence that the limiting factor is not only perception, but context retention and access structure. In that sense, SceneBench functions both as a benchmark and as a diagnostic lens on how current long-video VLM pipelines break down.
The benchmark’s significance is therefore tightly linked to the forgetting problem. The paper’s stated hope is that SceneBench will encourage research toward VLMs with “more robust, human-like video comprehension,” which places the benchmark in the broader line of work seeking long-context multimodal reasoning rather than short-horizon recognition.
6. Documentation limits, practical constraints, and open questions
The available material leaves several aspects of SceneBench unspecified. It does not provide the full dataset size, number of videos, source domains, scene segmentation pipeline, annotation instructions, quality-control procedure, split definitions, exact task prompts, evaluation metric definitions, human baseline, or the full experimental tables. An annotation pipeline figure is mentioned, but its operative details are not recoverable from the supplied text (Chen et al., 28 Mar 2026).
Comparable gaps remain for Scene-RAG. The available text does not specify the exact retrieval algorithm, top- selection rule, retrieval scoring function, learned versus non-learned memory, or formal training objective. No mathematical formulation of the retrieval mechanism is provided in the supplied material. The only explicit formulas recoverable for the paper’s main contribution are therefore limited to the descriptive statements in the abstract and the module/runtime tables in the fragment.
The material does, however, expose two concrete practical constraints. First, audio handling for long clips remains limited in currently available open-source omni models, forcing visual-only evaluation for SceneQA-Audio in that setting. Second, offline preprocessing is substantial for very long videos: on the reported 2,767-second example, offline preprocessing takes 276.72, while online inference takes 62.29, with 61.06 attributed to the LLM stage.
Taken together, the available evidence supports a precise but bounded characterization. SceneBench is a scene-aware long-video benchmark for testing whether VLMs can preserve and exploit scene-level context across multi-minute videos. The available quantitative results show that scene-level tasks are markedly harder than clip-level tasks, and that Scene-RAG improves performance by introducing dynamic scene memory. At the same time, many foundational details of the benchmark’s dataset construction and full protocol remain unavailable in the supplied material, so its current encyclopedic description is necessarily centered on its conceptual framing, visible task suite, reported performance patterns, and documented implementation fragments.
