Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidence

Abstract: The growing interest in embodied agents increases the demand for spatiotemporal video understanding, yet existing benchmarks largely emphasize extractive reasoning, where answers can be explicitly presented within spatiotemporal events. It remains unclear whether multimodal LLMs can instead perform abstractive spatiotemporal reasoning, which requires integrating observations over time, combining dispersed cues, and inferring implicit spatial and contextual structure. To address this gap, we formalize abstractive spatiotemporal reasoning from videos by introducing a structured evaluation taxonomy that systematically targets its core dimensions and construct a controllable, scenario-driven synthetic egocentric video dataset tailored to evaluate abstractive spatiotemporal reasoning capabilities, spanning object-, room-, and floor-plan-level scenarios. Based on this framework, we present VAEX-BENCH, a benchmark comprising five abstractive reasoning tasks together with their extractive counterparts. Our extensive experiments compare the performance of state-of-the-art MLLMs under extractive and abstractive settings, exposing their limitations on abstractive tasks and providing a fine-grained analysis of the underlying bottlenecks. The dataset will be released soon.

• The paper introduces VAEX-Bench, a controlled benchmark of 300 queries across 10 egocentric videos that compares extractive tasks with abstractive reasoning involving memory, maps, distance, navigation, and global counting.
• The paper finds a substantial accuracy decline on abstractive tasks: Gemini-3 Flash leads at 40.3%, while human annotators reach 81.7%, exposing weaknesses in temporal memory, spatial reconstruction, and cross-room aggregation.
• The paper shows that free-form answers reduce performance by up to 14.9 points and identifies trajectory-order memory, duplicate-aware counting, and consistent global coordinate mapping as key priorities for improving MLLMs.

Motivation and problem statement

The paper addresses a gap in how multimodal LLMs (MLLMs) are evaluated on spatiotemporal video understanding. Existing video spatiotemporal benchmarks, most notably VSI-Bench and VSTI-Bench, are characterized by the authors as extractive: the answer can be localized to a moment or viewpoint in the video and read off from directly visible evidence (e.g., object appearance order, nearest-object identification). While such tasks measure perceptual grounding, they do not test whether a model can synthesize fragmented egocentric observations into a coherent global representation of an environment — the capability the authors call abstractive spatiotemporal reasoning. This includes reconstructing floor plans, reasoning about allocentric directions between rooms, estimating metric distances under a shared scale, simulating navigation over an inferred layout, and aggregating object counts across rooms.

A key methodological observation motivates the benchmark design: when questions are annotated post-hoc on pre-captured real-world footage (e.g., ARKitScenes, ScanNet++), the visual evidence is fixed, making it difficult to systematically construct queries whose decisive evidence is dispersed across time and space. The paper instead adopts a query-conditioned construction pipeline in which scenarios and queries are designed first, and environments are built to contain the required evidence.

Taxonomy and task design

The authors formalize the extractive/abstractive distinction through a one-to-one expansion principle over five representative extractive task types:

Category	Extractive	Abstractive	Expanded capability
Chronology	Appearance Order	Memory–Action	Frame-local perception → memory-grounded perception
Direction	Relative Direction	Map Direction	Frame-local direction → allocentric orientation
Distance	Relative Distance	Map Scale	Local proximity → global metric estimation
Planning	Route Plan	Simulation	Local route planning → allocentric simulation
Counting	Object Counting	Global Counting	Single-room counting → multi-room aggregation

Each abstractive variant preserves the semantic intent of its extractive counterpart while removing the assumption that the decisive cue is observable at a single moment. The stated goal is not to replace extractive evaluation but to jointly assess both regimes, enabling controlled comparison of performance shifts.

Dataset construction

VAEX-Bench is built via a three-step pipeline. First, human annotators specify scenarios (environment typology, room composition, candidate objects) and design queries whose answers are uniquely determined by the resulting environment configuration. Second, scenes are materialized in SketchUp with query-driven object placements, and camera trajectories are designed under two constraints for abstractive tasks: temporal cue separation (evidence placed in distant parts of the traversal so short-window recognition is insufficient) and spatial mapping (trajectories unsolvable from local navigation cues alone). Third, scenes are rendered photorealistically with Enscape and recorded as egocentric videos at 30 fps, $640\times480$ resolution.

The resulting dataset comprises 10 egocentric navigation videos (average 6.7 rooms and 73.5 objects per scenario, average duration ~37 s), each grounded with 30 queries (5 abstractive + 5 extractive tasks × 3 queries), for 300 queries total. All annotation is human-driven: ten graduate-level annotators with top-tier computer vision publications constructed and cross-validated queries, with each scenario requiring roughly two to three weeks to produce. The authors are explicit that this manual process limits scale; they position VAEX-Bench as a high-precision diagnostic testbed rather than a large corpus, and note partial automation as future work. A further caveat is the synthetic-to-real domain gap: although environments are manually designed (not generatively synthesized) and validated by human baselines achieving high accuracy, results may not transfer directly to real-world footage.

Main evaluation results

Fourteen state-of-the-art MLLMs (five proprietary, nine open-source) were evaluated zero-shot with unified frame sampling (32 frames, except Gemini due to API constraints), fixed decoding configuration, and Accuracy@5 over five runs. Human annotators achieve 88.0% average accuracy on extractive tasks and 81.7% on abstractive tasks, confirming solvability while indicating genuine cognitive demand — particularly on Map Scale (60.0% human).

The central finding is a substantial drop from extractive to abstractive settings across all models. The best proprietary model on abstractive tasks, Gemini-3 Flash, reaches only 40.3% versus 50.0% extractive; GPT-5.2 drops from 44.5% to 30.1%; Claude 4.5 Haiku falls to 15.3%. Notably, the ranking shifts between regimes: Gemini-3 Pro leads on extractive tasks (55.6%) but Gemini-3 Flash substantially outperforms it on abstractive tasks (40.3% vs. 29.7%), which the authors interpret as evidence that gains in short-horizon recognition do not translate into abstractive spatiotemporal reasoning. Open-source models are competitive with proprietary ones on extractive tasks but show larger gaps on abstractive tasks, and scaling within the Qwen3-VL family does not yield monotonic abstractive gains (Qwen3-VL-235B scores 26.7% vs. 29.9% for the 32B variant). Global Counting is a consistent weakness across all models, far below the human level of 82.7%, pointing to failures in entity persistence and duplicate-aware aggregation under partial observability.

Free-form generation and diagnostic analyses

Converting MCQs to free-form generation degrades performance across all six top-performing models evaluated, with average drops up to 14.9 points (Gemini-3 Flash) and individual task drops as large as 22 points (GPT-5.2 on Map Scale). This indicates MCQ evaluation partially overestimates capability through option elimination and chance selection.

Fine-grained diagnostics attribute failures to three bottlenecks. Perceptual: error breakdowns for Global Counting show miscounting is the dominant error mode, followed by omission; hallucination is comparatively rare, implying the difficulty lies in maintaining and aggregating counts rather than detecting nonexistent objects. Temporal: two-hop analysis of Memory–Action shows the dominant failure is a global memory error at Hop 1 (incorrectly reconstructing the visited-room sequence); conditional pass rates at Hop 2 remain moderate even when Hop 1 succeeds, identifying trajectory-order memory as the primary bottleneck. Spatial: floor-plan reconstruction on a $10\times10$ grid reveals distorted room boundaries, inconsistent metric proportions, rotational misalignment, incorrect adjacencies, and fragmented layouts — indicating models fail to anchor a consistent global coordinate frame from egocentric observations.

Limitations and open questions

The paper concedes several limitations directly. Scale is modest (10 videos, 300 queries) owing to the fully manual construction pipeline (~USD 600 per tool license, ~USD 350 per scenario in annotator compensation, two to three weeks per scenario); the authors argue this trades coverage for rigor and diagnostic control. The synthetic setting, while deliberately chosen for controllability and validated against human baselines, leaves open whether observed failure modes persist on real captured footage. Statistical variance is nontrivial for some tasks given the limited instance counts, though relative rankings remain stable across five runs. Open questions include whether automated pipeline generation can preserve the controllability guarantees (unambiguous answers, distributed evidence, targeted scenario factors), and whether the identified bottlenecks — trajectory-order memory, duplicate-aware counting, and globally consistent spatial reconstruction — can be addressed without architectural changes beyond current MLLM designs.

Conclusion

VAEX-Bench contributes a controlled, scenario-driven benchmark that jointly evaluates extractive and abstractive spatiotemporal reasoning in egocentric video, enabled by a query-conditioned construction pipeline that places decisive evidence before video synthesis. Its evaluation of fourteen frontier MLLMs quantifies a persistent gap between frame-grounded extraction and integration-based inference, isolates specific failure modes in temporal memory, cross-room aggregation, and spatial reconstruction, and demonstrates that MCQ formats and short-horizon perceptual strength both overstate apparent spatiotemporal competence.