--- title: 'VSI-Bench: Vision-Language Spatial Reasoning' url: https://www.emergentmind.com/topics/vsi-bench type: topic --- # VSI-Bench: Vision-Language Spatial Reasoning Visual Spatial Intelligence Benchmark (VSI-Bench) is a comprehensive evaluation suite for vision-language models (VLMs) centered on physical spatial reasoning, relational judgment, and navigation in 3D visual environments. VSI-Bench focuses specifically on assessing models’ ability to parse, conceptualize, and operate on egocentric video data through a set of objectively-marked, spatially-informed question answering (QA) tasks. By standardizing datasets, metrics, and sub-task divisions, VSI-Bench enables detailed, quantitative comparisons of spatial reasoning capacity across a wide range of multimodal models and approaches [2509.24473]. ## 1. Dataset Structure and Task Taxonomy VSI-Bench comprises approximately 5,130 video question–answer pairs, each constructed from short egocentric video sequences and sourced from three major real-world 3D scan datasets: ARKitScenes, ScanNet, and ScanNet++. Each sample consists of a 32-frame video, uniformly sampled across varied trajectories and environments, together with a natural language question targeting fine-grained spatial properties, relations, or events within the scene. The overall QA tasks are partitioned into eight sub-tasks, organized into two broad classes: - **Numerical Sub-Tasks (regression, mean-relative-accuracy as metric):** 1. Object counting 2. Absolute distance estimation 3. Object-size estimation 4. Room-size estimation - **Multiple-Choice Sub-Tasks (classification, exact-match accuracy as metric):** 5. Relative distance (“which object is closer?”) 6. Relative direction (“which direction is X from Y?”) 7. Route planning (“which path?”) 8. Appearance order (“which object was seen first/last?”) This sub-task partition covers both metric spatial computation and relational reasoning, as well as pseudo-temporal aspects via appearance order. ## 2. Evaluation Metrics and Protocols Numerical sub-tasks use **mean relative accuracy (MRA)**, computed across a set of confidence thresholds $\mathcal{C} = \{0.5, 0.55, ..., 0.95\}$ to quantify proximity between model prediction and ground-truth answer. For classification sub-tasks, exact-match accuracy is reported: \[ \text{Acc} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}[a_i = \hat{a}_i] \] where $a_i$ is the true answer and $\hat{a}_i$ is the model’s prediction. Zero-shot evaluation is employed for most benchmarked systems, where models receive only the video frames and associated question as input. The standard evaluation pipeline uses lmms-eval toolkit with a temperature of 0 and a maximum output sequence of 1024 tokens per QA instance, and employs prompt engineering with explicit chain-of-thought (\, \boxed{} tags) to elicit stepwise spatial inference [2509.24473]. ## 3. Baseline Model Performance VSI-Bench enables direct comparison of proprietary, open-source, and specialized spatially-tuned VLMs: | Model (family/size) | Zero-Shot VSI-Bench Accuracy (%) | |-------------------------------|:--------------------------------:| | GPT-4o (OpenAI, proprietary) | 34.0 | | Gemini-1.5 Pro | 48.8 | | Gemini-2.0 Flash | 45.4 | | LongVA-7B | 29.2 | | Qwen2.5VL-3B (open-source) | 29.2 | | Qwen2.5VL-7B | 24.8 | | RoboBrain2.0-7B | 43.0 | | RoboBrain2.0-32B | 43.1 | | VILA-1.5-40B | 31.2 | | LLaVA-OneVision-72B | 40.2 | | LLaVA-Video-72B | 40.9 | | Spatial-MLLM-4B (prior SOTA) | 48.4 | | M2-Reasoning-7B | 42.3 | Results exhibit substantial variability, with proprietary or specially fine-tuned models attaining near 49% accuracy and most general-purpose VLMs remaining under 40%. Route planning and spatial relation subtasks were comparatively strong for spatially tuned systems, while appearance order (temporal) was not as improved [2509.24473]. ## 4. Impact of Geometry-Centric Surrogate Task Fine-Tuning Fine-tuning with geometry-focused surrogate tasks, specifically via the Euclid30K dataset (≈30,000 multimodal geometry problems with symbolic and numeric answers), leads to statistically robust gains. Models such as RoboBrain2.0-Euclid-7B achieve 49.6% accuracy, surpassing prior state-of-the-art Spatial-MLLM-4B (48.4%) and proprietary Gemini-1.5 Pro (48.8%). Improvements are consistent across all parameter scales, averaging +2 to +7 percentage points [2509.24473]. Per-subtask gains for RoboBrain2.0-Euclid-7B after Euclid30K fine-tuning: | VSI-Bench Sub-Task | Pretrain Accuracy (%) | Post-Euclid30K Accuracy (%) | |----------------------|:--------------------:|:--------------------------:| | Object Counting | 46.0 | 66.4 | | Abs. Distance | 32.7 | 36.9 | | Object Size | 58.9 | 66.3 | | Room Size | 35.9 | 40.5 | | Rel. Distance | 45.9 | 48.3 | | Rel. Direction | 41.5 | 45.3 | | Route Planning | 30.9 | 35.6 | | Appearance Order | 55.2 | 57.8 | These improvements are causally linked to the geometry-specific curriculum: ablation studies with matched-size, non-geometry spatial QA (Clevr-CoGenT) show 1–4 percentage point lower gains. The results suggest that transfer arises from the acquisition of fundamental Euclidean axioms and spatial priors (congruence, parallelism, metric inference), rather than generic multi-task RL or broader dataset scaling [2509.24473]. A notable limitation emerges for sub-tasks with an explicitly temporal component (e.g., appearance order), which saw only marginal improvement from static-geometry pretraining—indicating the need for further inclusion of spatio-temporal or 3D-rotation prior data for comprehensive coverage. ## 5. Technical Implementation and Model Training Regime VSI-Bench-compliant fine-tuning pipelines for geometry-centric transfer employ Group Relative Policy Optimization (GRPO) with the following settings: - **Reward Design:** - Symbolic answers: MathVerify for LaTeX equivalence. - Numeric answers: score of 1 if relative error ≤1%, 0 otherwise. - Multiple-choice: 1 for exact match. - **Optimization Objective:** \[ \mathcal{L}_{\rm GRPO}(\theta) = \mathbb{E}_{q,o_i}\left[ \mathcal{J}(\theta) - \beta \,\mathrm{KL}[ \pi_\theta \| \pi_{\rm ref} ] \right] \] $\mathcal{J}(\theta)$: token-level PPO reward function. - **Training Details:** - 10 epochs on 64 × NVIDIA H100 GPUs - Adam optimizer (1e-6 learning rate, 1e-2 weight decay) - PPO clip $\epsilon=0.2$, KL-coefficient $\beta=10^{-2}$ - 8 rollouts per question, actor batch 128, max-grad-norm 1.0 - Image resolution between $512 \times 512$ and $2048 \times 2048$ - **Evaluation:** - All VSI-Bench results are zero-shot—no further adaptation is performed after surrogate task fine-tuning. - Prompting with explicit chain-of-thought (CoT) format is critical for extracting stepwise reasoning. ## 6. Role in the Broader Spatial Reasoning Landscape VSI-Bench sits alongside other spatial and spatial–temporal benchmarks (e.g., Super-CLEVR, Omni3DBench, MindCube), but distinguishes itself via its strong orientation toward egocentric 3D video, real-world scan data, and the amalgamation of metric, relational, and route-planning subtasks. The observed transfer from geometry-based pretraining via Euclid30K is robust even when compared to large-scale non-geometry multimodal QA (as shown in causal ablation, Table 5) [2509.24473]. These findings suggest that fundamental geometric axioms and deductive reasoning skills provide transferrable priors directly conducive to a wide array of spatial inference and navigation tasks presented in VSI-Bench. ## 7. Limitations and Future Directions Despite robust gains from geometry-centric surrogate learning, VSI-Bench evaluations suggest persistent open challenges. Specifically, mental-rotation tasks and subtasks requiring explicit temporal memory benefit less from geometry-heavy pretraining, likely due to the predominance of plane-geometry problems and the lack of temporal or 3D-rotation representation within the current Euclid30K corpus. A plausible implication is that an augmented curriculum incorporating more solid (3D) geometry and temporally-dependent spatial tasks may yield further advances in spatio-temporal intelligence [2509.24473]. Source: https://www.emergentmind.com/topics/vsi-bench