---
title: SpatialScore-Hard Collection
url: https://www.emergentmind.com/topics/spatialscore-hard-collection
type: topic
---

# SpatialScore-Hard Collection

SpatialScore-Hard Collection is a rigorously curated suite designed to characterize the frontier of multimodal spatial reasoning in vision-language models (VLMs) and multimodal large language models (MLLMs). Distinct from generic spatial benchmarks, SpatialScore-Hard aggregates the most challenging samples—those that elude both open-source and commercial models—across diverse spatial tasks, with validated ground truth and category balance. Its construction, evaluation protocols, and experimental impact on state-of-the-art agents mark it as a central resource for diagnosing and advancing 3D spatial perception in modern AI systems [2505.17012][2512.20934].

## 1. Purpose and Formal Definition

SpatialScore-Hard isolates samples representing genuine failure cases for leading MLLMs in 3D spatial understanding. Traditional benchmarks such as SpatialScore (28,000 samples) encompass a wide range of difficulty, including many items solvable by current models. SpatialScore-Hard’s central mandate is to spotlight edge cases that resist solution by almost all models—making it an ultimate stress test for spatial reasoning, especially in vision-based tasks involving geometric complexity.

Formally, for a voting pool of $M$ models ($M=20$, parameter scale: 1B–78B), and a subset of "large" models $\mathcal{L}$ ($|\mathcal{L}|=4$, $\geq$32B parameters), for each candidate sample $i$:
\[
D_i = \sum_{m=1}^M \mathbf{1}[\text{model } m \text{ fails on } i], \quad
L_i = \sum_{m\in \mathcal{L}} \mathbf{1}[ \text{model } m \text{ fails on } i ]
\]
A sample is placed in SpatialScore-Hard if:
\[
D_i \geq \tau, \quad L_i \geq \lambda
\]
with $\tau = 16$ (at least 16/20 models fail) and $\lambda = 2$ (at least 2/4 large models fail).

Human annotators then verify label correctness and ensure balanced coverage across task categories. After filtering for ambiguity and category oversampling, the final collection size is $N_{\text{Hard}} = 1,400$ samples [2505.17012].

## 2. Dataset Composition and Category Structure

SpatialScore-Hard contains 1,400 validated samples covering eight super-categories:

- **Counting**
- **Object Localization** (2D/3D bounding box grounding)
- **3D Positional Relations** (e.g., above/behind relationships)
- **Depth & Distance Estimation**
- **Object Properties** (size, orientation)
- **Camera & Image Transformation** (homography, pose estimation)
- **Point/Object Tracking**
- **Others** (e.g., route planning in video)

Modalities are distributed as:
- Single-image: ~600
- Multi-image (pairs/sequences): ~400
- Video: ~400

Three QA formats are supported:
- Judgment (yes/no): ~250
- Multiple-choice: ~800
- Open-ended (numeric, free-form): ~350

The data sources span 12 distinct constituents, notably VGBench, SpatialSense, SpatialBench, QSpatialBench, CV-Bench, 3DSRBench, VSI-Bench, MMIU, BLINK, MMVP, RealWorldQA, CA-1M/ScanNet subsets. Per-category representation is balanced for diversity [2505.17012][2512.20934].

## 3. Selection and Curation Process

SpatialScore-Hard employs a hybrid algorithmic-human methodology:

1. **Difficulty Screening:**
   - Compute $D_i$ and $L_i$ for every sample in SpatialScore ($|S|=28,000$).
   - Select samples with $D_i \geq 16$ and $L_i \geq 2$ ($\sim$4,300 initial candidates).

2. **Expert Verification:**
   - Human experts validate ground truth and discard ambiguous/low-quality items.
   - Category rebalance eliminates distortion from oversampled failure types.

3. **Finalization:**
   - Retain $N_{\text{Hard}}=1,400$ samples, each meeting strict error and diversity criteria.

The process ensures that remaining samples expose authentic weaknesses in modelled spatial understanding rather than annotation flaws or dataset artifacts [2505.17012].

## 4. Evaluation Protocols and Benchmarks

SpatialScore-Hard supports rigorous evaluation across answer types:

- **Accuracy (Judgment & Multiple-choice):**
  \[
  \mathrm{Acc} = \frac{1}{N} \sum_{i=1}^N \mathbf{1} [\hat{y}_i = y_i] \times 100\%
  \]

- **Numeric Tolerance (Distance/Size, open-ended):**
  \[
  0.5\,y_i \leq \hat{y}_i \leq 2\,y_i
  \]
  or, in TVP’s protocol for its reduced set,
  \[
  \text{Accuracy}_{\pm 10\%}(\hat{y},y) = 1 \left\{ \frac{|\hat{y}-y|}{y} < 0.1 \right\}
  \]

- **Mean Absolute Error (Pose-angle):**
  \[
  \mathrm{MAE}_\theta = \frac{1}{N} \sum_{i=1}^N |\hat{\theta}_i - \theta_i| \quad [\text{degrees}]
  \]

Benchmark results on the full collection:

| Method                       | Overall Accuracy (%) |
|------------------------------|---------------------|
| InternVL3-78B                | 21.79               |
| GPT-4o (API)                 | 30.57               |
| SpatialAgent (Intern-PE)      | 46.08               |
| SpatialAgent (Qwen-ReAct)     | 30.29               |

On the reduced 256-question subset (TVP study):

| Method              | 3DSR-B (%) | SpatialSense (%) | VG-Bench (%) | Overall (%) |
|---------------------|------------|------------------|--------------|-------------|
| GPT-4o              |    52.1    |      46.5        |     20.3     |    42.6     |
| TVP (zero-shot)     |   52.9     |     59.2         |    43.8      |   52.3      |

TVP’s transductive tools deliver a +9.5 percentage point overall improvement over GPT-4o, and major per-category gains, with particularly strong zero-shot transfer on difficult metric estimation and 3D relation queries [2505.17012][2512.20934].

## 5. Task Examples and Difficulty Analysis

Representative SpatialScore-Hard samples include:

- **Homography Matrix Matching:** Identify which warped image matches a provided $3\times 3$ matrix $H$—requiring keypoint correspondence estimation and mental inversion of $H$.
- **Deep Distance Estimation:** Infer metric distances between partially occluded objects, lacking explicit calibration, necessitating contextual inference and multi-step reasoning.
- **Relative Camera Pose:** Distinguish between candidate $(R,t)$ pairs for inter-view transformation, necessitating decomposition of optical flow and translation to axis-angle parameterization.

A key property is that all such examples are answered incorrectly by $\geq$16/20 models, including at least two "giant" ($\geq$32B parameter) models. This isolates high-impact failure modes for subsequent model improvements [2505.17012].

## 6. Impact, Generalization, and Experimental Insights

SpatialScore-Hard distinguishes itself from the full SpatialScore benchmark in both scale and function:

- **Scope:** SpatialScore includes 28,000 samples, ranging from trivial recognition to medium-complexity geometric reasoning. Hard is restricted to 1,400 cases presenting authentic and persistent difficulty for current models.
- **Role:** SpatialScore tracks overall progress and model scaling trends. SpatialScore-Hard is used to stress-test and drive innovation in architectural, tool-based, or compositional enhancements to spatial reasoning.

TVP (Transductive Visual Programming) demonstrates substantial zero-shot generalization to SpatialScore-Hard, achieving state-of-the-art performance via experience-grounded tool libraries trained solely on Omni3D-Bench. Abstracted tools (such as compute_objects_size_ratio and find_largest_by_3d_metric) account for much of the accuracy and program simplification relative to traditional inductive tool pipelines. Programs relying solely on learned abstractions represent 36.3% of TVP’s solutions and confer complexity reductions and accuracy gains [2512.20934].

Failure cases typically arise in queries demanding entirely novel spatial abstractions, e.g., angle computations outside learned patterns, forcing fallback to generic tools and reduced accuracy.

## 7. Research Significance and Directions

SpatialScore-Hard acts as a focal resource for benchmark-driven diagnosis and advancement in multimodal spatial understanding. By concentrating on persistent model blind spots rather than generic tasks, it provides a challenging substrate for evaluating VLM compositionality, geometric reasoning, and tool-based learning agents. Its adoption by advanced frameworks such as SpatialAgent and TVP suggests that future work will increasingly emphasize experience-driven abstraction and targeted error minimization strategies. *A plausible implication is* that continual tool evolution and hybrid validation pipelines will be essential for reaching human-level 3D spatial reasoning in AI systems [2505.17012][2512.20934].

Source: https://www.emergentmind.com/topics/spatialscore-hard-collection