CRISP: Consistency in Spatial Perception

• CRISP is a diagnostic framework for visual spatial intelligence that evaluates the alignment between implicit perception and explicit reasoning through metric 3D scene graphs and spatial QA.
• It exposes failure modes in vision-language models by revealing semantic shortcutting and perception–reasoning disconnect through structured spatial analysis.
• CRISP employs a dual-protocol combining externalized scene graph generation with deterministic QA derivation to benchmark model consistency and diagnose latent reasoning performance.

CRISP, short for Consistency of Reasoning In Spatial Perception, is a structural-diagnostic evaluation paradigm for visual spatial intelligence that pairs traditional spatial question answering with explicit 3D Scene Graph Construction (SGC) and measures their cross-task alignment through consistency (Li et al., 25 Jun 2026). It is designed to expose a failure mode in contemporary vision-LLMs (VLMs): correct answers produced by exploiting language priors and positional biases rather than by grounding reasoning in metric geometry and topology. By forcing a model to externalize its implicit spatial understanding into a metric-aware 3D structure and then testing whether that structure supports the model’s own answer, CRISP shifts evaluation from black-box answer correctness toward grounded process diagnosis.

1. Motivation and conceptual scope

CRISP targets the gap between atomic semantic perception and structural spatial perception (Li et al., 25 Jun 2026). Atomic semantic perception concerns “what is where” in terms of 2D labels and boxes. Structural spatial perception concerns implicit 3D geometry, including distances, directions, and viewpoint transformations. Conventional visual question answering obscures this distinction because a model can answer questions such as relative proximity or object ordering by relying on priors like “drawers are near wardrobes,” without perceiving the underlying metric geometry.

This motivates CRISP’s central claim that a VLM must not only answer spatial questions but also expose the spatial structure that allegedly justifies those answers. Textual rationales and 2D cognitive maps are insufficient in this setting because they either hallucinate or lack metric fidelity. CRISP therefore couples two tasks: Spatial QA, which evaluates explicit reasoning over distances, directions, rankings, and related competencies; and 3D Scene Graph Construction, which acts as a generative probe of the model’s latent structural representation.

A frequent misconception in multimodal evaluation is that strong QA accuracy implies strong spatial intelligence. CRISP directly rejects that equivalence. Its design allows a model to be correct for the wrong reason, or wrong despite having partially adequate structural knowledge. This suggests that the relevant object of study is not answer correctness alone, but the alignment between perception, externalized structure, and reasoning.

2. Metric 3D Scene Graphs as the structural substrate

CRISP represents scenes as metric 3D Scene Graphs $G=(V,E)$ whose objects are identified by numeric IDs to prevent semantic leakage (Li et al., 25 Jun 2026). Each node $i \in V$ carries intrinsic metric attributes $s_i=(w_i,l_i,h_i)$, corresponding to 3D dimensions, together with an ego-centric depth $d_i$, defined as distance to camera. Positions are not given as explicit coordinates; instead, structure is expressed through distances, while camera intrinsics are withheld to test whether a model can recover metric relations from physical scale priors.

Edges encode two kinds of pairwise information. First, the graph includes metric Euclidean distances $d_{ij}$ between objects. Second, it includes directional predicates from mutually exclusive sets such as left versus right and front versus behind. To reduce generation drift, CRISP does not ask for a full global graph in one step. Instead, SGC uses an object-centric star topology, in which the model generates local topology and metrics centered on a specified object and its known neighbors. The list of known objects is provided, thereby decoupling spatial reasoning from object discovery.

Ground-truth graphs are derived from high-quality 3D annotations in nuScenes and ScanNet++, projected into the 2D image plane using sensor calibration and then filtered using strict visibility and resolution criteria. Each sample therefore has a Master 3D Scene Graph keyed by numeric IDs. This explicit structural representation is what makes it possible to test whether a model’s answer is derivable from its own externalized scene model rather than from latent semantic shortcuts.

3. Consistency: formal definition and evaluation procedure

The defining quantity in CRISP is consistency, the alignment between implicit perception and explicit reasoning (Li et al., 25 Jun 2026). Implicit perception is operationalized as the model’s externalized structure via SGC. Explicit reasoning is operationalized as the model’s direct QA answer.

CRISP implements a two-step protocol. In the first step, the generated SGC is parsed into a structured graph, and a deterministic rule-based solver derives an answer $A_{\text{derived}}$. If the graph lacks the information needed for derivation, the solver emits a $[FAILED]$ token. In the second step, the model’s direct QA answer $A_{\text{QA}}$ is treated as the anchor, and the derived answer is compared against it using the standard QA metric. Formally,

$$C \;=\; \text{QA\_Score}\!\left(\{A_{\text{derived}, s}\},\, \{A_{\text{QA}, s}\}\right).$$

A crucial feature of this definition is that consistency is not ground-truth correctness. A model can therefore be consistently wrong, which CRISP terms consistent hallucination. For diagnosis, CRISP jointly reports QA Score, SGC Score, Derived QA Score, and Consistency rather than collapsing them into a single scalar.

Spatial QA spans seven competencies: distance estimation, size estimation, directional perception, counting, spatial ranking, view transformation, and logical deduction. It includes both multiple-choice questions and numerical-answer questions. For numerical questions, CRISP uses Mean Relative Accuracy (MRA) from VSI-bench:

$\text{MRA} \;=\; \frac{1}{10}\sum_{\theta \in \mathcal{C}} \mathbbm{1}\!\left(\frac{|\hat{y}-y|}{y} < 1 - \theta\right),\quad \mathcal{C}=\{0.5,0.55,\cdots,0.95\}.$

The SGC Score combines metric and relational fidelity. Metric estimation aggregates object size, ego-centric distance, and object-centric distance. Relation score measures accuracy on mutually exclusive directional predicate pairs. The final SGC score is

$i \in V$0

This evaluation design makes it possible to distinguish at least four regimes: structurally grounded correct reasoning, semantic shortcutting, perception failure, and perception–reasoning disconnect.

4. Oracle interventions and causal-style diagnosis

CRISP includes an oracle intervention protocol that stages structured inputs to separate latent reasoning capacity from perceptual bottlenecks (Li et al., 25 Jun 2026). Four settings are used:

Setting	Input condition	Diagnostic role
Base	image + question	no structural guidance
Pred SG	model’s own predicted scene graph + image + question	tests robustness under modality conflict
Multimodal GT SG	ground-truth 3D scene graph + image	simulates idealized consistency
Text-Only GT SG	ground-truth scene graph only	symbolic ceiling

The intervention effects are summarized by

$i \in V$1

A large positive $i \in V$2 indicates underutilized latent reasoning: the model can reason when perfect structure is supplied, but fails to exploit its own implicit structural knowledge under end-to-end generation. A negative $i \in V$3 indicates modality trust bias, meaning that the model over-trusts its own erroneous structural text over image evidence.

Empirically, injecting ground-truth scene graphs causes large QA gains, especially for proprietary models and especially on compositional logic. Gemini-2.5-Flash improves by +30.3% in the multimodal GT SG setting, while Text-Only GT SG reaches approximately 80% accuracy, showing that the 3D scene graph contains sufficient information for the tasks (Li et al., 25 Jun 2026). Conversely, feeding predicted scene graphs broadly collapses QA performance, with Gemini-2.5-Flash as a notable exception. This suggests a threshold effect: when the predicted structure is good enough, the reasoning engine can benefit from it; when it is not, self-generated structure becomes a source of harmful interference.

5. Benchmark construction and task design

CRISP-Bench contains 1,162 static, single-view scenes with a 1:1 indoor/outdoor split and 9,839 image-specific QA pairs generated procedurally from Master 3D Scene Graphs (Li et al., 25 Jun 2026). Outdoor scenes come from nuScenes and indoor scenes from ScanNet++. Ground-truth construction uses calibration matrices to align 3D primitives with 2D observations, and filtering enforces visibility score $i \in V$4, 2D bounding box longest edge $i \in V$5 pixels, and a diversity filter to reduce semantic redundancy while balancing scene complexity.

The benchmark deliberately uses unique numeric object IDs rather than semantic object names, again to suppress language priors. It also withholds camera intrinsics. Numerical-answer questions require precise quantitative estimates; multiple-choice questions use balanced answer distributions to reduce positional bias.

These design decisions matter because CRISP is intended not as a generic VQA benchmark, but as a probe of grounded 3D structural competence. The restriction to static single-view scenes is deliberate. It isolates spatial grounding without confounding factors from temporal reasoning, multi-view parallax, or explicit motion cues. A plausible implication is that CRISP sacrifices some ecological breadth in order to sharpen failure diagnosis.

6. Empirical picture and diagnosed failure modes

CRISP reports a differentiated empirical landscape across proprietary, open-source, and specialized spatial models (Li et al., 25 Jun 2026).

Model	QA	SGC	Consistency
Gemini‑2.5‑Pro	58.93	64.58	57.28
Gemini‑3‑Flash	53.41	68.79	55.20
GPT‑5‑Mini	54.15	66.70	56.75
Qwen3‑VL‑8B	55.34	62.35	52.16
Qwen3‑VL‑32B	53.98	62.45	51.56
InternVL3.5‑38B	53.98	58.90	48.70

Proprietary models show stronger QA and SGC overall, but CRISP diagnoses a perception–reasoning disconnect. Their topological relation score $i \in V$6 consistently exceeds metric estimation $i \in V$7, and Derived QA often exceeds Base QA, indicating that the models can construct reasonable scene structure yet fail to use it during end-to-end reasoning. A particularly sharp comparison is that Gemini‑2.5‑Flash matches Gemini‑2.5‑Pro on grounding (SGC 64.92 vs. 64.58) while trailing in QA by −11.25 points. This is evidence of a utilization failure rather than a purely perceptual one.

Open-source models present a different profile. When ground-truth scene graphs are injected, their gains concentrate on atomic access, whereas multi-hop compositional reasoning remains a bottleneck even with perfect structure. CRISP therefore characterizes them as fundamentally bottlenecked by reasoning depth rather than by structural input alone.

CRISP also introduces consistency as reliability. Across models, consistent hallucination occurs in approximately 10–12% of instances. Gemini models show lower coherent-error rates, around 3.8%, which supports the diagnosis that their main weakness is not the absence of latent structure, but failure to anchor reasoning to that structure.

The benchmark’s blind text-only tests isolate priors. Text-only SGC floors around $i \in V$8 for most models, showing that visual gains in SGC reflect genuine structural perception rather than pure semantic guessing. Yet visual gains in QA can still be deceptive. For Qwen2.5‑VL, QA improves by +16.31 with images while SGC increases by only +3.24, exposing what CRISP terms a Semantic–Geometric Gap: recognizing “what” without grounding “where.”

Two diagnostic examples illustrate the method. In one indoor scene, the model answers that object 17 is closer to the camera than object 12, but its own SGC predicts $i \in V$9 m and $s_i=(w_i,l_i,h_i)$0 m; the solver derives “No,” yielding zero consistency. In another outdoor scene, the SGC correctly encodes a car as closer than a traffic sign, but the direct QA answer says “traffic sign”; consistency is zero while Derived QA is correct. The former diagnoses semantic shortcutting or consistent hallucination, and the latter diagnoses perception–reasoning disconnect.

7. Significance, limitations, and recommended use

CRISP reframes multimodal evaluation around a three-part requirement: perceive, verify, and reason (Li et al., 25 Jun 2026). Perception requires construction of metric 3D structure beyond 2D semantics. Verification requires that answers be supported by the model’s own structure. Reasoning requires manipulation of that structure in multi-hop form rather than reliance on priors. In this formulation, high capability requires the joint presence of high QA accuracy, high SGC fidelity, and high consistency.

The framework also states its own limits. Diagnoses such as perception–reasoning disconnect and modality trust bias are described as behavioral syndromes, not proven causal mechanisms. The benchmark excludes containment and occlusion questions in order to minimize entanglement with object-specific semantic priors. It focuses on static single-view scenes, leaving dynamic, multi-view, and temporal spatial reasoning for future work. Oracle conditions with perfect ground-truth scene graphs are idealized; extending the method to noisy but realistic 3D reconstructions remains an open direction.

For practical use, the code and dataset are available at https://github.com/iiyamayuki/CRISP-Bench. The repository provides prompts and parsers for Spatial QA and SGC, a deterministic solver for Derived QA and consistency scoring, and integration with lmms-eval in zero-shot mode. The recommended diagnosis workflow is to baseline a model on QA, SGC, and consistency; run blind text-only ablations to quantify priors; apply Multimodal GT SG and Pred SG interventions; and then decompose results by atomic versus compositional categories to localize whether the bottleneck lies in metric estimation, structure utilization, or multi-hop reasoning.

In that sense, CRISP functions less as a single leaderboard and more as a diagnostic framework for multimodal alignment. Its central contribution is to make spatial intelligence falsifiable at the level of internalized structure rather than answer strings alone.