11Plus-Bench: Spatial Reasoning Benchmark
- 11Plus-Bench is a benchmark that assesses multimodal LLMs’ spatial reasoning by disentangling visual perception complexity from reasoning operations using realistic 11+ aptitude tests.
- It employs fine-grained annotations—covering perceptual load, reasoning steps, and image structure—to diagnose model behavior versus human cognitive profiles.
- The framework integrates predictive analytics like Random Forest classifiers and human response time measures to correlate complexity with performance.
Searching arXiv for the target paper and closely related multimodal/spatial reasoning benchmarks. 11Plus-Bench is a benchmark and evaluation framework for assessing the spatial reasoning abilities of multimodal LLMs (MLLMs) relative to human performance. Introduced in "11Plus-Bench: Demystifying Multimodal LLM Spatial Reasoning with Cognitive-Inspired Analysis" (Li et al., 27 Aug 2025), it is built from realistic standardized spatial aptitude tests at the 11+ level and is designed to move beyond aggregate accuracy toward cognitively grounded, instance-level analysis. Its defining feature is the explicit separation of visual perception complexity from reasoning process complexity, together with human comparison, private test materials for contamination control, and fine-grained expert annotations of perceptual complexity, reasoning operations, and image structure.
1. Scope, motivation, and conceptual position
11Plus-Bench was created to address a specific limitation in multimodal evaluation: coarse benchmark accuracy often obscures what kind of computation a model is actually performing. The paper argues that spatial reasoning in MLLMs cannot be understood well unless perceptual difficulty and reasoning difficulty are disentangled, and unless model behavior is compared directly against human cognitive profiles (Li et al., 27 Aug 2025). In this formulation, the benchmark is not merely a test set; it is an instrument for analyzing which cognitive factors drive success or failure.
The benchmark is grounded in realistic standardized spatial aptitude tests rather than synthetic reasoning puzzles alone. It therefore targets whether MLLMs can solve spatial problems in a way that resembles human spatial cognition. The paper frames this as a response to several deficiencies in earlier multimodal reasoning benchmarks: many report only aggregate accuracy, conflate low-level visual clutter with genuine spatial reasoning, provide shallow or no human baselines, and do not support principled comparison between model and human cognitive profiles. The benchmark also includes private, non-public test materials to reduce the risk that models have seen the items during training, which is especially relevant for contamination-sensitive MLLM evaluation (Li et al., 27 Aug 2025).
A common misconception is that 11Plus-Bench is simply another multiple-choice vision benchmark. The benchmark is instead organized around a cognitive analysis program: it separates perceptual load from reasoning load, records human response times as a proxy for cognitive load, and supports prediction of correctness and cognitive effort from annotated instance features. This makes it simultaneously a benchmark, an annotation resource, and an analysis framework.
2. Dataset construction, splits, and annotation protocol
11Plus-Bench is built from two data sources: a public split crawled from the web using spatial reasoning keywords and a private split purchased from certified test preparation or official test centers (Li et al., 27 Aug 2025). The public split was filtered heavily to remove irrelevant, ambiguous, repetitive, or non-spatial examples. The private split is kept private to reduce contamination and copyright issues. All examples are in English.
| Component | Content |
|---|---|
| Public split | 824 public examples |
| Private split | 91 private examples |
| Annotation workforce | Three experts annotated the data |
The paper reports 94.5% accuracy of expert-provided answers on the private set with gold labels. For subjective fields such as reasoning steps, annotator agreement is high, with Pearson correlations typically around or above 0.8; for objective pattern complexity counts, agreement is perfect (Li et al., 27 Aug 2025). These statistics are used to support the claim that the benchmark is both high quality and interpretable.
Each benchmark item is a multiple-choice spatial reasoning question consisting of a question image and several candidate option images. The annotation scheme adds two layers of structure. First, it annotates visual perception complexity for both the question and the answer options, operationalized as the number of atomic components in the pattern. Second, it annotates the general reasoning process as a sequence of atomic operations. The benchmark also includes bounding boxes or coordinates for question and option images, enabling analysis at the level of visual layout and input decomposition.
The notion of atomic components depends on pattern type. For referable shapes such as hearts or stars, annotators count symbolic elements; for abstract 2D patterns, they count lines or segments; for abstract 3D structures, they count surfaces or faces (Li et al., 27 Aug 2025). This annotation is used as a proxy for perceptual load rather than as a claim about a complete theory of vision.
3. Task formulation and spatial reasoning taxonomy
The benchmark focuses on three core spatial capabilities and a set of comprehensive tasks that combine them (Li et al., 27 Aug 2025). The first is Spatial Relation and Orientation (SRO), which covers distance, direction, position, orientation, perspective, rotation, folding, and projection. The second is Spatial Visualization (SV), which covers mental rotation, folding and unfolding, shape combination and completion, and 3D transformations. The third is Flexibility of Closure (FoC), defined here as detecting incomplete or hidden shapes and mentally completing them. In addition, the benchmark includes comprehensive tasks labeled Com.1 and Com.2.
The appendix-level taxonomy maps these capabilities to concrete task types. SRO includes 2D shape rotation, 2D shape reflection, and 3D shape rotation. SV includes shape completion, shape combination, building blocks, paper folding, and cube and nets. FoC is represented by hidden shape. The comprehensive tasks are cube and dice, and 3D-2D view (Li et al., 27 Aug 2025). This taxonomy is intended to cover a broad range of spatial cognition, from simple recognition to multi-step mental transformation.
The benchmark evaluates two input formats. In the first, the question and all candidate options are presented as a single composite image. In the second, the question image and each option image are provided as separate visual inputs, together with bounding box annotations. The paper treats the separate-image setting as preferable because it reduces the additional burden of parsing a crowded composite image and better isolates spatial reasoning proper (Li et al., 27 Aug 2025). This distinction is methodologically important: a system may fail on the composite format because of visual parsing difficulty rather than because of deficient spatial inference.
4. Cognitive annotation schema and analytical representation
One of the central contributions of 11Plus-Bench is its explicit representation of reasoning operations. The paper organizes the general reasoning process into four atomic operation categories: Pattern Matching, Spatial Relation Analysis, Spatial Manipulation, and Logical Deduction (Li et al., 27 Aug 2025). These are defined in code-style form as follows:
1 2 3 4 5 6 7 |
def pattern_match(entity_a: Object, entity_b: Object) -> bool def spatial_relation(entity_a: Object, entity_b: Object) -> statement: str def spatial_manipulate(entity: Object, statement: str) -> Union[Object, str] def logical_deduction(*statements: str) -> Union[str, bool] |
Spatial Relation Analysis includes the subtypes Position, Orientation, Perspective, Rotation, Folding, and Projection. Spatial Manipulation includes 2D rotation, translation, reflection, adding or removing shapes, 3D rotation, translation, symmetry, projection, folding, counting holes, and symbol tagging (Li et al., 27 Aug 2025). The benchmark’s analytical premise is that a reasoning trace can be decomposed into a sequence of such atomic steps, and that the counts or compositions of these steps can be used as explanatory variables for correctness and cognitive load.
The paper further defines the feature categories used in downstream prediction. Under Visual Perception, the relevant features are Pattern Complexity of question, Pattern Complexity of options, and Image Resolution. Image resolution is discretized into low, medium, and high. Under General Reasoning, the relevant features are the numbers of steps in Pattern Matching, Spatial Relation Analysis, Spatial Manipulation, and Logical Deduction (Li et al., 27 Aug 2025). These features serve as the basis for SHAP analyses, Random Forest prediction of correctness, and linear regression for response-time or token-length prediction.
This representation has a broader methodological implication. Rather than treating benchmark items as opaque examples, 11Plus-Bench turns them into structured cognitive objects. A plausible implication is that this makes failures more diagnosable than in conventional multimodal benchmarks, because one can ask whether an error is associated with option complexity, resolution, folding operations, or logical constraint chaining rather than with the item as an indivisible whole.
5. Evaluation protocol, models, humans, and metrics
The evaluation framework has three explicit goals: disentangle cognitive features, enable instance-wise prediction, and compare human and model cognitive profiles (Li et al., 27 Aug 2025). Accuracy is the main task metric for both models and humans. Human evaluation used three participants who were not involved in annotation. A total of 402 examples were sampled for human testing, and response times in seconds were recorded. The paper uses response time as a proxy for cognitive load.
The benchmark evaluates 14 MLLMs, including open-source systems—Qwen2.5-VL 3B, Qwen2.5-VL 7B, Gemma 3 12B, and Gemma 3 27B—and closed-source systems—GPT-4o, GPT-4.1 mini, GPT-4.1 nano, GPT-4.1, GPT-o1, GPT-o3, GPT-o4-mini, Gemini 2.0 Flash, Gemini 2.5 Flash, and Gemini 2.5 Pro (Li et al., 27 Aug 2025). The study therefore spans both open and proprietary families, as well as a range of parameter scales and reasoning styles.
For human reliability, the paper reports pairwise response-time correlation greater than 0.4 among participants, together with and , which it interprets as good reliability of mean response time. The correlation between correctness and response time is Pearson , indicating that higher accuracy tends to correspond to shorter response time (Li et al., 27 Aug 2025).
For instance-level prediction, the authors train Random Forest classifiers using weighted F1, AUC, 5-fold cross-validation, and class weights inversely proportional to class frequencies. They evaluate both binary classification, correct versus incorrect, and four-class classification corresponding to 0–3 correct human responses across the three participants. For cognitive load prediction, they use linear regression to predict human response time and MLLM token counts or “thinking” length (Li et al., 27 Aug 2025). This makes the benchmark unusual in that it does not stop at outcome measurement; it also models the predictability of those outcomes.
6. Empirical results, cognitive profiles, and significance
Human performance substantially exceeds model performance. On the 402-item human subset, participant accuracies were 72%, 87%, and 85% (Li et al., 27 Aug 2025). Difficulty is distributed across the benchmark rather than concentrated only at the extremes: 241 examples were answered correctly by all three participants, 115 by two participants, and 46 by one or zero participants. This distribution supports the paper’s claim that the benchmark has a useful spread of difficulty.
Among MLLMs, closed-source models outperform open-source models. The paper further reports that larger open-source models do not substantially outperform smaller ones, that performance is worse under the single composite image setting than under separate images, and that with separate images most closed-source models beat random guessing significantly, whereas all open-source models fail to significantly outperform random baseline (Li et al., 27 Aug 2025). For the GPT series, the average drop from separate images to the single composite image setting is about 4%, which the authors interpret as evidence that visual parsing confounds can depress apparent reasoning performance.
The paper’s most consequential finding concerns cognitive profiles rather than raw scores. Human correctness is significantly predictable from annotated features: Random Forest classifiers achieve weighted F1 scores of 0.631, 0.821, and 0.799, all with , and AUC scores of 0.579, 0.643, and 0.621 (Li et al., 27 Aug 2025). In the four-class aggregated setting, the model achieves versus 0.192 chance, with . By contrast, MLLM correctness is near-random at the instance level: prediction models generally fail to identify stable correctness patterns, with weighted F1 and AUC mostly not significantly above random and in most cases. The result is that average model performance may be non-trivial, but the pattern of which items are solved is poorly structured relative to the annotated cognitive factors.
The relationship between performance and complexity also differs between humans and models. For humans, Pattern Complexity—especially in the answer options—is the strongest predictor of correctness, while reasoning complexity more strongly affects response time than correctness. For response time specifically, Spatial Relation Analysis, Spatial Manipulation, and Logical Deduction dominate, whereas Pattern Matching tends to correlate with shorter response times, plausibly because it supports fast elimination heuristics (Li et al., 27 Aug 2025). MLLMs exhibit only partial alignment: Option Pattern Complexity is influential, but models are more sensitive than humans to low-level visual cues such as Image Resolution and spatial relations in the visual input. For model “thinking length,” reasoning-related features matter, but Question Pattern Complexity also contributes, and Spatial Relation Analysis is the least predictive factor in some models. For Gemini 2.5 Pro, the correlation between response length and accuracy is Pearson , indicating that longer responses do not correspond to better reasoning or higher accuracy (Li et al., 27 Aug 2025).
The paper summarizes these results as evidence that current MLLMs show early signs of spatial cognition but remain far from human-level robustness. Their cognitive effort correlates strongly with reasoning-related complexity, which resembles the human pattern at a coarse level, yet their instance-level correctness remains largely random, whereas human correctness is highly predictable and shaped by abstract pattern complexity (Li et al., 27 Aug 2025). This suggests that current models may be relying more on surface-level visual cues than on stable, compositional spatial reasoning strategies.
In relation to prior multimodal reasoning benchmarks, the paper contrasts 11Plus-Bench with CLEVR, NLVR2, SpatialSense, Spatial-MM, Comsa, CLEVRER, TopViewRS, VSI-Bench, MindTheGap, and SAT. The claimed distinction is not merely a different task inventory, but psychometric grounding, fine-grained annotations, human-centered analysis, contamination resistance, and cognitive profiling (Li et al., 27 Aug 2025). On that basis, 11Plus-Bench is best understood as an analysis-oriented benchmark for spatial cognition in MLLMs rather than as a conventional leaderboard-only dataset. It implies that future progress should be evaluated not only by higher accuracy, but also by whether perceptual and reasoning improvements can be separated, whether model behavior becomes instance-wise predictable in cognitively meaningful ways, and whether spatial reasoning becomes more aligned with human cognitive structure.
