- The paper introduces a diagnostic benchmark of 235 naturally occurring, human-easy/model-hard tasks across text, vision-language, and image-generation formats, using a fine-grained taxonomy and validated automated grading pipeline.
- The paper finds that closed models lead open-weight systems by roughly 10 percentage points on text-only tasks at comparable intelligence-index scores, while perceptual counting and fine-grained visual recognition remain below 60% for every evaluated system.
- The paper shows that tool use, parameter scaling, and model size produce inconsistent gains, highlighting the need to evaluate specific failure modes rather than rely solely on aggregate benchmark scores or cost metrics.
Motivation and positioning
Blind-Spots-Bench (2607.08317) addresses a gap between aggregate benchmark performance and residual, persistent failure modes in frontier AI systems. The authors observe that models near saturation on established evaluations—GPQA, SWE-bench, MMMU, MathArena—still fail on tasks that humans find trivial: producing a string of a specified length, drawing a clock at a requested time, or generating an image with an exact object count. The benchmark is built explicitly from such naturally occurring failures rather than from perturbations of existing problems, distinguishing it from robustness suites like GSM-Plus or GSM-Symbolic. Its purpose is diagnostic: to decompose reasoning into fine-grained sub-tasks and expose weaknesses that aggregate scores obscure.
Benchmark construction
The dataset originates from a graduate-level AI course (EPFL, October 2025) in which each student proposed five questions that frontier chatbots failed to answer. From roughly 287 raw submissions, a cleaning pipeline—filtering duplicates and overly difficult or trivially solvable items, normalizing formats, and verifying well-posedness—yielded 235 samples. Each sample carries a structured reference solution specifying the expected answer, necessary correctness conditions, and common failure modes elicited from student interactions plus annotations by at least three annotators per question.
The taxonomy comprises three top-level categories and twelve sub-tasks:
- Object-centric: attribute/pattern recognition, spatial reasoning, generative counting, perceptual counting, attribute binding.
- Abstract reasoning: logical reasoning, arithmetic reasoning, geometric and graph reasoning, constraint reasoning.
- Language and knowledge: irrelevant-context robustness, character-level manipulation, world knowledge.
Question formats split into text-only (46.2%), image-generation (35.6%), and multi-to-text (18.2%). Spatial reasoning is the most frequent sub-task (53 occurrences), followed by counting-related categories—a distribution the authors argue reflects genuine model difficulty rather than arbitrary dataset statistics, since it aligns with documented limitations in spatial transformation, visual counting, and subword-tokenization-induced failures on character-level manipulation. Fifteen questions carry multiple sub-task labels, with failure-mode attribution determined from the model's response.
Evaluation pipeline
Evaluation uses the Inspect AI framework in two stages: a solver answers each question without few-shot examples or chain-of-thought prompting, then gemini-3-flash grades binary correctness against the reference solution, with code execution enabled for hard constraints such as counting or substring checks. Text tasks are run with k=4 repetitions; image-generation tasks once, due to cost. All models run with thinking mode enabled and medium effort where available, 32,768 max output tokens, no tools unless stated.
Grader validation against human labels on 100+ sampled responses yields 96.6% agreement on textual outputs and 90.9% on image-generation outputs. A disaggregated bias analysis finds no pro-Google inflation despite using a Google grader; notably, the false-positive rate is higher for non-Google solvers (0.061 text, 0.150 image) than for Google ones, meaning the pipeline if anything overestimates non-Google performance—an assumption worth noting when comparing families.
Thirty-eight systems are evaluated: open-weight LLMs/VLMs (GLM-4.7 through 5.2, DeepSeek-V4 Flash/Pro, GPT-OSS, Qwen3/Qwen3-VL/Qwen3.5, Gemma-4, Kimi-K2.5/2.6) served via vLLM on EPFL infrastructure, closed frontier models (Gemini-2.5/3/3.1, GPT-5/5.2/5.4/5.5), and six image-generation models (GPT-Image variants, Gemini-Image variants). No open-weight image-generation model is included, a coverage constraint imposed by technical availability.
Main results
Closed–open gap. Gemini-3.1-Pro achieves 83.3% mean@4 on text-only and 66.9% on multimodal problems; GPT-5.5 reaches 84.0% on text but only 58.7% multimodal. The best open-weight model, GLM-5.2, attains 73.8% text-only—a gap of roughly 10 points versus the frontier leaders. Critically, this gap persists between models with comparable Artificial Analysis Intelligence Index (AAII) scores: open-weight models underperform closed models of similar AAII, suggesting public benchmarks may overestimate robustness on underrepresented task types, plausibly because open models are optimized toward widely used evaluations. This is one of the paper's strongest claims and its principal evidence that blind-spot performance is not redundant with general intelligence indices.
| Model |
Text mean@4 |
Multi-to-text mean@4 |
Image-gen mean@1 |
| Gemini-3.1-Pro |
83.3 |
66.9 |
– |
| GPT-5.5 |
84.0 |
58.7 |
– |
| GLM-5.2 (best open) |
73.8 |
– |
– |
| Qwen3.5-397B-A17B |
71.1 |
48.3 |
– |
| Gemini-3-Pro-Image |
– |
– |
54.8 |
| GPT-Image-2 |
– |
– |
51.2 |
Cost-performance trade-off. In text-only tasks, moving from 60% to 70% accuracy requires approximately an order-of-magnitude increase in inference cost. Open models—DeepSeek-V4, Qwen3.5, GLM-5.2—are more cost-effective than frontier alternatives at matched cost (e.g., Gemini-3.1-Flash-Lite, GPT-5.4-mini). Among closed models, the Gemini-3 family shows the best accuracy-cost profile, particularly multimodally; Gemini-3-Pro-Image leads image generation at 54.8% but costs about 4× more per sample than GPT-Image-2 (51.2%).
Tool use. Equipping eleven models with Python execution produces mixed outcomes: Gemini-3.1-Flash-Lite gains +9.03 points while reducing tokens, whereas GPT-5.4 loses 5.32 points and Qwen3.5-397B loses 2.08. Qualitative inspection attributes losses to input mishandling—for example, incorrect copy-pasting of strings into generated code. Tool use therefore reduces token consumption but cannot be assumed to improve accuracy.
Task-taxonomy analysis
Three findings emerge from the fine-grained breakdown. First, fine-grained visual perception is a shared bottleneck: even the strongest models reach only 41.67% on attribute/pattern recognition (6 questions) and 57.14% on perceptual counting (21 questions), with many VLMs at 30–40% on counting. No evaluated system exceeds 60% on these sub-tasks, indicating a capability deficit common to all current architectures rather than a model-specific weakness.
Second, no single model dominates across sub-tasks. GPT-5.5 scores 88.64% on arithmetic reasoning and 100% on irrelevant-context robustness, while Gemini-3.1-Pro leads on spatial reasoning (87.50%) and logical reasoning (92.65%). DeepSeek-V4 reaches 75% on character-level manipulation; Kimi-K2.5 attains 82.95% on arithmetic reasoning but trails leaders by ~10 points elsewhere. Image-generation models show complementary profiles: Gemini variants win abstract-reasoning items, GPT-Image variants language-and-knowledge items.
Third, scaling within a family is inconsistent. In the Qwen3.5 series, the 122B model drops 10–14 points relative to the 35B variant on several abstract-reasoning sub-tasks (logical, geometric/graph, arithmetic), and the 397B model recovers only partially without surpassing the 35B. In Gemma-4, the 26B model performs about 8 points worse than E2B on irrelevant-context robustness. These non-monotonicities imply that parameter count alone does not predict blind-spot competence, and that small-model results should not be extrapolated from larger siblings.
Limitations and open questions
The paper concedes three substantive limitations. The dataset is small (235 samples) and imbalanced across sub-tasks—several analyses rest on as few as 6 questions, so sub-task-level accuracies carry wide uncertainty. Questions were authored by students targeting specific frontier models available in October 2025, introducing bias toward those models' particular weaknesses and potentially excluding other failure classes. No human baseline is reported, so the claimed "easy for humans" property is asserted rather than quantified. Additionally, the authors note in their broader-impact discussion that the benchmark is unsuitable as a reliability or safety assessment and is susceptible to overfitting if used as a training target. An open question the paper leaves explicit is whether the observed closed–open gap at matched AAII reflects training-data composition or evaluation-targeted optimization; disentangling these would require controlled comparisons the benchmark alone cannot provide.
Conclusion
Blind-Spots-Bench contributes a curated, annotated set of 235 human-easy/model-hard problems spanning text, vision-language, and image-generation formats, together with a validated automatic grading pipeline and a fine-grained taxonomy. Its evaluation of 38 models establishes that closed-source systems hold a ~10-point advantage over open-weight counterparts even at parity on standard intelligence indices; that open models dominate on cost-adjusted accuracy; that tool use and scale yield inconsistent gains; and that perceptual counting and fine-grained recognition remain below 60% accuracy for every system tested. As a diagnostic stress test complementing aggregate benchmarks, it provides a reproducible instrument for locating concrete, persistent weaknesses in contemporary multimodal models.