Taxonomy of Visual Reasoning Errors

• Taxonomy of visual reasoning errors categorizes systematic failures in visual data interpretation, distinguishing perceptual, attribute, localization, logical, and knowledge-related errors.
• It synthesizes cross-domain frameworks from radiology, video reasoning, and VQA to benchmark, diagnose, and improve both AI and human visual inferences.
• The taxonomy informs targeted strategies such as specialized data augmentation, modular tool use, and consistency-driven loss functions to enhance model reliability.

Visual reasoning errors represent systematic failures of AI or human observers to generate correct, reliable inferences from visual data. Taxonomies of these errors are essential for diagnosing and rectifying specific reasoning failure modes, benchmarking new models, and designing robust visual intelligence systems. Contemporary taxonomies span domains as diverse as radiology, video understanding, puzzle-solving, visual question answering (VQA), and general large vision-LLMs (LVLMs). These frameworks converge on several fundamental axes: perceptual grounding, attribute extraction, spatial/temporal localization, symbolic reasoning, and logical or causal inference. Below, leading taxonomic proposals from major recent benchmarks are synthesized, revealing cross-domain regularities as well as unique domain-specific elaborations.

1. Core Categories of Visual Reasoning Errors

Most taxonomies of visual reasoning errors employ a hierarchical structure that distinguishes perceptual errors at the information extraction stage from interpretive/logical errors at subsequent reasoning stages. This two-level stratification is evident in both domain-specific (e.g., radiology) and general (e.g., video, VQA, LVLM) taxonomies.

Top-Level Category	Definition	Example
Perceptual/Detection Error	Incorrect extraction of salient visual features or evidence	Missed ureterocele in radiology (Datta et al., 29 Sep 2025)
Attribute Error	Misrecognition of color, material, or state	Mistaking a red mug for orange (Zhou et al., 9 Feb 2026)
Relationship/Localization	Misinterpretation of spatial/temporal/contextual relationships	Swapping left/right for objects (Chen et al., 18 Nov 2025)
Logical/Reasoning Error	Faulty inferences from otherwise correct perceptual inputs	Deductive fallacy in puzzle chain (Qian et al., 27 Oct 2025)
Knowledge/Comprehension	Deployment of incorrect background knowledge or misreading the question	Wrong formula in physics VQA (Shi et al., 6 Jan 2026)

In high-complexity domains, additional types—such as tool-use failures (Zhou et al., 9 Feb 2026), process errors (redundant or omitted steps) (Qian et al., 27 Oct 2025), or answer-reasoning inconsistency (Zhou et al., 9 Feb 2026)—serve as crucial subcategories.

2. Fine-Grained Error Subtypes Across Domains

Leading benchmarks provide further sub-typing. The following table synthesizes granular categories across several influential taxonomies:

Source	Subtypes
RadLE (Radiology)	Under-detection, Over-detection, Mislocalization (Datta et al., 29 Sep 2025)
MINERVA (Video)	Perceptual Correctness, Temporal Localization, Logical Reasoning, Completeness (Nagrani et al., 1 May 2025)
ThinkWithImages PRM	Object Detection Failure, Attribute Error, Relationship Bias, Logical Understanding, Tool Error, Redundant Reasoning, ARI (Zhou et al., 9 Feb 2026)
MMErroR (VLM)	Visual Perception Error, Reasoning Error, Question Comprehension Error, Knowledge Deployment Error (Shi et al., 6 Jan 2026)
MVI-Bench (Robustness)	Visual Resemblance, Representation Confusion, Material Confusion, Mirror Reflection, Occlusion, Visual Illusion (Chen et al., 18 Nov 2025)
PRISM-Bench (Puzzles)	Attribute Misinterpretation, Counting/Progression, Language/Ambiguity, Deductive Error, Over/Under-Generalization, Step-Process, Visual/Spatial Misperception (Qian et al., 27 Oct 2025)

This cross-domain harmonization demonstrates recurring motifs: feature extraction breakdowns, attribute assignment mistakes, spatial or temporal mislocalizations, logic or deduction failures, and domain-specific errors such as tool mis-invocation or redundancy.

3. Domain-Specific Taxonomies and Illustrative Examples

Taxonomies are tailored to the structure and needs of each domain:

Radiology (RadLE) (Datta et al., 29 Sep 2025):

• Perceptual errors: False negatives (under-detection), false positives (over-detection/hallucination), spatial miscoding (mislocalization).
• Example: Omission of subtle radiologic signs (e.g., missing a ureterocele leads to misdiagnosis).
• Underlying causes: Limited low-level feature extraction, attention biases, lack of 3D reasoning from 2D images.

Video Reasoning (MINERVA) (Nagrani et al., 1 May 2025):

• Perceptual correctness: OCR/text misreading, missed objects/actions.
• Temporal localization: Incorrect timestamping of evidence.
• Logical completeness: Reasoning chain gaps.
• Quantitative insights: Temporal grounding is most error-prone (mean MiRA score 0.58; lower score worse), whereas logical steps are least prone (mean 0.80).

PRISM-Bench (Puzzle Reasoning) (Qian et al., 27 Oct 2025):

• Categories: Attribute/feature misinterpretation, progression errors, language/ambiguity, deductive error, over/under-generalization, step/process errors, visual/spatial misperception.
• Detection mechanism: Diagnostic tasks require models to localize the first erroneous chain-of-thought step.

MMErroR (VLM Reasoning) (Shi et al., 6 Jan 2026):

• Visual Perception Error: Low-level misreadings (text, chart, object misassignment).
• Reasoning Error: Arithmetic/mathematical logic mistakes.
• Question Comprehension Error: Answering the wrong sub-question, entity mix-up.
• Knowledge Deployment Error: Incorrect application of external knowledge (formula, law).

Misleading Visual Input (MVI-Bench) (Chen et al., 18 Nov 2025):

• Visual Concept Level: Errors by visual resemblance or representation confusion.
• Visual Attribute Level: Attribute misclassification due to subtle material/texture cues.
• Visual Relationship Level: Mirror confusion, occlusion, and optical illusion errors.

4. Quantitative Metrics for Error Analysis

Traditional classification accuracy is insufficient for process-level error auditing. Multiple domains of research propose domain- and error-type-specific metrics:

• RadLE: Error rates per trace for under-detection, over-detection, mislocalization, but frequency counts not published.
• MINERVA: MiRA (reference-based) scores per error category; temporal localization (0.58), perceptual correctness (0.53), logical (0.80), completeness (0.76).
• PRISM-Bench: Error-detection accuracy (proportion of CoTs with first error correctly pinpointed); per-category shares: 16% attribute errors, 16% logic/deduction, 18% spatial misperception.
• MVI-Bench: MVI-Sensitivity, defined as

$\mathrm{MVI\mbox{-}Sensitivity}(c) = \frac{|Acc_n(c) - Acc_m(c)|}{Acc_n(c)}$

captures robustness to misleading cues per category.

• MMErroR: Knowledge deployment errors recognized most reliably (~70%), question comprehension errors least (~55–60%). Macro-averaged top-model accuracy: 66.7%.

These error-centric metrics enable researchers to isolate performance breakdowns by category, compare model weaknesses, and direct fine-tuning or architectural improvement.

5. Implications for Model Diagnosis, Training, and Evaluation

Systematic error-type taxonomies inform not only evaluation but also the design and training of robust visual reasoning systems. Recommendations from diverse benchmarks converge on:

• Specialized data augmentation: Synthesizing images exploiting known error categories (mirrors, occlusions, illusions) (Chen et al., 18 Nov 2025)
• Modular and calibrated tool invocation: Verifying bounding box/region proposals during tool use to avoid object-missing and tool invocation errors (Zhou et al., 9 Feb 2026)
• Consistency-driven loss functions: Penalizing answer-reasoning inconsistencies and logical incoherence, using COT entailment models (Zhou et al., 9 Feb 2026)
• Multi-step diagnosis training: Including error location and process-level feedback (as in PRISM-Bench) to close the gap between final-answer accuracy and process reliability (Qian et al., 27 Oct 2025)
• Multi-label supervision: Anticipating and modeling ambiguity, synonymy, and granularity differences, as revealed in the VQA disagreement taxonomy (Bhattacharya et al., 2019)

Fine-grained benchmarking across error subtypes also reveals that the most frequent system-level failures often stem from grounding failures rather than pure logical reasoning, guiding research emphasis toward perceptual and grounding robustness strategies.

6. Broader Taxonomies: Human Disagreement and Annotation

The phenomenon of divergent human answers in VQA and labeling workflows itself motivates comprehensive taxonomies of disagreement:

• Nine-category taxonomy of disagreement causes: Low-quality image, insufficient visual evidence, invalid question, high difficulty, ambiguous phrasing, subjectivity, synonymy, answer granularity, and annotator spam (Bhattacharya et al., 2019).
• Empirical findings: Ambiguity, granularity, and synonymy co-occur in >70% of disagreement instances; workflow enhancements can reduce much of this variability.

This emphasis on human–system symmetry in error typology motivates not only improved machine reasoning but also the critical interpretation of human labels in visual datasets.

7. Summary Table of Visual Reasoning Error Taxonomy Dimensions

Dimension	Example Errors	Key Benchmarks
Perceptual Grounding	Missed/false objects, OCR, attributes	RadLE, MINERVA, PRISM, ThinkWithImages
Spatial/Temporal	Mislocalization, occlusion, time errors	RadLE, MVI-Bench, MINERVA
Attribute Extraction	Shape, color, material confusions	MVI-Bench, PRISM, ThinkWithImages
Relational & Logical	Deductive, analogical, process flaws	PRISM, MMErroR, MINERVA
Tool/Process	Incorrect tool use, step omissions	ThinkWithImages, PRISM
Knowledge/Answering	External knowledge or answer mismatch	MMErroR
Human Disagreement	Ambiguity, subjectivity, granularity	VizWiz Disagreement Taxonomy

Systematic taxonomies of visual reasoning errors, grounded in empirical analysis and fine-grained benchmarks, are shaping not just evaluation but also the scientific and engineering agenda of visual intelligence research (Datta et al., 29 Sep 2025, Nagrani et al., 1 May 2025, Chen et al., 18 Nov 2025, Qian et al., 27 Oct 2025, Zhou et al., 9 Feb 2026, Shi et al., 6 Jan 2026, Bhattacharya et al., 2019).