How Do VLMs Behave When Blind or Misled? Behavioral Evaluation of VLMs on Scientific Figures

Abstract: Existing vision-LLM (VLM) benchmarks emphasize perception and reasoning accuracy (how well VLMs describe and reason about what they see in an image), with limited attention to behavioral reliability under uncertainty (how they behave when visual evidence is missing or misleading). We introduce SciFigBench, a diagnostic VLM benchmark for scientific figure understanding that jointly evaluates perception, reasoning, and behavioral reliability under uncertainty. It contains 250 figures with high-quality human annotations across three evaluation aspects, totaling 600+ hours of annotation effort. We further extend these figures via image transformations, reasoning questions, resistance probes, caption-bias probes, and confirmed selective-blur targets, producing over 34,000 evaluation setups for stress testing. We further propose the Admittance-Resistance-Inductance (A-R-I) framework to evaluate whether models acknowledge insufficient evidence, resist misleading context, and infer cautiously from partial information. Our results reveal substantial behavioral differences among models. GPT-5.2 achieves the highest description quality (MQM 91.6) with strong reasoning accuracy (78.4%), yet hallucinates unreadable content in 96% of cases, whereas Gemini 3.1 Pro, a comparably capable model (MQM 90.2, reasoning 81.0%), admits uncertainty in 71% of such cases and achieves the strongest resistance score (0.91). These findings show that high perception and reasoning accuracy alone do not guarantee behavioral reliability, a dimension critical for deployment in scientific workflows.

• The paper introduces SciFigBench, a benchmark of 250 expert-annotated scientific figures, more than 34,000 evaluation setups, and the A-R-I framework for measuring how VLMs respond to missing, false, or inferable evidence.
• The evaluation finds a major perception–behaviour disconnect: GPT-5.2 and Gemini 3.1 Pro differ by only 1.4 MQM points, yet Gemini actively admits uncertainty in 71% of blurred-element questions compared with GPT-5.2’s 8%, which fabricates answers in 96% of those cases.
• The results show that presupposed false elements, misleading captions, and direct-questioning pressure can trigger confident fabrication, so scientific workflows should assess admittance, resistance, and inductive inference—not accuracy alone.

SciFigBench is a diagnostic benchmark for scientific figure understanding that extends evaluation beyond perception and reasoning accuracy to behavioural reliability under uncertainty. The paper introduces 250 human-annotated figures from 187 arXiv papers, over 34,000 evaluation setups, and the Admittance–Resistance–Inductance (A-R-I) framework, and demonstrates that models with near-identical perception scores can exhibit radically different behaviour when visual evidence is missing or misleading (2608.13267).

Benchmark design and scale

SciFigBench comprises 250 English-language scientific figures (99 bar charts, 99 line plots, 52 pie charts) drawn from 2023–2025 arXiv papers, each paired with an expert description produced by two trained annotators with third-party adjudication. The annotation campaign consumed over 600 hours, including roughly 240 hours for descriptions, 200 for reasoning-question post-editing, 100 for blur-target confirmation, and 90 for MQM validation. Beyond the base figures, the benchmark expands to more than 34,000 evaluation setups through image transformations (Gaussian noise, low contrast, rotation, in-paper page context), 1,000 figure-grounded reasoning questions, 1,293 behavioural probes, and confirmed selective-blur targets (228 admittance, 215 inductance).

Perception is scored with a checklist-based adaptation of MQM, in which a GPT-4o judge evaluates coverage and correctness against expert groundtruth and a rule engine maps findings to Accuracy, Completeness, and Clarity penalties. The automated evaluation is validated against human annotators with Krippendorff's $\alpha = 0.91$ and model-level Spearman $\rho = 0.80$; notably, the judge systematically under-scores humans by a mean of 15 MQM points, and its recall for hallucinated content is only 0.07, a limitation the authors disclose. Reasoning questions cover counting, computation, comparison, and pattern analysis, seeded by GPT-4o, validated by Mistral Large 3, and human-reviewed.

The A-R-I framework

The paper's central methodological contribution is the Admittance–Resistance–Inductance framework, which decomposes behaviour under uncertainty into three regimes distinguished by whether the underlying evidence exists and is recoverable:

• Admittance: when a relevant element is present but unrecoverable (e.g., selectively blurred), the model should acknowledge the limitation rather than answer as if the evidence were visible.
• Resistance: the model should reject false premises, non-existent elements, unanswerable requests, and poisoned captions rather than incorporate them.
• Inductance: when an element is present but altered and contextually inferable (e.g., a value deducible from the axis scale), the model should infer the correct answer rather than abstain or fabricate.

The authors show these dimensions are empirically separable: models that resist false premises can still fail to acknowledge missing evidence, and vice versa. Resistance probes take three forms—inexist probes that presuppose absent chart elements, contra probes that anchor on values 20–30% off from actual, and unanswerable probes—plus caption-bias probes embedding 2–3 plausible false claims within otherwise accurate captions. Selective-blur targets are located via OCR, proposed by GPT-4o, matched with fuzzy matching, blurred with a two-stage grey-blend and Gaussian kernel, and confirmed by human review.

Perception and reasoning results

Eight frontier models were evaluated at temperature 0: GPT-5.2, Gemini 3.1 Pro, Llama 4 Maverick, Qwen3-VL (235B, 30B, 8B), Gemma 3 27B, and Phi-4 Multimodal. On baseline description quality, GPT-5.2 leads with MQM 91.6 [90.4, 92.8] and Gemini 3.1 Pro follows at 90.2 [88.9, 91.4]; the 1.4-point gap is statistically significant ($p < 0.01$) but practically small (Cliff's $\delta = 0.09$). Phi-4 trails at 62.2, driven mainly by completeness penalties. Rotation is the most damaging transform, costing an average 19.4 MQM points, while noise is negligible and in-paper embedding costs only 2–5 points for most models. On reasoning, Gemini 3.1 Pro leads at 81.0% and GPT-5.2 at 78.4%, with a sharp drop to the rest of the field—no other model exceeds 53.4% on any category, and Phi-4 scores 8.6% overall.

Model	MQM (base)	Reasoning (%)	Resistance	Active admittance (%)
GPT-5.2	91.6	78.4	0.81	8
Gemini 3.1 Pro	90.2	81.0	0.91	71
Llama 4	81.4	48.5	0.76	19
Qwen-235B	80.8	58.4	0.67	15
Phi-4	62.2	8.6	0.21	5

The perception–behaviour disconnect

The paper's strongest and most consequential claim is that high perception and reasoning accuracy do not guarantee behavioural reliability. GPT-5.2 and Gemini 3.1 Pro differ by only 1.4 MQM points, yet their active admittance rates differ by 63 percentage points: Gemini acknowledges uncertainty in 71% of targeted questions about blurred elements, while GPT-5.2 admits only 8% and fabricates answers in 96% of such cases—a "confident fabricator" profile. Gemini achieves the highest resistance score (0.91 vs. GPT-5.2's 0.81, significant at $p < 0.001$, $n = 750$). Split-half reliability of $\rho = 0.979$ over 100 random splits confirms this divergence is stable, meaning a benchmark reporting only MQM would rank the two models as near-equivalent while missing opposite behaviours under uncertainty. The practical implication is direct: model selection for scientific workflows based solely on accuracy benchmarks risks embedding confident fabrication into research pipelines.

Three further findings merit emphasis. First, presupposition embedding is the strongest deception vector: inexist probes are hardest to resist across all eight models (e.g., Llama 4 resists explicit false values at 0.76 but drops to 0.63 against presupposed absent elements), mirroring Loftus's eyewitness-testimony findings that definite articles induce false memories. Second, caption dependency appears to be a training artifact rather than a capability limitation: Phi-4 follows modified captions almost entirely ($R = 0.05$) while Gemma 3 27B resists at $R = 0.38$ despite a lower MQM score, suggesting instruction tuning that conditions models to trust provided context over visual evidence. Third, a "must answer" bias emerges across modes: GPT-5.2 acknowledges blurred elements 23% of the time in passive descriptions but only 8% under direct questioning, a pattern consistent with RLHF-induced helpfulness pressure; only Gemini maintains high admittance in both modes (59% passive, 71% active).

Inductance results validate the framework's third axis: when models fabricate answers for context-inferable elements, correctness ranges from 14% to 66%, versus 5–14% for unrecoverable elements, confirming that top models perform genuine contextual inference when context permits rather than uniform guessing. Methodological robustness is supported by a probe-designer ablation (GPT-4o vs. Mistral Large 3 probes preserve rankings) and cross-judge validation (GPT-4o vs. Mistral Large 3 on capability questions yields $\rho = 1.000$ with a uniform leniency offset).

Limitations and open questions

The paper concedes several constraints on its conclusions. The figure corpus is restricted to bar charts, line plots, and pie charts; scatter plots, heatmaps, network diagrams, and schematic figures are excluded, as are non-English corpora. Because top models achieve MQM $\rho = 0.80$0 on clean descriptions, the benchmark cannot conclusively separate instruction-following pressure from genuine visual limitations, nor attribute failures to specific components such as the vision encoder versus the LLM—controlled alignment and prompting interventions and internal probing of open-weight models are left as open work. Automated judging depends on GPT-4o, which under-detects hallucinated content (recall 0.07) and over-flags completeness; the authors mitigate this with human agreement studies and cross-judge checks, but item-level judge calibration remains a source of dispersion. Finally, commercial API models may drift across snapshots, so absolute scores may not reproduce even though the paper pins identifiers and API versions.

Conclusion

SciFigBench demonstrates that perception and reasoning scores mask a distinct behavioural dimension of VLM competence on scientific figures. The A-R-I framework exposes failure modes—invisible to conventional quality metrics—such as confident fabrication under missing evidence, susceptibility to presupposition-embedded deception, and caption dependency, and the empirical reversals between GPT-5.2 and Gemini 3.1 Pro show these behaviours must be measured separately from accuracy. The benchmark's coverage of chart types, languages, and causal attribution of behavioural failures remains incomplete, and resolving whether these behaviours stem from alignment choices or architectural factors is the principal question the paper leaves open.