Beyond Shortcuts: Mitigating Visual Illusions in Frozen VLMs via Qualitative Reasoning

Abstract: While Vision-LLMs (VLMs) have achieved state-of-the-art performance in general visual tasks, their perceptual robustness remains remarkably brittle when confronted with optical illusions. These failures are often attributed to shortcut heuristics, where models prioritize linguistic priors and memorized prototypes over direct visual evidence. In this work, we propose Structured Qualitative Inference (SQI), a training-free, data-centric framework designed to fortify visual grounding in frozen VLMs. SQI addresses perceptual anomalies through three systematic modules: (1) Axiomatic Constraint Injection, which suppresses erroneous metric estimations and quantitative hallucinations; (2) Hierarchical Scene Decomposition, which decouples target visual manifolds from complex background distractors; and (3) Counterfactual Self-Verification, an adversarial reasoning step that mitigates confirmation bias. By orchestrating these qualitative constraints at inference time, SQI effectively aligns high-level linguistic reasoning with low-level visual perception. Our framework was evaluated on the DataCV 2026 Challenge (Task I: Classic Illusion Understanding), where it ranked 2nd place overall. Experimental results demonstrate that SQI not only significantly enhances accuracy across diverse illusion categories but also provides superior diagnostic interpretability without any model fine-tuning. Our success underscores the potential of structured qualitative grounding as a robust paradigm for developing next-generation, illusion-resistant vision-language systems.

• The paper introduces the SQI framework to mitigate VLM failures on illusion tasks by applying qualitative reasoning modules at inference time.
• It improves robustness through axiomatic constraint injection, hierarchical scene decomposition, and counterfactual self-verification.
• Empirical results show balanced accuracy around 69%, demonstrating SQI’s effectiveness without requiring model fine-tuning or data augmentation.

Structured Qualitative Inference for Robust Illusion Understanding in Frozen VLMs

Introduction

Standard vision-LLMs (VLMs) are susceptible to failures on tasks involving optical illusions, despite their high accuracy on classical visual benchmarks. This limitation persists because these models tend to adopt shortcut heuristics, predominantly relying on linguistic priors and memorized prototypes at the expense of direct, grounded visual reasoning. The resulting perceptual brittleness manifests as metric hallucination, background interference, and confirmation bias—failure modes that undermine interpretability and reliability, particularly in adversarial or ambiguous visual contexts. The paper "Beyond Shortcuts: Mitigating Visual Illusions in Frozen VLMs via Qualitative Reasoning" (2604.26250) addresses these challenges by proposing the Structured Qualitative Inference (SQI) framework, implemented exclusively as an inference-time augmentation, requiring no fine-tuning or data augmentation.

Figure 1: Overview of Structured Qualitative Inference (SQI) with its three qualitative reasoning modules applied to a frozen VLM.

Structured Qualitative Inference Framework

SQI is architected as a sequence of qualitative reasoning modules applied to frozen VLMs. The aim is to shift away from unreliable metric-based predictions towards robust, systematically constrained inference. SQI operates by executing the following modules:

• Axiomatic Constraint Injection: This module explicitly suppresses the model’s inclination for unreliable quantitative estimation (e.g., measurements of length, angle, or count) in contexts where optical illusions may distort such features. By enforcing qualitative, rather than metric, reasoning, the framework reduces the incidence of metric hallucination.
• Hierarchical Scene Decomposition: The input scene is partitioned into target objects and background, and the model is directed to ignore distracting contextual elements (e.g., occluders, grids, shading) commonly leveraged by visual illusions. This module supports robust isolation of visual evidence germane to the prompt, directly addressing background interference.
• Counterfactual Self-Verification: The model is guided to perform adversarial reevaluation of its initial answer, exploring alternative, counterfactual explanations to combat confirmation bias. This produces more reliable, evidence-driven final predictions.

These modules are orchestrated via a lightweight, domain-adaptive heuristic dispatcher, allowing localized adaptation of constraint sets to different visual illusion taxa, such as alignment, color, and geometric illusions.

Formal Problem Definition and Inference Restructuring

Given a frozen VLM $\mathcal{M}(I, Q) \rightarrow A$, receiving image $I$ and query $Q$ as input, the problem is to output a robust perceptual judgment $A$ in settings where deceptive visual patterns (illusions) may conflict with ground truth. The SQI process is formalized as

$A = \text{SQI}(\mathcal{M}, I, Q, \mathcal{C}),$

with structured qualitative constraints $\mathcal{C}$ guiding each modular step. Rather than propagating the raw query to $\mathcal{M}$, SQI decomposes the inference into constraint-driven steps, systematically suppressing unreliable cues and reinforcing local visual evidence. Importantly, this design is orthogonal to any model fine-tuning and imposes negligible inference-time overhead.

Empirical Results

Evaluation is performed on the DataCV 2026 Challenge (Task I: Classic Illusion Understanding), a benchmark that rigorously tests the robustness of VLMs under classic and adversarial illusion scenarios. SQI achieves 2nd place overall, registering an Overall Accuracy of 69.05%, with particularly well-balanced results on original images (70.48%) and perturbed (adversarial) images (67.62%). This performance demonstrates strong robustness unsupported by alternative methods relying solely on model-centric interventions. The empirical distribution also highlights SQI’s capability for stable accuracy under visual perturbations, where most competing methods experience larger performance variance.

Case Study: Alleviating Background Interference

A qualitative analysis of the SQI modules is presented via the occluded alignment illusion task. In this scenario, standard VLMs are typically misled by occluding elements, often reporting false misalignments due to overreliance on contextual appearance.

Figure 2: Example of SQI application to an occluded alignment illusion, with hierarchical decomposition and counterfactual reasoning.

Here, SQI first isolates the occluded segments from background features, then enforces qualitative directionality analysis, and finally applies counterfactual self-verification via hypothesis extension (such as virtual segment extrapolation). This multi-step protocol enables suppression of misleading visual cues and enhances interpretability, providing a transparent rationale for the model’s final answer.

Implications and Future Directions

The principal implication of SQI is that robust visual grounding in VLMs, particularly under illusion-rich conditions, is fundamentally a reasoning-level—not solely a representational—challenge. The use of structured, qualitative constraints at inference time constitutes a generalizable and lightweight defense against illusion-induced failures, without the inflexibilities and costs of task-specific fine-tuning or data augmentation. The diagnostic interpretability provided by SQI’s modular outputs is further advantageous for model auditing and failure analysis.

Practically, SQI can be deployed as a wrapper for existing foundation VLM APIs, enabling plug-and-play integration for downstream tasks involving ambiguous or adversarial stimuli. Theoretically, advances in this line may inform the design of future VLMs that natively incorporate qualitative, structured reasoning into their architectures, potentially closing the gap between human and machine visual perception under deceptively ambiguous conditions.

Conclusion

The Structured Qualitative Inference framework represents a systematic and practical method for mitigating shortcut-driven failures in frozen VLMs, particularly on visual illusion tasks. By sequencing explicit qualitative constraints on reasoning, SQI achieves strong robustness and interpretability without model retraining. Future research may extend these ideas towards universally-aligned qualitative reasoning layers and more elaborate forms of counterfactual visual self-evaluation.