More Than Meets the Eye: Measuring the Semiotic Gap in Vision-Language Models via Semantic Anchorage

Abstract: Vision-LLMs (VLMs) excel at photorealistic generation, yet often struggle to represent abstract meaning such as idiomatic interpretations of noun compounds. To study whether high visual fidelity interferes with idiomatic compositionality under visual abstraction, we introduce DIVA, a controlled benchmark that replaces high-fidelity visual detail with schematic iconicity by generating paired, sense-anchored visualizations for literal and idiomatic readings. We further propose Semantic Alignment Gap ($Δ$), an architecture-agnostic metric that quantifies divergence between literal and idiomatic visual grounding. We additionally introduce a directional signed bias $b(t)$ to separately measure the direction and strength of literal preference. Evaluating 8 recent VLMs, we reveal a consistent Literal Superiority Bias: model scale alone does not resolve literal preference, and increased visual fidelity is associated with weaker symbolic alignment, suggesting cognitive interference from hyper-realistic imagery. Our findings suggest that improving compositional understanding requires iconographic abstraction of visual input and anchoring interpretation and generation in intended meaning.

• The paper introduces Iconographic Abstraction and the DIVA benchmark to address literal superiority bias in interpreting idiomatic noun compounds.
• It defines unified metrics—Semantic Alignment Gap and signed literal bias—that show significant improvements, including near-zero gaps in models like GPT-5.
• Experimental results across model architectures demonstrate that controlled visual abstraction enhances idiomatic-literal balance, informing future pretraining strategies.

Measuring the Semiotic Gap in Vision-LLMs via Iconographic Abstraction

Introduction

The paper "More Than Meets the Eye: Measuring the Semiotic Gap in Vision-LLMs via Semantic Anchorage" (2604.17354) investigates a central challenge in vision-language modeling: the difficulty models have in transcending the literal interpretation of visual stimuli, particularly when tasked with semantic abstraction such as interpreting idiomatic noun compounds. The authors propose that high visual fidelity in input images can exacerbate a "Literal Superiority Bias," impeding the model's ability to align visual representations with abstract, idiomatic meaning. Addressing this, the work introduces the DIVA benchmark—a dataset built on iconographic abstraction—to systematically evaluate the impact of visual simplification on semantic alignment in leading Vision-LLMs (VLMs). Furthermore, the paper formalizes the Semantic Alignment Gap ($\Delta$) and a signed literal bias metric as unified, architecture-agnostic measures of visio-semantic divergence.

Figure 1: Overview of the Iconographic Abstraction Framework. The transition from iconic (high-fidelity simulation) to symbolic (abstract code) visualizations operationalizes "Literal Bias" measurement in VLMs.

Problem Statement and Theoretical Framework

The Semiotic Gap and Literal Superiority Bias

While modern text-to-image and vision-LLMs excel at photorealistic synthesis and literal visual decoding, they frequently fail to interpret images as symbols—signifiers whose meaning is established by convention rather than resemblance. This pitfall is acute for noun compounds (NCs) and idioms, where conventional meaning diverges from compositional/literal interpretation (e.g., "eye candy" as "something superficially attractive" rather than "literal candy shaped like an eye"). The authors attribute this deficiency to a "Literal Superiority Bias": an overreliance on visually plausible, high-fidelity details, yielding a marked gap between literal and idiomatic semantic interpretation.

Iconographic Abstraction and Semantic Anchorage

To disrupt this literal pull, the authors define an "Iconographic Abstraction" framework, operationalizing a spectrum from iconicity (detailed simulation) to symbolism (schematic representation). Central to this is "Semantic Anchorage": the process of systematically simplifying images while anchoring their content to specific sense labels (idiomatic vs. literal). The underlying hypothesis is that abstraction suppresses irrelevant high-frequency visual cues ("semiotic noise"), thus promoting symbolic alignment and improving the model's capacity for idiomatic understanding.

Figure 2: High Visual Fidelity ("semiotic noise") in photorealistic images promotes literal preference.

Figure 3: Iconographic Abstraction: Original high-fidelity images (AdMIRe) and their minimalist DIVA counterparts demonstrate the spectrum from literal to idiomatic interpretation for each noun compound.

DIVA Benchmark and Methodology

Dataset Construction

DIVA (Distilled Idiomatic Visual Abstraction) is introduced as a paired, sense-anchored benchmark derived from the AdMIRe idiomatic reasoning set. For each noun compound, the pipeline generates both literal and idiomatic iconographic images using an automated abstraction process followed by human verification. This pipeline enforces constraints on geometric simplicity, color palette, and context minimization to yield stylistically uniform and semantically transparent icons.

Unified Metric: Semantic Alignment Gap and Signed Literal Bias

To quantify the gap between literal and idiomatic interpretation, the authors formalize:

• Signed Literal Bias ($b(t)$): The directional difference in model preference for literal over idiomatic realizations for an NC.
• Semantic Alignment Gap ($\Delta(t)$): The absolute difference, reflecting the magnitude of preference independent of direction.

These metrics are deployed across architectural paradigms using matched scoring functions:

• Cosine similarity (for discriminative models, e.g., CLIP)
• Logit-calibrated binary decision probabilities (for open generative models, e.g., LLaVA, InternVL3)
• Self-reported confidence scores and forced-choice accuracy (for closed-source models, e.g., GPT-5, Claude)

Experimental Results

Figure 4: Across all architectural families, the Semantic Alignment Gap ($\Delta$) is systematically reduced when switching from high-fidelity (AdMIRe) to iconographic (DIVA) imagery.

Quantitative benchmarking on eight leading VLMs reveals that:

• All models exhibit significant literal bias ($b > 0$) under high-fidelity images.
• Semantic Alignment Gap $\Delta$ is largest for discriminative models (e.g., CLIP-style), intermediate for open-generative architectures, and lowest for proprietary (frontier) LLMs.
• Across all families, iconographic abstraction yields a significant, paired reduction in $\Delta$ (Wilcoxon signed-rank, $p < 0.001$).
• For GPT-5, $\Delta$ is reduced to near-zero ($\Delta_{icon} \approx 0.02$), indicating almost fully balanced idiomatic/literal alignment under icon conditions.
• Explicit sense specification coupled with iconographic input boosts 5-way selection accuracy (Discriminative: 42.3%→58.7%, Open-Generative: 61.8%→74.2%, Proprietary: 78.5%→91.3%).

Analysis and Implications

Cognitive Interference and Semiotic Cost of Fidelity

High visual fidelity introduces "cognitive interference" by amplifying irrelevant surface cues, impairing the system's ability to engage in abstract reasoning. This supports the contention that VLMs optimize for analog scene reconstruction rather than symbol-grounded interpretation—a limitation with direct theoretical and practical ramifications.

Architecture-Agnostic Evaluation and Model Development

The Semantic Alignment Gap framework enables meaningful intra-paradigm model comparison, robust to differences in scoring mechanics (cosine, logit probability, behavioral proxy). The findings implicate pretraining objectives and visual complexity as key modifiable factors in enhancing semantic compositionality.

Practical and Theoretical Directions

The practical upshot is that controlled visual abstraction is a tractable and effective intervention for benchmarking—and, potentially, training—models for deeper semantic understanding. Theoretically, the study motivates the development of pretraining and adaptation strategies that explicitly target symbolic, metaphorical, and idiomatic alignment, rather than mere simulation fidelity.

Limitations

The analysis is bounded by its focus on English noun compounds and specific iconographic styles (e.g., flat design), with results demonstrating correlation not strict causality between visual abstraction and alignment improvement. Generalization to other languages, idiom structures, and alternative abstraction protocols remains an open avenue. Furthermore, the self-reported confidence and forced-choice metrics in proprietary models may still partially reflect instruction-following artifacts.

Conclusion

The research establishes that high-fidelity visual realism in current vision-LLM pipelines enforces a systematic literal interpretation bias, quantifiable via the proposed Semantic Alignment Gap. Iconographic abstraction robustly reduces this bias and improves idiomatic compositionality across all major model architectures. This work underscores the utility of benchmarking—and potentially training—models with minimalist, semantically-anchored iconography to catalyze genuine symbolic reasoning in multimodal AI. Future directions include multilingual and cross-cultural expansion, as well as model fine-tuning paradigms that directly exploit paired literal/idiomatic contrast for enhanced compositional intelligence.