Attention Alignment Between Humans and Vision-Language Models

Abstract: Visual perception depends on top-down goals and bottom-up sensory mechanisms. Vision-LLMs implement both, allowing us to treat each component as a separable hypothesis about what drives where we look. We compared spatial attention maps from six vision-LLMs against human fixation heatmaps recorded on 200 images during two tasks (general description and social captioning). The six models spanned a 2$\times$2 factorial of CNN vs.\ ViT encoders crossed with LSTM vs.\ Transformer decoders, plus Molmo 7B-D and Qwen3.5 9B. We found that both decoder and encoder architecture shaped alignment, but decoder choice dominated. LSTM vs.\ Transformer decoders increased alignment by 40--50 percentage points (80--87\% vs.\ 40--59\% of the human noise ceiling). In contrast, CNN vs.\ ViT encoders contributed a secondary 5--20 point advantage depending on decoder family, with CNN-LSTM the most aligned model overall (85--87\%). Despite their alignment advantage, LSTM-decoder attention maps were spatially diffuse and minimally task-differentiated; ViT-Transformer, the weakest in alignment, showed the sharpest spatial concentration and strongest task differentiation. A hemispatial-neglect simulation confirmed that ablating attention impacted LSTM decoders more than Transformer decoders. In an exploratory extension using TRIBE-simulated synthetic neural responses, fixation alignment and neural relevance dissociate: CNN-Transformer attention maps better predicted synthetic brain activity despite lower fixation alignment, with attention maps best predicting early visual cortex. Together, top-down and bottom-up components trade off what they predict in behavioral and synthetic neural data.

• The paper demonstrates that decoder architecture, particularly LSTM-based designs, is the primary driver of alignment between model attention maps and human fixation patterns.
• It reveals that while CNN encoders outperform ViT encoders in spatial alignment, high alignment does not necessarily translate to superior neural predictivity.
• The study highlights a trade-off where diffuse LSTM attention achieves closer gaze alignment but lacks adaptability, whereas Transformer models show robust performance against spatial perturbations.

Architectural and Algorithmic Determinants of Attention Alignment Between Humans and Vision-LLMs

Introduction

This work conducts a systematic assessment of how architectural choices in VLMs determine the spatial alignment between model-derived attention maps and human visual fixation patterns during image captioning tasks. By employing a 2$\times$2 factorial manipulation of encoder type (CNN vs. ViT) and decoder type (LSTM vs. Transformer), alongside two large SOTA VLMs (Molmo 7B-D and Qwen3.5 9B), the study isolates the contributions of bottom-up and top-down components to human-model agreement in attention. Human behavioral data (fixation heatmaps) were acquired in both a generic "describe scene" and social attention-commentary setting, providing a robust benchmark. The analysis foregrounds decoder architecture as the primary determinant of model-human spatial alignment, with encoder family exerting a subordinate but nontrivial effect. Notably, the study demonstrates that high spatial alignment with human gaze does not universally yield superior predictivity of synthetic neural responses, indicating a dissociation between behavioral and neural alignment metrics.

Model-Human Alignment: Decoder Dominance and Architectural Effects

Decoder choice exerts a pronounced influence on human-model spatial alignment: LSTM-decoder variants (CNN-LSTM and ViT-LSTM) reach 80–87% of the human-human noise ceiling, substantially surpassing Transformer-decoder variants (40–59%) on both Describe and Social tasks (Figure 1).

Figure 1: Decoder and encoder effects on alignment—(A) Describe, (B) Social, and (C) between-task; alignment scores expressed relative to human-human ceiling.

CNN encoders outperform ViT encoders within both decoder families, with a 3–7 percentage point advantage. Among SOTA VLMs, Molmo exhibits alignment exceeding Transformer architectures and approaching LSTM-based models, while Qwen3.5 yields intermediate scores. Importantly, quantitative gains in model-human fixation agreement are robust to confounding by caption quality; all architectures achieve comparable semantic similarity to human captions, localizing observed differences specifically to spatial attention strategies.

Spatial Diffuseness, Task Adaptivity, and Attention Topography

Despite strong alignment with prioritized locations, LSTM-based decoders distribute attention diffusely with maximal entropy ($H \approx 1.0$), in sharp contrast with humans, whose attention is sharply focused ($H \sim 0.55$–$0.62$). Transformer decoders, regardless of encoder, produce more spatially concentrated attention but with reduced alignment to where humans look. ViT-Transformer exhibits the highest spatial acuity and strongest sensitivity to task demands, reflected in lower between-task attention similarity (Figure 2).

Figure 2: Representative attention maps for humans and all models on both Social and Describe tasks, highlighting inter- and intra-task flexibility.

LSTM decoders, although superior in spatial alignment, are less flexible: their attention maps are more invariant across tasks compared to human and Transformer-decoder attention, indicating a lack of context dependence.

Robustness to Spatial Lesion and Distributed Attention Dynamics

Simulated hemispatial neglect via progressive ablation of left-sided attention reveals that distributed Transformer-based models (specifically ViT-Transformer) are most robust to spatial lesion, maintaining caption similarity under severe ablation. LSTM architectures, in contrast, degrade more rapidly, underscoring a trade-off between broad spatial alignment and functional adaptation to spatial perturbation (Figure 3).

Figure 3: Caption similarity under increasing left-side spatial ablation; LSTM decoders are more sensitive to lesion, Transformers display greater robustness.

These results suggest that distributed attention mechanisms afford resilience to spatial loss at the cost of precision in alignment to human fixations.

Behavioral and Neural Alignment Dissociation

Encoding analyses leveraging TRIBE v2 synthetic cortical responses demonstrate that model gaze alignment and neural predictivity dissociate: CNN-Transformer yields the best whole-brain prediction ($R^2$), surpassing even LSTM decoders with higher behavioral alignment (Figure 4A). Variance partitioning indicates that CLIP-derived "what" features dominate explained variance, but unique "where" variance from model attention maps is detectable in occipital and dorsal regions, though minor in magnitude (Figure 4B).

Figure 4: (A) Whole-brain mean $R^2$ for encoding models, (B) Variance partition for CNN-Transformer; unique and shared contributions of "where" and "what" features visualized across cortical surfaces.

This decoupling of spatial and semantic-driven neural signals implies that spatial correspondence with human gaze does not guarantee biological validity at the level of neural activation patterns.

Implications and Future Directions

The quantitative separation between behavioral alignment (where models direct attention) and neural alignment (how well model-derived attention predicts population-level brain responses) reframes the criteria for evaluating human-likeness in VLMs. Decoder architecture, specifically the use of LSTM-based compressive attention, is optimal for capturing human spatial priors, but Transformer-based decompositions—the architectural backbone of large, distributed models—afford increased robustness and, counterintuitively, improved neural predictivity. The encoder family effect (CNN > ViT for alignment) challenges trends suggesting equivalence post-training, advocating for further investigation into the interplay of architectural inductive biases and training dynamics.

Practically, this work suggests that optimization of human attention modeling requires architectural choices distinct from those maximizing neural correspondence or robustness to spatial lesion. The dissociation argues for multi-metric evaluation in model selection and urges caution in interpreting alignment with gaze as sole evidence of biological plausibility.

Theoretically, the results favor a view in which "what" and "where" signals are interleaved across cortex, rather than strictly segregated, implicating the need for models that integrate both spatial priority and semantic content in flexible, context-driven ways.

Further empirical work is needed to determine whether neural results observed with TRIBE v2 synthetic data are replicable in measured fMRI data, and to extend architectural manipulations to larger, more diverse VLMs and downstream interactive settings.

Conclusion

This study delineates the architectural determinants of attention alignment in VLMs, establishing decoder mechanisms as primary drivers, with encoder family exerting a secondary but consistent effect. Strong spatial alignment with human fixations is mediated by diffuse LSTM-based attention, at the expense of context-specific reconfiguration and robustness. Critically, alignment with human gaze does not imply superior neural predictivity, highlighting a decoupling between behavioral and synthetic neural metrics. These findings urge reconsideration of single-metric evaluation and underscore the need for architecture-aware, multi-modal benchmarks in assessing human-like visual attention in artificial systems.