Gaze Heads: How VLMs Look at What They Describe

Abstract: How a vision-LLM internally solves the task of describing an image is far from obvious. We find that the model develops a specific mechanism for this: a small set of attention heads in its language-model backbone, which we call gaze heads, whose attention tracks the image region the model is currently describing. We find them with a simple correlation score from a few forward passes, using comic strips as a controlled testbed where narrative order is laid out spatially. These gaze heads do not just track the image tokens being described: redirecting their attention to a chosen region forces the VLM to describe that region instead. A single attention-mask intervention on the top-100 gaze heads, fewer than 9% of all heads, steers the model's answer to any chosen comic panel at 83.1% accuracy, while the same intervention on random heads fails to redirect the answer, and intervening on all heads destroys generation. The same lever also extends to continuous control: switching the gaze target mid-generation makes the model wrap up its current panel description and move to the new one within a few tokens. Beyond comics, the same intervention redirects answers to chosen regions in natural COCO images. The mechanism further recurs across model sizes from 2B to 32B parameters and across other VLM architectures, although some frozen-encoder families show no comparable head set. More broadly, this shows that targeted edits identified through mechanistic analysis can serve as practical inference-time levers for steering multimodal model behavior, without any retraining. Our code, interactive demo, and datasets are available at https://gaze.baulab.info/

• The paper demonstrates that a small subset of 'gaze heads' in VLMs is key to aligning language tokens with specific image regions.
• The study uses targeted bias interventions on mid-to-late attention heads to causally steer visual descriptions with up to 83.1% accuracy.
• The method generalizes across natural images and architectures, offering practical interventions for controlled, region-specific VQA and narration.

Mechanistic Dissection and Causal Steering of Visual Grounding in VLMs

Introduction

This paper, "Gaze Heads: How VLMs Look at What They Describe" (2606.14703), addresses the mechanistic basis for visual grounding in vision-LLMs (VLMs). The central contribution is the identification, characterization, and causal manipulation of a small, highly localized subset of language-model attention heads—named gaze heads—that are directly responsible for aligning VLM output tokens with specific image regions during multimodal narrative tasks. Unlike prior works that analyze global image attention or use head ablation as an interpretability probe, this study demonstrates that manipulating the attention patterns of a select set of gaze heads is sufficient to steer what the model describes, without retraining, reweighting, or major model surgery.

Localizing Visual Grounding: Layer and Head Analysis

The authors' analysis centers on Qwen3-VL-8B, a contemporary open-weight VLM with a ViT-based vision encoder and a 36-layer, 32-head/layer causal LM backbone. Using composite comic strip images with explicit spatial structure as the evaluation substrate, they probe and perturb model internals to dissect where and how visual grounding emerges.

Layerwise analysis using difference-of-means "reverse reading" directions reveals that only layers 20–28 support sharp, causal control over spatial referents in the residual stream (Figure 1).

Figure 1: A per-layer probe shows that steering the residual stream with a "read-in-reverse" vector in layers 20–28 reliably flips the predicted panel, localizing visual attention routing to a mid-layer band.

However, exhaustive permutation probing shows these residual directions only capture the binary left-right ordering; arbitrary narrative sequencing is not encoded as a simple vector direction but requires a more fine-grained mechanism.

Gaze Score: Discovering the Gaze Head Set

The theory that attention heads serve as the routing mechanism for spatial focus is substantiated via the gaze score: for each attention head, the score quantifies how its image-token attention distribution shifts as the conditioned panel query varies. The top-100 heads with the highest gaze score (out of 1152 total) are strictly localized to layers 20–28.

Figure 2: The gaze heads' attention profile forms a diagonal structure, tracking the queried panel index; non-gaze heads show diffused, prompt-independent patterns.

Under free-form generation—absent any explicit panel query—these same heads follow the generated narrative, showing a staircase attentional trajectory that aligns with the position in the described comic strip.

Causal Control: Steering What the Model Describes

The functional importance of gaze heads is demonstrated using a masked bias intervention: adding a large positive bias to targeted panel tokens, negative elsewhere, for the top-100 gaze heads through both prefill and decode. This hard attention steering is causally sufficient to make the VLM describe the targeted panel with 83.1% accuracy in VQA and 79.4% in static narration across a diverse held-out set (chance 16.7%), while the same intervention on randomly chosen non-gaze heads is ineffective.

Figure 3: Gaze-head redirection produces high accuracy in aligning the answer to the chosen panel, demonstrating the causal sufficiency and specificity of the mechanism.

Qualitative examples reinforce that the model, given a panel-generic question, defaults to a global/summary answer unless the gaze heads are steered, in which case the output shifts to describe the steered region precisely.

Figure 4: When gaze heads are pointed at the panel of interest, the answer content follows; absent steering, the output is a summary or conflates multiple panels.

Dynamic, Fine-Grained, and Generalizable Control

The gaze head control surface also supports dynamic redirection. By switching the bias schedule every 50 tokens during freeform narration, the model cleanly wraps up one panel's description and transitions to the newly targeted panel's content, obeying the external steering signal even when it conflicts with the default left-to-right layout.

Figure 5: In dynamic switching, the model's panel description order dissociates from serial order and precisely follows the sequence given by the gaze head bias schedule.

As measured by segment-level Spearman correlation ($\rho = 0.87$), gaze head steering tightly controls the order of described panels; non-gaze interventions do not (Figure 6).

Figure 6: Dynamic gaze steering shows a high correlation with the steering schedule, proving high-precision generative control.

A sweep over the number of redirected heads reveals sharp tradeoff behavior: accuracy peaks near $K=100$, representing fewer than 9% of heads; more heads harm fluency, while fewer do not provide full control. This supports the hypothesis that only a compact set of mid-late heads encode spatial grounding (cf. Figure 7).

Beyond Comics: Natural Images and Cross-Architecture Recurrence

The same set of gaze heads, discovered using spatially-structured comic panels, generalize to natural images. Free-form descriptions show gaze-head attention heatmaps that precisely track the described image region, even in the absence of explicit panel structure (Figures 18–20). Causally steering the gaze heads to a chosen bounding-box region on COCO images, the model describes the correct object with 80.3% accuracy for large, 76.2% for medium, and 61.7% for small regions—significantly above the non-gaze baseline.

Figure 8: Gaze-head steering on a natural image restricts the output to the selected region's objects, generalizing the mechanism beyond panel-structured inputs.

Further, the gaze head mechanism is not unique to Qwen3-VL-8B. Using the same procedure, gaze heads can be identified and steered effectively in Ovis, Qwen2-VL, and InternVL3.5, among others—provided the vision encoder is trained together with the LM. In frozen-encoder models like LLaVA and Bunny-3B, gaze head steering is absent or weak. This points to a necessary condition of end-to-end vision backbone training for robust, localizable visual grounding.

Theoretical and Practical Implications

Mechanistically, these results establish that "what is described" in current VLMs is determined, at the granularity of narrative time, by a small, mid-to-late subpopulation of LM attention heads. The selective causal role of gaze heads supports a modular perspective: global narrative structure and region-to-language grounding are separable, and steering the latter does not destroy fluency or text coherence as long as interventions are focused and anatomically precise.

Practically, the findings suggest that fine, precisely tensorized intervention on attention mechanisms can implement controlled visual focus at inference, with no retraining or gradient updates. This enables VLMs to be repurposed or audited for spatial grounding fidelity, user-directed attention, or robust region-wise VQA simply by post-hoc manipulation. The method is simple, scalable, and easily composable with various decoding or alignment techniques.

Theoretically, the research raises questions about the emergence conditions and limits for such modular controllers: Are gaze heads universal in end-to-end trained VLMs? Do they play a similar role in hallucination, spatial reasoning, or compositional tasks? Can they be harnessed as control dials for grounding explanations, region-specific captioning, or dataset debiasing?

Conclusion

The investigation convincingly demonstrates that visual grounding in state-of-the-art VLMs is mediated and causally steerable via a compact subset of mid-late attention heads, termed gaze heads. These heads are discoverable via simple correlation measures, can be manipulated to exert fine control over narrative focus, and generalize across tasks, image types, and architectures conditioned on end-to-end vision backbone training. This points to new directions for mechanistic interpretability, inference-time controllability, and architectural design in multimodal sequence models.