Gaze Heads: Steering Vision-Language Models

• Gaze heads are a specific set of attention heads in vision-language models that track the image region being described using a panel-tracking correlation score.
• The discovery pipeline uses six-panel comic strips and a simple attention-mass correlation to identify fewer than 9% of heads that exert causal control over model outputs.
• Targeted intervention on gaze heads enables inference-time steering with up to 83.1% accuracy, demonstrating practical control over model-generated descriptions on both comics and natural images.

Gaze heads are a small subset of attention heads in a vision-LLM’s language-model backbone whose cross-modal attention tracks the image region the model is currently describing. In the formulation introduced in “Gaze Heads: How VLMs Look at What They Describe” (Gandikota et al., 12 Jun 2026), these heads are not merely correlational markers of visual grounding: redirecting their attention to a chosen region forces the VLM to describe that region instead. The mechanism is identified with a simple correlation score computed from a few forward passes on six-panel comic strips, and it supports inference-time steering without retraining. On held-out strips, a single attention-mask intervention on the top-100 gaze heads, fewer than 9% of all heads, steers the model’s answer to any chosen comic panel at $83.1\%$ accuracy; the same lever also extends to continuous control and to natural COCO images (Gandikota et al., 12 Jun 2026).

1. Formal characterization

The core intuition is that gaze heads are attention heads whose image-attention re-routes to whichever region is being queried or described. Comic strips provide a controlled testbed because narrative order is laid out spatially from left to right. In that setting, a gaze head will, token by token, shift its image-attention to the panel currently under description (Gandikota et al., 12 Jun 2026).

The formal definition is based on a panel-tracking correlation. For a strip of $N=6$ panels, six forward passes are issued, each with the prompt asking what is happening in the $k$-th panel from the left. For each head $(\ell,h)$, the post-softmax attention mass $A^{(\ell,h)}_{k,j}$ is recorded from the final prompt token to the image tokens of panel $j$. This yields a $6\times 6$ attention matrix whose rows index the queried panel $k$ and whose columns index the attended panel $j$. A perfect gaze head would place its mass on the diagonal $j=k$, and the paper defines

$N=6$0

Heads with the highest $N=6$1 are therefore the ones whose attention tracks the queried panel most strongly (Gandikota et al., 12 Jun 2026).

This definition is deliberately operational. It does not require supervision over internal circuitry, nor any retraining objective tailored to grounding. Instead, it isolates a head set through observed routing behavior under controlled prompting. A plausible implication is that the paper treats visual grounding not as a diffuse property of the entire backbone, but as a mechanism concentrated in a comparatively small and measurable subset of heads.

2. Discovery pipeline on comic strips

The detection procedure exploits the spatial regularity of six-panel strips as ground truth for narrative order. Input preparation resizes each strip to uniform height and concatenates the panels horizontally. For discovery, the model is prompted six times, once per panel index, with the instruction: “Look carefully at this six-panel comic strip. What is happening in the $N=6$2-th panel from the left? Answer briefly.” For each forward pass, post-softmax attention from the final generation token to all image tokens is recorded, then grouped by panel and summed to obtain $N=6$3 for $N=6$4 (Gandikota et al., 12 Jun 2026).

After averaging over strips, each head receives a $N=6$5, the heads are sorted, and the top-$N=6$6 are designated as gaze heads; the default is $N=6$7. The discovery procedure is notable for its low data requirement: the abstract describes finding gaze heads “with a simple correlation score from a few forward passes” (Gandikota et al., 12 Jun 2026).

The methodological significance is twofold. First, the setup makes “narrative order” externally legible, so that internal attention patterns can be compared against a clear target structure. Second, the metric is head-local rather than model-global, allowing the paper to isolate a sparse candidate set before any causal intervention. This suggests a workflow in mechanistic interpretability where simple behavioral probes identify units for targeted editing.

3. Attention-mask steering as a causal intervention

Once the top-$N=6$8 gaze heads are identified, the paper tests causality by “pointing” them at an arbitrary target panel during generation. The intervention is applied to prefill and/or decode steps. For each selected head $N=6$9 and each image-token position $k$0, a constant bias $k$1 is added to the raw attention mask before softmax: $k$2 With $k$3, implemented in practice as $k$4 in bfloat16, the intervention hard-reassigns all attention mass to the target panel’s tokens. Text-token attention is unchanged, and the VLM then continues generation for tasks such as VQA or narration (Gandikota et al., 12 Jun 2026).

The causal result is central: redirecting only these $k$5 gaze heads is sufficient to steer what the model describes. The abstract states that the heads “do not just track the image tokens being described”; rather, redirecting them “forces the VLM to describe that region instead” (Gandikota et al., 12 Jun 2026). That distinction separates gaze heads from a passive diagnostic phenomenon. They are an actionable control surface inside the model.

The same lever also supports continuous control. When the gaze target is switched mid-generation, the model “wrap[s] up its current panel description and move[s] to the new one within a few tokens” (Gandikota et al., 12 Jun 2026). This behavior indicates that the intervention is not limited to initial answer selection; it can alter the active grounding locus during ongoing decoding.

4. Quantitative behavior on comics

The main quantitative evaluation uses 500 held-out strips, giving $k$6 targets, with a forced 1-of-6 LLM judge and chance at $k$7. Under a top-100-gaze-head intervention, VQA redirection reaches $k$8 and static narration reaches $k$9. The same intervention on random non-gaze heads fails to redirect, at approximately $(\ell,h)$0 on VQA and near chance, while intervening on all $(\ell,h)$1 heads destroys generation, below $(\ell,h)$2. Baselines using the identical mask intervention but selecting heads by Image-Heads and Localization-Heads criteria obtain intermediate performance, approximately $(\ell,h)$3–$(\ell,h)$4, below gaze heads (Gandikota et al., 12 Jun 2026).

Experiment	Gaze-head result	Comparison
VQA redirection, $(\ell,h)$5	$(\ell,h)$6	Random non-gaze $(\ell,h)$7; all heads $(\ell,h)$8
Static narration, $(\ell,h)$9	$A^{(\ell,h)}_{k,j}$0	Image-Heads / Localization-Heads $A^{(\ell,h)}_{k,j}$1–$A^{(\ell,h)}_{k,j}$2
Dynamic control	$A^{(\ell,h)}_{k,j}$3	Image-Heads $A^{(\ell,h)}_{k,j}$4; Localization-Heads $A^{(\ell,h)}_{k,j}$5; random non-gaze $A^{(\ell,h)}_{k,j}$6

A head-count sweep clarifies the sparsity regime. Redirection accuracy rises steeply from approximately $A^{(\ell,h)}_{k,j}$7 at $A^{(\ell,h)}_{k,j}$8 to $A^{(\ell,h)}_{k,j}$9 at $j$0, then declines as too many heads break fluency; random non-gaze heads remain near chance throughout (Gandikota et al., 12 Jun 2026). This is consistent with the paper’s broader claim that the mechanism is concentrated in a small subset of heads rather than uniformly distributed across the network.

Dynamic control is measured during a 300-token narration in which the target panel is switched every 50 tokens according to a random derangement. The Spearman correlation between the steering schedule and the described-panel sequence is $j$1 for gaze heads, versus approximately $j$2 for the Image-Heads baseline, approximately $j$3 for Localization-Heads, and approximately $j$4 for random non-gaze heads, where the model falls back to default left-to-right order (Gandikota et al., 12 Jun 2026). This result shows that the intervention is not merely changing the initial answer distribution; it is controlling the temporal progression of grounded description.

5. Extension beyond comics and recurrence across architectures

The intervention generalizes beyond the comic-strip setting to natural images from COCO val2017. There the method steers gaze heads to an object’s bounding box and asks, “What is the main object in this part of the image?” An LLM judge checks whether the answer names the target category. Using the top-100 gaze heads, overall accuracy is $j$5, compared with $j$6 for non-gaze heads. By COCO object size, the reported accuracies are $j$7 versus $j$8 for large objects, $j$9 versus $6\times 6$0 for medium objects, and $6\times 6$1 versus $6\times 6$2 for small objects (Gandikota et al., 12 Jun 2026).

The same discovery-plus-intervention pipeline also recurs across model sizes and architectures. For Qwen3-VL, the peak redirection values reported are $6\times 6$3 at 2B, $6\times 6$4 at 4B, $6\times 6$5 at 8B, and $6\times 6$6 at 32B. Other architectures include Ovis1.5-8B at $6\times 6$7, Qwen2-VL-7B at $6\times 6$8, InternVL3.5-8B at $6\times 6$9, LLaVA-1.5-13B at $k$0, LLaVA-NeXT-7B at $k$1, and Bunny-3B at $k$2 (Gandikota et al., 12 Jun 2026).

The reported limitation is architectural: families with frozen vision encoders, including LLaVA and Bunny, show weak or no comparable gaze-head mechanism, suggesting that joint fine-tuning of the encoder with the LM is critical for gaze heads to emerge (Gandikota et al., 12 Jun 2026). The paper therefore places the mechanism within a specific class of multimodal integration regimes rather than treating it as a universal property of all VLMs.

6. Relation to other uses of “gaze” and “head”

In (Gandikota et al., 12 Jun 2026), a “head” is an attention head in a VLM backbone. In adjacent gaze literature, by contrast, a “head” is often a human head localized in an image. GazeHTA predicts multiple head-target instances, each containing a head heatmap, a gaze-target heatmap, a binary out-of-frame flag, and a connection map linking a head location to its target (Lin et al., 2024). GazeDETR formulates the problem as set prediction of head bounding boxes, 2D gaze points, and in/out labels, using two disentangled decoders for head localization and gaze estimation (Belen et al., 18 Aug 2025).

Other work studies head–gaze coordination in humans rather than latent routing in VLMs. “Measuring and modeling the perception of natural and unconstrained gaze in humans and machines” reports that head orientation alone yields poor exact accuracy, while the eyes region—including nose bridge and temple cues—can be sufficient for accurate gaze judgments (Harari et al., 2016). The Gaze-in-the-Wild dataset shows that pursuit detection degrades markedly without head-movement information and that absolute eye and head velocities provide most of the classification power (Kothari et al., 2019). In human–robot interaction, Schreiter et al. quantify task-conditioned coordination between gaze direction and head rotation, including a head-contribution metric for gaze shifts (Schreiter et al., 2024).

These literatures address different entities and different levels of analysis. One line predicts where people are looking from visual, pose, or motion cues; another isolates internal attention heads that govern where a VLM “looks” while generating text. A plausible implication is that the term “gaze heads” now spans both external gaze estimation and internal multimodal routing, but the object under study is fundamentally different in each case.

7. Interpretability, control, and open questions

The broader claim of (Gandikota et al., 12 Jun 2026) is that mechanistic interpretability can do more than describe latent function. The paper states that targeted edits identified through mechanistic analysis can serve as practical inference-time levers for steering multimodal model behavior, without retraining. Gaze heads therefore function as a precise, low-overhead handle to redirect what a VLM looks at and hence what it says (Gandikota et al., 12 Jun 2026).

Several open questions are explicitly posed. The paper asks why gaze heads emerge only when the encoder is trained end-to-end; whether gaze heads mediate other visual tasks such as spatial reasoning or hallucination suppression; whether similar discovery-plus-intervention pipelines can uncover levers for other modalities or for internal reasoning steps; and how robust gaze steering is to adversarial inputs or more complex scene layouts (Gandikota et al., 12 Jun 2026).

In summary, gaze heads are a formally identifiable set of attention heads whose raw attention patterns both correlate with and causally control the visual grounding of VLM text generation. Their significance lies not only in explaining how multimodal description is internally routed, but also in demonstrating that simple correlation scores over multiple forward passes, coupled with targeted attention-mask interventions, can expose a usable control interface inside large multimodal models (Gandikota et al., 12 Jun 2026).