Determine whether gaze and visual-retrieval heads share one mechanism

Determine whether the gaze heads identified by Gandikota and Bau and the visual retrieval heads identified by Park et al. constitute the same underlying mechanism in vision-language models.

Background

The paper contrasts two families of visual heads in vision-LLMs: Gaze Heads, identified through attention-mass scores and validated by steering, and visual retrieval heads, identified through region-attention scores and validated by masking. Because the prior studies use different interventions, tasks, metrics, and random controls, their findings cannot be directly compared. The paper explicitly notes that neither study evaluates the other family’s heads, leaving unresolved whether the two families reflect a common mechanism or distinct mechanisms.