Scalable detection and causal evaluation of aggregation heads in modern vision–language models

Develop computationally viable methods to automatically detect aggregation attention heads in large-scale vision–language models and to apply causal interventions that validate the role of these heads in achieving symbol grounding between environmental tokens (e.g., visual patch embeddings) and linguistic tokens in the language backbone across diverse architectures.

Background

The paper provides mechanistic evidence that symbol grounding in autoregressive models is implemented via aggregate attention heads that retrieve environmental grounds and support prediction of corresponding linguistic forms, with strongest effects in middle layers. These observations are confirmed through saliency analysis and causal ablations in controlled text-only and multimodal settings.

When extending the grounding-as-aggregation hypothesis to full-scale vision–LLMs like LLaVA-1.5-7B, the authors note practical complications: multiple sets of visual tokens (including CLIP-derived embeddings that encode language priors), global information stored in redundant artifact tokens, and a large number of visual tokens that increase the computational cost and difficulty of isolating genuine aggregation heads. These challenges motivate the need for scalable, systematic detection and causal evaluation methods across diverse VLMs.