Explaining cross-family variation in visual-grounding layer distributions

Determine the underlying architectural or training factors that cause the observed variation in which Transformer layers most accurately localize visual evidence (as measured by AUROC and NDCG alignment with VisualCOT evidence labels) across Vision-Language Model families such as LLaVA, Qwen2.5-VL, and Gemma3.

Background

The paper analyzes layer-wise attention dynamics in Vision-LLMs and finds that deeper layers tend to act as visual grounders, focusing sparsely but reliably on evidence regions. Using VisualCOT, the authors quantify evidence attribution with AUROC and NDCG, showing strong alignment of deep-layer attention with ground-truth evidence, even when answers are incorrect.

However, the distribution of layers that maximize evidence attribution differs across model families: LLaVA’s best-performing layers cluster around middle depths, Qwen2.5-VL’s peaks concentrate near the final output layers, and Gemma3 shows a periodic pattern with recurring "good attribution" layers. The authors hypothesize these differences may stem from family-specific design choices or training strategies, but emphasize that a deeper understanding of the underlying causes remains open, motivating this problem.