Explaining cross-family variation in visual-grounding layer distributions

Determine the underlying architectural or training factors that cause the observed variation in which Transformer layers most accurately localize visual evidence (as measured by AUROC and NDCG alignment with VisualCOT evidence labels) across Vision-Language Model families such as LLaVA, Qwen2.5-VL, and Gemma3.

Background

The paper analyzes layer-wise attention dynamics in Vision-LLMs and finds that deeper layers tend to act as visual grounders, focusing sparsely but reliably on evidence regions. Using VisualCOT, the authors quantify evidence attribution with AUROC and NDCG, showing strong alignment of deep-layer attention with ground-truth evidence, even when answers are incorrect.

However, the distribution of layers that maximize evidence attribution differs across model families: LLaVA’s best-performing layers cluster around middle depths, Qwen2.5-VL’s peaks concentrate near the final output layers, and Gemma3 shows a periodic pattern with recurring "good attribution" layers. The authors hypothesize these differences may stem from family-specific design choices or training strategies, but emphasize that a deeper understanding of the underlying causes remains open, motivating this problem.

References

We hypothesize that such differences may stem from family-specific design choices or training strategies, though a deeper understanding of their underlying causes remains open for future investigation.

— Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs  (2510.17771 - Liu et al., 20 Oct 2025) in Appendix, Section "Additional Results and Analysis" — Subsection "Full Layer-wise Attention Dynamics Visualization" (sec:ap-res-att)

However, generalizability of entropy-based profiling across more diverse visual backbones and to different families of vision-LLMs remains to be studied more systematically.

— Seeing Before Answering: Training-Free Visual Layer Profiling for Vision-Language Models  (2608.16263 - Liu et al., 17 Aug 2026) in Conclusion

Whether the same shallow-to-deep transition pattern holds universally for other encoders or multimodal architectures requires further investigation.

— Shallow to Deep: Aligning Token Pruning with Stage-wise Roles in LVLMs  (2609.25635 - Zhang et al., 22 Sep 2026) in Section Limitations

The informative experiment compares against closed-form alignment of last-layer states~\citep{statebridge}, which succeeds cross-model where our intermediate-layer injection fails; separating the alignment method from the injection depth would tell us whether the wall is real.

— Augment the Cache, Replace the Activation: A 2x2 Study of Internal-State Sharing  (2610.22026 - Deng et al., 15 Oct 2026) in Section 5, “Further work”