Seeing Before Answering: Training-Free Visual Layer Profiling for Vision-Language Models

Abstract: LLaVA-style Vision-LLMs (VLMs) pass visual tokens from a fixed late layer of the vision backbone, typically the penultimate one, to the LLM. We first show that this hidden convention is fragile: across 2 VLMs and 7 image and video benchmarks, the default layer is sub-optimal in 13 of 14 model-task pairs, and the best layer shifts with both task and visual backbone. Finding that layer by exhaustive layer-wise inference is prohibitively expensive, and no better fixed default exists. We therefore ask whether layer usefulness can instead be predicted from representation geometry. We study matrix-based entropy, introduced for unimodal layer analysis, which we compute over sample-level visual embeddings as Visual Dataset Entropy (VDE); and Gromov-Wasserstein (GW) distance, introduced for encoder-level VLM model selection, which we repurpose as a layer-wise visual--language alignment signal. Transferring these to LLaVA-based models is not obvious a priori: the vision tower is frozen while the multimodal projector is trained, so we profile both sides of the projector. We find that VDE transfers, and GW does not. Computed from 100 unlabeled task samples without downstream inference, pre-projector VDE tracks layer-wise accuracy and its top-ranked layers cover the oracle best layer on every task for the SigLIP-based LLaVA-Video, while giving region-level guidance for the CLIP-based Video-LLaVA. Post-projector profiles show that the projector reshapes visual geometry but does not erase the performance-relevant trend, leaving $\mathrm{VDE}_{\mathrm{pre}}$ the stronger signal. GW instead flattens after projection and is best read as an alignment diagnostic rather than a selector. VDE thus offers an interpretable, training-free policy that narrows the visual-layer search to a handful of candidates for limited downstream verification.

• The paper introduces a training-free profiling method that uses Visual Dataset Entropy from 100 unlabeled samples to rank visual-backbone layers before downstream answer generation.
• The paper finds that pre-projector VDE correlates strongly with layer-wise accuracy and covers the oracle layer in all seven tasks for the SigLIP-based LLaVA-Video model, while offering region-level guidance for CLIP-based Video-LLaVA.
• The paper shows that the penultimate-layer convention is unreliable across tasks and backbones, and recommends using VDE to narrow layer searches before limited downstream verification while treating Gromov-Wasserstein distance as an alignment diagnostic.

The fragility of the fixed late-layer convention

LLaVA-style vision-LLMs (VLMs) pass visual tokens to the LLM from a single fixed layer of the frozen vision backbone, almost always the penultimate layer. The paper's first contribution is a systematic demonstration that this hidden architectural assumption is unreliable. Across two VLMs — LLaVA-Video-7B-Qwen2 with a 27-layer SigLIP tower and Video-LLaVA-7B-hf with a 24-layer CLIP tower — and seven image and video benchmarks (ScienceQA, POPE adversarial, CV-Bench 2D/3D, MMStar, and HD-EPIC Action and Gaze), the default layer is sub-optimal in 13 of 14 model-task pairs. Gains from switching layers are modest but consistent: for LLaVA-Video on ScienceQA, the default Layer 26 yields 87.62% versus 88.90% at Layer 27; for Video-LLaVA on CV-Bench 2D, the default Layer 23 yields 49.86% versus 53.20% at Layer 20. Critically, the best layer shifts with both task and backbone — SigLIP-based models favor the deepest layers, while CLIP-based models often peak in mid-to-upper layers with degradation at the final layer.

Identifying the best layer by exhaustive layer-wise inference requires a full downstream evaluation per candidate layer, which is prohibitively expensive for large video VLMs, and no better universal fixed default exists. The paper therefore reframes visual-layer choice as an interpretability problem: can layer usefulness be predicted from representation geometry alone, before any answer generation?

Profiling framework

The method computes two training-free geometric signals over sample-level embeddings extracted from each candidate layer, using only $N = 100$ unlabeled task samples. Visual Dataset Entropy (VDE) applies matrix-based Rényi entropy to the trace-normalized Gram matrix of mean-pooled, centered, row-normalized sample embeddings; its max-entropy-normalized form measures how many directions the sample representations span. Because the vision tower is frozen while the multimodal projector is trained during instruction tuning, VDE is computed on both sides of the projector: $\mathrm{VDE}_{\mathrm{pre}}$ reflects intrinsic backbone geometry, while $\mathrm{VDE}_{\mathrm{post}}$ captures what is actually delivered to the LLM. Gromov-Wasserstein (GW) distance, originally introduced for encoder-level VLM model selection, is repurposed as a coordinate-free measure of structural compatibility between visual sample geometry and question-side language geometry (last-token penultimate-layer LLM representations), computed pre- and post-projection after median normalization.

The evaluation protocol is deliberately parameter-free: layers are ranked by a single signal (descending for VDE, ascending for GW), Top-3 candidates are selected, and a hit is counted when the oracle layer from exhaustive inference falls within that set. No learned or task-specific combination of signals is used.

VDE tracks downstream accuracy; GW does not select

On ScienceQA, $\mathrm{VDE}_{\mathrm{pre}}$ correlates strongly with exhaustive layer-wise accuracy despite using only 100 samples: Pearson $r = 0.85$, Spearman $\rho = 0.86$, $R^2 = 0.73$ for LLaVA-Video, and $r = 0.63$, $\rho = 0.74$, $R^2 = 0.40$ for Video-LLaVA. For LLaVA-Video, both VDE variants peak at Layer 27 — the true best layer, one above the default. This correlation implies that a practitioner can rank visual layers from raw forward passes alone, at negligible cost relative to full benchmark sweeps.

The projector comparison yields a clear asymmetry. Post-projector entropy profiles are less regular than their pre-projector counterparts, yet their high-value regions remain aligned with high-performing layers: multimodal projection reshapes visual geometry without erasing the performance-relevant depth trend, leaving $\mathrm{VDE}_{\mathrm{pre}}$0 as the stronger signal. GW behaves differently: pre-projector GW decreases monotonically with depth, suggesting increasing visual-language structural compatibility, but post-projector GW flattens to uniformly low values across all layers, destroying the contrast needed to localize strong layers. The authors accordingly position GW as an alignment diagnostic rather than a selector — a negative result that usefully bounds the transferability of GW from encoder-level model selection to layer-level profiling.

These patterns generalize across four additional image benchmarks and both video tasks. For Video-LLaVA, $\mathrm{VDE}_{\mathrm{pre}}$1 forms a broad plateau around Layers 18–23 followed by a final-layer decline that mirrors the accuracy drop; for LLaVA-Video, entropy rises progressively toward the deepest layers, matching the stable high-performance region at Layers 22–27. On HD-EPIC Action Recognition, even a reduced-budget profile of 50 examples with a single seed reproduces the coarse structure, with the best layer shifting to Layer 25 for LLaVA-Video while remaining inside the high-VDE region.

Candidate retrieval results

Model	Signal	Oracle covered in Top-3	Notable failure mode
LLaVA-Video (SigLIP)	$\mathrm{VDE}_{\mathrm{pre}}$2	All 7 tasks	None reported
LLaVA-Video (SigLIP)	$\mathrm{VDE}_{\mathrm{pre}}$3	5 of 7 tasks	Misses oracle on CV-Bench 3D, HD-EPIC Action
Video-LLaVA (CLIP)	$\mathrm{VDE}_{\mathrm{pre}}$4	4 of 7 tasks	Broad upper-layer plateau blurs ranking
Video-LLaVA (CLIP)	$\mathrm{VDE}_{\mathrm{pre}}$5	6 of 7 tasks	Misses oracle on HD-EPIC Gaze

For the SigLIP-based model, $\mathrm{VDE}_{\mathrm{pre}}$6's Top-3 covers the oracle best layer on every task, and the Top-1 selection matches oracle accuracy exactly on all seven tasks (e.g., 70.20% on CV-Bench 2D, 61.00% on HD-EPIC Action). For the CLIP-based model, strict retrieval is less reliable — on HD-EPIC Gaze, the Top-1 $\mathrm{VDE}_{\mathrm{pre}}$7 pick reaches only 26.50% against an oracle 37.00%, because adjacent CLIP layers have nearly identical entropy and comparable accuracy. In these cases VDE still filters out clearly weak shallow layers and concentrates candidates in the correct upper-layer region, supporting its use as a region-level policy followed by limited downstream verification rather than a guaranteed exact selector.

Limitations and open questions

The paper is explicit about three constraints. First, validation covers only two VLMs with two vision backbones; generalization to other model families, larger towers, and alternative projector designs remains untested. Second, VDE is a profiling signal, not a guaranteed selector — its instability as a strict Top-1/Top-3 ranker on the CLIP-based model means it should be paired with small verification budgets. Third, GW is diagnostic only; no dependable standalone selection rule follows from it. An additional open question concerns why the projector preserves the entropy–accuracy trend despite reshaping geometry: the paper observes the preservation empirically but offers no mechanistic account.

Conclusion

This work establishes that the penultimate-layer convention in LLaVA-style VLMs is a fragile default whose optimality varies systematically with task and backbone, and that Visual Dataset Entropy computed from roughly 100 unlabeled samples provides a training-free, interpretable signal that localizes high-performing visual layers before any answer generation. Pre-projector entropy emerges as the strongest practical signal — exact for SigLIP-based models, region-level for CLIP-based ones — while post-projector profiles show the multimodal projector reshapes but does not erase performance-relevant geometry, and GW distance serves as an alignment diagnostic rather than a selector. The practical consequence is that visual-layer search can be narrowed from all $\mathrm{VDE}_{\mathrm{pre}}$8 layers to a handful of candidates at negligible cost, though broader validation across architectures remains necessary.