Vision Function Layers in Multimodal Models

• Vision Function Layers (VFLs) are narrow decoder blocks in multimodal models that localize specific visual functions such as recognition, counting, grounding, and OCR.
• They reveal a clear layerwise hierarchy where distinct visual tasks are isolated through techniques like visual token swapping and dropping.
• Applications of VFL insights include parameter-efficient fine-tuning, targeted data selection, and effective model pruning to enhance multimodal performance.

Searching arXiv for papers on Vision Function Layers and closely related layer-localization in multimodal models. Vision Function Layers (VFLs) are decoder-layer substructures in multimodal LLMs (MLLMs) and large vision-LLMs (LVLMs) in which specific visual capabilities become causally concentrated rather than remaining uniformly distributed across depth. In the most explicit formulation, a VFL is a narrow layer block, typically 2–3 decoder layers, specialized for a particular visual function such as recognition, counting, grounding, or OCR (Shi et al., 29 Sep 2025). Closely related work broadens the idea from contiguous function-specific blocks to a distributed “visual region” inside the LLM backbone, arguing that visual capability is layer-localized but distributed, with a sparse and uniform subset of layers carrying most multimodal adaptation (Wang et al., 2024). A further adjacent line of work shows that some layers are not merely less useful but actively task-interfering, so layerwise visual functionality must be understood together with layerwise liability (Liu et al., 1 Feb 2026).

1. Definition and conceptual scope

The core claim behind VFLs is that visual-related functionality is not spread uniformly across decoder layers. Instead, each core visual ability tends to be concentrated in a small, contiguous block of layers, and those blocks can be causally identified by targeted intervention during decoding (Shi et al., 29 Sep 2025). In this formulation, a “visual function” is a distinct perceptual capability essential for solving a visual task, including object recognition, counting, text reading, and spatial grounding.

A related but not identical formulation appears in work on LVLM training efficiency. There, the relevant structure is a layer-wise “visual region” inside the LLM backbone: certain layers are especially important for visual alignment and visual reasoning, but they are not contiguous. Rather, they are sparsely and uniformly distributed through the network (Wang et al., 2024). This difference in operational definition is important. One literature isolates narrow function-specific blocks during inference-time probing, whereas another identifies a distributed subset of layers that is sufficient for multimodal fine-tuning.

The term therefore denotes a family of layer-localization hypotheses rather than a single canonical object. A plausible implication is that “VFL” is best treated as a mesoscopic concept: finer than whole-model modality specialization, but broader than individual neuron attribution.

2. Reported functional stratification across decoder depth

The most direct VFL analysis studies four visual functions—recognition, counting, grounding, and OCR—and reports that they occupy different depth regions in the decoder, with a consistent ordering across model families and scales (Shi et al., 29 Sep 2025). The repeated pattern is:

1. Recognition occurs earliest.
2. Counting appears in middle layers.
3. Grounding appears later than counting.
4. OCR appears latest.

For the 28-layer Qwen2.5-VL-7B example, recognition shows high sensitivity in early layers, roughly 0–10; counting peaks around layer 12, with functional concentration around layers 14–16 also reported; grounding peaks around layer 18; and OCR is especially concentrated at layers 22–24 (Shi et al., 29 Sep 2025). Recognition is described as a partial exception because it is strongest early but remains more distributed than the other three functions.

Function	Reported depth pattern	Diagnostic change criterion
Recognition	Earliest; roughly 0–10 in Qwen2.5-VL-7B	Model switches to “No” when the object is removed
Counting	Middle layers; peak around 12, concentration around 14–16	Predicted number changes
Grounding	Later than counting; peak around 18	Predicted box IoU with the swapped ground truth exceeds 0.5
OCR	Latest; especially concentrated at 22–24	Output text changes

The ordering is reported across LLaVA-v1.5 and Qwen2.5-VL, and across model sizes ranging from 3B to 70B (Shi et al., 29 Sep 2025). The paper interprets this as progressive abstraction and explicitly states that the ordering is well-aligned with human behavior: recognition first, then counting, then grounding, then OCR.

This hierarchy should not be confused with a claim that all visual processing is late-stage. On the contrary, the evidence places coarse object identity early, numerical and spatial interpretation in middle depth, and specialized symbolic extraction deepest.

3. Identification methods and causal diagnostics

The principal mechanistic tool for identifying VFLs is Visual Token Swapping. At a chosen decoder layer $k$, the method replaces the vision-token KV-cache entries corresponding to one image with those from another image, while keeping the rest of the model unchanged (Shi et al., 29 Sep 2025). The layerwise multimodal computation is written as

$$[U^{(l)}; W^{(l)}] = \Phi^{(l)}([U^{(l-1)}; W^{(l-1)}]).$$

Here, $I$ is the input image, $V$ are vision encoder outputs, $U$ are projected vision embeddings in the language space, $W$ are text embeddings, and $\Phi^{(l)}$ is the $l$-th transformer layer (Shi et al., 29 Sep 2025). The intervention asks whether replacing the visual information at layer $k$ changes the output in a function-specific way. If it does, that layer contains the relevant information for the function being probed.

When swapping is not feasible, the same work uses Vision Token Dropping, in which visual tokens are propagated only up to a given layer and then removed thereafter (Shi et al., 29 Sep 2025). The point is not merely to delete one layer’s visual tokens, but progressively to drop all vision tokens from a given layer onward so that the impact becomes easier to observe.

The diagnostic datasets are paired so that each image pair differs only in one attribute: OCR text, number of objects, object presence or identity, or object location (Shi et al., 29 Sep 2025). Layer involvement is quantified by a “Results Change Rate”: higher change rate means stronger causal involvement in the probed function, while low change rate elsewhere indicates negligible contribution for that function.

For data valuation, the same framework defines a layerwise utility score

$$R_k(x,y)= \frac{P(y \mid U(x)(\le k), W)}{P(y \mid U(x)(\le k-1), W)}.$$

If keeping layer $$[U^{(l)}; W^{(l)}] = \Phi^{(l)}([U^{(l-1)}; W^{(l-1)}]).$$0’s vision information boosts the probability of the correct answer substantially, the sample is treated as strongly associated with that layer’s function (Shi et al., 29 Sep 2025). This turns layer-localized causal evidence into a data-selection signal.

4. Distributed visual regions and task-interfering layers

A broader layer-selection view appears in work on the “visual region” of LVLM backbones. Instead of contiguous 2–3-layer blocks for individual functions, this line argues that visual ability is concentrated in a sparse, uniformly distributed subset of LLM layers (Wang et al., 2024). In Bunny-Llama-3-8B-V, the best-performing 8-layer configuration among 32 layers is $$[U^{(l)}; W^{(l)}] = \Phi^{(l)}([U^{(l-1)}; W^{(l-1)}]).$$1. Updating these 8 layers, i.e. 25% of the backbone, yields 62.88% average performance versus 63.51% for tuning all 32 layers, reported as 99.00% relative performance (Wang et al., 2024). Cross-model validation reports 58.14% versus 58.71% on LLaVA-1.5-7B, and 59.88% versus 61.27% on LLaVA-1.5-13B when 10 tuned layers are used, with a 9-layer setting also reported at 60.37% (Wang et al., 2024).

This distributed-region result directly contradicts the heuristic that visual adaptation is best confined to the top layers. The same paper states that consecutive blocks underperform, top-only tuning is worst among heuristic layouts, and sparse-uniform tuning is best (Wang et al., 2024). It also reports that selective tuning can better preserve language ability: for Bunny-LLaMA3-8B-V, full multimodal tuning gives MMLU 60.27% and BIG-bench-Hard 30.93%, whereas partial training on 8 layers gives MMLU 63.36% and BIG-bench-Hard 31.50% (Wang et al., 2024).

An adjacent line of work complicates the picture further by showing that some layers hinder downstream tasks. In pretrained VLMs, intervening on a single layer by zeroing the self-attention parameters can improve performance, revealing Task-Interfering Layers (Liu et al., 1 Feb 2026). The corresponding Task-Layer Interaction Vector for task $$[U^{(l)}; W^{(l)}] = \Phi^{(l)}([U^{(l-1)}; W^{(l-1)}]).$$2 is

$$[U^{(l)}; W^{(l)}] = \Phi^{(l)}([U^{(l-1)}; W^{(l-1)}]).$$3

with

$$[U^{(l)}; W^{(l)}] = \Phi^{(l)}([U^{(l-1)}; W^{(l-1)}]).$$4

Positive values indicate that a layer is interfering because removing it improves accuracy; negative values indicate that the layer is helpful (Liu et al., 1 Feb 2026). On LLaVA-Next-LLaMA3-8B evaluated on MMMU, 54.1% of tasks showed $$[U^{(l)}; W^{(l)}] = \Phi^{(l)}([U^{(l-1)}; W^{(l-1)}]).$$5 improvement after zeroing a single layer; on Qwen2-VL, the fraction rises to 75.6% (Liu et al., 1 Feb 2026). TaLo, a training-free test-time method that selects and bypasses the most interfering layer, reports gains up to 10.4% for LLaVA, 16.6% for Qwen-VL, and 10.0% for InternVL (Liu et al., 1 Feb 2026).

Taken together, these findings suggest that layerwise visual structure has at least two aspects: capability concentration and capability interference. VFLs are therefore not merely “important layers”; they are part of a structured functional organization in which some layers are specialized, some are redundant for a given task, and some can be liabilities.

5. Adaptation, fine-tuning, data curation, and pruning

The VFL hypothesis has been used directly for parameter-efficient fine-tuning. VFL-LoRA applies LoRA only to layers identified as critical for the target visual function rather than across all adaptable layers (Shi et al., 29 Sep 2025). For spatial reasoning, the selected layers on Qwen2.5-VL-7B are 10–17, 20, 21, 22, and 23, chosen because the count-related VFL overlaps with spatial reasoning needs. In the reported setup, VFL-LoRA uses about 155M tunable parameters versus 309M for standard LoRA, with CV-Bench spatial reasoning average 85.0% versus 84.4%, and out-of-domain average 75.0% versus 74.3% (Shi et al., 29 Sep 2025). The stated interpretation is that updating only function-relevant layers reduces waste and avoids catastrophic forgetting.

The same paper proposes VFL-select for data selection. It computes the layerwise functional value $$[U^{(l)}; W^{(l)}] = \Phi^{(l)}([U^{(l-1)}; W^{(l-1)}]).$$6, assigns each sample to the layer where the score is highest, and then samples uniformly from each layer-based partition (Shi et al., 29 Sep 2025). Using a 20 million vision instruction sample pool and subsets of 150k, 250k, 350k, and 665k, VFL-select is reported to outperform Random and Expert selection. With only 20% of the LLaVA-665k dataset, it achieves 99.5% of full-data performance on shallow-layer tasks and 97.4% on deep-layer tasks; the abstract summarizes this as 98% of full-data performance with 20% of the data (Shi et al., 29 Sep 2025). Because VFL patterns are described as consistent across MLLMs, the expensive classification stage can be run on a smaller proxy model such as TinyLLaVA-0.5B and transferred to a larger target model.

The distributed visual-region framework also motivates pruning. After targeted training on the visual region, layers outside that region can be pruned using angular distance-based importance, specifically by removing the 0–4 layers with the lowest angular distance outside the visual region (Wang et al., 2024). On LLaVA-1.5-7B, this visual-region-based pruning paradigm is reported to maintain higher performance, especially when pruning 3–4 layers, than pruning the fully trained model directly (Wang et al., 2024).

These applications shift VFLs from a descriptive claim about internal organization to an operational design principle for training, data curation, and inference-time compression.

6. Misconceptions, ambiguity, and limits of generalization

A common misconception is that VFLs imply a single contiguous late-layer “visual block.” The evidence does not support that simplification. One line of work identifies narrow, contiguous, function-specific decoder blocks for recognition, counting, grounding, and OCR (Shi et al., 29 Sep 2025). Another finds that the layers most useful for multimodal fine-tuning are sparsely and uniformly distributed across depth rather than contiguous (Wang et al., 2024). These claims are not mutually exclusive; they operate at different granularities.

A second misconception is that all pretrained layers are beneficial if left untouched. Layer-intervention results explicitly reject this, showing that bypassing one self-attention layer can improve task performance and that similar tasks have similar task-layer interaction vectors (Liu et al., 1 Feb 2026). A plausible implication is that VFL analysis should include both supporting and interfering structures.

There is also a real generalization tension. The function ordering Recognition $$[U^{(l)}; W^{(l)}] = \Phi^{(l)}([U^{(l-1)}; W^{(l-1)}]).$$7 Counting $$[U^{(l)}; W^{(l)}] = \Phi^{(l)}([U^{(l-1)}; W^{(l-1)}]).$$8 Grounding $$[U^{(l)}; W^{(l)}] = \Phi^{(l)}([U^{(l-1)}; W^{(l-1)}]).$$9 OCR is reported across architectures, successive model versions, and scales from 3B to 70B (Shi et al., 29 Sep 2025). By contrast, task-interfering layer indices are reported to differ across models such as LLaVA, Qwen, and InternVL (Liu et al., 1 Feb 2026). This suggests that coarse functional hierarchy may transfer across model families, while exact actionable layer indices remain model-specific.

Finally, the acronym “VFL” is overloaded. In another well-established literature it denotes Vertical Federated Learning, including Multi-VFL for $I$0 data owners and $I$1 label owners (Mugunthan et al., 2021) and BlindFL, which introduces federated source layers for secure VFL training and inference (Fu et al., 2022). In yet another nearby usage, vision “function layers” can refer to optimization-based layers such as total variation minimization inserted into deep networks (Yeh et al., 2022). For precision, the MLLM literature’s Vision Function Layers should therefore be distinguished from both Vertical Federated Learning and optimization-layer design.

Overall, the current literature supports a layered functional view of multimodal vision processing, but not a single universal ontology. What is established is the existence of non-uniform, causally meaningful layerwise organization; what remains open is how to reconcile contiguous function blocks, distributed visual regions, and task-interfering layers within one model-agnostic theory.