VFL-LoRA: Selective Multimodal Fine-Tuning

• The paper introduces VFL-LoRA as a method that restricts LoRA updates to key Vision Function Layers, achieving nearly 50% fewer tunable parameters while maintaining performance.
• VFL-LoRA is defined as a parameter-efficient approach that specifically targets visual functions—recognition, counting, grounding, and OCR—in multimodal decoders.
• Empirical results on Qwen2.5-VL models demonstrate improved in-domain and out-of-domain performance, preserving non-target capabilities by aligning updates with visual functions.

VFL-LoRA is a parameter-efficient fine-tuning method for multimodal LLMs in which LoRA adapters are trained only on decoder layers identified as Vision Function Layers (VFLs), rather than being applied uniformly across all adaptable layers. In the formulation introduced in "Vision Function Layer in Multimodal LLMs" (Shi et al., 29 Sep 2025), VFLs are narrow decoder-layer blocks where specific visual functions such as recognition, counting, grounding, or OCR are predominantly executed during multimodal decoding. The method is motivated by an interpretability claim about multimodal decoders: visual computation is not uniformly distributed across depth, but is localized to small sets of layers, typically two or three at peak. VFL-LoRA therefore changes the layer-selection policy for LoRA, not the low-rank parameterization itself, and is presented as a way to improve task-aligned specialization while reducing out-of-domain function forgetting (Shi et al., 29 Sep 2025).

1. Conceptual definition and scope

The defining premise of VFL-LoRA is that downstream multimodal tasks depend disproportionately on a restricted subset of decoder layers associated with particular visual functions. Instead of applying LoRA across all adaptable layers, VFL-LoRA restricts adaptation to the set of layers empirically implicated in the target function. The paper states this as: “Vision-Function LoRA (VFL), a PEFT method that selectively applies LoRA updates only to layers critical for the target visual function(s), thereby enhancing desired skills while preserving overall model capability” (Shi et al., 29 Sep 2025).

Under standard LoRA, a weight matrix $W$ is updated as

$W' = W + \Delta W, \qquad \Delta W = BA,$

with low-rank factors $A$ and $B$. VFL-LoRA preserves this parameterization but changes where it is instantiated: $$W'_{\ell} = \begin{cases} W_{\ell} + \frac{\alpha}{r} B_{\ell}A_{\ell}, & \ell \in \mathcal{L}_{\mathrm{VFL}} \ W_{\ell}, & \ell \notin \mathcal{L}_{\mathrm{VFL}} . \end{cases}$$ Here $\mathcal{L}_{\mathrm{VFL}}$ denotes the selected Vision Function Layers. The method is therefore not a new adapter family, optimizer, or loss; it is a function-aware restriction on the support of LoRA updates (Shi et al., 29 Sep 2025).

A recurrent source of confusion is the acronym VFL. In this literature, VFL denotes Vision Function Layer, not vertical federated learning. Recent federated-LoRA papers such as "FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCA" (Jhunjhunwala et al., 1 Jun 2025) and "LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinement" (Bian et al., 2024) study horizontal federated learning and are explicitly not VFL-LoRA in the Vision Function Layer sense. Likewise, "On Performance of LoRa Fluid Antenna Systems" (Mu et al., 21 Feb 2025) concerns wireless LoRa rather than low-rank adaptation. Within the present topic, VFL-LoRA refers specifically to selective LoRA tuning in multimodal LLM decoders (Shi et al., 29 Sep 2025).

2. Vision Function Layers and their identification

The mechanistic basis of VFL-LoRA is the claim that visual functions in multimodal decoders are depth-localized. The paper studies four functions: recognition, counting, grounding, and OCR. Its central analytical tool is Visual Token Swapping, implemented through targeted KV-cache intervention during decoding (Shi et al., 29 Sep 2025).

The multimodal decoder is written as

$$[U^{(l)}; W^{(l)}] = \Phi^{(l)}([U^{(l-1)}; W^{(l-1)}]),$$

where $U$ are aligned visual embeddings, $W$ are text token embeddings, and $\Phi^{(l)}$ is the $W' = W + \Delta W, \qquad \Delta W = BA,$0-th transformer layer. Decoding is

$W' = W + \Delta W, \qquad \Delta W = BA,$1

Visual Token Swapping replaces the visual representation at a specific layer $W' = W + \Delta W, \qquad \Delta W = BA,$2: $W' = W + \Delta W, \qquad \Delta W = BA,$3 Operationally, the model is run on a target image and on a minimally different source image; at decoder layer $W' = W + \Delta W, \qquad \Delta W = BA,$4, the target image’s vision-token cache entries are replaced with those from the source image, while the remaining state is left unchanged. If the output changes in the expected function-specific way, the layer is treated as causally relevant to that function (Shi et al., 29 Sep 2025).

The paired-image construction is function-specific. OCR pairs differ only in the rendered word; counting pairs, adapted from CLEVR, differ mainly in object count; grounding pairs contain the same object at different locations; recognition pairs, drawn from COCO, contrast an image containing the target object with a blank canvas. The prompt is aligned to the manipulated attribute, so the intervention is intended to be function-specific rather than a generic perturbation (Shi et al., 29 Sep 2025).

The paper complements swapping with Vision Token Dropping. Since dropping visual tokens at a single layer produced effects that were often too subtle, it uses a cumulative variant: $W' = W + \Delta W, \qquad \Delta W = BA,$5 where visual tokens are propagated only up to layer $W' = W + \Delta W, \qquad \Delta W = BA,$6, and all visual tokens from layer $W' = W + \Delta W, \qquad \Delta W = BA,$7 onward are dropped. This is used on broader VQA-style benchmarks to validate the depth sensitivity suggested by swapping (Shi et al., 29 Sep 2025).

3. Layer-function organization in multimodal decoders

The main detailed analysis is reported for Qwen2.5-VL-7B, which has 28 decoder layers. The paper’s stable qualitative conclusion is that visual functions are ordered by depth: recognition is earliest, counting occupies middle layers, grounding follows, and OCR is latest (Shi et al., 29 Sep 2025).

Function	Qwen2.5-VL-7B localization	Note
Recognition	strongest in layers 0–10	somewhat distributed
Counting	layers 14–16; peak around layer 12	broader non-zero range used later
Grounding	around layer 18	peak change rate 100.0%
OCR	layers 22–24	late-layer specialization

The paper reports for Qwen2.5-VL-7B that counting peaks around layer 12 with 87.4% change rate, grounding peaks at layer 18 with 100.0% change rate, and OCR peaks at layer 22 with 92.8% change rate; recognition is strongest in layers 0–10, though some effect persists deeper (Shi et al., 29 Sep 2025). It also states that functions usually occupy only 2–3 layers at peak, with recognition being the most distributed exception.

The layer assignments are used in two related but distinct ways. First, the paper describes sharp peak layers or narrow peak bands, such as counting in layers 14–16 and OCR in 22–24. Second, for fine-tuning it sometimes selects the broader set of layers with non-zero change rate for the relevant function. For Qwen2.5-VL-7B, the VFL-LoRA spatial-reasoning experiment uses layers 10–17, 20, 21, 22, 23, derived from count-function analysis rather than from downstream data access (Shi et al., 29 Sep 2025). This distinction matters: the mechanistic interpretation is sharply localized, but the practical adapter mask may be broader.

The authors further claim that the ordering of functions is consistent across model families and scales, including LLaVA-v1.5 and Qwen2.5-VL, and summarize the hierarchy as recognition first, counting next, grounding after that, and OCR last (Shi et al., 29 Sep 2025). The paper interprets this as a decoder-side progression of visual abstraction.

4. VFL-LoRA as selective adaptation

In VFL-LoRA, the target task is first associated with one or more visual functions, and LoRA is then applied only to the decoder layers identified with those functions. For the main experiment, the target is spatial reasoning, trained on SAT (“Dynamic Spatial Aptitude Training for Multimodal LLMs”), specifically its single-image QA tasks probing spatial understanding (Shi et al., 29 Sep 2025).

A notable methodological feature is that layer selection is not tuned on the downstream training set. The paper states that, to evaluate generalizability and the robustness of the identified Vision Function Layers, it directly selects the layers with non-zero change rate of count-function from the probing analysis, “without any access to the training or test data of the downstream spatial reasoning benchmarks” (Shi et al., 29 Sep 2025). For Qwen2.5-VL-7B, that yields the layer set

$W' = W + \Delta W, \qquad \Delta W = BA,$8

The paper compares three regimes on Qwen2.5-VL-3B and Qwen2.5-VL-7B: standard LoRA over all adaptable layers, Reversed-VFL over layers excluding the count-function layer range, and VFL-LoRA over the selected VFL layers only (Shi et al., 29 Sep 2025). It does not specify in the provided text which exact submodules receive adapters—such as $W' = W + \Delta W, \qquad \Delta W = BA,$9, $A$0, or MLP projections—so the method should be understood as a layer-selection policy over the model’s existing “adaptable layers,” not as a submodule-specific innovation.

The parameter reduction is substantial. The paper reports approximately 309M tunable parameters for standard LoRA and approximately 155M for VFL-LoRA, describing this as nearly 50% fewer tunable parameters (Shi et al., 29 Sep 2025). It also provides model-specific parameter percentages:

Model	Standard LoRA	Reversed-VFL	VFL-LoRA
Qwen2.5-VL-3B	3.1%	2.1%	0.9%
Qwen2.5-VL-7B	1.9%	0.9%	0.9%

The paper’s conceptual claim is not merely that fewer layers suffice, but that restricting updates to function-relevant layers reduces interference with unrelated capabilities. In its own interpretation, VFL-LoRA aligns adaptation with the internal location of the target computation and thereby avoids the broader perturbation induced by full-LoRA (Shi et al., 29 Sep 2025).

5. Empirical performance and function retention

The main in-domain evaluation uses CV-Bench sub-tasks—Count, Relation, Depth, and Distance—while out-of-domain evaluation uses ChartQA, OCRBench / OCR, MMMU, and POPE (Shi et al., 29 Sep 2025). The reported pattern is that VFL-LoRA matches or slightly exceeds full-LoRA in-domain while improving out-of-domain retention.

For Qwen2.5-VL-3B, the paper reports:

Method	Params	In-domain avg	OOD avg
LoRA	3.1%	82.7	71.8
Reversed-VFL	2.1%	82.0	71.9
VFL-LoRA	0.9%	83.5	72.9

For Qwen2.5-VL-7B, it reports:

Method	Params	In-domain avg	OOD avg
LoRA	1.9%	84.4	74.3
Reversed-VFL	0.9%	82.7	74.0
VFL-LoRA	0.9%	85.0	75.0

The finer-grained scores reinforce the same pattern. On Qwen2.5-VL-7B, VFL-LoRA improves Count from 70.9 to 72.6, Relation from 91.3 to 91.4, and ChartQA from 86.2 to 86.4, while OCR rises from 50.1 to 51.7 and MMMU from 86.6 to 86.9. The main exception is Distance, where full-LoRA scores 88.3 and VFL-LoRA scores 86.8 (Shi et al., 29 Sep 2025). On Qwen2.5-VL-3B, the paper reports analogous improvements, including 72.3 versus 70.6 on Count and 47.3 versus 46.1 on OCR (Shi et al., 29 Sep 2025).

The paper interprets these results as evidence against a naïve “more trainable layers is always better” view. The poor performance of Reversed-VFL, which explicitly excludes the function-aligned layers, supports the claim that the effect is not merely sparsity or regularization, but layer-function alignment (Shi et al., 29 Sep 2025). The authors’ interpretation is that VFL-LoRA helps most on vision-heavy tasks and less on tasks with stronger language priors, such as Distance.

The out-of-domain results are used to motivate the phrase out-of-domain function forgetting. Here the term denotes degradation on capabilities or benchmarks outside the fine-tuning target after specialization on spatial reasoning. The gains are modest but consistent: for example, OOD average rises from 74.3 to 75.0 on Qwen2.5-VL-7B and from 71.8 to 72.9 on Qwen2.5-VL-3B (Shi et al., 29 Sep 2025). The evidence suggests that restricting adaptation to function-aligned VFLs preserves unrelated visual functions better than broad LoRA updates.

6. Relation to VFL-select, misconceptions, and limitations

VFL-LoRA is part of a broader framework in which the same layer-function map also supports VFL-select, a data-selection method. VFL-select assigns samples to function-associated layer partitions using the score

$A$1

The procedure computes $A$2 over layers, assigns each sample to the partition of its highest-scoring layer, and then samples uniformly from partitions to build balanced training subsets (Shi et al., 29 Sep 2025). On LLaVA-665k subset selection, the paper reports that using only 20% of the training data, VFL-select achieves 99.5% of full-data performance on shallow-layer tasks and 97.4% on deep-layer tasks, summarized as about 98% of full-data performance with only 20% of the original dataset (Shi et al., 29 Sep 2025). This positions VFL-LoRA and VFL-select as parallel parameter-selection and data-selection consequences of the same mechanistic analysis.

Several misconceptions are explicitly ruled out by the evidence presented. VFL-LoRA is not a new low-rank factorization; it is standard LoRA with a selective layer mask. It is not synonymous with vertical federated learning; papers such as FedRPCA (Jhunjhunwala et al., 1 Jun 2025) and LoRA-FAIR (Bian et al., 2024) address horizontal federated LoRA aggregation and initialization issues rather than Vision Function Layers. Nor is it a universal claim that every visual capability lives in exactly two or three perfectly isolated layers: recognition remains somewhat distributed, and practical VFL-LoRA selection can use broader non-zero-change-rate layer sets rather than only the sharpest peaks (Shi et al., 29 Sep 2025).

The paper also leaves visible limitations. The architectural scope is limited to a small set of open MLLMs, principally LLaVA-v1.5 and Qwen2.5-VL. Function boundaries are not perfectly sharp. The identified VFLs depend on controlled paired datasets and intervention methodology, so the resulting layer map may be sensitive to probing design. Token swapping manipulates layer-specific visual token representations or KV-cache entries, raising the usual possibility of off-manifold intervention artifacts. The reported gains are task-dependent and weakest on tasks with stronger language priors. Finally, the reproducibility checklist explicitly states that no error bars or significance tests were reported, and the provided text does not specify optimizer, learning rate, LoRA rank, $A$3, dropout, target submodules, or compute setup for the VFL-LoRA experiments (Shi et al., 29 Sep 2025).

Within those limits, VFL-LoRA represents a mechanistically grounded form of selective multimodal fine-tuning: if specific visual functions are localized to restricted decoder depths, then updating only those layers can improve parameter efficiency and reduce collateral degradation relative to full-LoRA (Shi et al., 29 Sep 2025).