---
title: 'VFL-LoRA: Selective Multimodal Fine-Tuning'
url: https://www.emergentmind.com/topics/vfl-lora
type: topic
---

# VFL-LoRA: Selective Multimodal Fine-Tuning

VFL-LoRA is a parameter-efficient fine-tuning method for multimodal large language models in which LoRA adapters are trained only on decoder layers identified as **Vision Function Layers** (VFLs), rather than being applied uniformly across all adaptable layers. In the formulation introduced in "Vision Function Layer in Multimodal LLMs" [2509.24791], VFLs are narrow decoder-layer blocks where specific visual functions such as recognition, counting, grounding, or OCR are predominantly executed during multimodal decoding. The method is motivated by an interpretability claim about multimodal decoders: visual computation is not uniformly distributed across depth, but is localized to small sets of layers, typically two or three at peak. VFL-LoRA therefore changes the **layer-selection policy** for LoRA, not the low-rank parameterization itself, and is presented as a way to improve task-aligned specialization while reducing out-of-domain function forgetting [2509.24791].

## 1. Conceptual definition and scope

The defining premise of VFL-LoRA is that downstream multimodal tasks depend disproportionately on a restricted subset of decoder layers associated with particular visual functions. Instead of applying LoRA across all adaptable layers, VFL-LoRA restricts adaptation to the set of layers empirically implicated in the target function. The paper states this as: “Vision-Function LoRA (VFL), a PEFT method that selectively applies LoRA updates only to layers critical for the target visual function(s), thereby enhancing desired skills while preserving overall model capability” [2509.24791].

Under standard LoRA, a weight matrix \(W\) is updated as
\[
W' = W + \Delta W, \qquad \Delta W = BA,
\]
with low-rank factors \(A\) and \(B\). VFL-LoRA preserves this parameterization but changes where it is instantiated:
\[
W'_{\ell} =
\begin{cases}
W_{\ell} + \frac{\alpha}{r} B_{\ell}A_{\ell}, & \ell \in \mathcal{L}_{\mathrm{VFL}} \\
W_{\ell}, & \ell \notin \mathcal{L}_{\mathrm{VFL}} .
\end{cases}
\]
Here \(\mathcal{L}_{\mathrm{VFL}}\) denotes the selected Vision Function Layers. The method is therefore not a new adapter family, optimizer, or loss; it is a function-aware restriction on the support of LoRA updates [2509.24791].

A recurrent source of confusion is the acronym **VFL**. In this literature, VFL denotes **Vision Function Layer**, not vertical federated learning. Recent federated-LoRA papers such as "FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCA" [2506.01194] and "LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinement" [2411.14961] study horizontal federated learning and are explicitly not VFL-LoRA in the Vision Function Layer sense. Likewise, "On Performance of LoRa Fluid Antenna Systems" [2502.15258] concerns wireless LoRa rather than low-rank adaptation. Within the present topic, VFL-LoRA refers specifically to selective LoRA tuning in multimodal LLM decoders [2509.24791].

## 2. Vision Function Layers and their identification

The mechanistic basis of VFL-LoRA is the claim that visual functions in multimodal decoders are depth-localized. The paper studies four functions: recognition, counting, grounding, and OCR. Its central analytical tool is **Visual Token Swapping**, implemented through targeted KV-cache intervention during decoding [2509.24791].

The multimodal decoder is written as
\[
[U^{(l)}; W^{(l)}] = \Phi^{(l)}([U^{(l-1)}; W^{(l-1)}]),
\]
where \(U\) are aligned visual embeddings, \(W\) are text token embeddings, and \(\Phi^{(l)}\) is the \(l\)-th transformer layer. Decoding is
\[
P(y \mid U, W) = \prod_{t=1}^{N_{\mathrm{gen}}} P(y_t \mid y_{<t}, U, W).
\]
Visual Token Swapping replaces the visual representation at a specific layer \(k\):
\[
P_{\mathrm{swap}}(y \mid U^{(k)}, \tilde{U}^{(k)}, W)
=
\prod_{t=1}^{N_{\mathrm{gen}}}
P(y_t \mid y_{<t}, U^{(k)}, \tilde{U}^{(k)}, W).
\]
Operationally, the model is run on a target image and on a minimally different source image; at decoder layer \(k\), the target image’s vision-token cache entries are replaced with those from the source image, while the remaining state is left unchanged. If the output changes in the expected function-specific way, the layer is treated as causally relevant to that function [2509.24791].

The paired-image construction is function-specific. OCR pairs differ only in the rendered word; counting pairs, adapted from CLEVR, differ mainly in object count; grounding pairs contain the same object at different locations; recognition pairs, drawn from COCO, contrast an image containing the target object with a blank canvas. The prompt is aligned to the manipulated attribute, so the intervention is intended to be function-specific rather than a generic perturbation [2509.24791].

The paper complements swapping with **Vision Token Dropping**. Since dropping visual tokens at a single layer produced effects that were often too subtle, it uses a cumulative variant:
\[
P_{\mathrm{drop}}(y \mid U^{(<k)}, W)
=
\prod_{t=1}^{N_{\mathrm{gen}}}
P(y_t \mid y_{<t}, U^{(<k)}, W),
\]
where visual tokens are propagated only up to layer \(k\), and all visual tokens from layer \(k\) onward are dropped. This is used on broader VQA-style benchmarks to validate the depth sensitivity suggested by swapping [2509.24791].

## 3. Layer-function organization in multimodal decoders

The main detailed analysis is reported for **Qwen2.5-VL-7B**, which has 28 decoder layers. The paper’s stable qualitative conclusion is that visual functions are ordered by depth: **recognition** is earliest, **counting** occupies middle layers, **grounding** follows, and **OCR** is latest [2509.24791].

| Function | Qwen2.5-VL-7B localization | Note |
|---|---|---|
| Recognition | strongest in layers 0–10 | somewhat distributed |
| Counting | layers 14–16; peak around layer 12 | broader non-zero range used later |
| Grounding | around layer 18 | peak change rate 100.0% |
| OCR | layers 22–24 | late-layer specialization |

The paper reports for Qwen2.5-VL-7B that counting peaks around **layer 12** with **87.4%** change rate, grounding peaks at **layer 18** with **100.0%** change rate, and OCR peaks at **layer 22** with **92.8%** change rate; recognition is strongest in **layers 0–10**, though some effect persists deeper [2509.24791]. It also states that functions usually occupy only **2–3 layers** at peak, with recognition being the most distributed exception.

The layer assignments are used in two related but distinct ways. First, the paper describes sharp peak layers or narrow peak bands, such as counting in layers **14–16** and OCR in **22–24**. Second, for fine-tuning it sometimes selects the broader set of layers with **non-zero change rate** for the relevant function. For **Qwen2.5-VL-7B**, the VFL-LoRA spatial-reasoning experiment uses **layers 10–17, 20, 21, 22, 23**, derived from count-function analysis rather than from downstream data access [2509.24791]. This distinction matters: the mechanistic interpretation is sharply localized, but the practical adapter mask may be broader.

The authors further claim that the ordering of functions is consistent across model families and scales, including **LLaVA-v1.5** and **Qwen2.5-VL**, and summarize the hierarchy as recognition first, counting next, grounding after that, and OCR last [2509.24791]. The paper interprets this as a decoder-side progression of visual abstraction.

## 4. VFL-LoRA as selective adaptation

In VFL-LoRA, the target task is first associated with one or more visual functions, and LoRA is then applied only to the decoder layers identified with those functions. For the main experiment, the target is **spatial reasoning**, trained on **SAT** (“Dynamic Spatial Aptitude Training for Multimodal Language Models”), specifically its single-image QA tasks probing spatial understanding [2509.24791].

A notable methodological feature is that layer selection is not tuned on the downstream training set. The paper states that, to evaluate generalizability and the robustness of the identified Vision Function Layers, it directly selects the layers with **non-zero change rate of count-function** from the probing analysis, “without any access to the training or test data of the downstream spatial reasoning benchmarks” [2509.24791]. For Qwen2.5-VL-7B, that yields the layer set
\[
10\text{–}17, 20, 21, 22, 23.
\]

The paper compares three regimes on **Qwen2.5-VL-3B** and **Qwen2.5-VL-7B**: standard LoRA over all adaptable layers, **Reversed-VFL** over layers excluding the count-function layer range, and **VFL-LoRA** over the selected VFL layers only [2509.24791]. It does not specify in the provided text which exact submodules receive adapters—such as \(q_{\mathrm{proj}}\), \(v_{\mathrm{proj}}\), or MLP projections—so the method should be understood as a layer-selection policy over the model’s existing “adaptable layers,” not as a submodule-specific innovation.

The parameter reduction is substantial. The paper reports approximately **309M** tunable parameters for standard LoRA and approximately **155M** for VFL-LoRA, describing this as **nearly 50% fewer** tunable parameters [2509.24791]. It also provides model-specific parameter percentages:

| Model | Standard LoRA | Reversed-VFL | VFL-LoRA |
|---|---:|---:|---:|
| Qwen2.5-VL-3B | 3.1% | 2.1% | 0.9% |
| Qwen2.5-VL-7B | 1.9% | 0.9% | 0.9% |

The paper’s conceptual claim is not merely that fewer layers suffice, but that restricting updates to function-relevant layers reduces interference with unrelated capabilities. In its own interpretation, VFL-LoRA aligns adaptation with the internal location of the target computation and thereby avoids the broader perturbation induced by full-LoRA [2509.24791].

## 5. Empirical performance and function retention

The main in-domain evaluation uses **CV-Bench** sub-tasks—**Count**, **Relation**, **Depth**, and **Distance**—while out-of-domain evaluation uses **ChartQA**, **OCRBench / OCR**, **MMMU**, and **POPE** [2509.24791]. The reported pattern is that VFL-LoRA matches or slightly exceeds full-LoRA in-domain while improving out-of-domain retention.

For **Qwen2.5-VL-3B**, the paper reports:

| Method | Params | In-domain avg | OOD avg |
|---|---:|---:|---:|
| LoRA | 3.1% | 82.7 | 71.8 |
| Reversed-VFL | 2.1% | 82.0 | 71.9 |
| VFL-LoRA | 0.9% | 83.5 | 72.9 |

For **Qwen2.5-VL-7B**, it reports:

| Method | Params | In-domain avg | OOD avg |
|---|---:|---:|---:|
| LoRA | 1.9% | 84.4 | 74.3 |
| Reversed-VFL | 0.9% | 82.7 | 74.0 |
| VFL-LoRA | 0.9% | 85.0 | 75.0 |

The finer-grained scores reinforce the same pattern. On Qwen2.5-VL-7B, VFL-LoRA improves **Count** from **70.9** to **72.6**, **Relation** from **91.3** to **91.4**, and **ChartQA** from **86.2** to **86.4**, while **OCR** rises from **50.1** to **51.7** and **MMMU** from **86.6** to **86.9**. The main exception is **Distance**, where full-LoRA scores **88.3** and VFL-LoRA scores **86.8** [2509.24791]. On Qwen2.5-VL-3B, the paper reports analogous improvements, including **72.3** versus **70.6** on Count and **47.3** versus **46.1** on OCR [2509.24791].

The paper interprets these results as evidence against a naïve “more trainable layers is always better” view. The poor performance of **Reversed-VFL**, which explicitly excludes the function-aligned layers, supports the claim that the effect is not merely sparsity or regularization, but layer-function alignment [2509.24791]. The authors’ interpretation is that VFL-LoRA helps most on **vision-heavy tasks** and less on tasks with stronger language priors, such as **Distance**.

The out-of-domain results are used to motivate the phrase **out-of-domain function forgetting**. Here the term denotes degradation on capabilities or benchmarks outside the fine-tuning target after specialization on spatial reasoning. The gains are modest but consistent: for example, OOD average rises from **74.3** to **75.0** on Qwen2.5-VL-7B and from **71.8** to **72.9** on Qwen2.5-VL-3B [2509.24791]. The evidence suggests that restricting adaptation to function-aligned VFLs preserves unrelated visual functions better than broad LoRA updates.

## 6. Relation to VFL-select, misconceptions, and limitations

VFL-LoRA is part of a broader framework in which the same layer-function map also supports **VFL-select**, a data-selection method. VFL-select assigns samples to function-associated layer partitions using the score
\[
R_k(x, y)
=
\frac{P(y \mid U^{(\le k)}(x), W)}
{P(y \mid U^{(\le k-1)}(x), W)}.
\]
The procedure computes \(R_k(x,y)\) over layers, assigns each sample to the partition of its highest-scoring layer, and then samples uniformly from partitions to build balanced training subsets [2509.24791]. On **LLaVA-665k** subset selection, the paper reports that using only **20%** of the training data, VFL-select achieves **99.5%** of full-data performance on shallow-layer tasks and **97.4%** on deep-layer tasks, summarized as about **98% of full-data performance with only 20% of the original dataset** [2509.24791]. This positions VFL-LoRA and VFL-select as parallel parameter-selection and data-selection consequences of the same mechanistic analysis.

Several misconceptions are explicitly ruled out by the evidence presented. VFL-LoRA is **not** a new low-rank factorization; it is standard LoRA with a selective layer mask. It is **not** synonymous with vertical federated learning; papers such as FedRPCA [2506.01194] and LoRA-FAIR [2411.14961] address horizontal federated LoRA aggregation and initialization issues rather than Vision Function Layers. Nor is it a universal claim that every visual capability lives in exactly two or three perfectly isolated layers: recognition remains somewhat distributed, and practical VFL-LoRA selection can use broader non-zero-change-rate layer sets rather than only the sharpest peaks [2509.24791].

The paper also leaves visible limitations. The architectural scope is limited to a small set of open MLLMs, principally **LLaVA-v1.5** and **Qwen2.5-VL**. Function boundaries are not perfectly sharp. The identified VFLs depend on controlled paired datasets and intervention methodology, so the resulting layer map may be sensitive to probing design. Token swapping manipulates layer-specific visual token representations or KV-cache entries, raising the usual possibility of off-manifold intervention artifacts. The reported gains are task-dependent and weakest on tasks with stronger language priors. Finally, the reproducibility checklist explicitly states that **no error bars or significance tests** were reported, and the provided text does not specify optimizer, learning rate, LoRA rank, \(\alpha\), dropout, target submodules, or compute setup for the VFL-LoRA experiments [2509.24791].

Within those limits, VFL-LoRA represents a mechanistically grounded form of selective multimodal fine-tuning: if specific visual functions are localized to restricted decoder depths, then updating only those layers can improve parameter efficiency and reduce collateral degradation relative to full-LoRA [2509.24791].

Source: https://www.emergentmind.com/topics/vfl-lora