VFL-select: Function-Aware Data Selection

• VFL-select is a function-aware data selection method that curates training examples based on their reliance on specific vision function layers in multimodal LLMs.
• It identifies narrow, function-specific layers through causal probing techniques like visual token swapping and token dropping to map tasks such as Recognition, Counting, Grounding, and OCR.
• Empirical results show that VFL-select achieves up to 99.5% performance using only 20% of the data, enhancing training efficiency and targeted capability improvement.

VFL-select is a function-aware data selection method introduced in “Vision Function Layer in Multimodal LLMs” (Shi et al., 29 Sep 2025). In that usage, it denotes a training-data curation rule for multimodal LLMs (MLLMs): examples are scored by how strongly correct prediction depends on narrow, function-specific decoder layer blocks called Vision Function Layers (VFLs), and data are then selected to strengthen the corresponding visual capability. A plausible implication is that the term is now polysemous across adjacent literatures, because “selection” under the VFL label also appears in vertical federated learning and broader federated systems, where the selected object may be participants, features, samples, inference instances, or client-specific model slices rather than multimodal training examples (Charles et al., 2022).

1. Vision Function Layers and the meaning of “VFL” in VFL-select

In the MLLM setting, the paper argues that visual-related functional decoding is not distributed uniformly across depth. Instead, visual functions are localized into narrow layer blocks, usually only 2–3 layers per function, and these blocks are termed Vision Function Layers. The functions explicitly studied are Recognition, Counting, Grounding, and OCR. The reported layerwise pattern is consistent across multiple MLLMs and sizes: Recognition appears earliest, then Counting, then Grounding, and OCR appears latest. The paper describes this ordering as aligned with human perception: first identify objects, then quantify them, then localize them, and finally read text (Shi et al., 29 Sep 2025).

The same study also reports that many general tasks do not require all VFLs, and that some models even perform slightly better when certain late visual layers are dropped, suggesting redundancy. This point is central to VFL-select, because the method assumes that capability-relevant supervision should be matched to the layers where the associated visual computation is concentrated. In this formulation, “select” does not refer to selecting clients or participants; it refers to selecting training examples whose predictive success depends on the VFLs aligned with the target function.

2. Identification of function-specific layers

The discovery mechanism for VFLs is Visual Token Swapping, a causal probing framework that modifies targeted KV-cache entries during decoding. Let $I$ be the image, $T$ the text prompt, $U$ the vision embeddings after the vision encoder and connector, $W$ the text embeddings, and let the MLLM have layers $\{f^{(l)}\}_{l=1}^L$. Multimodal decoding is written as

$$P(y \mid U, W) = \prod_{t=1}^{N_{\text{gen}}} P(y_t \mid y_{<t}, U, W).$$

At a specific layer $k$, the method replaces the vision token states $U^{(k)}$ from a target image with those from a source image:

$$P_{\text{swap}}(y \mid U^{(k)}, \tilde U^{(k)}, W) = \prod_t P(y_t \mid y_{<t}, U^{(k)}, \tilde U^{(k)}, W).$$

This is implemented as a targeted KV-cache modification, so only the chosen layer’s vision-related memory is altered while all other layers remain unchanged. The model therefore decodes with mixed internal visual states; if swapping at layer $k$ changes the output in a function-specific way, that layer is treated as functionally relevant (Shi et al., 29 Sep 2025).

Function-specific layers are identified through paired images that differ in only one attribute: OCR pairs use the same blank canvas with different words; Counting pairs use the same scene except that the number changes; Recognition pairs compare object present versus blank canvas; Grounding pairs keep the same object but change its location. The paper defines a change rate that measures whether the output flips when swapping tokens at layer $T$0. For Grounding, the predicted box is considered matched when $T$1. Layers with the highest change rate are taken as the VFLs for that function (Shi et al., 29 Sep 2025).

For Qwen2.5-VL-7B, the reported peaks are sharp rather than diffuse: Recognition appears in early layers around $T$2–$T$3; Counting appears around layer 12 or roughly layers 14–16; Grounding appears around layer 18; and OCR appears around layers 22–24. A complementary probe, Vision Token Dropping, removes visual tokens from layer $T$4 onward and shows that OCR tasks degrade first, then spatial reasoning, then recognition, while general knowledge tasks are often much less sensitive to late visual layers. Together, these results support a layered specialization view rather than a monolithic vision pipeline (Shi et al., 29 Sep 2025).

3. Construction of VFL-select

VFL-select converts the VFL analysis into a data curation rule. For each training example $T$5, the paper defines a layer score

$T$6

This ratio compares the correct-answer probability when including layer $T$7 versus excluding it. The operational interpretation given by the paper is straightforward: if $T$8 is high, the sample relies strongly on layer $T$9, and the sample is therefore associated with the visual function represented by that layer (Shi et al., 29 Sep 2025).

Automatic data classification proceeds in four steps. The method computes $U$0 across layers, finds the layer with the highest score, assigns the sample to that layer’s function group, and then samples from each group to build a balanced training subset. The paper stresses that this allows automatic classification by function without explicit semantic labels. It does not require a human to label a sample as OCR, Counting, or Grounding; instead, the assignment is inferred from the model’s own layer sensitivity.

The key assumption behind the method is that the data most useful for improving a capability are the data that activate the corresponding VFLs. The reported practical consequence is twofold: targeted improvement, because selected data match the function that needs strengthening, and less wasted training, because examples unrelated to the target function are less likely to dominate the update. The paper also notes that the VFL classification step can be performed with a small proxy model such as TinyLLaVA-0.5B and then transferred to a larger model, which is presented as a scalability mechanism (Shi et al., 29 Sep 2025).

4. Reported empirical behavior and relation to VFL-LoRA

The paper reports that VFL-select outperforms human expert data selection, outperforms random selection, and achieves 98% of full-data performance using only 20% of the data (Shi et al., 29 Sep 2025). On a large 20M-sample pool, it is reported to perform consistently better than both Random and Expert selection across subset sizes 150k, 250k, 350k, and 665k. On LLaVA-665k, when only 20% of the dataset is selected for fine-tuning, VFL-select reaches 99.5% of full-data performance on shallow-layer task benchmarks and 97.4% on deep-layer task benchmarks. In the paper’s framing, this is the concrete efficiency result: nearly full performance from a small fraction of the data.

The same work uses the VFL concept for parameter-efficient adaptation through VFL-LoRA. The comparison is between Full-LoRA, which applies adapters across all layers, and VFL-LoRA, which applies adapters only to the functionally relevant layers. For the spatial-reasoning setup, the relevant layers are chosen from the count-function VFL. The reported findings are that VFL-LoRA uses about one-third the tunable parameters of full-LoRA, matches or slightly improves in-domain performance, performs better out of domain, and reduces catastrophic forgetting of unrelated abilities (Shi et al., 29 Sep 2025).

These results position VFL-select as more than an interpretability artifact. In the paper’s formulation, the same layer-function map supports both selective data curation and selective fine-tuning. The article’s central claim is therefore mechanistic rather than heuristic: because visual computation is concentrated in narrow layer blocks, training should preferentially target the data and parameters associated with those blocks.

5. Related meanings of “select” in vertical federated learning and federated systems

A separate literature uses “selection” in the context of vertical federated learning (VFL), where multiple parties hold different features for the same aligned samples. In that literature, the object of selection is typically not a multimodal training example but some element of the federated pipeline itself. A concise comparison is given below.

Setting	Selected object	Basis of selection
VFL-select in MLLMs	Training examples	Layer-wise dependency on Vision Function Layers
VFLens	Features, training samples, inference instances	Visualization, clustering, and host-side confidence gating
VFL-RPS	Passive participants	Secure Spearman correlation, redundancy detection, forward selection
LESS-VFL / ICAFS	Feature or embedding components	Group-lasso stages or synthetic gated selectors
Federated Select	Client-specific model slices	Keys specifying relevant submodels

“VFLens: Co-design the Modeling Process for Efficient Vertical Federated Learning via Visualization” (Tian et al., 2022) treats selection as an interactive workflow spanning feature engineering, sample selection, and inference-time gating. It uses LR-based VFL, private set intersection, and additively homomorphic encryption plus masking for gradient exchange. For sample selection, it first projects samples with t-SNE and groups them with KMeans++, then guides within-cluster sampling through statistical homogeneity and content diversity. For inference-time selection, the host first runs its local model, then compares each prediction instance to a nearest training sample in host feature space; if the local prediction, the nearest training label, and the VFL model’s prediction for that training sample agree, the instance is treated as high-confidence and the host model is used. In the reported case study, out of 15,000 prediction records, 8,202 did not require VFL inference, a rejection rate of 54.6%, and 7,782 of those matched the VFL labels, a 94% hit rate (Tian et al., 2022).

“VFL-RPS: Relevant Participant Selection in Vertical Federated Learning” (Khan et al., 20 Feb 2025) addresses pre-training participant selection. It assumes one active party with labels and passive parties contributing feature subsets, computes secure Spearman correlation, detects redundancy, scores parties by relevance, and applies forward selection before federated training. The paper reports that selecting only a small subset, often 50% of parties, can match the performance of using all parties; for example, on California Housing, VFL-RPS selects top 3 of 5 passive parties and reports MSE $U$1 and $U$2, close to the all-party result, while on Credit Card Default its selection time is about 17–19 s versus over 300–400 s for VFLMG (Khan et al., 20 Feb 2025).

“LESS-VFL: Communication-Efficient Feature Selection for Vertical Federated Learning” (Castiglia et al., 2023) selects features through a three-stage pipeline: short VFL pre-training, server-side group-lasso over fixed embeddings to identify significant embedding components, and local feature selection at each party with no further communication. The paper provides both a theoretical spurious-feature removal guarantee and empirical communication savings; on Phishing, reported communication cost is 3.99 MB for LESS-VFL versus 95.22 MB for group lasso, about 20× lower (Castiglia et al., 2023). “ICAFS: Inter-Client-Aware Feature Selection for Vertical Federated Learning” (Jin et al., 15 Apr 2025) shifts the emphasis to inter-client interactions, using conditional feature synthesis with a federated conditional GAN, multiple gated selectors, and synthetic embeddings so that selection can be learned without sharing private gradients. It reports the best or near-best test accuracy across several real-world datasets, including 100.00% on ALLAML at $U$3 and $U$4 (Jin et al., 15 Apr 2025).

The more general primitive “Federated Select” (Charles et al., 2022) extends the selection idea from data or features to server-model slices. It defines

$U$5

which generalizes broadcast by allowing each client to receive a client-specific submodel. The paper argues that this can reduce communication and client memory substantially and potentially enable the training of models too large to fit on-device (Charles et al., 2022). A plausible implication is that “VFL-select” now sits at the intersection of at least two naming traditions: mechanistic data selection in MLLMs and selection primitives or selection policies in federated systems.

6. Significance, misconceptions, and unresolved issues

One recurring misconception is to treat VFL-select as a generic heuristic sampler. In the MLLM paper, it is explicitly grounded in a mechanistic criterion: examples are selected because their correct prediction depends strongly on function-specific layer blocks. Another misconception is that every visual task must engage all discovered VFLs. The paper states the opposite: many general tasks do not require all VFLs, and some late visual layers may even be redundant for certain models (Shi et al., 29 Sep 2025).

A second source of confusion is terminological. In multimodal LLM research, VFL denotes Vision Function Layers. In vertical federated learning, VFL denotes Vertical Federated Learning. The two usages are unrelated in mechanism and objective. This suggests that disambiguation by domain is necessary when the term appears without expansion. In the federated literature, selection may concern robust collaborative inference under device and communication faults, as in “Robust Collaborative Inference with Vertically Split Data Over Dynamic Device Environments” (Ganguli et al., 2023), where VFL is treated as a brittle baseline and robustness improves through server replication, gossip averaging, and training with simulated faults; at 30% fault rate on StarCraftMNIST with 16 devices, reported average accuracy rises from 0.430 for VFL to 0.717 for MVFL-G4 (Ganguli et al., 2023).

The unresolved issues also differ by domain. For the MLLM formulation, the central open question is how far function-aware selection generalizes across models, tasks, and data regimes beyond those reported in the paper. In the vertical federated learning literature, the open issues are often systems and privacy constraints: VFLens notes that inference-time sampling depends heavily on user experience and domain knowledge (Tian et al., 2022); VFL-RPS is designed mainly for numerical tabular data and may require tuning thresholds such as $U$6, $U$7, and $U$8 (Khan et al., 20 Feb 2025); ICAFS argues that synthetic data reduce information leakage but does not provide a formal privacy guarantee in the main method (Jin et al., 15 Apr 2025); and the robust collaborative inference framework with MAGS explicitly notes the absence of communication latency modeling and more realistic streaming or asynchronous settings (Ganguli et al., 2023).

Taken narrowly, VFL-select denotes the layer-aware data selection method derived from Vision Function Layers in MLLMs. Taken broadly, it names a family resemblance among methods that convert internal structure, participant relevance, feature interaction, or systems constraints into a selection rule. The common thread is not a single algorithmic template but a shared methodological stance: selection should be aligned with the structure that actually governs prediction.