--- title: 'VFL-select: Function-Aware Data Selection' url: https://www.emergentmind.com/topics/vfl-select type: topic --- # VFL-select: Function-Aware Data Selection VFL-select is a function-aware data selection method introduced in “Vision Function Layer in Multimodal LLMs” [2509.24791]. In that usage, it denotes a training-data curation rule for multimodal large language models (MLLMs): examples are scored by how strongly correct prediction depends on narrow, function-specific decoder layer blocks called Vision Function Layers (VFLs), and data are then selected to strengthen the corresponding visual capability. A plausible implication is that the term is now polysemous across adjacent literatures, because “selection” under the VFL label also appears in vertical federated learning and broader federated systems, where the selected object may be participants, features, samples, inference instances, or client-specific model slices rather than multimodal training examples [2208.09432]. ## 1. Vision Function Layers and the meaning of “VFL” in VFL-select In the MLLM setting, the paper argues that visual-related functional decoding is not distributed uniformly across depth. Instead, visual functions are localized into narrow layer blocks, usually only **2–3 layers** per function, and these blocks are termed **Vision Function Layers**. The functions explicitly studied are **Recognition**, **Counting**, **Grounding**, and **OCR**. The reported layerwise pattern is consistent across multiple MLLMs and sizes: **Recognition appears earliest**, then **Counting**, then **Grounding**, and **OCR** appears latest. The paper describes this ordering as aligned with human perception: first identify objects, then quantify them, then localize them, and finally read text [2509.24791]. The same study also reports that many general tasks do **not** require all VFLs, and that some models even perform slightly better when certain late visual layers are dropped, suggesting redundancy. This point is central to VFL-select, because the method assumes that capability-relevant supervision should be matched to the layers where the associated visual computation is concentrated. In this formulation, “select” does not refer to selecting clients or participants; it refers to selecting training examples whose predictive success depends on the VFLs aligned with the target function. ## 2. Identification of function-specific layers The discovery mechanism for VFLs is **Visual Token Swapping**, a causal probing framework that modifies targeted KV-cache entries during decoding. Let \(I\) be the image, \(T\) the text prompt, \(U\) the vision embeddings after the vision encoder and connector, \(W\) the text embeddings, and let the MLLM have layers \(\{f^{(l)}\}_{l=1}^L\). Multimodal decoding is written as $$ P(y \mid U, W) = \prod_{t=1}^{N_{\text{gen}}} P(y_t \mid y_{ 0.5\). Layers with the highest change rate are taken as the VFLs for that function [2509.24791]. For Qwen2.5-VL-7B, the reported peaks are sharp rather than diffuse: **Recognition** appears in early layers around \(0\)–\(10\); **Counting** appears around layer **12** or roughly layers **14–16**; **Grounding** appears around layer **18**; and **OCR** appears around layers **22–24**. A complementary probe, **Vision Token Dropping**, removes visual tokens from layer \(k\) onward and shows that **OCR tasks degrade first**, then **spatial reasoning**, then **recognition**, while general knowledge tasks are often much less sensitive to late visual layers. Together, these results support a layered specialization view rather than a monolithic vision pipeline [2509.24791]. ## 3. Construction of VFL-select VFL-select converts the VFL analysis into a data curation rule. For each training example \((x,y)\), the paper defines a layer score $$ R_k(x,y)=\frac{P(y \mid U(x)_{\le k}, W)}{P(y \mid U(x)_{\le k-1}, W)}. $$ This ratio compares the correct-answer probability when including layer \(k\) versus excluding it. The operational interpretation given by the paper is straightforward: if \(R_k\) is high, the sample relies strongly on layer \(k\), and the sample is therefore associated with the visual function represented by that layer [2509.24791]. Automatic data classification proceeds in four steps. The method computes \(R_k\) across layers, finds the layer with the highest score, assigns the sample to that layer’s function group, and then samples from each group to build a balanced training subset. The paper stresses that this allows **automatic classification by function without explicit semantic labels**. It does not require a human to label a sample as OCR, Counting, or Grounding; instead, the assignment is inferred from the model’s own layer sensitivity. The key assumption behind the method is that the data most useful for improving a capability are the data that activate the corresponding VFLs. The reported practical consequence is twofold: **targeted improvement**, because selected data match the function that needs strengthening, and **less wasted training**, because examples unrelated to the target function are less likely to dominate the update. The paper also notes that the VFL classification step can be performed with a **small proxy model** such as **TinyLLaVA-0.5B** and then transferred to a larger model, which is presented as a scalability mechanism [2509.24791]. ## 4. Reported empirical behavior and relation to VFL-LoRA The paper reports that VFL-select **outperforms human expert data selection**, **outperforms random selection**, and achieves **98% of full-data performance using only 20% of the data** [2509.24791]. On a large **20M-sample pool**, it is reported to perform consistently better than both Random and Expert selection across subset sizes **150k**, **250k**, **350k**, and **665k**. On **LLaVA-665k**, when only **20%** of the dataset is selected for fine-tuning, **VFL-select** reaches **99.5%** of full-data performance on shallow-layer task benchmarks and **97.4%** on deep-layer task benchmarks. In the paper’s framing, this is the concrete efficiency result: nearly full performance from a small fraction of the data. The same work uses the VFL concept for parameter-efficient adaptation through **VFL-LoRA**. The comparison is between **Full-LoRA**, which applies adapters across all layers, and **VFL-LoRA**, which applies adapters only to the functionally relevant layers. For the spatial-reasoning setup, the relevant layers are chosen from the **count-function VFL**. The reported findings are that VFL-LoRA uses about **one-third the tunable parameters** of full-LoRA, matches or slightly improves **in-domain** performance, performs **better out of domain**, and reduces catastrophic forgetting of unrelated abilities [2509.24791]. These results position VFL-select as more than an interpretability artifact. In the paper’s formulation, the same layer-function map supports both selective data curation and selective fine-tuning. The article’s central claim is therefore mechanistic rather than heuristic: because visual computation is concentrated in narrow layer blocks, training should preferentially target the data and parameters associated with those blocks. ## 5. Related meanings of “select” in vertical federated learning and federated systems A separate literature uses “selection” in the context of **vertical federated learning (VFL)**, where multiple parties hold different features for the same aligned samples. In that literature, the object of selection is typically not a multimodal training example but some element of the federated pipeline itself. A concise comparison is given below. | Setting | Selected object | Basis of selection | |---|---|---| | VFL-select in MLLMs | Training examples | Layer-wise dependency on Vision Function Layers | | VFLens | Features, training samples, inference instances | Visualization, clustering, and host-side confidence gating | | VFL-RPS | Passive participants | Secure Spearman correlation, redundancy detection, forward selection | | LESS-VFL / ICAFS | Feature or embedding components | Group-lasso stages or synthetic gated selectors | | Federated Select | Client-specific model slices | Keys specifying relevant submodels | “VFLens: Co-design the Modeling Process for Efficient Vertical Federated Learning via Visualization” [2210.00472] treats selection as an interactive workflow spanning feature engineering, sample selection, and inference-time gating. It uses LR-based VFL, private set intersection, and additively homomorphic encryption plus masking for gradient exchange. For sample selection, it first projects samples with **t-SNE** and groups them with **KMeans++**, then guides within-cluster sampling through **statistical homogeneity** and **content diversity**. For inference-time selection, the host first runs its local model, then compares each prediction instance to a nearest training sample in host feature space; if the local prediction, the nearest training label, and the VFL model’s prediction for that training sample agree, the instance is treated as high-confidence and the host model is used. In the reported case study, out of **15,000** prediction records, **8,202** did not require VFL inference, a rejection rate of **54.6%**, and **7,782** of those matched the VFL labels, a **94%** hit rate [2210.00472]. “VFL-RPS: Relevant Participant Selection in Vertical Federated Learning” [2502.14375] addresses pre-training participant selection. It assumes one active party with labels and passive parties contributing feature subsets, computes **secure Spearman correlation**, detects redundancy, scores parties by relevance, and applies **forward selection** before federated training. The paper reports that selecting only a small subset, often **50% of parties**, can match the performance of using all parties; for example, on California Housing, VFL-RPS selects top **3 of 5** passive parties and reports **MSE \(0.33\)** and \(R^2=0.67\), close to the all-party result, while on Credit Card Default its selection time is about **17–19 s** versus over **300–400 s** for VFLMG [2502.14375]. “LESS-VFL: Communication-Efficient Feature Selection for Vertical Federated Learning” [2305.02219] selects features through a three-stage pipeline: short VFL pre-training, server-side group-lasso over fixed embeddings to identify significant embedding components, and local feature selection at each party with no further communication. The paper provides both a theoretical spurious-feature removal guarantee and empirical communication savings; on **Phishing**, reported communication cost is **3.99 MB** for LESS-VFL versus **95.22 MB** for group lasso, about **20× lower** [2305.02219]. “ICAFS: Inter-Client-Aware Feature Selection for Vertical Federated Learning” [2504.10851] shifts the emphasis to inter-client interactions, using **conditional feature synthesis** with a federated conditional GAN, multiple gated selectors, and synthetic embeddings so that selection can be learned **without sharing private gradients**. It reports the best or near-best test accuracy across several real-world datasets, including **100.00%** on **ALLAML** at \(K=2\) and \(K=5\) [2504.10851]. The more general primitive “Federated Select” [2208.09432] extends the selection idea from data or features to server-model slices. It defines $$ \mathrm{FEDSELECT}(x@S,\{z_1,\dots,z_N\}@C,v) = \Big\{ [v(x,z_{n,1}),\dots,v(x,z_{n,m})] : n\in[N] \Big\}@C, $$ which generalizes broadcast by allowing each client to receive a client-specific submodel. The paper argues that this can reduce communication and client memory substantially and potentially enable the training of models too large to fit on-device [2208.09432]. A plausible implication is that “VFL-select” now sits at the intersection of at least two naming traditions: mechanistic data selection in MLLMs and selection primitives or selection policies in federated systems. ## 6. Significance, misconceptions, and unresolved issues One recurring misconception is to treat VFL-select as a generic heuristic sampler. In the MLLM paper, it is explicitly grounded in a mechanistic criterion: examples are selected because their correct prediction depends strongly on function-specific layer blocks. Another misconception is that every visual task must engage all discovered VFLs. The paper states the opposite: many general tasks do **not** require all VFLs, and some late visual layers may even be redundant for certain models [2509.24791]. A second source of confusion is terminological. In multimodal LLM research, **VFL** denotes **Vision Function Layers**. In vertical federated learning, **VFL** denotes **Vertical Federated Learning**. The two usages are unrelated in mechanism and objective. This suggests that disambiguation by domain is necessary when the term appears without expansion. In the federated literature, selection may concern robust collaborative inference under device and communication faults, as in “Robust Collaborative Inference with Vertically Split Data Over Dynamic Device Environments” [2312.16638], where **VFL** is treated as a brittle baseline and robustness improves through **server replication**, **gossip averaging**, and **training with simulated faults**; at **30% fault rate** on StarCraftMNIST with 16 devices, reported average accuracy rises from **0.430** for VFL to **0.717** for **MVFL-G4** [2312.16638]. The unresolved issues also differ by domain. For the MLLM formulation, the central open question is how far function-aware selection generalizes across models, tasks, and data regimes beyond those reported in the paper. In the vertical federated learning literature, the open issues are often systems and privacy constraints: VFLens notes that inference-time sampling depends heavily on user experience and domain knowledge [2210.00472]; VFL-RPS is designed mainly for numerical tabular data and may require tuning thresholds such as \(0.9\), \(0.95\), and \(\delta\) [2502.14375]; ICAFS argues that synthetic data reduce information leakage but does not provide a formal privacy guarantee in the main method [2504.10851]; and the robust collaborative inference framework with MAGS explicitly notes the absence of communication latency modeling and more realistic streaming or asynchronous settings [2312.16638]. Taken narrowly, VFL-select denotes the layer-aware data selection method derived from Vision Function Layers in MLLMs. Taken broadly, it names a family resemblance among methods that convert internal structure, participant relevance, feature interaction, or systems constraints into a selection rule. The common thread is not a single algorithmic template but a shared methodological stance: selection should be aligned with the structure that actually governs prediction. Source: https://www.emergentmind.com/topics/vfl-select