FedVLM: Federated Vision-Language Fine-Tuning

• FedVLM is a federated fine-tuning framework for vision-language models that adapts a frozen pre-trained backbone using parameter-efficient LoRA and a personalized pLoRA mechanism.
• The framework trains only low-rank adapters on the final decoder layer, communicating shared global parameters while retaining client-specific components to reduce communication costs by about 50%.
• Empirical results on visual question answering show that FedVLM achieves faster convergence, higher accuracy, and improved handling of data heterogeneity compared to standard federated and centralized approaches.

FedVLM denotes a federated vision-language modeling paradigm in which a pre-trained vision-LLM is adapted across decentralized clients without centralizing raw image–text data. In its named formulation, FedVLM is a federated fine-tuning framework for vision-LLMs that combines parameter-efficient tuning via LoRA with a personalization mechanism, pLoRA, to handle non-iid client data efficiently and with low communication cost. The framework is instantiated on Florence-2, freezes the vision encoder and language encoder/decoder, and trains LoRA adapters only in the final layer of the language decoder, with a global shared component and a client-local personalized component (Mitra et al., 23 Jul 2025). This suggests that FedVLM is best understood both as a specific federated LoRA framework and, more broadly, as part of a research area concerned with privacy-preserving multimodal adaptation under heterogeneity, communication limits, and resource constraints (Zheng et al., 11 Jun 2025).

1. Problem setting and base model

In the FedVLM formulation, vision-LLMs are transformer-based generative models that take both images and text as input and generate text autoregressively for tasks such as visual question answering, image captioning, and visual reasoning and comprehension. The concrete base model is Florence-2, described as a relatively lightweight VLM with fewer than 1B parameters, a DaViT vision transformer encoder, a BART text encoder/decoder, and a linear projection aligning visual embeddings to the text embedding space (Mitra et al., 23 Jul 2025).

The multimodal computation is written with a vision encoder output $V \in \mathbb{R}^{N_v \times D_v}$, a text encoder output $T \in \mathbb{R}^{N_t \times D_t}$, and a projection $P : \mathbb{R}^{D_v} \to \mathbb{R}^{D_t}$. The unified sequence is

$\mathbf{X} = [P(V), T],$

and the language decoder $D_l$ generates output tokens $O = D_l(X)$. Fine-tuning uses token-level cross-entropy,

$$\mathcal{L} = - \sum_{i=1}^{|o|} \log P_{\theta}(o_i \mid o_{<i}, x),$$

with $\theta$ restricted to the trainable adaptation parameters rather than the full backbone (Mitra et al., 23 Jul 2025).

The motivation for federated fine-tuning is explicitly multimodal and operational. Raw image–text pairs may be sensitive, clients may hold non-iid data from very different domains, VLMs are expensive to transmit and update in full, edge devices have limited memory and compute, and multimodal optimization is more complex than unimodal federated learning. The paper therefore positions FedVLM as a response to decentralization and privacy requirements, data heterogeneity, communication cost, computational constraints, and multimodal complexity (Mitra et al., 23 Jul 2025).

2. Federated fine-tuning protocol

FedVLM is organized around a frozen pre-trained backbone $\mathcal{W}_0$, $K$ clients with local datasets $T \in \mathbb{R}^{N_t \times D_t}$0, and a central server that maintains a global LoRA matrix. Classical federated learning is written as

$T \in \mathbb{R}^{N_t \times D_t}$1

with $T \in \mathbb{R}^{N_t \times D_t}$2, but in FedVLM the effective trainable parameters are the LoRA components while $T \in \mathbb{R}^{N_t \times D_t}$3 remains fixed (Mitra et al., 23 Jul 2025).

The client-level loss under pLoRA remains token-level cross-entropy:

$T \in \mathbb{R}^{N_t \times D_t}$4

A communication round proceeds by distributing the pre-trained base weights once, initializing $T \in \mathbb{R}^{N_t \times D_t}$5 to zeros and each $T \in \mathbb{R}^{N_t \times D_t}$6 randomly with a Gaussian, broadcasting the current global $T \in \mathbb{R}^{N_t \times D_t}$7, performing local training of LoRA parameters for a fixed number of local epochs, uploading only the client-side $T \in \mathbb{R}^{N_t \times D_t}$8, averaging those updates on the server, and broadcasting the new global matrix back to clients (Mitra et al., 23 Jul 2025).

The aggregation rule is simple averaging:

$T \in \mathbb{R}^{N_t \times D_t}$9

In the reported implementation, FedVLM uses Florence-2, LoRA rank $P : \mathbb{R}^{D_v} \to \mathbb{R}^{D_t}$0, scaling factor $P : \mathbb{R}^{D_v} \to \mathbb{R}^{D_t}$1, dropout $P : \mathbb{R}^{D_v} \to \mathbb{R}^{D_t}$2, learning rate $P : \mathbb{R}^{D_v} \to \mathbb{R}^{D_t}$3, batch size $P : \mathbb{R}^{D_v} \to \mathbb{R}^{D_t}$4, and $P : \mathbb{R}^{D_v} \to \mathbb{R}^{D_t}$5 local epochs per round. Only LoRA matrices are trained; the backbone encoders and decoder are frozen. The framework is implemented in PyTorch with the Flower FL framework (Mitra et al., 23 Jul 2025).

3. LoRA parameterization and personalized LoRA

FedVLM adopts standard LoRA, in which a frozen weight matrix $P : \mathbb{R}^{D_v} \to \mathbb{R}^{D_t}$6 is augmented by a low-rank update $P : \mathbb{R}^{D_v} \to \mathbb{R}^{D_t}$7 with $P : \mathbb{R}^{D_v} \to \mathbb{R}^{D_t}$8, $P : \mathbb{R}^{D_v} \to \mathbb{R}^{D_t}$9, and $\mathbf{X} = [P(V), T],$0. The effective layer becomes

$\mathbf{X} = [P(V), T],$1

so the layer output is

$\mathbf{X} = [P(V), T],$2

Initialization follows the usual practice that $\mathbf{X} = [P(V), T],$3 is initialized to zero and $\mathbf{X} = [P(V), T],$4 with random Gaussian entries, so initially $\mathbf{X} = [P(V), T],$5 and the model behaves exactly as the pre-trained model (Mitra et al., 23 Jul 2025).

The central methodological contribution is personalized LoRA, or pLoRA. Rather than aggregating a fully shared LoRA adapter, pLoRA decomposes the update into a global matrix $\mathbf{X} = [P(V), T],$6 and a personalized matrix $\mathbf{X} = [P(V), T],$7 that remains on client $\mathbf{X} = [P(V), T],$8. The adapted decoder weights are

$\mathbf{X} = [P(V), T],$9

with

$D_l$0

Equivalently,

$D_l$1

The personalization mechanism is therefore asymmetric: each client trains both $D_l$2 and $D_l$3 locally, but only $D_l$4 is shared and aggregated, while $D_l$5 is never uploaded (Mitra et al., 23 Jul 2025).

The motivation is explicitly tied to the limitations of standard LoRA in federated settings. A single global adapter cannot capture diverse client distributions under heavy heterogeneity; existing federated LoRA variants such as FLoRA, HetLoRA, FDLoRA, and FFA-LoRA are described as lacking a simple systematic personalization mechanism, or trading personalization for lower communication, or increasing complexity and communication. FedVLM therefore uses a partly shared and partly personalized decomposition. The paper further interprets $D_l$6 as a global “output-side” adaptation and $D_l$7 as an input-side, client-specific adaptation, although it does not present an explicit bi-level or regularized objective beyond the round-wise local optimization and server aggregation (Mitra et al., 23 Jul 2025).

4. Tasks, datasets, and empirical results

The main downstream task is visual question answering. Experiments center on RLAIF-V, a large multimodal dataset constructed by integrating samples from 14 established datasets, including VQAv2, OK-VQA, MSCOCO, ART500K, and LCS-558K, and augmenting image–question pairs with high-quality AI-generated answers. The reported subset contains 15,956 samples from LCS-558K, 15,199 from COCO, 14,802 from OK-VQA, 12,942 from VQAv2, and 1,096 from ART500K. Non-iid conditions are simulated by allocating clients different sub-datasets or domains, so each client receives samples primarily from a specific subset, thereby inducing label and feature skew (Mitra et al., 23 Jul 2025).

The main comparison on RLAIF-V uses four clients, three local epochs, and five rounds. In that setting, pLoRA substantially outperforms standard federated LoRA baselines.

Method	Non-IID acc.	IID acc.
pLoRA	0.867	0.745
FLoRA	0.696	0.647
FFA-LoRA	0.343	0.337

The paper interprets the non-iid result as a 24.5% improvement of pLoRA over standard LoRA/FLoRA in client-specific performance, and the iid result as a 15.1% improvement over FLoRA. It also reports that pLoRA yields faster and smoother convergence in both accuracy and loss (Mitra et al., 23 Jul 2025).

Additional experiments broaden the picture. With four clients and three local epochs, FedVLM converges faster and achieves higher accuracy than centralized LoRA fine-tuning on OK-VQA in the iid setting, and on full RLAIF-V centralized LoRA reaches 89% accuracy while FedVLM converges faster and achieves even better accuracy. Under non-iid client-count ablations with 2, 4, 6, and 8 clients, pLoRA maintains approximately 0.98 accuracy, approximately 0.97 F1, and low loss around 0.11–0.13, whereas performance before fine-tuning is much lower, with accuracy around 0.33 and loss around 5.92. A rank ablation over $D_l$8 indicates that performance is relatively insensitive to rank when LoRA is applied only to the final decoder layer (Mitra et al., 23 Jul 2025).

The paper also includes a unimodal transfer experiment on CIFAR-10 using MobileNetV3 plus LoRA or pLoRA in federated learning. CIFAR-10 is partitioned into 80 shards, two shards per client for 40 clients, creating non-iid label skew. In this setting, pLoRA consistently achieves higher accuracy and lower loss than LoRA throughout training, which the authors use as additional evidence that the personalization mechanism addresses heterogeneity rather than only a specific VQA benchmark (Mitra et al., 23 Jul 2025).

5. Communication efficiency, privacy, and limitations

FedVLM is designed around a frozen backbone and a minimal communication surface. Because only LoRA parameters are trained, the large Florence-2 backbone remains fixed, which reduces client-side compute and memory. Because LoRA is applied only to the final decoder layer, the number of trainable parameters and FLOPs is further reduced. Because only the matrix $D_l$9 is communicated, while the personalized matrix $O = D_l(X)$0 is retained locally, communication volume is reduced by roughly 50% compared to sending both $O = D_l(X)$1 and $O = D_l(X)$2 (Mitra et al., 23 Jul 2025).

The efficiency argument is both architectural and algebraic. The low-rank multiplication costs $O = D_l(X)$3 rather than $O = D_l(X)$4 for full updates, and the communicated object has size $O = D_l(X)$5 rather than the full model state. In the paper’s qualitative comparison, pLoRA preserves low communication similar to FFA-LoRA while also supporting personalized federated learning, unlike FLoRA, HetLoRA, FDLoRA, and FFA-LoRA (Mitra et al., 23 Jul 2025).

Privacy is treated in the standard federated-learning sense: raw images, questions, and answers never leave the clients, and only model updates for LoRA $O = D_l(X)$6 are communicated. At the same time, the paper explicitly states that no differential privacy mechanism, secure aggregation, or cryptographic protocol is implemented, and no formal $O = D_l(X)$7-DP guarantees are provided. Potential privacy leakage from model updates, hallucinations or unreliable outputs, local bias reinforcement, and limited experimental scope are all acknowledged limitations. The empirical evaluation is concentrated on a single base VLM, a main VQA task, and LoRA only on the final decoder layer, although the framework is described as conceptually general (Mitra et al., 23 Jul 2025).

6. Broader research landscape and alternative uses of the term

FedVLM also names, or is closely connected to, a broader body of work on federated multimodal modeling. “FedVLMBench” is presented as the first systematic benchmark for federated fine-tuning of VLMs, integrating two mainstream VLM architectures, four fine-tuning strategies, five FL algorithms, and six multimodal datasets spanning four cross-domain single-task scenarios and two cross-domain multitask settings. Its abstract reports that a 2-layer multilayer perceptron connector with concurrent connector and LLM tuning emerges as the optimal configuration for encoder-based VLMs in FL, and that current FL methods exhibit significantly higher sensitivity to data heterogeneity in vision-centric tasks than text-centric ones (Zheng et al., 11 Jun 2025).

A separate fairness line treats federated VLMs as prompt-tuned multimodal systems with frozen backbones. “Mitigating Group-Level Fairness Disparities in Federated Visual LLMs” introduces FVL-FP, combining FL with fair prompt tuning through Cross-Layer Demographic Fair Prompting, Demographic Subspace Orthogonal Projection, and Fair-aware Prompt Fusion. The reported effect is an average 45% reduction in demographic disparity compared to standard FL approaches while maintaining task performance within 6% of state-of-the-art results (Chen et al., 3 May 2025).

In surveillance-oriented video analysis, the term is used more generically for federated LoRA tuning of a video-capable VLM rather than for the Florence-2 pLoRA framework. That study compares zero-shot VLM inference, LoRA-based fine-tuning of LLaVA-NeXT-Video-7B under FedAvg, and personalized federated learning of a 65.8M-parameter 3D CNN. On a mixed RLVS + RWF validation setting, LoRA FL improves LLaVA from 90.62% to 90.87% accuracy, raises ROC AUC from 85.93% to 91.24%, and reduces log loss from 0.706 to 0.535, but requires 570 Wh, compared with 240 Wh for the personalized 3D CNN, which attains ROC AUC 92.59%. On UCF-Crime, hierarchical grouping boosts Ovis-8B zero-shot multiclass accuracy from 65.31% to 81.0%, indicating that VLMs can be effective for coarse-grained semantic anomaly categorization while carrying substantial energy costs (Thuau et al., 10 Nov 2025).

Historically adjacent work predates the named FedVLM framework. “FedVMR” formulates federated video moment retrieval with a visual encoder, text encoder, cross-modal fusion module, and output layer, and introduces grouped sequential federated learning, selective aggregation via a c-validation set, and a temporal-distribution-gap loss for non-iid video-language data. On Charades-STA and ActivityNet Captions, FedVMR matches or slightly surpasses centralized performance while outperforming FedAvg, FedProx, and FedCMR under non-iid partitions (Wang et al., 2022). A common source of confusion is the similarity between FedVLM and “F-VLM,” but F-VLM denotes “Frozen Vision–LLMs” for open-vocabulary object detection and is not a federated framework; its core design is to freeze the VLM and train only a detector head (Kuo et al., 2022).

Taken together, these works position FedVLM as a concrete personalized LoRA framework and as a broader research direction in privacy-preserving multimodal learning. The named FedVLM framework is distinguished by its decomposition of LoRA into a shared $O = D_l(X)$8 and a client-local $O = D_l(X)$9, whereas benchmark, fairness, and surveillance papers show that the central questions extend beyond personalization to include benchmark standardization, demographic disparity, calibration, energy use, and deployment strategy across heterogeneous multimodal environments (Mitra et al., 23 Jul 2025).