Federated Learning for LLMs

• Federated learning for LLMs is a decentralized method enabling collaborative model training on sensitive, non-IID data without sharing raw information.
• It leverages parameter-efficient techniques like LoRA and prompt tuning to minimize communication overhead and improve scalability.
• The approach incorporates robust privacy measures and aggregation protocols to address client heterogeneity and ensure secure, personalized updates.

Federated learning for LLMs is a paradigm in which multiple clients collaboratively fine-tune or train LLMs on decentralized, often sensitive, datasets—without exchanging raw data. This approach is motivated by stringent privacy requirements across domains such as healthcare, finance, law, and multilingual settings. Federated LLM training combines advances in parameter-efficient fine-tuning, communication-efficient protocols, robust aggregation, and privacy-preserving mechanisms, but it faces challenges arising from model scale, non-IID data, and resource heterogeneity. Recent research has systematized the mathematical formulation, surveyed variants of aggregation and tuning, evaluated empirical and system-level trade-offs, and charted multiple avenues for practical deployment and future study (Yao et al., 2024).

1. Mathematical Formulation and Core Algorithms

The federated LLM objective is to learn a global parameter vector $w$ (potentially the entire LLM or a parameter-efficient submodule) across $K$ clients, each indexed by $k=1,\ldots,K$ and holding a local dataset $\mathcal{D}_k$. Each client minimizes its local loss: $$F_k(w) = \mathbb{E}_{(x,y)\sim\mathcal{D}_k}[\ell(w; x,y)]$$ with aggregation weights $p_k = |\mathcal{D}_k| / (\sum_j |\mathcal{D}_j|)$. The FL meta-objective is: $F(w) = \sum_{k=1}^K p_k F_k(w)$ The canonical optimization is the FedAvg protocol: in each communication round $t$, the server broadcasts $w_t$ to participating clients. Each client performs $E$ local SGD steps, updating $K$0 via: $K$1 and returns $K$2; the server aggregates: $K$3 FedProx introduces a quadratic proximal penalty $K$4 to stabilize local updates under data heterogeneity.

For prompt learning, only soft prompt matrices $K$5 are optimized (the backbone $K$6 is frozen). Global prompt aggregation and personalization are performed by averaging and interpolation: $K$7 These mathematical structures underpin nearly all contemporary federated LLM protocols (Yao et al., 2024).

2. Communication-Efficient and Parameter-Efficient Techniques

Due to the immense size of LLMs, naive federated fine-tuning is impractical. Parameter-efficient fine-tuning (PEFT) and communication reduction strategies are crucial.

PEFT Mechanisms:

• LoRA (Low-Rank Adaptation): For each targeted weight $K$8, introduce $K$9 with trainable $k=1,\ldots,K$0, $k=1,\ldots,K$1, $k=1,\ldots,K$2, and freeze $k=1,\ldots,K$3. Extra parameters per matrix: $k=1,\ldots,K$4.
• SLoRA: Further imposes structured sparsity to reduce update volume.
• Prefix-Tuning and Adapters: Only small prompt vectors or MLP bottleneck layers are learned.
• Layer-skipping: Only a subset of layers (e.g., top 8/32) are fine-tuned, while the rest are frozen, yielding up to 69% bandwidth reduction with ≤2% performance loss versus centralized models (Zhang et al., 13 Apr 2025).
• Hybrid approaches combine adapters, LoRA ranks, or even client-personalized structures for further efficiency.

Compression and Transport:

• Quantization of model deltas to 8-bit or 4-bit—sometimes using norm-float or zeroth-order (FedKSeed) methods—reduces transmitted bytes by factors of 4–8, often with negligible loss in convergence (Xu et al., 20 Nov 2025).
• Top-$k=1,\ldots,K$5 gradient sparsification is also effective.
• Chunked transport (e.g., RDMA, file/container streaming) significantly improves throughput and reduces peak memory.

Empirical benchmarks such as FedLLM-Bench and FLASH demonstrate that, with LoRA or prompt-based methods, communication rounds can use <2% of full model parameters, with final performance within 1–2% of centralized fine-tuning (Yao et al., 2024, Puppala et al., 2024).

3. Addressing Data/Client Heterogeneity and Personalization

Heterogeneity in data distributions (topics, language, style) induces client drift and impedes model convergence. Solutions include:

• Algorithmic stabilization: FedProx adds a proximal term; SCAFFOLD corrects drift with control variates; curriculum methods (e.g., Fisher information-guided scheduling) adapt batch difficulty and parameter importance (Liu et al., 2024).
• Model/architecture heterogeneity: FedAMoLE deploys a mixture-of-LoRA-experts design, where clients use locally relevant experts, selected via an embedding-based reverse assignment scheme, and only synchronize experts with substantial shared utility (Zhang et al., 2024). FedP$k=1,\ldots,K$6EFT automates per-client LoRA rank selection via Bayesian sparse optimization (Lee et al., 5 Feb 2025).
• Multilingual FL: Increasing within-client language diversity improves global, cross-lingual performance and equity, though more rounds are required as drift decreases (Sant et al., 25 Mar 2026). Personalized FL architectures (personal adapters, local heads) are also effective for domain and resource adaptation.

Empirically, personalization and architectural heterogeneity provide up to +5% absolute and up to 45% relative improvement in accuracy over vanilla federated tuning baselines in highly non-IID settings (Zhang et al., 2024, Liu et al., 2024).

4. Privacy, Security, and Memorization

Privacy guarantees in federated LLMs go beyond the absence of raw data sharing. Emerging issues are:

• Unintended memorization: Even FL-tuned LLMs can memorize and regurgitate sensitive client data. LoRA drastically reduces memorization—up to 10× in empirical BLEU-based metrics—compared to full-model fine-tuning, while incurring negligible utility cost (Bossy et al., 7 Feb 2025).
• Differential privacy (DP): Gradient clipping and DP-SGD (noise on local adapter gradients) improve record-level privacy. Layer-skipping and LoRA adapters make DP integration more robust because fewer parameters are perturbed (Zhang et al., 13 Apr 2025).
• Secure aggregation: Fully homomorphic encryption and secure multiparty computation (SMPC) for LoRA updates is computationally feasible at LLM scales (e.g., secure aggregation for 25 million LoRA parameters in ~11s) (Bossy et al., 7 Feb 2025).
• Fine-grained disclosure: SecureGate proposes token-gated cross-client dual-adapter architectures, training “secure” and “revealing” LoRA modules on sanitized and raw data, respectively, and gating access at inference. Unauthorized inference attack accuracy and extraction recall are reduced by factors of up to 31.66× and 17.07×, respectively, with 100% routing reliability (Shaaban et al., 13 Feb 2026).
• Prompt-based FL: For black-box LLMs, differentially private synthetic prompts can support FL with strong $k=1,\ldots,K$7-privacy where $k=1,\ldots,K$8 (Wu et al., 2024).

These innovations collectively advance privacy, utility, and robustness guarantees in real-world deployments.

5. System and Optimization Challenges

Scalability to billion-parameter models in realistic settings enforces stringent requirements on FL for LLMs:

• Communication overhead remains a dominant cost. Synchronizing LoRA, adapters, or selected full layers achieves communication reduction factors ∼50–100× relative to transmitting all parameters (Yao et al., 2024).
• System heterogeneity: Device capabilities and network reliability differ widely. Algorithms such as FwdLLM use backpropagation-free protocols; scheduling and split-learning variants optimize bandwidth and compute utilization (Chen et al., 3 Jun 2025).
• Convergence rates: Under smoothness and variance assumptions, FedAvg on LLMs can achieve $k=1,\ldots,K$9 global regret (e.g., as in FedPEAT and FedMeZO) (Yao et al., 2024). Adaptive techniques (e.g., additional momentum, local early stopping) can reduce the number of communication rounds or client steps by up to 4.9× and 20–30%, respectively (Yao et al., 2024, Sant et al., 25 Mar 2026).
• Edge deployment: On-device FL for LLMs is feasible on embedded AI SoCs using PEFT and communication-efficient protocols, albeit subject to compute/memory bottlenecks and energy constraints (Woisetschläger et al., 2023, Ding et al., 2024).
• Aggregation: Reliable FedAvg, FedProx, SCAFFOLD, and adaptive optimizers (FedAdamW, FedYogi) each have specific stability/performance advantages for LLMs (Ye et al., 2024).

Empirical studies confirm that system-level engineering—quantized streaming, hierarchical memory, model partitioning—can reduce memory by >50%, bandwidth by up to 70%, with insignificant impact on convergence or downstream accuracy (Xu et al., 20 Nov 2025, Ding et al., 2024).

6. Empirical Benchmarks, Applications, and Future Directions

Benchmarks such as FedLLM-Bench, FLASH, and OpenFedLLM systematically evaluate federated LLMs on instruction-tuning, question answering, multilinguality, code generation, and value alignment. Key empirical findings include:

• LoRA-based FL achieves within 1–2% of the accuracy of centralized fine-tuning with $\mathcal{D}_k$0 parameter transmission (Yao et al., 2024, Puppala et al., 2024).
• Layer-skipping FL in healthcare NLP yields $\mathcal{D}_k$1 lower bandwidth draw and only $\mathcal{D}_k$2 F1 gap on clinical NER/ICD classification (Zhang et al., 13 Apr 2025).
• Federated value alignment and preference modeling (RLHF, DPO) can be implemented with privacy-preserving updates and achieve comparable or improved group fairness and convergence rates versus centralized pipelines (Srewa et al., 13 Mar 2025, Ye et al., 2024).
• Communication-efficient (container/file streaming, quantization) and parameter-efficient schemes make FL practical for 1B+ parameter models within available compute/memory budgets (Xu et al., 20 Nov 2025).
• Advanced privacy controls (SecureGate, prompt-based FL with synthetic samples) are practical and compatible with high utility (Shaaban et al., 13 Feb 2026, Wu et al., 2024).

Significant open research avenues include federated LLM pre-training across private corpora, leveraging LLMs themselves to enhance FL (e.g., synthetic data generation, reasoning for client scheduling), developing responsible cross-client data/knowledge transfer protocols, privacy and legal-ethical frameworks, and robust multimodal FL paradigms (Yao et al., 2024).

7. Outlook: Towards Practical Federated Training of LLMs

Federated learning for LLMs has matured into a discipline with theoretically principled algorithms, parameter- and communication-optimized techniques, privacy-preserving and personalization mechanisms, and strong empirical validation across application domains. Best-practice recommendations include:

• Default to LoRA, prompt, or adapter methods for FL to maximize efficiency and minimize memorization leakage.
• Prefer partial or adaptive layer tuning (layer-skipping, mixture-of-experts) for high-dimensional or heterogeneous data scenarios.
• Integrate quantization, streaming, and bandwidth-aware protocols at scale.
• Combine cryptographically secure aggregation, DP, and per-access control for privacy compliance.
• Employ fine-grained evaluation on privacy leakage, fairness, personalization, and downstream accuracy across non-IID clients.

These foundational elements, validated by large-scale and domain-specific benchmarks, enable practitioners to deploy LLMs in federated infrastructures with robust privacy, scalability, and performance (Yao et al., 2024, Zhang et al., 13 Apr 2025, Shaaban et al., 13 Feb 2026, Puppala et al., 2024, Ye et al., 2024).