---
title: Adapter-based Federated LLMs
url: https://www.emergentmind.com/topics/adapter-based-federated-large-language-models-fedllms
type: topic
---

# Adapter-based Federated LLMs

Adapter-based Federated Large Language Models (FedLLMs) are a class of methodologies for privacy-preserving and computation/communication-efficient fine-tuning of large language models (LLMs) in federated learning (FL) environments. These approaches integrate parameter-efficient modules—typically low-rank adapters such as LoRA—into frozen transformer backbones, enabling distributed collaborative model adaptation across heterogeneous clients without sharing raw data or transmitting full model weights. Adapter-based FedLLMs form a central paradigm in contemporary research, supporting robust personalization, scalable deployment, and compliance with privacy and resource constraints in domains ranging from generic NLP to specialized fields such as medicine.

## 1. Formal Setting and Motivation

Adapter-based FedLLMs address two critical barriers to federated adaptation of large pre-trained LLMs: the prohibitive communication cost of exchanging full model weights and the significant local computation/memory requirements inherent to multi-billion-parameter models. In the FL context, clients iterate between local training—updating small adapter modules inserted into a frozen global backbone—and periodic aggregation of these adapters at a central server.

Formally, let $M$ clients possess their own datasets $\mathcal{D}_k$, and let $f_0$ denote a frozen LLM. Each client trains only its adapter parameters (e.g., low-rank matrices $A$, $B$ or per-adapter parameters $\phi_k$), minimizing a local loss:
\[
\min_{\phi_k} L_k(\phi_k) := \mathbb{E}_{(x,y)\in\mathcal{D}_k}[\ell(f_0(x) + A_l(x;\phi_k), y)]
\]
A global adapter $\theta$ is simultaneously fine-tuned via federated averaging (FedAvg), and clients periodically synchronize to aggregate their local adapter updates into an improved central model [2403.19211][2409.15723][2503.12016].

Adapter modules (e.g., LoRA, bottleneck adapters) typically account for $0.1\%$–$1\%$ of the full model parameters, reducing per-round communication by $>90\%$ relative to classic FedAvg. This design also confines train-time resource utilization, making federated LLM fine-tuning tractable on edge hardware or institutionally partitioned compute [2409.15723][2601.22124].

## 2. Adapter Architectures and Insertion Strategies

Adapter-based FedLLMs utilize well-defined parameter-efficient fine-tuning (PEFT) modules:

- **Bottleneck Adapters:** Two linear projections with a nonlinearity between frozen transformer sublayers. For $d$-dimensional hidden states and bottleneck width $r$, each adapter contains $\approx 2dr$ trainable parameters per layer [2205.10162][2503.12016].
- **LoRA Adapters:** Low-rank decomposition of selected weight matrices (e.g., attention projections) as $\Delta W = BA$, with $A\in \mathbb{R}^{r\times d}$, $B\in \mathbb{R}^{d\times r}$ and $r\ll d$, inserted in parallel to $W$ [2409.15723][2601.22124][2505.24773].
- **Tensor-Train Adapters:** Decomposition of projection matrices into a sequence of small TT-cores for further communication reduction and factor sharing [2410.13097].
- **Dual/Expert Adapters:** Dual adapters (global+local) [2403.19211] and architectures with mixtures of LoRA experts assigned via data-driven criteria for personalization [2411.19128].

Typical insertion points include all transformer layers, selective attention/FFN submodules, or even only the top layers for extreme efficiency configurations [2205.10162][2409.15723][2406.17706]. Recent frameworks support per-layer dynamic adapter allocation and rank selection matched to client resource budgets or data heterogeneity [2505.24773][2411.19128].

## 3. Federated Training Protocols and Aggregation

Adapter-only federated optimization is typically instantiated via variants of FedAvg:

- At each round $t$, the server broadcasts the current global adapter state (or both global and frozen backbone for new clients).
- Each selected client downloads global parameters, initializes local adapter(s), and runs $E$ epochs of local SGD or Adam to minimize its partial data loss.
- Upon completion, the client uploads its updated adapters. For certain methods (e.g., dual adapters or expert mixtures), only the shared/global component or a selective subset is communicated, while highly personalized parameters remain private [2403.19211][2411.19128].

Aggregation strategies include:
- Plain FedAvg (data-weighted mean of adapters) [2409.15723][2503.12016].
- Adaptive aggregation: Eg, in Fed-MedLoRA+, client contributions are weighted by both sample count and validation loss on a held-out dataset [2601.22124].
- Singular Value Thresholding (FLoRIST): Server-side SVD on stacked LoRA updates with a tunable singular value cutoff to obtain an optimal rank-$r$ global low-rank adapter [2506.09199].
- Rank-aware aggregation: Contributions proportional to both data volume and effective adapter rank (client-specific) [2505.24773].

Personalization is supported via local adapters or expert assignments never uploaded to the server [2403.19211][2411.19128].

## 4. Handling Data Heterogeneity, Personalization, and System Heterogeneity

Federated LLM adaptation faces substantial data heterogeneity, including label skew, feature skew, and diverse annotation structure. Adapter-based FedLLMs address these through:

- **Dual or Heterogeneous Adapters:** FedDPA [2403.19211] trains both a personalized adapter per client and a federated global adapter, enabling test-time selection/combination via a data-driven instance weighting function $\alpha(x)$ computed from similarity of test examples to local data priors.
- **Mixtures of Adapters/Experts:** FedAMoLE [2411.19128] introduces client-specific assignment of LoRA expert adapters via a reverse selection mechanism optimizing both adaptation capacity and load balancing.
- **Sparse/Selective Updates:** Curriculum techniques based on Fisher Information [2410.00131] select only “easy-to-learn” samples/batches early in training and adaptively update only the most informative layers or individual neurons, reducing both communication and compute.
- **Resource-Awareness:** Algorithms such as AFLoRA [2505.24773] dynamically prune low-importance adapter ranks or adapt aggregation weighting based on client capability, while SflLLM [2504.14667] jointly optimizes split points, LoRA rank, and network resource allocation for edge deployments.

Empirically, these approaches yield +1–5% absolute improvement over plain FedAvg-LoRA under extreme heterogeneity or limited resource settings, and can match or surpass centralized fine-tuning in moderate regimes [2403.19211][2411.19128][2506.09199][2410.13097].

## 5. Communication, Computation, and System Complexity

Adapters reduce per-round network and local resource demands by several orders of magnitude:

| Method                 | Relative Param Fraction | Typical Comm/round (MB) | Typical Speed-up |
|------------------------|------------------------|-------------------------|------------------|
| Full Fine-Tuning       | 100%                   | >10,000                 | 1×               |
| LoRA, Adapter-Based    | ~0.1–1%                | <100 (often <10)        | 10–100×          |
| Tensor-Train Adapter   | ~0.02–0.06%            | <2                      | up to 100×       |
| FedAMoLE (Expert Mixt.)| 0.01–0.5%              | 10–15                   | 5–50×            |

These factors depend on model size, adapter configuration, and client population. For example, Fed-MedLoRA achieves ≈98.5% reduction in transmitted bytes per round—dropping from 32 GB to 168 MB (8B LLaMA)—and attains computation feasibility on consumer GPUs (14 GB for training, 6.8 GB for inference) [2601.22124].

Adapter-based FedLLMs are fundamentally modular: clients require only frozen backbone weights and adapter updates, supporting flexible onboarding/dropout scenarios and device capability stratification [2409.15723][2205.10162][2406.17706][2504.14667].

## 6. Empirical Performance, Personalization, and Test-Time Robustness

Extensive experiments on standard NLP (GLUE, Flan, AGNews, SNLI), medical IE (MIMIC-III, i2b2, YNHH), and instructional finetuning (Alpaca, WizardLM) benchmarks report that:

- Adapter-based methods reach within 0.1–1% of full fine-tuning accuracy in classification and generation, often with as few as 2–4 communication rounds for stable domains [2403.19211][2503.12016][2601.22124].
- Dual-personalizing architectures (FedDPA) outperform classic FedLoRA baselines by 1–2 ROUGE/accuracy points on test-time distribution shift, and test-time dynamic mixing of local/global adapters improves out-of-domain robustness by 3–15 points [2403.19211].
- Mixtures of LoRA experts with dynamic assignment (FedAMoLE) achieve improvements of +3–5% mean test accuracy per client under label/feature skew or multi-task distribution [2411.19128].
- In medical IE, Fed-MedLoRA+ outperforms single-site fine-tuning and domain-specific BERT by 10–70% absolute F1 in external validation, achieving near-centralized performance in real-world cross-institutional scenarios [2601.22124].
- Adaptive communication policies, curriculum, and rank-pruning yield an additional 70–98% speedup and effectively track optimal trade-offs under tight constraints [2410.00131][2506.09199][2505.24773].

## 7. Security, Privacy, and Open Challenges

Contrary to earlier assumptions, small adapter sizes do not guarantee privacy: gradient inversion attacks (e.g., UTR attack [2601.17533]) can reconstruct training data with near-perfect accuracy (ROUGE-1/2 ≈ 99–100), even from low-rank adapter gradients in federated settings. Differential privacy and extreme gradient pruning only become effective at the cost of severe accuracy degradation; no parameter-efficient adapter mechanism currently provides both strong utility and provable privacy guarantees [2601.17533][2409.15723].

Open challenges include:

- Designing aggregation and update mechanisms robust to severe heterogeneity, both in data and system resources [2505.24773][2411.19128][2410.13097].
- Achieving privacy guarantees (differential privacy, secure aggregation) tailored to adapter structure, and defending against leakage attacks without destroying learning signal [2601.17533][2409.15723].
- Extending adapter-based FL LLM methodology to cross-modal and highly personalized domains (e.g., vision-language, retrieval-assisted models) and real-world legal/compliance frameworks [2409.15723][2503.12016].
- Addressing the memory wall (full LLM backbone needs to reside on client) via split learning or server-aided inference [2406.17706][2504.14667].

Adapter-based FedLLMs represent a critical direction enabling scalable, resource-aware, and (potentially) privacy-preserving adaptation of LLMs in decentralized, heterogeneous, and sensitive environments, but require further advances in privacy and security to be robust for real-world deployment [2403.19211][2409.15723][2601.22124][2601.17533][2503.12016].

Source: https://www.emergentmind.com/topics/adapter-based-federated-large-language-models-fedllms