---
title: Federated Tuning for LLMs
url: https://www.emergentmind.com/topics/federated-tuning-for-llms
type: topic
---

# Federated Tuning for LLMs

Searching arXiv for recent federated LLM tuning papers to ground the article.
to=arxiv.search  北京赛车如何json code ￣第四色{
  "query": "\"federated\" AND (\"large language model\" OR LLM) AND (fine-tuning OR tuning OR instruction)",
  "max_results": 10,
  "sort_by": "submittedDate"
}
to=arxiv.search ／久久িদ code  天天中彩票中奖了oml
query: "federated tuning large language models instruction tuning LoRA"
max_results: 10
sort_by: submittedDate

Federated tuning for large language models (LLMs) adapts a pretrained model to new user instructions or downstream tasks using decentralized private data without moving raw data off-device. In the current literature, it spans federated instruction tuning, parameter-efficient fine-tuning (PEFT), split learning, asynchronous optimization, multilingual personalization, and resource-aware adaptation for heterogeneous edge systems. The first explicit exploration of federated instruction tuning, FedIT, established the basic pattern of freezing the backbone, training LoRA adapters locally, and aggregating adapter parameters with FedAvg; subsequent work extended that recipe to data-efficient coreset selection, heterogeneous model architectures, tensorized adapters, and communication-aware aggregation [2305.05644][2410.10926].

## 1. Historical emergence and problem formulation

A common starting point is the federated objective
$$
\min_{\theta} \sum_{i=1}^N p_i \,\mathbb{E}_{x\sim D_i}\big[\ell(f_{\theta}(x))\big],
$$
with $p_i$ determined by client participation or data size, local optimization on private data, and server-side aggregation of updates rather than raw samples [2410.10926]. In early instruction-tuning work, client data were explicitly non-IID, often differing in instruction category, domain, or sample volume, and PEFT was adopted because full-model fine-tuning was infeasible on edge devices [2305.05644]. A later comparative analysis organized federated LLM fine-tuning into three broad frameworks: FedLLMs, where clients upload tunable parameters or gradients; KD-FedLLMs, where clients and server exchange logits on public data; and Split-FedLLMs, where the model is partitioned between client and server to trade communication for lower client compute [2501.04436].

| Framework | Main mechanism | Representative outcome |
|---|---|---|
| FedIT [2305.05644] | Federated instruction tuning with LoRA and FedAvg | First exploration of FL-based instruction tuning |
| FedHDS [2410.10926] | Hierarchical intra-/inter-client coreset selection | +10.72% average Rouge-L over best full-data baseline with <1.5% of samples |
| FedPipe [2404.06448] | Importance-driven LoRA placement, heterogeneous rank/batch sizing, quantization | Higher accuracy and no inference latency |
| FLoRG [2602.17095] | Single-factor low-rank updates, Gram aggregation, Procrustes alignment | Up to $2041\times$ lower communication |
| FedAMoLE [2411.19128] | Heterogeneous mixtures of LoRA experts with reverse selection | 5.14% average client-side performance gain |
| AlignFed [2606.08197] | Alignment-aware asynchronous aggregation | Stable async optimization under update staleness |

The scope of federated tuning is correspondingly broad. It includes instruction-following generation on Dolly-15K and Natural Instructions, classification on GLUE and Banking77, question answering on SQuAD and medical QA benchmarks, summarization with FLAN-T5, and domain-specific adaptation in code, medical, financial, and mathematical settings [2410.10926][2501.04436][2310.03150][2409.20135].

## 2. PEFT parameterization and aggregation mechanics

Most contemporary systems rely on LoRA or closely related PEFT modules. In its standard form, a frozen weight matrix is adapted as
$$
W = W_0 + BA,
$$
with only the low-rank factors trained and communicated. FedPipe uses this design to avoid added inference latency, because LoRA does not insert extra layers, and automates which attention projections to adapt, how large each rank should be, and which batch size each client can afford under its local FLOPs budget [2404.06448]. In the same line, FedIT applied LoRA to all linear layers of LLaMA 7B with rank $r=8$, yielding 17.9M trainable parameters, or 0.26% of the full model [2305.05644].

Aggregation, however, is not a trivial detail. FLoRG formalizes two LoRA-specific pathologies in federated settings. First, separate aggregation of factors is biased because
$$
\left(\frac{1}{N}\sum_n B_n\right)\left(\frac{1}{N}\sum_n A_n\right)\neq \frac{1}{N}\sum_n (B_nA_n).
$$
Second, even if the server aggregates products and refactorizes them, the decomposition is non-unique and induces decomposition drift across rounds. FLoRG replaces the two-factor update with a single trainable low-rank factor whose Gram matrix is aggregated,
$$
Q^{t+1}=\frac{1}{N}\sum_{n=1}^N (A_n^{t+1/2})^\top A_n^{t+1/2},
$$
and then uses orthogonal Procrustes alignment to keep the reconstructed factor consistent with the previous round. The paper proves that this alignment removes the drift term from its convergence bound and reports up to $2041\times$ lower communication overhead while outperforming five federated PEFT baselines [2602.17095].

Alternative parameterizations extend beyond plain FedAvg. FedTT replaces dense adapter matrices with Tensor-Train cores and inserts tensorized adapters after both attention and MLP sublayers. FedTT is designed for both cross-silo and cross-device FL, while FedTT+ further freezes subsets of TT cores to mitigate non-IID aggregation mismatch. Across BERT and LLaMA models, the method achieves up to $10\times$ lower communication cost while matching or slightly exceeding federated LoRA baselines in several settings [2410.13097].

## 3. Data efficiency, instruction coverage, and sample selection

A distinct line of work argues that the primary inefficiency in federated tuning is not merely parameter count but the indiscriminate use of all local samples. FedHDS addresses this with a federated hierarchical data selection procedure. Each client extracts last-token hidden states from all Transformer layers, fuses them with t-SNE to $k=2$, clusters them with HDBSCAN, and selects one representative sample per cluster. The server then clusters the client centroids again, chooses non-redundant groups across clients, and instructs each client to form a coreset from the selected local representatives. In six scenarios, FedHDS and FedHDS-Turbo improved Rouge-L on held-out tasks relative to full-data federated baselines while using less than 1.5% of samples; the reported average relative gain over the best full-data baseline is +10.72%, and the method improves training efficiency by up to tens of times [2410.10926].

FedDCA targets a different bottleneck: domain coverage in federated domain-specific instruction tuning. Its central empirical claim is that cross-client domain coverage, rather than data heterogeneity, drives performance in FedDIT. The method therefore clusters local instruction embeddings into client centers, greedily selects centers that maximize coverage while avoiding redundancy, and retrieves public instructions from a server-side multi-domain corpus using cosine similarity. Under FedAvg, the average score across six tasks rises from 48.99 with local-only data and 51.71 with direct retrieval to 56.56 with FedDCA; FedDCA+FedProx reaches 57.54. The lighter FedDCA* replaces the client encoder with a smaller model and adds server-side feature alignment, reducing public dataset encoding time from 15 min 46 s to 5 min 02 s while retaining most of the coverage gain [2409.20135].

These results make two methodological points. First, federated tuning is increasingly coupled to client-side or server-guided data selection rather than uniform sample traversal. Second, representativeness is treated in several papers as a privacy-compatible proxy for informativeness because it can be implemented with embeddings, centroids, or public retrieval rather than shared gradients or raw examples [2410.10926][2409.20135].

## 4. Personalization, heterogeneity, and fairness

Uniform adapter architectures are often inadequate when clients differ strongly in domain, language, label skew, or resource budget. HierFedLoRA addresses this by partitioning devices into near-IID groups and jointly optimizing two group-level controls: intra-group aggregation frequency and fine-tuning depth, defined as the number of continuous transformer layers from the output to which LoRA is applied. On a physical platform with 80 commercial devices, the method improves final accuracy by 1.6% to 4.2% and speeds up fine-tuning by at least $2.1\times$ compared to strong baselines [2503.21213].

FedP$^2$EFT tackles personalization from the perspective of PEFT structure learning. Instead of manually setting LoRA ranks and placements, it trains a global personalization-strategy generator that maps per-layer feature statistics to client-specific rank scales and combines it with Bayesian sparse rank selection on-device. Only the small PSG parameters are federated, amounting to about 0.2% of mBERT and about 0.16% of Llama-3.2-3B parameters, while the resulting client-specific structures remain local. Across XNLI, MasakhaNEWS, and Fed-Aya multilingual instruction tuning, the method consistently outperforms both non-federated structure search baselines and federated learning-rate personalization [2502.04387].

FedAMoLE introduces architectural personalization through heterogeneous mixtures of LoRA experts. Each module contains a shared expert and a variable number of domain experts; routing depends on a shared token projection, and expert assignment is updated through a reverse selection-based expert assignment strategy that lets experts choose clients according to data-driven relevance. Across five scenarios, FedAMoLE improves client-side performance by an average of 5.14% while keeping per-client upload and download near 12.5 MB per round for a LLaMA-3.2-1B backbone [2411.19128].

Other personalization strategies are more explicitly multi-task or fairness-oriented. MIRA replaces global averaging with Laplacian regularization over a client graph,
$$
J(\mathbf{W}) = \sum_{i=1}^{N} f_i(\mathbf{w}_i) + \lambda\, \mathbf{W}^\top (\mathbf{L}\otimes \mathbf{I}_d)\mathbf{W},
$$
thereby pulling related client adapters together without collapsing them into a single global model; it reports lower local loss for each client while maintaining comparable global performance [2410.15524]. In healthcare, federated LoRA is evaluated with Macro-Acc, Min-Acc, and H-mean to operationalize fairness as improvement of the weakest-performing client. On Qwen3-1.7B, federated LoRA increases Macro-Acc from 52.44 to 53.56, Min-Acc from 44.68 to 46.40, and H-mean from 50.45 to 52.03 relative to the best single-client LoRA baseline under an extreme non-IID, multilingual medical QA setup [2510.00543].

## 5. Systems co-design: communication, edge hardware, split learning, and asynchrony

A large fraction of the literature treats federated tuning as a systems problem as much as an optimization problem. FedPipe is representative: it automates selection of the most important attention weights, assigns heterogeneous LoRA ranks and batch sizes under client-specific compute budgets, aggregates only $\Delta W_i=B_iA_i$, and quantizes frozen pretrained weights with NF4, NF8, or FP16 to fit memory budgets. On GPT-2 medium under a heterogeneous setting, it reports BLEU 69.4 with 0.37M trainable parameters, outperforming LoRA, FedAdapter, and full fine-tuning while preserving inference speed because LoRA adds no extra layers [2404.06448].

Split architectures push the compute burden further off device. FedsLLM combines split federated learning with LoRA, jointly optimizing subchannel allocation, power control, split position, and LoRA rank to minimize total training delay over wireless links. On GPT-2 small, the method matches centralized LoRA within 0.001 perplexity across tested ranks while substantially reducing client-side computation and lowering training latency relative to unoptimized baselines [2504.14667]. FedCoLLM uses a different two-model split: a server-side LLM and client-side SLMs are co-tuned with lightweight adapters and bidirectional distillation on an auxiliary server dataset. Its reported payloads are 0.23–0.24% of the corresponding full-model sizes, and the server LLM approaches direct fine-tuning on client data while the client SLMs improve over Standalone and FedAvg baselines [2411.11707].

Asynchrony becomes essential once stragglers and version skew dominate. AlignFed is an asynchronous framework for 8B-class LLMs that groups updates by originating version, aligns stale updates to the current semantic space using a mini-batch calibration set, and aggregates them with weights that combine freshness, participation frequency, and update strength. In experiments on GSM8K, CodeAlpaca, and Dolly, it stabilizes asynchronous optimization under high staleness and improves on buffered asynchronous baselines [2606.08197].

Hardware-centric measurement studies show why these systems choices matter. On ten NVIDIA Jetson AGX Orin 64 GB devices, federated fine-tuning of FLAN-T5 with LoRA made communication viable where full-model FL was not: PEFT reduced communicated parameters by more than 99% and improved the compute-to-communication granularity ratio by up to roughly $110\times$, while FedAdamW materially outperformed FedAvg in convergence speed. The same study also found that FLAN-T5 XL was not feasible on current Orin-class devices and that communication energy can dominate total energy consumption even when client-side compute is optimized [2310.03150].

Specialized architectures push these ideas further. FLUX targets MoE-based LLMs on consumer-grade GPUs by profiling expert activation with quantized models, merging non-tuning experts in a layer-aware way, and dynamically selecting which experts to tune; it reports up to 4.75X speedup in time-to-accuracy [2508.19078].

## 6. Privacy, IP protection, and open research directions

Most federated LLM tuning papers rely first on architectural privacy: raw data remain local, and only adapters, factors, logits, activations, or centroids are exchanged. Formal privacy mechanisms are less uniformly adopted. FedHDS is one of the few methods that explicitly supports a differential privacy variant: clients share only low-dimensional centroids, tanh-normalized to $[-1,1]$, and the released centroid vector can be perturbed with Gaussian noise under an $(\varepsilon,\delta)$-DP guarantee [2410.10926]. By contrast, several frameworks explicitly note that they do not implement secure aggregation or DP in their reported experiments, even when they are compatible with such add-ons [2305.05644][2404.06448].

Another strand treats model intellectual property as a first-class constraint. FedProxy frames federated LLM adaptation as a trilemma involving LLM IP protection, client privacy, and non-IID performance. Its solution is to compress the proprietary LLM into a proxy small language model, federate only that proxy, and then fuse the updated proxy back into the full LLM by direct parameter replacement. On homogeneous tasks with LLaMA2-7B at 50% compression, FedProxy reaches 90.1% of centralized fine-tuning on QA and 90.7% on GLUE while substantially outperforming Offsite-Tuning baselines [2604.19015].

Knowledge reuse is also emerging as a privacy- and efficiency-compatible direction. SmartFed decomposes existing LoRA modules into rank-wise experts, activates them sparsely through a Mixture of Rank-Wise Experts router, and allocates expert budgets elastically across matrices with EEQA. Because only lightweight routers are trained on clients, the framework reduces communication by up to 31.47× and converges up to 3.95× faster while reusing public LoRA knowledge rather than retraining from scratch [2512.00902]. This suggests that future federated tuning systems may increasingly combine PEFT reuse, sparse activation, and personalization rather than treating every task as a new federated optimization problem.

The open problems are recurrent across the literature. They include stronger privacy mechanisms such as secure aggregation for centroids or adapter updates, private clustering, and DP noise that does not erase utility; better robustness to noisy data, adversarial clients, and severe non-IID drift; fairer contribution mechanisms for minority-task or low-resource clients; stronger theory for coreset selection, asynchronous semantics, and non-IID convergence; and broader scaling to multimodal, MoE, and very-large-model settings [2410.10926][2606.08197][2604.19015][2508.19078]. Across these directions, federated tuning has moved from a straightforward “FedAvg plus LoRA” recipe to a research area defined by fine-grained control over data selection, aggregation geometry, adapter architecture, communication format, and client heterogeneity.

Source: https://www.emergentmind.com/topics/federated-tuning-for-llms