Federated Tuning for LLMs

• Federated tuning for LLMs is defined as adapting pre-trained models using decentralized private data and parameter efficient techniques like LoRA.
• It employs aggregation methods such as FedAvg alongside advanced strategies like tensorized adapters, data selection, and asynchronous optimization.
• Recent studies report improvements in communication efficiency, personalization, and fairness across diverse domains such as medical, financial, and coding applications.

Searching arXiv for recent federated LLM tuning papers to ground the article. to=arxiv.search 北京赛车如何json code ￣第四色{ "query": "\"federated\" AND (\"LLM\" OR LLM) AND (fine-tuning OR tuning OR instruction)", "max_results": 10, "sort_by": "submittedDate" } to=arxiv.search ／久久িদ code 天天中彩票中奖了oml query: "federated tuning LLMs instruction tuning LoRA" max_results: 10 sort_by: submittedDate

Federated tuning for LLMs adapts a pretrained model to new user instructions or downstream tasks using decentralized private data without moving raw data off-device. In the current literature, it spans federated instruction tuning, parameter-efficient fine-tuning (PEFT), split learning, asynchronous optimization, multilingual personalization, and resource-aware adaptation for heterogeneous edge systems. The first explicit exploration of federated instruction tuning, FedIT, established the basic pattern of freezing the backbone, training LoRA adapters locally, and aggregating adapter parameters with FedAvg; subsequent work extended that recipe to data-efficient coreset selection, heterogeneous model architectures, tensorized adapters, and communication-aware aggregation (Zhang et al., 2023, Qin et al., 2024).

1. Historical emergence and problem formulation

A common starting point is the federated objective

$$\min_{\theta} \sum_{i=1}^N p_i \,\mathbb{E}_{x\sim D_i}\big[\ell(f_{\theta}(x))\big],$$

with $p_i$ determined by client participation or data size, local optimization on private data, and server-side aggregation of updates rather than raw samples (Qin et al., 2024). In early instruction-tuning work, client data were explicitly non-IID, often differing in instruction category, domain, or sample volume, and PEFT was adopted because full-model fine-tuning was infeasible on edge devices (Zhang et al., 2023). A later comparative analysis organized federated LLM fine-tuning into three broad frameworks: FedLLMs, where clients upload tunable parameters or gradients; KD-FedLLMs, where clients and server exchange logits on public data; and Split-FedLLMs, where the model is partitioned between client and server to trade communication for lower client compute (Yan et al., 8 Jan 2025).

Framework	Main mechanism	Representative outcome
FedIT (Zhang et al., 2023)	Federated instruction tuning with LoRA and FedAvg	First exploration of FL-based instruction tuning
FedHDS (Qin et al., 2024)	Hierarchical intra-/inter-client coreset selection	+10.72% average Rouge-L over best full-data baseline with <1.5% of samples
FedPipe (Fang et al., 2024)	Importance-driven LoRA placement, heterogeneous rank/batch sizing, quantization	Higher accuracy and no inference latency
FLoRG (Meng et al., 19 Feb 2026)	Single-factor low-rank updates, Gram aggregation, Procrustes alignment	Up to $2041\times$ lower communication
FedAMoLE (Zhang et al., 2024)	Heterogeneous mixtures of LoRA experts with reverse selection	5.14% average client-side performance gain
AlignFed (Wang et al., 6 Jun 2026)	Alignment-aware asynchronous aggregation	Stable async optimization under update staleness

The scope of federated tuning is correspondingly broad. It includes instruction-following generation on Dolly-15K and Natural Instructions, classification on GLUE and Banking77, question answering on SQuAD and medical QA benchmarks, summarization with FLAN-T5, and domain-specific adaptation in code, medical, financial, and mathematical settings (Qin et al., 2024, Yan et al., 8 Jan 2025, Woisetschläger et al., 2023, Wang et al., 2024).

2. PEFT parameterization and aggregation mechanics

Most contemporary systems rely on LoRA or closely related PEFT modules. In its standard form, a frozen weight matrix is adapted as

$W = W_0 + BA,$

with only the low-rank factors trained and communicated. FedPipe uses this design to avoid added inference latency, because LoRA does not insert extra layers, and automates which attention projections to adapt, how large each rank should be, and which batch size each client can afford under its local FLOPs budget (Fang et al., 2024). In the same line, FedIT applied LoRA to all linear layers of LLaMA 7B with rank $r=8$, yielding 17.9M trainable parameters, or 0.26% of the full model (Zhang et al., 2023).

Aggregation, however, is not a trivial detail. FLoRG formalizes two LoRA-specific pathologies in federated settings. First, separate aggregation of factors is biased because

$$\left(\frac{1}{N}\sum_n B_n\right)\left(\frac{1}{N}\sum_n A_n\right)\neq \frac{1}{N}\sum_n (B_nA_n).$$

Second, even if the server aggregates products and refactorizes them, the decomposition is non-unique and induces decomposition drift across rounds. FLoRG replaces the two-factor update with a single trainable low-rank factor whose Gram matrix is aggregated,

$$Q^{t+1}=\frac{1}{N}\sum_{n=1}^N (A_n^{t+1/2})^\top A_n^{t+1/2},$$

and then uses orthogonal Procrustes alignment to keep the reconstructed factor consistent with the previous round. The paper proves that this alignment removes the drift term from its convergence bound and reports up to $2041\times$ lower communication overhead while outperforming five federated PEFT baselines (Meng et al., 19 Feb 2026).

Alternative parameterizations extend beyond plain FedAvg. FedTT replaces dense adapter matrices with Tensor-Train cores and inserts tensorized adapters after both attention and MLP sublayers. FedTT is designed for both cross-silo and cross-device FL, while FedTT+ further freezes subsets of TT cores to mitigate non-IID aggregation mismatch. Across BERT and LLaMA models, the method achieves up to $10\times$ lower communication cost while matching or slightly exceeding federated LoRA baselines in several settings (Ghiasvand et al., 2024).

3. Data efficiency, instruction coverage, and sample selection

A distinct line of work argues that the primary inefficiency in federated tuning is not merely parameter count but the indiscriminate use of all local samples. FedHDS addresses this with a federated hierarchical data selection procedure. Each client extracts last-token hidden states from all Transformer layers, fuses them with t-SNE to $k=2$, clusters them with HDBSCAN, and selects one representative sample per cluster. The server then clusters the client centroids again, chooses non-redundant groups across clients, and instructs each client to form a coreset from the selected local representatives. In six scenarios, FedHDS and FedHDS-Turbo improved Rouge-L on held-out tasks relative to full-data federated baselines while using less than 1.5% of samples; the reported average relative gain over the best full-data baseline is +10.72%, and the method improves training efficiency by up to tens of times (Qin et al., 2024).

FedDCA targets a different bottleneck: domain coverage in federated domain-specific instruction tuning. Its central empirical claim is that cross-client domain coverage, rather than data heterogeneity, drives performance in FedDIT. The method therefore clusters local instruction embeddings into client centers, greedily selects centers that maximize coverage while avoiding redundancy, and retrieves public instructions from a server-side multi-domain corpus using cosine similarity. Under FedAvg, the average score across six tasks rises from 48.99 with local-only data and 51.71 with direct retrieval to 56.56 with FedDCA; FedDCA+FedProx reaches 57.54. The lighter FedDCA* replaces the client encoder with a smaller model and adds server-side feature alignment, reducing public dataset encoding time from 15 min 46 s to 5 min 02 s while retaining most of the coverage gain (Wang et al., 2024).

These results make two methodological points. First, federated tuning is increasingly coupled to client-side or server-guided data selection rather than uniform sample traversal. Second, representativeness is treated in several papers as a privacy-compatible proxy for informativeness because it can be implemented with embeddings, centroids, or public retrieval rather than shared gradients or raw examples (Qin et al., 2024, Wang et al., 2024).

4. Personalization, heterogeneity, and fairness

Uniform adapter architectures are often inadequate when clients differ strongly in domain, language, label skew, or resource budget. HierFedLoRA addresses this by partitioning devices into near-IID groups and jointly optimizing two group-level controls: intra-group aggregation frequency and fine-tuning depth, defined as the number of continuous transformer layers from the output to which LoRA is applied. On a physical platform with 80 commercial devices, the method improves final accuracy by 1.6% to 4.2% and speeds up fine-tuning by at least $p_i$0 compared to strong baselines (Liu et al., 27 Mar 2025).

FedP$p_i$1EFT tackles personalization from the perspective of PEFT structure learning. Instead of manually setting LoRA ranks and placements, it trains a global personalization-strategy generator that maps per-layer feature statistics to client-specific rank scales and combines it with Bayesian sparse rank selection on-device. Only the small PSG parameters are federated, amounting to about 0.2% of mBERT and about 0.16% of Llama-3.2-3B parameters, while the resulting client-specific structures remain local. Across XNLI, MasakhaNEWS, and Fed-Aya multilingual instruction tuning, the method consistently outperforms both non-federated structure search baselines and federated learning-rate personalization (Lee et al., 5 Feb 2025).

FedAMoLE introduces architectural personalization through heterogeneous mixtures of LoRA experts. Each module contains a shared expert and a variable number of domain experts; routing depends on a shared token projection, and expert assignment is updated through a reverse selection-based expert assignment strategy that lets experts choose clients according to data-driven relevance. Across five scenarios, FedAMoLE improves client-side performance by an average of 5.14% while keeping per-client upload and download near 12.5 MB per round for a LLaMA-3.2-1B backbone (Zhang et al., 2024).

Other personalization strategies are more explicitly multi-task or fairness-oriented. MIRA replaces global averaging with Laplacian regularization over a client graph,

$p_i$2

thereby pulling related client adapters together without collapsing them into a single global model; it reports lower local loss for each client while maintaining comparable global performance (Elbakary et al., 2024). In healthcare, federated LoRA is evaluated with Macro-Acc, Min-Acc, and H-mean to operationalize fairness as improvement of the weakest-performing client. On Qwen3-1.7B, federated LoRA increases Macro-Acc from 52.44 to 53.56, Min-Acc from 44.68 to 46.40, and H-mean from 50.45 to 52.03 relative to the best single-client LoRA baseline under an extreme non-IID, multilingual medical QA setup (Chen et al., 1 Oct 2025).

5. Systems co-design: communication, edge hardware, split learning, and asynchrony

A large fraction of the literature treats federated tuning as a systems problem as much as an optimization problem. FedPipe is representative: it automates selection of the most important attention weights, assigns heterogeneous LoRA ranks and batch sizes under client-specific compute budgets, aggregates only $p_i$3, and quantizes frozen pretrained weights with NF4, NF8, or FP16 to fit memory budgets. On GPT-2 medium under a heterogeneous setting, it reports BLEU 69.4 with 0.37M trainable parameters, outperforming LoRA, FedAdapter, and full fine-tuning while preserving inference speed because LoRA adds no extra layers (Fang et al., 2024).

Split architectures push the compute burden further off device. FedsLLM combines split federated learning with LoRA, jointly optimizing subchannel allocation, power control, split position, and LoRA rank to minimize total training delay over wireless links. On GPT-2 small, the method matches centralized LoRA within 0.001 perplexity across tested ranks while substantially reducing client-side computation and lowering training latency relative to unoptimized baselines (Zhao et al., 20 Apr 2025). FedCoLLM uses a different two-model split: a server-side LLM and client-side SLMs are co-tuned with lightweight adapters and bidirectional distillation on an auxiliary server dataset. Its reported payloads are 0.23–0.24% of the corresponding full-model sizes, and the server LLM approaches direct fine-tuning on client data while the client SLMs improve over Standalone and FedAvg baselines (Fan et al., 2024).

Asynchrony becomes essential once stragglers and version skew dominate. AlignFed is an asynchronous framework for 8B-class LLMs that groups updates by originating version, aligns stale updates to the current semantic space using a mini-batch calibration set, and aggregates them with weights that combine freshness, participation frequency, and update strength. In experiments on GSM8K, CodeAlpaca, and Dolly, it stabilizes asynchronous optimization under high staleness and improves on buffered asynchronous baselines (Wang et al., 6 Jun 2026).

Hardware-centric measurement studies show why these systems choices matter. On ten NVIDIA Jetson AGX Orin 64 GB devices, federated fine-tuning of FLAN-T5 with LoRA made communication viable where full-model FL was not: PEFT reduced communicated parameters by more than 99% and improved the compute-to-communication granularity ratio by up to roughly $p_i$4, while FedAdamW materially outperformed FedAvg in convergence speed. The same study also found that FLAN-T5 XL was not feasible on current Orin-class devices and that communication energy can dominate total energy consumption even when client-side compute is optimized (Woisetschläger et al., 2023).

Specialized architectures push these ideas further. FLUX targets MoE-based LLMs on consumer-grade GPUs by profiling expert activation with quantized models, merging non-tuning experts in a layer-aware way, and dynamically selecting which experts to tune; it reports up to 4.75X speedup in time-to-accuracy (Chen et al., 26 Aug 2025).

6. Privacy, IP protection, and open research directions

Most federated LLM tuning papers rely first on architectural privacy: raw data remain local, and only adapters, factors, logits, activations, or centroids are exchanged. Formal privacy mechanisms are less uniformly adopted. FedHDS is one of the few methods that explicitly supports a differential privacy variant: clients share only low-dimensional centroids, tanh-normalized to $p_i$5, and the released centroid vector can be perturbed with Gaussian noise under an $p_i$6-DP guarantee (Qin et al., 2024). By contrast, several frameworks explicitly note that they do not implement secure aggregation or DP in their reported experiments, even when they are compatible with such add-ons (Zhang et al., 2023, Fang et al., 2024).

Another strand treats model intellectual property as a first-class constraint. FedProxy frames federated LLM adaptation as a trilemma involving LLM IP protection, client privacy, and non-IID performance. Its solution is to compress the proprietary LLM into a proxy small LLM, federate only that proxy, and then fuse the updated proxy back into the full LLM by direct parameter replacement. On homogeneous tasks with LLaMA2-7B at 50% compression, FedProxy reaches 90.1% of centralized fine-tuning on QA and 90.7% on GLUE while substantially outperforming Offsite-Tuning baselines (Fan et al., 21 Apr 2026).

Knowledge reuse is also emerging as a privacy- and efficiency-compatible direction. SmartFed decomposes existing LoRA modules into rank-wise experts, activates them sparsely through a Mixture of Rank-Wise Experts router, and allocates expert budgets elastically across matrices with EEQA. Because only lightweight routers are trained on clients, the framework reduces communication by up to 31.47× and converges up to 3.95× faster while reusing public LoRA knowledge rather than retraining from scratch (Wu et al., 30 Nov 2025). This suggests that future federated tuning systems may increasingly combine PEFT reuse, sparse activation, and personalization rather than treating every task as a new federated optimization problem.

The open problems are recurrent across the literature. They include stronger privacy mechanisms such as secure aggregation for centroids or adapter updates, private clustering, and DP noise that does not erase utility; better robustness to noisy data, adversarial clients, and severe non-IID drift; fairer contribution mechanisms for minority-task or low-resource clients; stronger theory for coreset selection, asynchronous semantics, and non-IID convergence; and broader scaling to multimodal, MoE, and very-large-model settings (Qin et al., 2024, Wang et al., 6 Jun 2026, Fan et al., 21 Apr 2026, Chen et al., 26 Aug 2025). Across these directions, federated tuning has moved from a straightforward “FedAvg plus LoRA” recipe to a research area defined by fine-grained control over data selection, aggregation geometry, adapter architecture, communication format, and client heterogeneity.