FedLEASE: Federated Fine-Tuning Framework

• FedLEASE is a federated fine-tuning framework that uses parameter-efficient LoRA experts to address heterogeneous client domains.
• It employs agglomerative clustering on LoRA B matrices to adaptively allocate experts based on cross-client similarity.
• An adaptive top-M routing mechanism allows flexible expert selection during training, achieving notable gains on GLUE and FLAN benchmarks.

Searching arXiv for FedLEASE and closely related papers to ground the article with current records. FedLEASE is a federated fine-tuning framework for LLMs that combines parameter-efficient adaptation with expert specialization under heterogeneous client distributions. The name expands to Federated adaptive LoRA Expert Allocation and SElection, and the method is designed for a setting in which multiple organizations hold private domain-specific data, a central server coordinates training, and only model updates—not raw data—are exchanged. Its two stated targets are adaptive allocation of LoRA experts across heterogeneous clients and adaptive selection of those experts during inference and training, with the aim of improving performance in multi-domain federated fine-tuning while maintaining communication efficiency (Wang et al., 18 Sep 2025).

1. Definition and problem setting

FedLEASE addresses federated fine-tuning rather than full-model federated optimization. The underlying motivation is that full fine-tuning of LLMs is expensive in memory, computation, and communication, while a single shared LoRA module is often inadequate when clients differ substantially in task or domain. The paper therefore places FedLEASE in a heterogeneous federated learning setting with $N$ clients, local datasets

$$\mathcal{D}_i=\{(x_j^i,y_j^i)\}_{j=1}^{|\mathcal{D}_i|},$$

and a central server coordinating training over private client data (Wang et al., 18 Sep 2025).

The method adopts LoRA as its parameter-efficient fine-tuning substrate. For a frozen pre-trained weight $W_0\in\mathbb{R}^{l\times d}$, LoRA introduces trainable low-rank matrices

$$A\in\mathbb{R}^{r\times d},\qquad B\in\mathbb{R}^{l\times r}, \quad r\ll \min(d,l),$$

and computes

$y = W_0x + BAx.$

Within FedLEASE, the frozen backbone remains shared, while the trainable adaptation is organized into multiple experts rather than a single global adapter (Wang et al., 18 Sep 2025).

The paper frames the central design question as a double failure of simple baselines. A single global LoRA module can be harmful when clients perform very different tasks, but one LoRA per client is also not generally desirable. FedLEASE is proposed as a middle regime: similar clients should share adaptation through clustered experts, while clients should still be able to use more than one expert when their input or task characteristics warrant it (Wang et al., 18 Sep 2025).

2. Adaptive expert allocation through client clustering

FedLEASE begins with a short local warm-up phase. Each client independently trains its own LoRA module for $E$ local epochs, after which the server receives the local LoRA parameters and computes cross-client similarity. The paper states that the LoRA $B$ matrix captures task- or domain-specific information better than $A$, whereas $A$ appears to encode more general linguistic features. For that reason, client similarity is computed from the per-layer $B$ matrices rather than from $$\mathcal{D}_i=\{(x_j^i,y_j^i)\}_{j=1}^{|\mathcal{D}_i|},$$0 or from a fixed single global embedding (Wang et al., 18 Sep 2025).

For clients $$\mathcal{D}_i=\{(x_j^i,y_j^i)\}_{j=1}^{|\mathcal{D}_i|},$$1 and $$\mathcal{D}_i=\{(x_j^i,y_j^i)\}_{j=1}^{|\mathcal{D}_i|},$$2, the distance metric is the average cosine dissimilarity across LoRA-equipped layers: $$\mathcal{D}_i=\{(x_j^i,y_j^i)\}_{j=1}^{|\mathcal{D}_i|},$$3 Lower values indicate more similar adaptation patterns. The server then applies Agglomerative Hierarchical Clustering to the clients. For each candidate number of clusters $$\mathcal{D}_i=\{(x_j^i,y_j^i)\}_{j=1}^{|\mathcal{D}_i|},$$4, clustering quality is evaluated by the average silhouette coefficient

$$\mathcal{D}_i=\{(x_j^i,y_j^i)\}_{j=1}^{|\mathcal{D}_i|},$$5

and the number of experts is selected as

$$\mathcal{D}_i=\{(x_j^i,y_j^i)\}_{j=1}^{|\mathcal{D}_i|},$$6

This makes the expert pool size adaptive at initialization, but not dynamic across communication rounds (Wang et al., 18 Sep 2025).

Once clustering is fixed, each cluster is assigned one shared LoRA expert. Expert initialization is done by averaging the LoRA modules of the clients in that cluster: $$\mathcal{D}_i=\{(x_j^i,y_j^i)\}_{j=1}^{|\mathcal{D}_i|},$$7 Each client is assigned to exactly one cluster and hence one expert, while all experts are later distributed to all clients. The appendix reports that in the main 16-client GLUE experiment, clustering naturally selected 4 experts, matching the 4 task groups in that setting (Wang et al., 18 Sep 2025).

This allocation mechanism distinguishes FedLEASE from both globally shared federated LoRA methods and clustered FL methods that stop at hard partitioning. The clustering stage determines which clients jointly train an expert, but it does not force inference or forward propagation to use only that expert (Wang et al., 18 Sep 2025).

3. Adaptive expert selection with top-$$\mathcal{D}_i=\{(x_j^i,y_j^i)\}_{j=1}^{|\mathcal{D}_i|},$$8 routing

The second major component of FedLEASE is its adaptive expert-selection mechanism. After expert allocation, every client receives all $$\mathcal{D}_i=\{(x_j^i,y_j^i)\}_{j=1}^{|\mathcal{D}_i|},$$9 experts, but during local training it updates only its assigned expert together with a router. The purpose of the router is to permit cross-expert knowledge use without turning the system into a fully individualized or fully global model (Wang et al., 18 Sep 2025).

The paper contrasts this with a conventional MoE-style top-$W_0\in\mathbb{R}^{l\times d}$0 rule,

$W_0\in\mathbb{R}^{l\times d}$1

which requires manual choice of $W_0\in\mathbb{R}^{l\times d}$2 and may fail to include the client’s assigned expert. FedLEASE modifies the routing space rather than the aggregation weights. For a client assigned to expert $W_0\in\mathbb{R}^{l\times d}$3, the router output has dimension

$W_0\in\mathbb{R}^{l\times d}$4

and the first $W_0\in\mathbb{R}^{l\times d}$5 routing outputs all map to the assigned expert $W_0\in\mathbb{R}^{l\times d}$6, while the remaining $W_0\in\mathbb{R}^{l\times d}$7 outputs map to the non-assigned experts (Wang et al., 18 Sep 2025).

The resulting routing weights are

$W_0\in\mathbb{R}^{l\times d}$8

and the forward computation is written as

$W_0\in\mathbb{R}^{l\times d}$9

Because top-$$A\in\mathbb{R}^{r\times d},\qquad B\in\mathbb{R}^{l\times r}, \quad r\ll \min(d,l),$$0 is applied over positions rather than over distinct experts, multiple selected positions can correspond to the assigned expert. The number of distinct experts actually used can therefore vary from 1 to $$A\in\mathbb{R}^{r\times d},\qquad B\in\mathbb{R}^{l\times r}, \quad r\ll \min(d,l),$$1. This is the sense in which FedLEASE makes expert use adaptive: the expert pool size $$A\in\mathbb{R}^{r\times d},\qquad B\in\mathbb{R}^{l\times r}, \quad r\ll \min(d,l),$$2 is chosen by clustering, while the number of distinct experts used by a client is decided implicitly by the router during the forward pass (Wang et al., 18 Sep 2025).

The paper reports that different clients prefer different effective expert counts under fixed-top-$$A\in\mathbb{R}^{r\times d},\qquad B\in\mathbb{R}^{l\times r}, \quad r\ll \min(d,l),$$3 baselines; for example, QQP clients perform best with top-2, whereas MRPC clients perform best with top-4. FedLEASE’s adaptive top-$$A\in\mathbb{R}^{r\times d},\qquad B\in\mathbb{R}^{l\times r}, \quad r\ll \min(d,l),$$4 routing outperforms all fixed top-$$A\in\mathbb{R}^{r\times d},\qquad B\in\mathbb{R}^{l\times r}, \quad r\ll \min(d,l),$$5 settings in the reported experiments. Visualizations further show that deeper layers tend to activate more experts, harder tasks tend to use more experts, and clients in the same cluster exhibit similar but not identical routing patterns (Wang et al., 18 Sep 2025).

4. Federated training workflow and mathematical structure

FedLEASE has a two-phase training protocol. In the initialization phase, the server distributes the initial LoRA setup; each client performs brief local training; the server computes pairwise similarities from the $$A\in\mathbb{R}^{r\times d},\qquad B\in\mathbb{R}^{l\times r}, \quad r\ll \min(d,l),$$6 matrices; hierarchical clustering and silhouette scoring determine $$A\in\mathbb{R}^{r\times d},\qquad B\in\mathbb{R}^{l\times r}, \quad r\ll \min(d,l),$$7; and one expert is initialized per cluster by averaging client LoRAs. In the iterative federated phase, the server broadcasts all experts, each client trains only its assigned expert and router, and the server aggregates updates within cluster rather than globally across all clients (Wang et al., 18 Sep 2025).

The paper states the heterogeneous federated fine-tuning objective as

$$A\in\mathbb{R}^{r\times d},\qquad B\in\mathbb{R}^{l\times r}, \quad r\ll \min(d,l),$$8

with $$A\in\mathbb{R}^{r\times d},\qquad B\in\mathbb{R}^{l\times r}, \quad r\ll \min(d,l),$$9. Operationally, the important point is that FedLEASE does not enforce a single shared personalized model for all clients; instead it permits cluster-specific expert parameters and router-mediated cross-expert use (Wang et al., 18 Sep 2025).

Cluster-wise aggregation is performed by averaging expert parameters over the clients assigned to that expert: $y = W_0x + BAx.$0 The appendix also defines cluster/expert parameters

$y = W_0x + BAx.$1

cluster-level aggregation

$y = W_0x + BAx.$2

and a convergence claim of the form

$y = W_0x + BAx.$3

Under the condition $y = W_0x + BAx.$4, the cluster models converge to a stable point according to the appendix’s analysis (Wang et al., 18 Sep 2025).

The framework is presented as communication-efficient because only LoRA parameters are trained and transmitted. Reported trainable-parameter fractions are 0.2075% for FedLEASE on GLUE / RoBERTa-Large, compared with 0.2213% for FedIT/FedSA/FedDPA and 0.1107% for FFA-LoRA; and 0.0584% for FedLEASE on FLAN / LLaMA-2-7B, compared with 0.0622% for FedIT/FedSA/FedDPA/IFCA+LoRA and 0.0311% for FFA-LoRA. The paper explicitly claims that the gains are achieved without additional computational or communication overhead relative to comparable LoRA-based baselines (Wang et al., 18 Sep 2025).

Several operational details are left less formal than the core clustering and routing rules. The text states that router networks are aggregated within cluster and that this outperforms maintaining separate routers per client, but the main equations do not provide a separate router-aggregation formula. The clustering is also static after initialization: the paper does not specify an online reclustering rule or a distance threshold for updating assignments during training (Wang et al., 18 Sep 2025).

5. Experimental evaluation

FedLEASE is evaluated on both natural language understanding and natural language generation workloads. The NLU experiments use SST-2, QNLI, MRPC, and QQP, with 16 clients total, 4 clients per dataset, and RoBERTa-Large (355M) as the backbone. The NLG experiments use FLAN task groups—Text Editing, Struct to Text, Sentiment Analysis, and Commonsense Reasoning—with 8 clients total, 2 clients per dataset, 600 training samples and 200 test samples per client, and LLaMA-2-7B, 8-bit quantized, as the backbone. The reported metrics are Accuracy for NLU and ROUGE-1 for NLG (Wang et al., 18 Sep 2025).

The baselines are FedIT, FedSA, FFA-LoRA, FedDPA, and IFCA + LoRA. Across the main benchmarks, FedLEASE reports the following average results:

Benchmark	Strongest baseline	FedLEASE
GLUE average accuracy	84.60 (FedSA)	87.76
FLAN average ROUGE-1	60.20 (FedSA)	61.70

These correspond to gains of 3.16 points over the strongest baseline on GLUE and 1.50 points on FLAN. Per-task GLUE results for FedLEASE are 93.33 on SST-2, 87.22 on QNLI, 86.93 on MRPC, and 83.57 on QQP. On the FLAN side, the reported gains over the strongest baseline are +2.26 for Text Editing, +0.46 for Struct to Text, +1.43 for Sentiment Analysis, and +1.85 for Commonsense Reasoning (Wang et al., 18 Sep 2025).

The ablation study attributes the performance gains to both expert allocation and adaptive selection. On GLUE, FedLoRA-Single ($y = W_0x + BAx.$5) reaches 82.00, FedLoRA-Single ($y = W_0x + BAx.$6) reaches 83.84, FedLoRA-Individual (16 experts) reaches 80.69, FedLEASE without adaptive top-$y = W_0x + BAx.$7 reaches 85.91, and full FedLEASE reaches 87.76. The reported interpretation is that one expert is too coarse, one expert per client is too fragmented, clustered expert sharing is substantially better, and adaptive top-$y = W_0x + BAx.$8 provides an additional improvement beyond clustering alone (Wang et al., 18 Sep 2025).

Sensitivity analyses further report that FedLEASE remains strongest across tested LoRA ranks, scales from $y = W_0x + BAx.$9 clients with average 87.72 to $E$0 clients with average 87.13, and remains ahead of baselines under varying heterogeneity levels and additional label non-IID. For the upper bound on the number of experts, the paper reports: $E$1 gives final experts 2 and average 85.49; $E$2 gives final experts 3 and average 87.21; $E$3 gives final experts 4 and average 87.85; and $E$4 still gives final experts 4 and average 87.76. This indicates that a generous upper bound need not force over-fragmentation, whereas too small an upper bound degrades performance (Wang et al., 18 Sep 2025).

6. Relation to adjacent methods, naming ambiguity, and limitations

FedLEASE belongs to a family of federated methods whose names can be confused but whose technical settings differ sharply. It is distinct from FedLE, which studies battery-aware client selection for edge IoT networks and lifespan extension under battery constraints (Wu et al., 2023), and from FedLEO, which studies federated learning in LEO satellite constellations through intra-plane model propagation and sink satellite scheduling (Elmahallawy et al., 2023). It is also unrelated in substance to “Functional Encryption with Secure Key Leasing”, which studies quantum-cryptographic leasing of functional decryption keys rather than federated LoRA fine-tuning (Kitagawa et al., 2022).

Within federated LLM adaptation, the relevant comparisons are with single-shared LoRA methods, personalized methods, clustered FL, and MoE-style sharing. The paper positions FedIT as a single shared LoRA baseline, FedDPA as a personalized FL baseline, and IFCA+LoRA as a clustered FL baseline. FedLEASE differs from IFCA-style clustered FL because it uses clustering for expert training but still permits cross-cluster knowledge transfer through adaptive MoE routing. It differs from naive individualized MoE sharing because it does not retain one private expert per client; instead it first learns a compact expert set through clustering (Wang et al., 18 Sep 2025).

The method also comes with explicit limitations. The appendix states that FedLEASE assumes a static client population and fixed expert assignments after initial clustering. The paper suggests dynamic clustering, meta-routing strategies for non-stationary settings, extension to other PEFT methods beyond LoRA, and further communication optimizations as future directions. Additional underspecified points include the lack of an exact router-aggregation formula, the absence of a separate inference-time deployment protocol beyond using the trained router, and the absence of explicit communication- or computation-complexity expressions (Wang et al., 18 Sep 2025).

A common misconception is therefore to read FedLEASE as a generic “federated MoE” or as a dynamic reclustering system. The paper does not present it that way. The expert pool size $E$5 is adaptive only at initialization, cluster assignment remains fixed thereafter, every client receives all experts, and only the assigned expert is updated locally. The novelty lies in combining data-driven expert allocation with a routing mechanism that can vary the number of distinct experts used from 1 to $E$6 without manually choosing a fixed top-$E$7 (Wang et al., 18 Sep 2025).