---
title: 'FedLEASE: Federated Fine-Tuning Framework'
url: https://www.emergentmind.com/topics/fedlease
type: topic
---

# FedLEASE: Federated Fine-Tuning Framework

Searching arXiv for FedLEASE and closely related papers to ground the article with current records.
FedLEASE is a federated fine-tuning framework for large language models that combines parameter-efficient adaptation with expert specialization under heterogeneous client distributions. The name expands to **Federated adaptive LoRA Expert Allocation and SElection**, and the method is designed for a setting in which multiple organizations hold private domain-specific data, a central server coordinates training, and only model updates—not raw data—are exchanged. Its two stated targets are adaptive allocation of LoRA experts across heterogeneous clients and adaptive selection of those experts during inference and training, with the aim of improving performance in multi-domain federated fine-tuning while maintaining communication efficiency [2509.15087].

## 1. Definition and problem setting

FedLEASE addresses federated fine-tuning rather than full-model federated optimization. The underlying motivation is that full fine-tuning of large language models is expensive in memory, computation, and communication, while a single shared LoRA module is often inadequate when clients differ substantially in task or domain. The paper therefore places FedLEASE in a heterogeneous federated learning setting with \(N\) clients, local datasets
\[
\mathcal{D}_i=\{(x_j^i,y_j^i)\}_{j=1}^{|\mathcal{D}_i|},
\]
and a central server coordinating training over private client data [2509.15087].

The method adopts **LoRA** as its parameter-efficient fine-tuning substrate. For a frozen pre-trained weight \(W_0\in\mathbb{R}^{l\times d}\), LoRA introduces trainable low-rank matrices
\[
A\in\mathbb{R}^{r\times d},\qquad B\in\mathbb{R}^{l\times r}, \quad r\ll \min(d,l),
\]
and computes
\[
y = W_0x + BAx.
\]
Within FedLEASE, the frozen backbone remains shared, while the trainable adaptation is organized into multiple experts rather than a single global adapter [2509.15087].

The paper frames the central design question as a double failure of simple baselines. A single global LoRA module can be harmful when clients perform very different tasks, but one LoRA per client is also not generally desirable. FedLEASE is proposed as a middle regime: similar clients should share adaptation through clustered experts, while clients should still be able to use more than one expert when their input or task characteristics warrant it [2509.15087].

## 2. Adaptive expert allocation through client clustering

FedLEASE begins with a short local warm-up phase. Each client independently trains its own LoRA module for \(E\) local epochs, after which the server receives the local LoRA parameters and computes cross-client similarity. The paper states that the **LoRA \(B\) matrix** captures task- or domain-specific information better than \(A\), whereas \(A\) appears to encode more general linguistic features. For that reason, client similarity is computed from the per-layer \(B\) matrices rather than from \(A\) or from a fixed single global embedding [2509.15087].

For clients \(i\) and \(j\), the distance metric is the average cosine dissimilarity across LoRA-equipped layers:
\[
d(i,j)=\frac{1}{|L|}\sum_{l\in L}\left(1-\frac{\mathbf{B}_i^l\cdot \mathbf{B}_j^l}{\|\mathbf{B}_i^l\|\cdot \|\mathbf{B}_j^l\|}\right).
\]
Lower values indicate more similar adaptation patterns. The server then applies **Agglomerative Hierarchical Clustering** to the clients. For each candidate number of clusters \(k\in\{2,3,\dots,M_{\max}\}\), clustering quality is evaluated by the average silhouette coefficient
\[
S(k)=\frac{1}{N}\sum_{i=1}^N s^k(i),
\qquad
s^k(i)=\frac{b^k(i)-a^k(i)}{\max(a^k(i),b^k(i))},
\]
and the number of experts is selected as
\[
M=\arg\max_{2\le k\le M_{\max}} S(k).
\]
This makes the expert pool size adaptive at initialization, but not dynamic across communication rounds [2509.15087].

Once clustering is fixed, each cluster is assigned one shared LoRA expert. Expert initialization is done by averaging the LoRA modules of the clients in that cluster:
\[
A_j^{\text{expert}}=\frac{1}{|C_j^M|}\sum_{i\in C_j^M} A_i,\qquad
B_j^{\text{expert}}=\frac{1}{|C_j^M|}\sum_{i\in C_j^M} B_i.
\]
Each client is assigned to exactly one cluster and hence one expert, while all experts are later distributed to all clients. The appendix reports that in the main 16-client GLUE experiment, clustering naturally selected **4 experts**, matching the **4 task groups** in that setting [2509.15087].

This allocation mechanism distinguishes FedLEASE from both globally shared federated LoRA methods and clustered FL methods that stop at hard partitioning. The clustering stage determines which clients jointly train an expert, but it does not force inference or forward propagation to use only that expert [2509.15087].

## 3. Adaptive expert selection with top-\(M\) routing

The second major component of FedLEASE is its adaptive expert-selection mechanism. After expert allocation, every client receives **all \(M\) experts**, but during local training it updates only its **assigned expert** together with a router. The purpose of the router is to permit cross-expert knowledge use without turning the system into a fully individualized or fully global model [2509.15087].

The paper contrasts this with a conventional MoE-style top-\(k\) rule,
\[
y = W_0 x + \sum_{i \in \text{TopK}(\omega, k)} \omega_i B_i A_i x,
\qquad
\omega_i = \text{softmax}(G_i x),
\]
which requires manual choice of \(k\) and may fail to include the client’s assigned expert. FedLEASE modifies the routing space rather than the aggregation weights. For a client assigned to expert \(j\), the router output has dimension
\[
\mathbb{R}^{(2M-1)\times d},
\]
and the first \(M\) routing outputs all map to the assigned expert \(j\), while the remaining \(M-1\) outputs map to the non-assigned experts [2509.15087].

The resulting routing weights are
\[
\hat{\omega}=\text{softmax}(G_i x)\in \mathbb{R}^{2M-1},
\]
and the forward computation is written as
\[
y = W_0 x + \sum_{i \in \text{TopK}(\hat{\omega}, M)} \hat{\omega}_i \cdot
\begin{cases}
B_jA_jx, & \text{if } i < M \\
B_{i-M+1}A_{i-M+1}x, & \text{if } i \ge M .
\end{cases}
\]
Because top-\(M\) is applied over **positions** rather than over **distinct experts**, multiple selected positions can correspond to the assigned expert. The number of distinct experts actually used can therefore vary from **1** to **\(M\)**. This is the sense in which FedLEASE makes expert use adaptive: the expert pool size \(M\) is chosen by clustering, while the number of distinct experts used by a client is decided implicitly by the router during the forward pass [2509.15087].

The paper reports that different clients prefer different effective expert counts under fixed-top-\(k\) baselines; for example, **QQP** clients perform best with **top-2**, whereas **MRPC** clients perform best with **top-4**. FedLEASE’s adaptive top-\(M\) routing outperforms all fixed top-\(k\) settings in the reported experiments. Visualizations further show that deeper layers tend to activate more experts, harder tasks tend to use more experts, and clients in the same cluster exhibit similar but not identical routing patterns [2509.15087].

## 4. Federated training workflow and mathematical structure

FedLEASE has a two-phase training protocol. In the initialization phase, the server distributes the initial LoRA setup; each client performs brief local training; the server computes pairwise similarities from the \(B\) matrices; hierarchical clustering and silhouette scoring determine \(M\); and one expert is initialized per cluster by averaging client LoRAs. In the iterative federated phase, the server broadcasts all experts, each client trains only its assigned expert and router, and the server aggregates updates **within cluster** rather than globally across all clients [2509.15087].

The paper states the heterogeneous federated fine-tuning objective as
\[
\min_{\mathcal{W}} \mathcal{L}(\mathcal{W})=\sum_{i=1}^{N}\frac{|\mathcal{D}_i|}{|\mathcal{D}|}\mathcal{L}_i(W_i),
\]
with \(|\mathcal{D}|=\sum_i |\mathcal{D}_i|\). Operationally, the important point is that FedLEASE does not enforce a single shared personalized model for all clients; instead it permits cluster-specific expert parameters and router-mediated cross-expert use [2509.15087].

Cluster-wise aggregation is performed by averaging expert parameters over the clients assigned to that expert:
\[
A_j^{\text{expert}} \leftarrow \frac{1}{|C_j|}\sum_{i\in C_j} A_j^i,\qquad
B_j^{\text{expert}} \leftarrow \frac{1}{|C_j|}\sum_{i\in C_j} B_j^i.
\]
The appendix also defines cluster/expert parameters
\[
\Theta_j^t=\{A_j^{\text{expert},t},B_j^{\text{expert},t},G_j^{\text{expert},t}\},
\]
cluster-level aggregation
\[
\Theta_j^t = \frac{1}{|\mathcal{C}_j|}\sum_{i\in \mathcal{C}_j}\theta_i^t,
\]
and a convergence claim of the form
\[
\|\Theta_j^{t+1} - \Theta_j^t\| \le \frac{2\epsilon}{1-\beta M} + (\beta M)^t \max_j \|\Theta_j^1-\Theta_j^0\|.
\]
Under the condition \(\beta M <1\), the cluster models converge to a stable point according to the appendix’s analysis [2509.15087].

The framework is presented as communication-efficient because only LoRA parameters are trained and transmitted. Reported trainable-parameter fractions are **0.2075%** for FedLEASE on **GLUE / RoBERTa-Large**, compared with **0.2213%** for **FedIT/FedSA/FedDPA** and **0.1107%** for **FFA-LoRA**; and **0.0584%** for FedLEASE on **FLAN / LLaMA-2-7B**, compared with **0.0622%** for **FedIT/FedSA/FedDPA/IFCA+LoRA** and **0.0311%** for **FFA-LoRA**. The paper explicitly claims that the gains are achieved **without additional computational or communication overhead** relative to comparable LoRA-based baselines [2509.15087].

Several operational details are left less formal than the core clustering and routing rules. The text states that router networks are aggregated within cluster and that this outperforms maintaining separate routers per client, but the main equations do not provide a separate router-aggregation formula. The clustering is also static after initialization: the paper does not specify an online reclustering rule or a distance threshold for updating assignments during training [2509.15087].

## 5. Experimental evaluation

FedLEASE is evaluated on both natural language understanding and natural language generation workloads. The NLU experiments use **SST-2**, **QNLI**, **MRPC**, and **QQP**, with **16 clients total**, **4 clients per dataset**, and **RoBERTa-Large (355M)** as the backbone. The NLG experiments use FLAN task groups—**Text Editing**, **Struct to Text**, **Sentiment Analysis**, and **Commonsense Reasoning**—with **8 clients total**, **2 clients per dataset**, **600 training samples** and **200 test samples** per client, and **LLaMA-2-7B**, **8-bit quantized**, as the backbone. The reported metrics are **Accuracy** for NLU and **ROUGE-1** for NLG [2509.15087].

The baselines are **FedIT**, **FedSA**, **FFA-LoRA**, **FedDPA**, and **IFCA + LoRA**. Across the main benchmarks, FedLEASE reports the following average results:

| Benchmark | Strongest baseline | FedLEASE |
|---|---:|---:|
| GLUE average accuracy | 84.60 (FedSA) | 87.76 |
| FLAN average ROUGE-1 | 60.20 (FedSA) | 61.70 |

These correspond to gains of **3.16 points** over the strongest baseline on GLUE and **1.50 points** on FLAN. Per-task GLUE results for FedLEASE are **93.33** on **SST-2**, **87.22** on **QNLI**, **86.93** on **MRPC**, and **83.57** on **QQP**. On the FLAN side, the reported gains over the strongest baseline are **+2.26** for **Text Editing**, **+0.46** for **Struct to Text**, **+1.43** for **Sentiment Analysis**, and **+1.85** for **Commonsense Reasoning** [2509.15087].

The ablation study attributes the performance gains to both expert allocation and adaptive selection. On GLUE, **FedLoRA-Single (\(r=4\))** reaches **82.00**, **FedLoRA-Single (\(r=16\))** reaches **83.84**, **FedLoRA-Individual (16 experts)** reaches **80.69**, **FedLEASE without adaptive top-\(M\)** reaches **85.91**, and full **FedLEASE** reaches **87.76**. The reported interpretation is that one expert is too coarse, one expert per client is too fragmented, clustered expert sharing is substantially better, and adaptive top-\(M\) provides an additional improvement beyond clustering alone [2509.15087].

Sensitivity analyses further report that FedLEASE remains strongest across tested LoRA ranks, scales from **\(N=8\)** clients with average **87.72** to **\(N=32\)** clients with average **87.13**, and remains ahead of baselines under varying heterogeneity levels and additional label non-IID. For the upper bound on the number of experts, the paper reports: **\(M_{\max}=2\)** gives final experts **2** and average **85.49**; **\(M_{\max}=3\)** gives final experts **3** and average **87.21**; **\(M_{\max}=4\)** gives final experts **4** and average **87.85**; and **\(M_{\max}=8\)** still gives final experts **4** and average **87.76**. This indicates that a generous upper bound need not force over-fragmentation, whereas too small an upper bound degrades performance [2509.15087].

## 6. Relation to adjacent methods, naming ambiguity, and limitations

FedLEASE belongs to a family of federated methods whose names can be confused but whose technical settings differ sharply. It is distinct from **FedLE**, which studies battery-aware client selection for edge IoT networks and lifespan extension under battery constraints [2302.07305], and from **FedLEO**, which studies federated learning in LEO satellite constellations through intra-plane model propagation and sink satellite scheduling [2302.13447]. It is also unrelated in substance to **“Functional Encryption with Secure Key Leasing”**, which studies quantum-cryptographic leasing of functional decryption keys rather than federated LoRA fine-tuning [2209.13081].

Within federated LLM adaptation, the relevant comparisons are with single-shared LoRA methods, personalized methods, clustered FL, and MoE-style sharing. The paper positions **FedIT** as a single shared LoRA baseline, **FedDPA** as a personalized FL baseline, and **IFCA+LoRA** as a clustered FL baseline. FedLEASE differs from IFCA-style clustered FL because it uses clustering for expert training but still permits **cross-cluster knowledge transfer** through adaptive MoE routing. It differs from naive individualized MoE sharing because it does not retain one private expert per client; instead it first learns a compact expert set through clustering [2509.15087].

The method also comes with explicit limitations. The appendix states that FedLEASE assumes a **static client population** and **fixed expert assignments** after initial clustering. The paper suggests **dynamic clustering**, **meta-routing strategies** for non-stationary settings, extension to other **PEFT methods** beyond LoRA, and further **communication optimizations** as future directions. Additional underspecified points include the lack of an exact router-aggregation formula, the absence of a separate inference-time deployment protocol beyond using the trained router, and the absence of explicit communication- or computation-complexity expressions [2509.15087].

A common misconception is therefore to read FedLEASE as a generic “federated MoE” or as a dynamic reclustering system. The paper does not present it that way. The expert pool size \(M\) is adaptive only at initialization, cluster assignment remains fixed thereafter, every client receives all experts, and only the assigned expert is updated locally. The novelty lies in combining data-driven expert allocation with a routing mechanism that can vary the number of distinct experts used from **1** to **\(M\)** without manually choosing a fixed top-\(k\) [2509.15087].

Source: https://www.emergentmind.com/topics/fedlease