Switching-Based Federated Fine-Tuning

• Switching-Based Federated Fine-Tuning is a dynamic adaptation approach where specific model parts are selectively activated during training to address non-IID data and resource limitations.
• It optimizes performance by adjusting components like transformer blocks, LoRA modules, and early exits to mitigate catastrophic forgetting and improve convergence speed.
• The paradigm balances personalized client adaptation with shared global aggregation by tailoring active trainable capacity to device capabilities and communication budgets.

Switching-Based Federated Fine-Tuning denotes a class of federated adaptation methods in which the trainable portion of a pre-trained model is not fixed throughout training. Instead, the system dynamically changes which blocks, factors, experts, exits, layers, ranks, or LoRA modules are active, trainable, aggregated, or mixed across rounds, phases, clients, or inputs. In the literature, this design is used to address non-IID data, catastrophic forgetting, device heterogeneity, communication constraints, decentralized topology effects, and personalization, in settings ranging from server-based LLM fine-tuning to decentralized LoRA training, multilingual PEFT, SSL speech adaptation, and wireless edge inference (Huo et al., 6 Jun 2025, Wang et al., 31 Jan 2026, Lee et al., 5 Feb 2025, Wu et al., 30 Nov 2025).

1. Conceptual scope and lineage

A minimal precursor to later switching formulations appears in the analysis of FedAvg with local updates followed by global averaging and then a fine-tuning phase. In the multi-task linear representation setting, local computation and server averaging already constitute a switch between client-specific adaptation and shared representation consolidation; after training, the system can switch again to head-only or light joint fine-tuning for each client. That work shows that, under its assumptions, FedAvg learns a common representation, that the representation recovery rate is linear in communication rounds, and that $\tau \ge 2$ local steps are essential, since $\tau = 1$ does not guarantee recovery of the shared subspace (Collins et al., 2022).

Subsequent work makes the switching variable explicit. The switched object may be the set of expanded transformer blocks trainable on a client, the active early-exit depth, the width and depth of adapters, the LoRA factor updated in a phase, the subset of ranks retained by a sparse PEFT policy, the mixture weights over local and rest-of-world adapters, the top-$K$ or top-$M$ expert set, or the LoRA module subscribed by a wireless edge device. The resulting literature does not describe a single algorithmic family so much as a common control principle: selective, adaptive exposure of trainable capacity under federated constraints.

Granularity	Representative formulations	Switched object
Structural capacity	FedBE, adaptive early exits, DevFT, FedAdapter, HierFedLoRA	blocks, exits, stage depth, adapter depth/width, group depth/frequency
Expert or module routing	SmartFed, FedLEASE, FedALT, FedHFT, wireless LoRA subscription	rank-wise experts, expert sets, local-vs-RoW adapters, cluster adapters, LoRA modules
Alternating optimization	TAD-LoRA, H2Tune	LoRA factors, shared vs specific parameter subsets

This breadth matters because switching-based federated fine-tuning is often misidentified with sparse MoE routing alone. The cited works use the term more broadly, covering structural reassignment, phase alternation, partial aggregation, and personalized PEFT structure learning.

2. Structural switching over blocks, depths, and stages

A canonical structural formulation is FedBE, which addresses catastrophic forgetting in federated fine-tuning of LLMs by combining adaptive transformer block expansion with dynamic trainable-block allocation. The backbone remains frozen, and zero-initialized trainable blocks are appended after selected backbone layers. For a residual layer $\phi_l$, the expanded update is

$$x_{l+1} = x_l + \phi_l(x_l) + \phi_{\text{expand}_l}(\phi_l(x_l)),$$

with $W_{\text{expand}_l} \leftarrow 0$ and $b_{\text{expand}_l} \leftarrow 0$. The server selects expansion positions using a gradient-informed score

$$S(l) = \frac{g(l)}{\max_{l'} g(l')} + \lambda \cdot \frac{d(l, E_k)}{L},$$

subject to $\Delta P(k) \le \Delta P_{\max}$ and $\tau = 1$0, and then reassigns trainable blocks across rounds by client heterogeneity, training history, and resource capability. Aggregation is masked so that only clients that trained a given expanded block contribute to that block’s update. Under non-IID MRPC partitioning with $\tau = 1$1, fine-tuned models suffered 13–31% drops on RTE, QNLI, and SST-2, compared with 4.5–12% in centralized settings; FedBE reported 12–74% higher accuracy retention on general tasks, 1.9–3.1$\tau = 1$2 convergence acceleration, 93.2% on AG News and 93.6% on IMDB with RoBERTa-base, and downstream accuracy preservation while keeping the backbone fixed (Huo et al., 6 Jun 2025).

A related structural switch appears in adaptive federated fine-tuning of SSL speech models. Here, clients choose an exit depth $\tau = 1$3, train only layers up to $\tau = 1$4, and upload only those layers plus the corresponding exit head. The server aggregates each layer $\tau = 1$5 over the subset $\tau = 1$6, which yields depth-aware partial aggregation rather than full-model averaging. This mechanism directly couples computation depth to local hardware and task requirements. Reported federated results show that mid-depth exits can outperform full-depth training for several tasks: ASR test-clean WER was 8.81% at 9 layers versus 12.88% at 12 layers, KWS-35 error was 10.70% at 6 layers versus 15.30% at 12 layers, and layer-wise aggregation improved heterogeneous FL over naïve FedAvg while reducing client memory by 10–43% depending on task (Guo et al., 23 Mar 2026).

Progressive stage schedules generalize the same idea to longer horizons. FedAdapter progressively switches adapter depth and width, profiling current, deeper, and wider configurations with parallel trial groups and inheriting the winner configuration over time; it reports up to 155.5$\tau = 1$7 faster convergence than vanilla FedNLP, up to 48$\tau = 1$8 faster than strong baselines, 126.7$\tau = 1$9 average reduction in total network traffic, and up to 18.4$K$0 reduction in per-device energy (Cai et al., 2022). DevFT instead switches the active submodel capacity across developmental stages, using deconfliction-guided layer grouping and differential-based layer fusion to construct stage-specific representative layers; with stage schedules such as $K$1 or $K$2, it reports up to 4.59$K$3 faster convergence, 10.67$K$4 reduction in communication overhead, and 9.07% average performance improvement (Wu et al., 31 Jul 2025). HierFedLoRA switches group-level aggregation frequency $K$5 and fine-tuning depth $K$6 under resource budgets, with near-IID grouping and hierarchical aggregation; on 80 commercial devices it improves final accuracy by 1.6% to 4.2% and speeds fine-tuning by at least 2.1$K$7 (Liu et al., 27 Mar 2025).

3. Expert, adapter, and module routing

Another major branch of switching-based federated fine-tuning uses routing over PEFT experts. SmartFed reuses pre-existing LoRA modules by decomposing each low-rank update into rank-wise experts,

$K$8

and trains only a router $K$9 while keeping expert parameters fixed. The router computes $M$0, applies a top-$M$1 mask, and forms the output

$M$2

Elastic Expert Quota Allocation then redistributes activation capacity across parameter matrices under a fixed budget $M$3. Because only the router and lightweight importance statistics are communicated, SmartFed reports up to 31.47$M$4 lower communication overhead, 3.95$M$5 faster convergence, and 3.61$M$6 lower energy consumption; on LLaMA2-7B it achieves MGSM 35.48, DoIT 52.59, GSM-Hard 64.19, and Average 50.75 with Activated Rank 32 (Wu et al., 30 Nov 2025).

FedLEASE also uses LoRA experts, but allocates them by clustering clients according to cosine distance between their layerwise $M$7 matrices after brief local training. It selects the number of experts by silhouette score, yielding $M$8 experts for 16 NLU clients in the main setting, initializes each expert by cluster-average LoRA weights, and equips each client with an adaptive top-$M$9 router whose output space repeats the assigned expert so that the number of unique experts used per input can vary from 1 to $\phi_l$0. On NLU, FedLEASE reaches 87.76% average accuracy, versus 84.60% for the strongest reported baseline; on NLG with LLaMA-2-7B it reaches 61.70 average ROUGE-1, versus 60.20 for FedSA (Wang et al., 18 Sep 2025).

FedALT uses a two-expert personalized mixture rather than a larger MoE. Each client maintains its own LoRA $\phi_l$1, receives a frozen rest-of-world LoRA $\phi_l$2 formed by averaging other clients’ adapters, and learns a soft gate $\phi_l$3 to interpolate local and RoW paths. The key switch is therefore input-conditioned mixing between local specialization and non-invasive shared knowledge. On two heterogeneous Flan-derived settings, FedALT reports averages of 67.55 and 70.48, exceeding the best baseline by 2.38 and 2.88 points respectively (Bian et al., 14 Mar 2025).

FedHFT moves the routing decision to the client level by mixing cluster-specific masked adapters with soft responsibilities $\phi_l$4 from a GMM over client updates. Each client uses an importance-based binary mask $\phi_l$5 over adapter dimensions, merges cluster adapters as $\phi_l$6, and uploads only masked low-rank updates. The server aggregates per-cluster updates and re-projects them to fixed rank via SVD. Under GLUE and SQuAD settings, FedHFT reports up to $\phi_l$7 reduction in memory footprint and up to $\phi_l$8 reduction in communication per client per round while improving or matching accuracy (Ilhan et al., 15 Oct 2025).

4. Alternating optimization and knowledge disentanglement

In decentralized federated learning, switching interacts directly with topology. TAD-LoRA studies LoRA factorization in a serverless setting, where independent aggregation of $\phi_l$9 and $$x_{l+1} = x_l + \phi_l(x_l) + \phi_{\text{expand}_l}(\phi_l(x_l)),$$0 creates cross-client terms:

$$x_{l+1} = x_l + \phi_l(x_l) + \phi_{\text{expand}_l}(\phi_l(x_l)),$$1

These bilinear cross terms depend on consensus error, graph connectivity, and phase synchronization. TAD-LoRA therefore alternates which factor is updated every $$x_{l+1} = x_l + \phi_l(x_l) + \phi_{\text{expand}_l}(\phi_l(x_l)),$$2 rounds while jointly mixing both factors on every round, including the frozen one, to avoid block-wise drift. Its convergence result makes the trade-off explicit:

$$x_{l+1} = x_l + \phi_l(x_l) + \phi_{\text{expand}_l}(\phi_l(x_l)),$$3

so that the preferred switching interval grows as connectivity weakens, with $$x_{l+1} = x_l + \phi_l(x_l) + \phi_{\text{expand}_l}(\phi_l(x_l)),$$4 and, under random edge activation, $$x_{l+1} = x_l + \phi_l(x_l) + \phi_{\text{expand}_l}(\phi_l(x_l)),$$5. Empirically, TAD-LoRA remains competitive at $$x_{l+1} = x_l + \phi_l(x_l) + \phi_{\text{expand}_l}(\phi_l(x_l)),$$6, improves over baselines at $$x_{l+1} = x_l + \phi_l(x_l) + \phi_{\text{expand}_l}(\phi_l(x_l)),$$7, and in the weak regime $$x_{l+1} = x_l + \phi_l(x_l) + \phi_{\text{expand}_l}(\phi_l(x_l)),$$8 achieves 0.8050 average accuracy versus 0.7920 for LoRA, 0.7860 for FFA-LoRA, and 0.7739 for RoLoRA; on MNLI at $$x_{l+1} = x_l + \phi_l(x_l) + \phi_{\text{expand}_l}(\phi_l(x_l)),$$9, it reaches $W_{\text{expand}_l} \leftarrow 0$0 versus 0.6407 for LoRA and 0.5980 for RoLoRA (Wang et al., 31 Jan 2026).

H2Tune addresses a different alternating problem: simultaneous architecture heterogeneity and task heterogeneity. Standard LoRA aggregation fails when clients have different hidden dimensions, layer counts, and tasks, so H2Tune factorizes each local update as

$W_{\text{expand}_l} \leftarrow 0$1

where only the rank-consistent middle matrices $W_{\text{expand}_l} \leftarrow 0$2 are globally shared. A trainable relation matrix $W_{\text{expand}_l} \leftarrow 0$3 aligns local layers to a common global layer space, and local optimization alternates between a shared phase over $W_{\text{expand}_l} \leftarrow 0$4 and a specific phase over $W_{\text{expand}_l} \leftarrow 0$5. The method proves an ergodic convergence rate corresponding to $W_{\text{expand}_l} \leftarrow 0$6 for standard optimality measures and reports up to 15.4% accuracy improvement over strong baselines, with communication cost of approximately 4.59 MB per exchange (Guo et al., 30 Jul 2025).

These alternating formulations clarify an important point. Switching does not, by itself, guarantee stability. In centralized FL, alternating can suppress interference because frozen blocks remain synchronized. In DFL, naïve alternating fails because the frozen block drifts under partial mixing, whereas in hybrid-heterogeneous FL, alternating must be paired with explicit separation of shared and specific subspaces to avoid transferring task-private information (Wang et al., 31 Jan 2026, Guo et al., 30 Jul 2025).

5. Personalized parameter-subset selection and resource-aware control

Some switching-based methods do not route among experts at all; instead, they learn which ranks, parameters, or modules should be active for a client. FedP$W_{\text{expand}_l} \leftarrow 0$7EFT learns personalized PEFT structures for multilingual LLMs by assigning a latent scale vector $W_{\text{expand}_l} \leftarrow 0$8 to the ranks of each LoRA layer,

$W_{\text{expand}_l} \leftarrow 0$9

with Bayesian sparse rank selection penalties and a PS generator MLP that maps layer-input mean and standard deviation features to $b_{\text{expand}_l} \leftarrow 0$0. After local optimization, the client prunes to the top-$b_{\text{expand}_l} \leftarrow 0$1 ranks under the target budget, so the active PEFT structure is effectively switched on a per-client basis. Only the PS generator parameters $b_{\text{expand}_l} \leftarrow 0$2 are aggregated. On XNLI with a standard FL base model, FedP$b_{\text{expand}_l} \leftarrow 0$3EFT reports wins on 15/15 languages across budgets $b_{\text{expand}_l} \leftarrow 0$4; the communicated model is small, using about 0.2% of mBERT and 0.16% of Llama-3.2-3B parameters (Lee et al., 5 Feb 2025).

FedSelect makes the switched subset even more explicit. Each client maintains a binary mask $b_{\text{expand}_l} \leftarrow 0$5 where $b_{\text{expand}_l} \leftarrow 0$6 marks personalized parameters and $b_{\text{expand}_l} \leftarrow 0$7 marks shared parameters. The effective client model is

$b_{\text{expand}_l} \leftarrow 0$8

Masks are discovered by GradLTN, which repeatedly identifies parameters with large change magnitude as personalization candidates and freezes low-variation parameters for global aggregation. Server aggregation ignores indices with $b_{\text{expand}_l} \leftarrow 0$9. On CIFAR-10 with non-IID splits, FedSelect reaches 85.91% mean test accuracy for the $$S(l) = \frac{g(l)}{\max_{l'} g(l')} + \lambda \cdot \frac{d(l, E_k)}{L},$$0 setting and 65.88% for $$S(l) = \frac{g(l)}{\max_{l'} g(l')} + \lambda \cdot \frac{d(l, E_k)}{L},$$1, exceeding FedRep in both reported best cases (Tamirisa et al., 2023).

Resource-aware switching can also be coupled to communication control. In heterogeneous wireless networks, one formulation introduces binary decisions $$S(l) = \frac{g(l)}{\max_{l'} g(l')} + \lambda \cdot \frac{d(l, E_k)}{L},$$2 indicating whether device $$S(l) = \frac{g(l)}{\max_{l'} g(l')} + \lambda \cdot \frac{d(l, E_k)}{L},$$3 subscribes LoRA module $$S(l) = \frac{g(l)}{\max_{l'} g(l')} + \lambda \cdot \frac{d(l, E_k)}{L},$$4 at round $$S(l) = \frac{g(l)}{\max_{l'} g(l')} + \lambda \cdot \frac{d(l, E_k)}{L},$$5, and jointly optimizes switching, transmit power, and bandwidth allocation under a long-term objective combining inference risk and energy. The LoRA-module update is weighted by local data sizes and successful transmission indicators $$S(l) = \frac{g(l)}{\max_{l'} g(l')} + \lambda \cdot \frac{d(l, E_k)}{L},$$6. On SST-2 and QNLI, the proposed online algorithm reduces rounds to convergence by 42.7% on average versus vanilla FedLoRA, improves over HetLoRA by 34.1% and over FlexLoRA by 5.0%, yields 1.9% higher QNLI test accuracy than one-shot switching and 1.3% higher than greedy switching, and achieves 16.8% power savings with adaptive transmit power control (Wang et al., 5 Sep 2025).

6. Empirical patterns, limitations, and recurring misconceptions

Across the cited literature, the largest gains typically appear when heterogeneity is severe or communication is weak. FedBE’s retention advantage is reported under strong non-IID forgetting pressure (Huo et al., 6 Jun 2025). TAD-LoRA’s advantage grows as the communication graph becomes moderately or weakly connected (Wang et al., 31 Jan 2026). FedAdapter, DevFT, SmartFed, FedHFT, and HierFedLoRA all report the strongest systems gains when communication or device budgets are bottlenecks rather than secondary concerns (Cai et al., 2022, Wu et al., 31 Jul 2025, Wu et al., 30 Nov 2025, Ilhan et al., 15 Oct 2025, Liu et al., 27 Mar 2025).

The literature also contradicts a common assumption that “more capacity” is always better. In FedBE, benefits plateau beyond three expanded blocks and costs rise (Huo et al., 6 Jun 2025). In SSL speech fine-tuning, deeper exits are not uniformly better: ASR favored depth 9, while KWS-35 favored depth 6 (Guo et al., 23 Mar 2026). In FedLEASE, different clients prefer different fixed top-$$S(l) = \frac{g(l)}{\max_{l'} g(l')} + \lambda \cdot \frac{d(l, E_k)}{L},$$7 values, and adaptive top-$$S(l) = \frac{g(l)}{\max_{l'} g(l')} + \lambda \cdot \frac{d(l, E_k)}{L},$$8 exceeds any single fixed choice (Wang et al., 18 Sep 2025). A plausible implication is that switching is not merely a sparsity device; it is a mechanism for matching representational depth or expert multiplicity to local data and system conditions.

Several limitations recur. Structural methods depend on selection heuristics: FedBE relies on proxy gradient norms and accurate estimates of $$S(l) = \frac{g(l)}{\max_{l'} g(l')} + \lambda \cdot \frac{d(l, E_k)}{L},$$9, $\Delta P(k) \le \Delta P_{\max}$0, and $\Delta P(k) \le \Delta P_{\max}$1, and some extremely skewed clients may still overfit high-level expansions (Huo et al., 6 Jun 2025). Decentralized alternating methods depend on topology assumptions such as doubly-stochastic mixing and effective spectral gaps; very sparse or disconnected graphs require large switching intervals and incur representation bias (Wang et al., 31 Jan 2026). Expert-based methods often trade stronger personalization against extra coordination: FedLEASE uses static clustering and notes privacy implications of sharing initial LoRA parameters for similarity estimation, while SmartFed does not provide theoretical convergence guarantees or DP/secure aggregation mechanisms (Wang et al., 18 Sep 2025, Wu et al., 30 Nov 2025). H2Tune’s guarantees rely on smoothness and bounded-domain assumptions, and the wireless switching formulation assumes orthogonal intra-cell channels, channel-state observability, and strong convexity in theory (Guo et al., 30 Jul 2025, Wang et al., 5 Sep 2025).

A second recurring misunderstanding is to treat switching-based federated fine-tuning as synonymous with token-level expert routing. The surveyed work uses switching to mean dynamic reassignment of trainable transformer blocks, client-specific exit depth, stage-wise adapter capacity, alternating LoRA-factor optimization, shared-versus-specific disentanglement, sparse rank activation, client-specific subnetwork masks, and even module subscription under wireless constraints. What unifies these methods is not a single architecture, but a shared design choice: trainable capacity is exposed selectively rather than uniformly.

In that sense, switching-based federated fine-tuning is best understood as a control paradigm for federated adaptation. Its central question is not simply how to fine-tune a large model under privacy constraints, but which part of the model should be active, for whom, when, and under what aggregation rule. The diversity of answers in current work suggests that switching is becoming a general organizing principle for handling the joint statistical and systems heterogeneity of federated foundation-model adaptation.