Federated Mixture of Experts (FedMox)

• Federated Mixture of Experts (FedMox) is a decentralized learning framework that uses conditional expert routing to address non-IID data challenges.
• It leverages diverse architectural patterns—such as global-local mixtures, shared expert pools, and networked experts—to enable flexible and efficient model training.
• Practical implementations focus on optimizing routing, load balancing, and synchronization to reduce communication overhead and energy consumption.

Federated Mixture of Experts (FedMox) denotes a family of federated learning architectures that combine decentralized optimization with Mixture-of-Experts (MoE) sparsity: multiple experts are trained, assigned, or reused across clients, while a router or gating mechanism selects only a small subset of experts for a given input, token, client, or modality. The label is not standardized—many papers instead use names such as FedMoE, FedMix, FLEX-MoE, FLEx, SmartFed, or FlexMoRE—but the underlying design space is unified by three recurring goals: specialization under non-IID data, privacy preservation through decentralized training, and resource efficiency through conditional computation (Zhang et al., 28 Dec 2025, Pirchert et al., 9 Feb 2026).

1. Terminology, scope, and historical development

Early federated MoE work framed the idea primarily as personalization. "PFL-MoE: Personalized Federated Learning Based on Mixture of Experts" combines a global expert and a personalized expert through a client-specific gate, explicitly targeting the tension between personalization and generalization under non-IID data (Guo et al., 2020). "Specialized federated learning using a mixture of experts" uses a similar two-expert structure, mixing a local specialist with a global generalist and emphasizing a three-stage protocol: FedAvg training of the global model, local fine-tuning of the specialist, and local gate training with the global model frozen (Zec et al., 2020).

A second line of work moved from two-expert personalization to multi-expert federated specialization. FedMix trains an ensemble of specialized global experts with shard-specific gates and expert-specific aggregation weights derived from marginal responsibilities, rather than simple whole-model averaging (Reisser et al., 2021). "Adaptive Expert Models for Personalization in Federated Learning" extends clustered federated learning by combining multiple global cluster models with a local expert and introducing an $\epsilon$-greedy exploration strategy in client-to-expert assignment so that more experts converge and remain useful for personalization (Isaksson et al., 2022).

Subsequent systems broadened the meaning of federated MoE beyond client clustering. DDOME/FedJETs emphasizes joint gate-expert training and just-in-time personalization via a pretrained common expert and anchor-client regularization in decentralized settings (Farhat et al., 2023). FedMoE-DA, FedMoE, FLEX-MoE, FC-MoE, FedLEASE, FLEx, SmartFed, and EMS-FL each redefine the expert unit, assignment rule, or aggregation operator: experts may be dense subnetworks, LoRA modules, rank-wise components, domain-specific adapters, or cluster-owned expert subsets, and they may be aggregated centrally, synchronized peer-to-peer, or not averaged at all (Zhan et al., 2024, Mei et al., 2024, Wang et al., 18 Sep 2025, Lu et al., 14 Jun 2026, Xu et al., 23 Feb 2026).

This trajectory suggests that FedMox is best understood not as a single algorithm but as a research program: a set of federated architectures that exploit expert modularity to address statistical heterogeneity and systems heterogeneity simultaneously.

2. Architectural patterns

Across the literature, several architectural templates recur. The simplest form is a two-expert personalized mixture, where a client-specific gate interpolates between a global model and a local model. A representative formulation is

$$\hat{y}_k(x)=h^k(x;w_h^k)\,f_s^k(x;w_s^k)+\bigl(1-h^k(x;w_h^k)\bigr)\,f_g(x;w_g),$$

with the global parameters trained by FedAvg and the specialist and gate kept local (Guo et al., 2020). This design prioritizes personalization while leaving communication nearly identical to standard FedAvg.

A second pattern uses a shared pool of global experts and client- or shard-specific routing. FedMix conditions gating on shard identity and optimizes a variational lower bound with global auxiliary assignments, whereas FedMoE constructs a client-specific sub-MoE by selecting only the experts that fit a client’s activation statistics and memory budget (Reisser et al., 2021, Mei et al., 2024). In these systems, personalization comes from selective expert usage rather than from a single local specialist.

A third pattern anchors experts to a common frozen base. FlexMoRE is exemplary: it uses a public, frozen base expert within each Transformer FFN and allows each domain expert to be either a full-sized dense FFN or a low-rank adapter defined by

$W' = W + AB,$

with heterogeneous rank $r$ across experts (Pirchert et al., 9 Feb 2026). FLEx follows a related principle in MoE LLM fine-tuning: the pretrained MoE backbone remains frozen, each client retains one personalized side expert locally, and only dense shared modules such as attention LoRA adapters are aggregated (Liu et al., 1 Jun 2025).

A fourth pattern distributes experts across clients or devices rather than merely across parameters. NMoE places a shared feature extractor and shared gate on every client but keeps one personalized expert per client; at inference time, the gate can route a latent representation to experts hosted on neighboring devices, so expert utilization becomes networked rather than local (Gao et al., 3 Nov 2025). EMS-FL pushes this further in satellite-terrestrial networks by assigning non-overlapping expert subsets to clusters and allowing asynchronous training under intermittent connectivity (Xu et al., 23 Feb 2026).

A fifth pattern conditions routing on richer context than client identity alone. ProMoE-FL, designed for multimodal federated learning with missing modalities, uses a global client-aware prototype bank and direction-aware routing conditioned on modality indices and target-modality prototypes, so expert choice reflects the synthesis direction as well as the observed feature (Chhetri et al., 7 Jul 2026).

Pattern	Representative methods	Core mechanism
Global–local mixture	PFL-MoE, specialized FL using MoE	Gate mixes a global expert with a local expert
Shared expert pool	FedMix, FedMoE, Adaptive Expert Models	Clients use or train subsets of shared experts
Base-anchored experts	FlexMoRE, FLEx	Experts are deviations or adapters relative to a frozen base
Networked experts	NMoE, EMS-FL	Experts reside across clients or clusters
Context-conditioned experts	ProMoE-FL	Routing depends on prototypes, modalities, or direction

These patterns are not mutually exclusive. Later systems frequently combine several of them—for example, base anchoring plus sparse client-specific expert selection, or cluster ownership plus asynchronous aggregation.

3. Routing, assignment, and aggregation mechanisms

The main technical axis separating federated MoE methods is not the existence of experts but the rule by which clients and inputs are matched to them. In base-anchored LLM systems such as FlexMoRE, routing remains token-level and MoE-standard:

$$p_i(x)=\operatorname{softmax}(g(x))_i,\quad S(x)=\text{Top-}k\big(p_1(x),\ldots,p_E(x)\big),\quad y(x)=\sum_{i\in S(x)} p_i(x)f_i(x),$$

where a routed low-rank expert implicitly activates its corresponding base expert as well (Pirchert et al., 9 Feb 2026).

Other systems make assignment itself an optimization problem. FLEX-MoE formulates server-side client–expert assignment as an integer linear program over binary variables $X_{c,e}^{(t)}$, maximizing accumulated client–expert fitness scores under exact per-client capacity constraints and dynamic per-expert load bounds. Its explicit objective is to maximize $\sum_{c,e} Q_{t-1}(c,e)X_{c,e}$ while constraining each client to store at most $k_c$ experts and constraining the load assigned to each expert within a bounded interval (Zhang et al., 28 Dec 2025). This contrasts with greedy top-$k$ personalization methods that ignore global expert utilization.

FedMoE-DA takes a different route: it leaves experts and gates private to clients, aggregates only a shared embedding model centrally, and uses cosine similarity between gating proxies to drive peer-to-peer synchronization of similar experts. Expert updates are therefore similarity-weighted at the expert level rather than averaged model-wide, and synchronization occurs via client-to-client pulls every $I$ rounds rather than via server aggregation of expert parameters (Zhan et al., 2024). This is a fine-grained aggregation strategy rather than a pure routing strategy, but it serves the same purpose of aligning expertise with data heterogeneity.

FedMoE for heterogeneous LLM fine-tuning constructs a client-specific sub-MoE in two stages. First, it measures activation probabilities for experts and uses a binary-search heuristic to select a submodel satisfying a memory budget and an activation-coverage threshold. Second, it trains those client-specific submodels locally and aggregates dense layers by FedAvg while aggregating sparse experts only across the subset of clients that used them (Mei et al., 2024). FedLEASE adopts yet another assignment rule: it clusters clients using cosine dissimilarity between flattened LoRA $$\hat{y}_k(x)=h^k(x;w_h^k)\,f_s^k(x;w_s^k)+\bigl(1-h^k(x;w_h^k)\bigr)\,f_g(x;w_g),$$0 matrices, chooses the number of expert groups through silhouette scores, and then lets clients use an adaptive top-$$\hat{y}_k(x)=h^k(x;w_h^k)\,f_s^k(x;w_s^k)+\bigl(1-h^k(x;w_h^k)\bigr)\,f_g(x;w_g),$$1 router that guarantees inclusion of the client’s assigned cluster expert while allowing a variable number of additional experts (Wang et al., 18 Sep 2025).

Conflict-aware aggregation becomes central when multiple clients update same-index experts under incompatible non-IID distributions. FC-MoE addresses this by defining expert-level importance scores

$$\hat{y}_k(x)=h^k(x;w_h^k)\,f_s^k(x;w_s^k)+\bigl(1-h^k(x;w_h^k)\bigr)\,f_g(x;w_g),$$2

forming a consensus update direction $$\hat{y}_k(x)=h^k(x;w_h^k)\,f_s^k(x;w_s^k)+\bigl(1-h^k(x;w_h^k)\bigr)\,f_g(x;w_g),$$3, and projecting any conflicting update onto the normal plane whenever its cosine similarity with the consensus is negative. It then restores some client-specific specialization through a local residual retention mechanism with coefficient $$\hat{y}_k(x)=h^k(x;w_h^k)\,f_s^k(x;w_s^k)+\bigl(1-h^k(x;w_h^k)\bigr)\,f_g(x;w_g),$$4 (Lu et al., 14 Jun 2026).

Finally, some federated MoE systems depart from synchronous federated rounds altogether. EMS-FL assigns non-overlapping experts to clusters based on empirical gate-routing relevance probabilities and lets clusters train their assigned experts asynchronously during disconnected phases, uploading only when the satellite link is available (Xu et al., 23 Feb 2026). This suggests that “federated” need not imply round-synchronous optimization once expert modularity makes partial, asynchronous updates feasible.

4. Efficiency, parameterization, and systems trade-offs

A major reason FedMox has become prominent is that expert modularity creates several orthogonal compression levers: fewer active experts, smaller experts, fewer communicated parameters, or fewer synchronized components. The reported efficiencies are therefore heterogeneous in mechanism and not directly comparable across tasks, but they illustrate the breadth of the design space.

FlexMoRE replaces many full-sized domain experts with low-rank adapters derived post hoc through truncated SVD of the difference between a domain expert layer and the shared base layer. Using heterogeneous ranks chosen with an MC9 proxy, it reports improved downstream average score $$\hat{y}_k(x)=h^k(x;w_h^k)\,f_s^k(x;w_s^k)+\bigl(1-h^k(x;w_h^k)\bigr)\,f_g(x;w_g),$$5 relative to a FlexOlmo-style full-expert baseline at $$\hat{y}_k(x)=h^k(x;w_h^k)\,f_s^k(x;w_s^k)+\bigl(1-h^k(x;w_h^k)\bigr)\,f_g(x;w_g),$$6, while reducing the total parameter count from $$\hat{y}_k(x)=h^k(x;w_h^k)\,f_s^k(x;w_s^k)+\bigl(1-h^k(x;w_h^k)\bigr)\,f_g(x;w_g),$$7B to $$\hat{y}_k(x)=h^k(x;w_h^k)\,f_s^k(x;w_s^k)+\bigl(1-h^k(x;w_h^k)\bigr)\,f_g(x;w_g),$$8B (Pirchert et al., 9 Feb 2026). FLEx achieves efficiency differently: it freezes the pretrained MoE experts and communicates only attention LoRA adapters, reducing per-round communication on Qwen1.5-MoE-A2.7B from approximately $$\hat{y}_k(x)=h^k(x;w_h^k)\,f_s^k(x;w_s^k)+\bigl(1-h^k(x;w_h^k)\bigr)\,f_g(x;w_g),$$9 of total parameters, or about $W' = W + AB,$0M parameters, to about $W' = W + AB,$1, or about $W' = W + AB,$2M parameters (Liu et al., 1 Jun 2025).

SmartFed and its Mixture of Rank-Wise Experts push the expert unit itself to the rank-1 level. By decomposing LoRA modules into rank-wise experts and training only the routers, it reports up to $W' = W + AB,$3 lower communication, $W' = W + AB,$4 faster convergence, and $W' = W + AB,$5 lower energy consumption than baselines (Wu et al., 30 Nov 2025). In the semi-supervised detection setting of PSSFL, the method explicitly named FedMox keeps the foundation-model backbone frozen, trains only a sparse MoE head at low resolution on clients, and reduces client memory from $W' = W + AB,$6 GB for conventional FL at $W' = W + AB,$7 to $W' = W + AB,$8 GB at $W' = W + AB,$9; the routing overhead is reported as $r$0M FLOPs with $r$1 experts (Sun et al., 22 Aug 2025).

Method	Efficiency mechanism	Reported result
FlexMoRE	Heterogeneous low-rank experts	$r$2B vs $r$3B parameters; $r$4 vs $r$5 average score
FLEx	Aggregate only attention LoRA; keep side expert local	$r$6M vs $r$7M communicated parameters per round
SmartFed	Rank-wise experts; train only routers	Up to $r$8 lower communication
FedMox in PSSFL	Frozen backbone; low-resolution sparse head	$r$9 GB vs $$p_i(x)=\operatorname{softmax}(g(x))_i,\quad S(x)=\text{Top-}k\big(p_1(x),\ldots,p_E(x)\big),\quad y(x)=\sum_{i\in S(x)} p_i(x)f_i(x),$$0 GB client memory

Systems-oriented papers make clear that efficiency is not purely parameter count. FLEX-MoE addresses the inability of edge devices to store full expert sets by enforcing exact expert-capacity constraints per client (Zhang et al., 28 Dec 2025). NMoE trades local storage for peer-to-peer latent exchange at inference, thereby moving cost from memory to network traffic (Gao et al., 3 Nov 2025). EMS-FL exploits MoE structure to tolerate intermittent links by synchronizing only cluster-owned experts during connected phases (Xu et al., 23 Feb 2026). FedMox, in this broader sense, is therefore as much a systems abstraction as a modeling abstraction.

5. Empirical regimes and application domains

The empirical literature spans vision classification, LLM fine-tuning, multimodal medical learning, object detection, and networked edge inference. On classical image benchmarks, load balancing and capacity awareness materially affect outcomes. FLEX-MoE reports, on CIFAR-10 with heterogeneous capacities $$p_i(x)=\operatorname{softmax}(g(x))_i,\quad S(x)=\text{Top-}k\big(p_1(x),\ldots,p_E(x)\big),\quad y(x)=\sum_{i\in S(x)} p_i(x)f_i(x),$$1, that under IID loss-driven scoring Greedy reaches $$p_i(x)=\operatorname{softmax}(g(x))_i,\quad S(x)=\text{Top-}k\big(p_1(x),\ldots,p_E(x)\big),\quad y(x)=\sum_{i\in S(x)} p_i(x)f_i(x),$$2 accuracy with coefficient of variation $$p_i(x)=\operatorname{softmax}(g(x))_i,\quad S(x)=\text{Top-}k\big(p_1(x),\ldots,p_E(x)\big),\quad y(x)=\sum_{i\in S(x)} p_i(x)f_i(x),$$3, whereas FLEX-MoE reaches $$p_i(x)=\operatorname{softmax}(g(x))_i,\quad S(x)=\text{Top-}k\big(p_1(x),\ldots,p_E(x)\big),\quad y(x)=\sum_{i\in S(x)} p_i(x)f_i(x),$$4 with $$p_i(x)=\operatorname{softmax}(g(x))_i,\quad S(x)=\text{Top-}k\big(p_1(x),\ldots,p_E(x)\big),\quad y(x)=\sum_{i\in S(x)} p_i(x)f_i(x),$$5; under non-IID Dirichlet $$p_i(x)=\operatorname{softmax}(g(x))_i,\quad S(x)=\text{Top-}k\big(p_1(x),\ldots,p_E(x)\big),\quad y(x)=\sum_{i\in S(x)} p_i(x)f_i(x),$$6, Greedy yields $$p_i(x)=\operatorname{softmax}(g(x))_i,\quad S(x)=\text{Top-}k\big(p_1(x),\ldots,p_E(x)\big),\quad y(x)=\sum_{i\in S(x)} p_i(x)f_i(x),$$7 versus $$p_i(x)=\operatorname{softmax}(g(x))_i,\quad S(x)=\text{Top-}k\big(p_1(x),\ldots,p_E(x)\big),\quad y(x)=\sum_{i\in S(x)} p_i(x)f_i(x),$$8 for FLEX-MoE, again with near-uniform expert utilization (Zhang et al., 28 Dec 2025). FedMoE-DA reports robustness across homogeneous, pathological, and Dirichlet partitions on CIFAR-10, including $$p_i(x)=\operatorname{softmax}(g(x))_i,\quad S(x)=\text{Top-}k\big(p_1(x),\ldots,p_E(x)\big),\quad y(x)=\sum_{i\in S(x)} p_i(x)f_i(x),$$9 at Dirichlet $X_{c,e}^{(t)}$0 and $X_{c,e}^{(t)}$1 at $X_{c,e}^{(t)}$2, while reducing server-client communication by centralizing only the shared embedding (Zhan et al., 2024).

In LLM adaptation, the dominant themes are modular fine-tuning and conflict management. FlexMoRE’s regression analysis shows strong positive rank sensitivity on reasoning-heavy benchmarks such as BBH, with $X_{c,e}^{(t)}$3 up to $X_{c,e}^{(t)}$4 and Pearson $X_{c,e}^{(t)}$5–$X_{c,e}^{(t)}$6, while knowledge-heavy benchmarks such as GEN5 and MC9 can show weak or negative sensitivity, with $X_{c,e}^{(t)}$7 down to $X_{c,e}^{(t)}$8 and $X_{c,e}^{(t)}$9 (Pirchert et al., 9 Feb 2026). FC-MoE reports, under non-IID label skew with $\sum_{c,e} Q_{t-1}(c,e)X_{c,e}$0, accuracy of approximately $\sum_{c,e} Q_{t-1}(c,e)X_{c,e}$1 on AGNews and approximately $\sum_{c,e} Q_{t-1}(c,e)X_{c,e}$2 on MMLU, outperforming FedAvg, PFL-MoE, and FedMoE, with ablations showing that removing gradient consensus projection causes the largest degradation (Lu et al., 14 Jun 2026). FedLEASE, which clusters clients and allocates LoRA experts adaptively, reports average NLU accuracy $\sum_{c,e} Q_{t-1}(c,e)X_{c,e}$3 on heterogeneous GLUE-style settings, compared with strongest baselines around $\sum_{c,e} Q_{t-1}(c,e)X_{c,e}$4–$\sum_{c,e} Q_{t-1}(c,e)X_{c,e}$5, while using $\sum_{c,e} Q_{t-1}(c,e)X_{c,e}$6 of RoBERTa-Large parameters (Wang et al., 18 Sep 2025).

Multimodal and medical settings have produced distinct federated MoE variants. ProMoE-FL, for chest X-ray image–text learning with missing modalities, reports the best performance in $\sum_{c,e} Q_{t-1}(c,e)X_{c,e}$7 homogeneous configurations and consistent superiority in every heterogeneous configuration, including $\sum_{c,e} Q_{t-1}(c,e)X_{c,e}$8 macro AUC for the $\sum_{c,e} Q_{t-1}(c,e)X_{c,e}$9 heterogeneous split compared with $k_c$0 for FeatImp and $k_c$1 for PmcmFL (Chhetri et al., 7 Jul 2026). In object detection under Practical Semi-Supervised Federated Learning, FedMox reports total mAP@50 of $k_c$2 on BDD100K with $k_c$3 clients versus $k_c$4 for FedAvg, and $k_c$5 on Cityscapes versus $k_c$6 for FedAvg, while operating under frozen-backbone, low-resolution client constraints (Sun et al., 22 Aug 2025).

Deployment settings also matter. EMS-FL reports that on a Mixtral-8$k_c$77B-Instruct-v0.1 experiment with Top-2 routing, it reaches about $k_c$8 global accuracy within about $k_c$9 training rounds, whereas its baseline requires more than $k$0 rounds (Xu et al., 23 Feb 2026). NMoE reports that on CIFAR-10 with IID data, the FedCE+FedGate variant reaches accuracy $k$1, F1 $k$2, and AUC $k$3, while under stronger non-IID conditions the FedSC+FedGate combination is more robust than FedCE-based gating (Gao et al., 3 Nov 2025). These results reinforce that federated MoE performance is tightly coupled to routing design, synchronization regime, and the systems substrate on which training occurs.

6. Misconceptions, limitations, and open problems

A common misconception is that FedMox refers to a single canonical protocol. The literature does not support that view. Several papers explicitly note that they do not use the term “FedMox,” even when their method is functionally a federated MoE, and the underlying protocols differ sharply in whether experts are global, private, clustered, base-anchored, peer-hosted, or merely inference-time components (Pirchert et al., 9 Feb 2026, Mei et al., 2024). This diversity suggests that “FedMox” functions more as a family resemblance term than as a single algorithmic specification.

A second misconception is that federated MoE necessarily means averaging all client experts à la FedAvg. FlexMoRE does not aggregate across clients in the classical federated sense; instead, each client contributes an expert trained against a frozen public base, and experts are composed only at inference time (Pirchert et al., 9 Feb 2026). FedMoE-DA aggregates experts peer-to-peer rather than through the server (Zhan et al., 2024). FLEx keeps personalized experts entirely local and aggregates only dense shared modules (Liu et al., 1 Jun 2025). NMoE does not even keep inference local, since it can route latents to experts hosted on other clients (Gao et al., 3 Nov 2025).

Load balancing is another point of confusion. Sparse routing alone does not guarantee balanced expert utilization. FLEX-MoE was motivated precisely by severe expert load imbalance under non-IID data and limited client capacity, and addresses it with dynamic per-expert load bounds in an ILP (Zhang et al., 28 Dec 2025). By contrast, FlexMoRE inherits a domain-informed router and explicitly does not introduce additional balancing terms in the reported results (Pirchert et al., 9 Feb 2026). Whether balancing is enforced by auxiliary losses, assignment constraints, masking, or not at all remains a major methodological fault line.

Several limitations recur across the field. Many methods depend on a shared frozen base, shared routing scaffold, or a small public dataset, creating alignment assumptions that can fail under domain shift (Pirchert et al., 9 Feb 2026, Parsaeefard et al., 2021). Prototype-based systems preserve privacy better than raw-data sharing but still communicate latent structures that may encode distributional priors; ProMoE-FL identifies privacy leakage from prototype vectors as an open issue (Chhetri et al., 7 Jul 2026). Networked systems such as NMoE reduce memory pressure but expose inference-time latents to neighboring devices and assume stable connectivity (Gao et al., 3 Nov 2025). STN-oriented systems such as EMS-FL trade synchronous simplicity for operational complexity, including expert reassignment and contact-window scheduling (Xu et al., 23 Feb 2026).

Open questions remain technically sharp. FlexMoRE identifies adaptive rank selection and per-layer heterogeneity as unresolved directions (Pirchert et al., 9 Feb 2026). FedLEASE suggests that static clustering can become brittle under client drift and that dynamic clustering or meta-routing may be necessary (Wang et al., 18 Sep 2025). SmartFed points toward privacy-preserving importance aggregation and further router compression (Wu et al., 30 Nov 2025). System-level FedMox proposals argue for reinforcement-learning-based alignment policies, hardware-aware routing, and hierarchical scheduling across edge, fog, and cloud tiers (Chen et al., 8 Jul 2025). Across nearly all variants, formal convergence analysis lags behind architectural innovation, especially once sparse routing, private gating, or asynchronous connectivity are combined.

FedMox therefore occupies a broad and still-fragmented research area. Its central promise is clear: expert modularity creates a way to localize knowledge, communication, and computation simultaneously. Its central challenge is equally clear: once experts become modular, every choice about routing, ownership, aggregation, and synchronization becomes a first-class design decision rather than a detail of optimization.