---
title: Fine-Grained Mixture of Experts (FGMoE)
url: https://www.emergentmind.com/topics/fine-grained-mixture-of-experts-fgmoe
type: topic
---

# Fine-Grained Mixture of Experts (FGMoE)

Fine-Grained Mixture of Experts (FGMoE) denotes a family of sparse-parameter architectures that refine the standard Mixture-of-Experts (MoE) paradigm by increasing the number of experts while reducing their individual capacity and, in many formulations, by routing at a finer token-, spatial-, or task-level granularity. In the large-language-model setting, a common formalization decomposes each traditional expert of size $d_{ff}$ into $G$ smaller experts of size $d_{ff}/G$ and routes each token to $k\!\cdot\!G$ experts, thereby preserving total active parameters and non-router FLOPs per token while enlarging the routing space [2506.02890]. Related work uses the same broad label for several closely related ideas: treating granularity $G$ as an architectural hyperparameter controlling expert width [2402.07871], treating granularity $k$ as the number of experts activated per token [2505.06839], or implementing fine-graining through low-rank adapters, spatial routing, region-aware specialists, or fine-grained systems support [2510.00570].

## 1. Formal definition and terminological scope

In a standard MoE Transformer layer, a feed-forward layer is replaced by $N_E$ experts $E_i(x)\in\mathbb{R}^{d_{ff}}$, and a lightweight router computes logits $s_i(x)=w_i^\top x$, normalizes them, and selects the top-$k$ experts. The output is
$$
y(x)=\sum_{i\in \tau(x)} R_i(x)\,E_i(x),
$$
where $\tau(x)\subset\{1,\dots,N_E\}$ is the selected expert set and $R_i(x)$ are the normalized gating weights [2506.02890].

FGMoE generalizes this construction by replacing $N_E$ experts of width $d_{ff}$ with $G\!\cdot\!N_E$ experts of width $d_{ff}/G$, while routing each token to $k\!\cdot\!G$ experts:
$$
y(x)=\sum_{i\in \tau_G(x)} R_i(x)\,E_i(x),\qquad |\tau_G|=k\!\cdot\!G.
$$
Under this definition, the active computation budget is preserved, but the router can distribute computation across a larger pool of smaller units [2506.02890].

The literature does not use a single invariant meaning for “granularity.” Two usages recur most often. In scaling-law work, granularity is the ratio
$$
G=\frac{\text{ff}}{\text{expert}},
$$
so that each expert has hidden width $\text{expert}=\text{ff}/G$ and each token is routed to $G$ mini-experts, preserving the dense FFN’s active parameter count per token [2402.07871]. In expressivity analysis, granularity is the number of experts activated per token, denoted $k$, and the central question is how representational capacity changes as more, smaller experts are simultaneously active [2505.06839].

| Usage in the literature | Variable | Meaning |
|---|---:|---|
| LLM scaling-law FGMoE | $G$ | Expert width becomes $\text{ff}/G$ |
| Routing-granularity theory | $k$ | Number of activated experts per token |
| LoRA and MTL variants | $(N,r)$ or task/region partitions | More experts with smaller rank or smaller hidden partitions |

This terminological variation matters because papers that all describe “fine-grained MoE” may optimize different objects: some refine expert size, some refine activation patterns, and some refine the operational handling of experts after routing.

## 2. Theoretical foundations: scaling laws and expressivity

The most explicit scaling-law treatment models FGMoE loss as a joint function of non-embedding parameter count $N$, training tokens $D$, and granularity $G$:
$$
\mathcal{L}(N,D,G)
=
c+\Bigl(a+\frac{g}{G^\gamma}\Bigr)N^{-\alpha}+bD^{-\beta}.
$$
For expansion rate $E=64$, the reported best-fit coefficients are $c=0.47$, $a=18.1$, $\alpha=0.115$, $b=30.8$, $\beta=0.147$, $g=2.1$, and $\gamma=0.58$ [2402.07871]. Within that study, compute-optimal FGMoE with $G>1$ outperforms both dense Transformers and standard MoE with $G=1$ across budgets from $10^{18}$ to $10^{25}$ FLOPs, and the common practice of setting expert size equal to the full FFN is reported as “not optimal at almost any computational budget” [2402.07871].

The same work gives budget-dependent rules of thumb for optimal granularity: approximately $G^\*\!\approx 8$ at $F\sim 10^{18}$ FLOPs, $16$ at $10^{20}$, $32$ at $10^{22}$, and $64$ at $10^{24}$–$10^{25}$, while warning that routing costs scale with $G$ and very large values can become routing-bound [2402.07871]. A plausible implication is that FGMoE should be treated as a joint architecture–compute allocation problem, not merely as a drop-in replacement for dense FFNs.

A separate theoretical line studies granularity as the number of active experts. There, an MoE layer with routing vectors $r_i$, expert matrices $A_i,B_i$, and hard top-$k$ gating is written as
$$
y(x)=\sum_{i\in S(x)} A_i\,\sigma(B_i^\top x),
$$
and the main theorem establishes an exponential separation in expressivity as granularity increases [2505.06839]. In the reported result, for $k'<k$, there exist functions exactly realized by an $(m,k)$-MoE that every $(m',k')$-MoE with the same active-parameter count fails to approximate unless $m'$ grows on the order of $\exp(\Omega(k\log m))$ in the $k'=1$ case [2505.06839]. The paper’s student–teacher experiments further report that when a teacher has granularity $k=8$, students with $k'\in\{1,2,4\}$ do not drive test MSE to zero under the matched active-neuron budget, while $k'\in\{8,16\}$ does [2505.06839].

Together, these results frame FGMoE as more than a systems heuristic. One strand argues that finer experts improve the compute-optimal frontier [2402.07871]; another argues that activating more fine-grained experts enlarges the realizable function class exponentially [2505.06839].

## 3. Large-language-model practice and empirical scaling

A direct empirical comparison between standard and fine-grained MoE at scales up to 56B total parameters evaluates four variants: Switch-like Top-1 and fine-grained Top-8 with $G=8$ at $1\times$MoE FLOPs, and Mixtral-like Top-2 and fine-grained Top-16 with $G=8$ at $2\times$MoE FLOPs [2506.02890]. At approximately 11B total parameters, the $1\times$ configurations use $N_E=8,k=1,d_{model}=2048,d_{expert}=8192$ for $G=1$ and $N_E=64,k=8,d_{expert}=1024$ for $G=8$, both with active parameters of approximately 2.7B; the $2\times$ variants analogously use active parameters of approximately 3.9B. At approximately 56B total parameters, the same expert counts are retained while $d_{model}=4096$ and $d_{expert}=16384$ for $G=1$ or $2048$ for $G=8$, with active parameters of approximately 10.7B at $1\times$ FLOPs and 17.1B at $2\times$ FLOPs [2506.02890].

The reported training recipe uses AdamW with $\beta_1=0.9$, $\beta_2=0.95$, weight decay $0.1$, a batch of 4M tokens, sequence length 2048, learning rate $2\times 10^{-4}$, cosine decay with linear warmup over the first 1% of steps, initialization $N(0,0.01)$, and attention dropout $0.1$ [2506.02890]. MoE-specific settings include capacity\_factor $=1.5$, auxiliary load-balancing loss $L_{load}=\lambda\cdot \mathrm{Var}(\{R_i(x)\})$ with $\lambda=10^{-2}$, and $z$-loss $=10^{-3}$ [2506.02890]. The router is a single linear layer followed by softmax and top-$k$ selection; for $k>1$, the reported recommendation is to apply softmax after top-$k$ selection so that the selected experts’ weights sum to one, whereas for $k=1$ softmax must precede top-$k$ to preserve gradients, with Table 8 showing up to $0.036$ loss improvement [2506.02890].

The empirical results favor fine-grained configurations at both moderate and larger scales. At 11B, the fine-grained Switch configuration reaches final validation loss $2.183$ versus $2.233$ for the standard $1\times$ baseline and saves approximately 28% of steps to match the baseline loss at 50B tokens; the $2\times$ fine-grained model shows marginal gains at 50B tokens and 39% step savings at 100B tokens [2506.02890]. At 56B, $1\times$FLOPs-$G8$ matches $2\times$FLOPs-$G1$ at approximately $1.78$ validation loss, while $2\times$FLOPs-$G8$ achieves the best loss, $1.757$ [2506.02890].

The downstream benchmark results exhibit the same pattern. Average accuracy across 10 tasks at 11B is 50.6% for $1\times$FLOPs-$G8$ versus 48.4% for $1\times$FLOPs-$G1$, and 51.5% for $2\times$FLOPs-$G8$ versus 50.5% for $2\times$FLOPs-$G1$; at 56B, the corresponding figures are 59.0% versus 57.3% and 60.5% versus 58.8%, with gains described as consistent across ARC, MMLU, and HellaSwag [2506.02890]. The paper’s summary guideline is that when inference cost is constrained, $1\times$FLOPs-$G8$ often matches or exceeds standard Top-2 MoE at half the active parameters [2506.02890].

## 4. Communication, load balancing, and serving systems

FGMoE’s architectural benefits are coupled to a recurrent systems problem: increasing the number of experts and activated routes raises router overhead and All-to-All communication. In the 56B-scale H100 experiments above, a Megatron-LM expert-parallel setup with EP size 8 still sustains more than 46% MFU on MoE layers, but the same study explicitly identifies increased router and communication cost as a practical consideration and recommends tuning expert-parallel size against tensor and pipeline parallelism [2506.02890].

One line of work addresses this bottleneck by changing the communication pattern itself. BigMac argues that prior fine-grained MoEs, exemplified there by DeepSeekMoE, follow a communicate-descend-ascend-communicate (CDAC) pattern in which both All-to-All operations occur at the full hidden dimension $h$ [2502.16927]. BigMac introduces descend-communicate-communicate-ascend (DCCA): tokens are first projected from $h$ to $r\!\cdot\!h$, expert computation occurs in that lower dimension, and only then are outputs projected back. The resulting communication volume becomes
$$
C_{DCCA}=r\cdot C_{CDAC},
$$
so for $r=0.25$ the token-movement volume is reduced by 75% [2502.16927]. In the reported GPT3-XL comparison with $r=0.25$, $e=64$, and $ep=32$, fine-grained MoE uses 3.73B parameters, 3490.7T FLOPs, and 1488 GB of All-to-All per iteration, whereas BigMac uses 3.78B parameters, 3649.0T FLOPs, and 372 GB of All-to-All [2502.16927]. Training and inference results report end-to-end gains up to $3.09\times$ for training and $3.11\times$ for inference while matching or exceeding fine-grained MoE quality on most downstream evaluations [2502.16927].

A second line of work treats fine granularity as a load-balancing problem over micro-batches. MicroMoE introduces MicroEP, which merges $d$ expert-parallel groups into a larger MicroEP group and distributes expert loads across replicated experts by solving a per-micro-batch linear program minimizing the maximum GPU load [2511.16947]. The imbalance ratio is defined as
$$
\rho=\frac{\max_{g\in G} L_g}{(1/|G|)\sum_{g\in G}L_g},
$$
with large $\rho$ indicating a straggler GPU [2511.16947]. In the reported experiments on 4 nodes of 8 NVIDIA H100 80GB GPUs with 400 Gbps InfiniBand, MicroMoE improves end-to-end throughput by up to 47.6% over the state-of-the-art system, achieves an average improvement of 36.9%, and, in its adaptive form, maintains $\rho\approx 1$ across Zipf skewness $s\in[0,2]$ [2511.16947]. LP solve time is reported as at most 0.8 ms for 256 experts on 64 GPUs, and token routing at most 0.2 ms [2511.16947].

A third line focuses on inference-time memory management for sparse experts. The serving system fMoE records per-iteration expert probability maps $\mathrm{map}_i=\{\mathbf{P}^{(i)}_1,\dots,\mathbf{P}^{(i)}_L\}$, matches them with historical maps using a blend of semantic and trajectory similarity, and prefetches only enough experts to exceed a dynamic threshold $\delta_\ell=\mathrm{Clip}(1-\sigma,0,1)$ while never fetching fewer than the top-$K$ experts [2502.05370]. On a six-GPU RTX 3090 testbed, the reported averages across Mixtral-8B, Qwen1.5-MoE, and Phi-3.5-MoE show TTFT reductions of 44% versus DeepSpeed, 35% versus Mixtral-Offload, and 30% versus MoE-Infinity; TPOT reductions of 70% versus DeepSpeed, 55% versus ProMoE, and 48% versus MoE-Infinity; and expert hit-rate improvements of 147% versus DeepSpeed and 63% versus MoE-Infinity [2502.05370]. The abstract summarizes these results as a 47% inference-latency reduction and a 36% hit-rate improvement over state-of-the-art solutions [2502.05370].

## 5. Variants in multi-task learning, vision, and spatial modeling

Outside autoregressive language modeling, FGMoE appears in several domain-specific forms that preserve the same core idea: finer decomposition of capacity under a constrained parameter or compute envelope.

In LoRA-based multi-task learning, fine-graining is implemented by increasing the number of LoRA experts $N$ while decreasing each expert’s rank $r$ so that the total LoRA parameter budget remains roughly constant, using the heuristic constraint $N\!\cdot\!r=\text{const}$ [2510.00570]. The reported configurations are Base $(16/3/1/4)$, Medium $(32/6/2/2)$, and High $(64/12/4/1)$ for $(N/k/S/r)$, where $S$ denotes adaptive shared experts jointly normalized with sparse experts [2510.00570]. On PASCAL-Context with a ViT-base backbone at $224\times 224$ for 40 epochs, the baseline LoRA-MoE $(16/4/0/4)$ achieves $\Delta_m=+6.06\%$ over single-task ViT, ASE plus fine-grained $(16/3/1/4)$ yields $\Delta_m=+7.49\%$, and the medium fine-grained setting $(32/6/2/2)$ gives the best overall result, $\Delta_m=+7.58\%$ [2510.00570].

For dense prediction fine-tuning, FGMoE is defined through intra-task experts that partition along intermediate hidden dimensions of MLPs, shared experts that capture common information within a task, and a global expert layer that mediates cross-task transfer [2507.19077]. The encoder is frozen and only the decoder is fine-tuned. On PASCAL-Context with a Swin-L backbone, the decoder-only model FGMoE+Swin-L reports 77.34 for mIoU, 64.16 for PartSeg, 64.13 for maxF, 16.40 for mErr, $+7.89$ Am, and 4.70M trainable parameters [2507.19077]. On NYUD-v2 with Swin-T, FGMoE-tune reports 44.71 semantic-segmentation mIoU, 0.618 depth rmse, $+21.02$ Am, and 2.50M trainable parameters [2507.19077]. The ablations further show that adding Deformable Mixer plus Global MoE, then Task-specific MoE, improves a multi-task baseline from 38.78/0.6312/21.05/75.6/−3.74 to 47.12/0.5616/20.17/77.1/+5.17 across SemSeg/Depth/Normal/Boundary/Am on NYUD-v2 [2507.19077].

In real-time open-vocabulary object detection, Dynamic-DINO decomposes each decoder FFN of hidden size $H$ into $k$ blocks of size $H/k$ so that
$$
\mathrm{FFN}(x)=\sum_{j=1}^{k} E_j(x),
$$
then replicates this split across $N$ FFNs to obtain $E=N\!\cdot\!k$ experts [2507.17436]. The method also initializes experts by slicing pretrained FFN weights and initializes routers so that groups of $k$ experts reconstruct the original FFN at step zero, preventing an initial performance drop [2507.17436]. With 16 experts and top-2 routing, zero-shot results at $640\times 640$ improve COCO AP\_box from 42.6 to 43.7 and LVIS AP\_all from 31.1 to 33.6, while measured speed on A100 declines from 20.2 FPS to 17.1 FPS in PyTorch and from 108.9 FPS to 98.0 FPS in TensorRT FP32; on Jetson Orin NX, speed changes from 10.2 FPS to 9.4 FPS [2507.17436].

In face-forgery detection, the SFAM framework uses a facial-region mixture-of-experts (FaRMoE) in which patches are assigned to semantic regions such as eyes, nose, or mouth by landmark-based hard routing, and each region expert replaces CLIP’s shared key projection with a region-specific linear projection $W_k^{(r_i)}$ [2604.21478]. In the five-dataset ablation, Cross-AUC Avg improves from approximately 0.740 for baseline CLIP to approximately 0.885 for the full system, while Cross-AUC Min improves from approximately 0.601 to approximately 0.747 [2604.21478].

In spatial modeling, SMoE routes each spatial location independently by selecting the top-$E$ experts per $(i,j)$ location in a feature map, optionally using a learned tensor-routing gate $D\in\mathbb{R}^{E_{total}\times H\times W}$ that is independent of the input [2211.13491]. The output is formed by gathering the selected expert outputs at each spatial site [2211.13491]. On a 64×64 location-dependent heat-diffusion task, SMoE with 3 experts and $E=1$ reaches 100% of gridpoints within 1% error in 8 epochs, while a locally connected network also reaches 100% but uses 3× more parameters and requires 110 epochs [2211.13491]. On WeatherBench, replacing all 3×3 convolutions with SMoE reduces Z500 RMSE at 3 days from 316 to 278, then to 270 with a gate prior, 253 with CMIP6 pretraining, 232 with extra ERA5 back-extension, and 198 with higher-resolution data [2211.13491].

## 6. Trade-offs, misconceptions, and research directions

A recurrent misconception is that “fine-grained” has a single canonical implementation. The literature instead spans soft token-level routing with top-$k$ gating in LLMs [2506.02890], joint normalization of sparse and shared experts in LoRA-based MTL [2510.00570], deterministic one-hot routing by facial region [2604.21478], per-pixel top-$E$ spatial routing [2211.13491], and fine-grained micro-batch scheduling or expert offloading at the systems layer [2511.16947; 2502.05370]. These are related by the granularity of specialization and dispatch, but not by a single router design.

A second misconception is that fine-graining necessarily increases the active compute budget. In the main LLM formalization, FGMoE preserves total non-router FLOPs and active parameters per token by shrinking expert size as expert count and the number of selected experts increase [2506.02890]. BigMac similarly aims to preserve quality while shifting communication to a lower-dimensional representation, accepting a reported FLOP increase of 4.54% in exchange for a 75% communication reduction in its illustrative configuration [2502.16927].

The trade-offs are explicit across the literature. Fine-grained routing raises router cost and All-to-All pressure [2506.02890; 2502.16927]. Larger expert counts can create micro-batch load imbalance unless auxiliary losses, capacity factors, replica scheduling, or migration mechanisms are added [2506.02890; 2511.16947]. In LoRA-based MTL, very fine settings such as $(64/12/4/1)$ are reported as similar or slightly higher performing but with more router cost [2510.00570]. In Dynamic-DINO, splitting each FFN into $k=2$ is reported as optimal, while $k>2$ can overfit on limited data [2507.17436]. The scaling-law work likewise warns that very large $G$ can become routing-bound even when it is statistically favorable [2402.07871].

Several directions are already named in the current literature. MicroMoE proposes heuristics in place of LP solving for low-latency inference scenarios, hierarchical scheduling for very large clusters, integration with FSDP or ZeRO-3, and more communication-aware objectives [2511.16947]. fMoE identifies cold-start behavior, prefetch-distance tuning, richer similarity metrics, and adaptive thresholding as open system-level issues [2502.05370]. Theoretical and empirical work together suggest that the central unresolved design question is not whether granularity matters, but how to allocate it among expert size, number of active experts, routing normalization, communication topology, and training horizon for a fixed budget [2402.07871; 2505.06839; 2506.02890].

FGMoE is therefore best understood as a design principle rather than a single architecture. Its common claim is that under sparse conditional computation, finer decomposition of expert capacity can improve approximation, convergence, specialization, or deployment efficiency, provided that routing, balance, and communication are co-designed with the expert layout.

Source: https://www.emergentmind.com/topics/fine-grained-mixture-of-experts-fgmoe