Fine-Grained Mixture of Experts (FGMoE)

• Fine-Grained Mixture of Experts (FGMoE) is a family of sparse architectures that decompose traditional experts into smaller units, enabling token-, spatial-, or task-level routing.
• The design preserves compute budgets by shrinking expert sizes while increasing the number of activated experts, leading to improved convergence and expressivity.
• Empirical studies and scaling-law analyses show FGMoE can outperform standard MoE models, though increased routing overhead requires careful system-level tuning.

Fine-Grained Mixture of Experts (FGMoE) denotes a family of sparse-parameter architectures that refine the standard Mixture-of-Experts (MoE) paradigm by increasing the number of experts while reducing their individual capacity and, in many formulations, by routing at a finer token-, spatial-, or task-level granularity. In the large-language-model setting, a common formalization decomposes each traditional expert of size $d_{ff}$ into $G$ smaller experts of size $d_{ff}/G$ and routes each token to $k\!\cdot\!G$ experts, thereby preserving total active parameters and non-router FLOPs per token while enlarging the routing space (Krajewski et al., 3 Jun 2025). Related work uses the same broad label for several closely related ideas: treating granularity $G$ as an architectural hyperparameter controlling expert width (Krajewski et al., 2024), treating granularity $k$ as the number of experts activated per token (Boix-Adsera et al., 11 May 2025), or implementing fine-graining through low-rank adapters, spatial routing, region-aware specialists, or fine-grained systems support (Yang et al., 1 Oct 2025).

1. Formal definition and terminological scope

In a standard MoE Transformer layer, a feed-forward layer is replaced by $N_E$ experts $E_i(x)\in\mathbb{R}^{d_{ff}}$, and a lightweight router computes logits $s_i(x)=w_i^\top x$, normalizes them, and selects the top-$k$ experts. The output is

$G$0

where $G$1 is the selected expert set and $G$2 are the normalized gating weights (Krajewski et al., 3 Jun 2025).

FGMoE generalizes this construction by replacing $G$3 experts of width $G$4 with $G$5 experts of width $G$6, while routing each token to $G$7 experts:

$G$8

Under this definition, the active computation budget is preserved, but the router can distribute computation across a larger pool of smaller units (Krajewski et al., 3 Jun 2025).

The literature does not use a single invariant meaning for “granularity.” Two usages recur most often. In scaling-law work, granularity is the ratio

$G$9

so that each expert has hidden width $d_{ff}/G$0 and each token is routed to $d_{ff}/G$1 mini-experts, preserving the dense FFN’s active parameter count per token (Krajewski et al., 2024). In expressivity analysis, granularity is the number of experts activated per token, denoted $d_{ff}/G$2, and the central question is how representational capacity changes as more, smaller experts are simultaneously active (Boix-Adsera et al., 11 May 2025).

Usage in the literature	Variable	Meaning
LLM scaling-law FGMoE	$d_{ff}/G$3	Expert width becomes $d_{ff}/G$4
Routing-granularity theory	$d_{ff}/G$5	Number of activated experts per token
LoRA and MTL variants	$d_{ff}/G$6 or task/region partitions	More experts with smaller rank or smaller hidden partitions

This terminological variation matters because papers that all describe “fine-grained MoE” may optimize different objects: some refine expert size, some refine activation patterns, and some refine the operational handling of experts after routing.

2. Theoretical foundations: scaling laws and expressivity

The most explicit scaling-law treatment models FGMoE loss as a joint function of non-embedding parameter count $d_{ff}/G$7, training tokens $d_{ff}/G$8, and granularity $d_{ff}/G$9:

$k\!\cdot\!G$0

For expansion rate $k\!\cdot\!G$1, the reported best-fit coefficients are $k\!\cdot\!G$2, $k\!\cdot\!G$3, $k\!\cdot\!G$4, $k\!\cdot\!G$5, $k\!\cdot\!G$6, $k\!\cdot\!G$7, and $k\!\cdot\!G$8 (Krajewski et al., 2024). Within that study, compute-optimal FGMoE with $k\!\cdot\!G$9 outperforms both dense Transformers and standard MoE with $G$0 across budgets from $G$1 to $G$2 FLOPs, and the common practice of setting expert size equal to the full FFN is reported as “not optimal at almost any computational budget” (Krajewski et al., 2024).

The same work gives budget-dependent rules of thumb for optimal granularity: approximately $G$3 at $G$4 FLOPs, $G$5 at $G$6, $G$7 at $G$8, and $G$9 at $k$0–$k$1, while warning that routing costs scale with $k$2 and very large values can become routing-bound (Krajewski et al., 2024). A plausible implication is that FGMoE should be treated as a joint architecture–compute allocation problem, not merely as a drop-in replacement for dense FFNs.

A separate theoretical line studies granularity as the number of active experts. There, an MoE layer with routing vectors $k$3, expert matrices $k$4, and hard top-$k$5 gating is written as

$k$6

and the main theorem establishes an exponential separation in expressivity as granularity increases (Boix-Adsera et al., 11 May 2025). In the reported result, for $k$7, there exist functions exactly realized by an $k$8-MoE that every $k$9-MoE with the same active-parameter count fails to approximate unless $N_E$0 grows on the order of $N_E$1 in the $N_E$2 case (Boix-Adsera et al., 11 May 2025). The paper’s student–teacher experiments further report that when a teacher has granularity $N_E$3, students with $N_E$4 do not drive test MSE to zero under the matched active-neuron budget, while $N_E$5 does (Boix-Adsera et al., 11 May 2025).

Together, these results frame FGMoE as more than a systems heuristic. One strand argues that finer experts improve the compute-optimal frontier (Krajewski et al., 2024); another argues that activating more fine-grained experts enlarges the realizable function class exponentially (Boix-Adsera et al., 11 May 2025).

3. Large-language-model practice and empirical scaling

A direct empirical comparison between standard and fine-grained MoE at scales up to 56B total parameters evaluates four variants: Switch-like Top-1 and fine-grained Top-8 with $N_E$6 at $N_E$7MoE FLOPs, and Mixtral-like Top-2 and fine-grained Top-16 with $N_E$8 at $N_E$9MoE FLOPs (Krajewski et al., 3 Jun 2025). At approximately 11B total parameters, the $E_i(x)\in\mathbb{R}^{d_{ff}}$0 configurations use $E_i(x)\in\mathbb{R}^{d_{ff}}$1 for $E_i(x)\in\mathbb{R}^{d_{ff}}$2 and $E_i(x)\in\mathbb{R}^{d_{ff}}$3 for $E_i(x)\in\mathbb{R}^{d_{ff}}$4, both with active parameters of approximately 2.7B; the $E_i(x)\in\mathbb{R}^{d_{ff}}$5 variants analogously use active parameters of approximately 3.9B. At approximately 56B total parameters, the same expert counts are retained while $E_i(x)\in\mathbb{R}^{d_{ff}}$6 and $E_i(x)\in\mathbb{R}^{d_{ff}}$7 for $E_i(x)\in\mathbb{R}^{d_{ff}}$8 or $E_i(x)\in\mathbb{R}^{d_{ff}}$9 for $s_i(x)=w_i^\top x$0, with active parameters of approximately 10.7B at $s_i(x)=w_i^\top x$1 FLOPs and 17.1B at $s_i(x)=w_i^\top x$2 FLOPs (Krajewski et al., 3 Jun 2025).

The reported training recipe uses AdamW with $s_i(x)=w_i^\top x$3, $s_i(x)=w_i^\top x$4, weight decay $s_i(x)=w_i^\top x$5, a batch of 4M tokens, sequence length 2048, learning rate $s_i(x)=w_i^\top x$6, cosine decay with linear warmup over the first 1% of steps, initialization $s_i(x)=w_i^\top x$7, and attention dropout $s_i(x)=w_i^\top x$8 (Krajewski et al., 3 Jun 2025). MoE-specific settings include capacity_factor $s_i(x)=w_i^\top x$9, auxiliary load-balancing loss $k$0 with $k$1, and $k$2-loss $k$3 (Krajewski et al., 3 Jun 2025). The router is a single linear layer followed by softmax and top-$k$4 selection; for $k$5, the reported recommendation is to apply softmax after top-$k$6 selection so that the selected experts’ weights sum to one, whereas for $k$7 softmax must precede top-$k$8 to preserve gradients, with Table 8 showing up to $k$9 loss improvement (Krajewski et al., 3 Jun 2025).

The empirical results favor fine-grained configurations at both moderate and larger scales. At 11B, the fine-grained Switch configuration reaches final validation loss $G$00 versus $G$01 for the standard $G$02 baseline and saves approximately 28% of steps to match the baseline loss at 50B tokens; the $G$03 fine-grained model shows marginal gains at 50B tokens and 39% step savings at 100B tokens (Krajewski et al., 3 Jun 2025). At 56B, $G$04FLOPs-$G$05 matches $G$06FLOPs-$G$07 at approximately $G$08 validation loss, while $G$09FLOPs-$G$10 achieves the best loss, $G$11 (Krajewski et al., 3 Jun 2025).

The downstream benchmark results exhibit the same pattern. Average accuracy across 10 tasks at 11B is 50.6% for $G$12FLOPs-$G$13 versus 48.4% for $G$14FLOPs-$G$15, and 51.5% for $G$16FLOPs-$G$17 versus 50.5% for $G$18FLOPs-$G$19; at 56B, the corresponding figures are 59.0% versus 57.3% and 60.5% versus 58.8%, with gains described as consistent across ARC, MMLU, and HellaSwag (Krajewski et al., 3 Jun 2025). The paper’s summary guideline is that when inference cost is constrained, $G$20FLOPs-$G$21 often matches or exceeds standard Top-2 MoE at half the active parameters (Krajewski et al., 3 Jun 2025).

4. Communication, load balancing, and serving systems

FGMoE’s architectural benefits are coupled to a recurrent systems problem: increasing the number of experts and activated routes raises router overhead and All-to-All communication. In the 56B-scale H100 experiments above, a Megatron-LM expert-parallel setup with EP size 8 still sustains more than 46% MFU on MoE layers, but the same study explicitly identifies increased router and communication cost as a practical consideration and recommends tuning expert-parallel size against tensor and pipeline parallelism (Krajewski et al., 3 Jun 2025).

One line of work addresses this bottleneck by changing the communication pattern itself. BigMac argues that prior fine-grained MoEs, exemplified there by DeepSeekMoE, follow a communicate-descend-ascend-communicate (CDAC) pattern in which both All-to-All operations occur at the full hidden dimension $G$22 (Jin et al., 24 Feb 2025). BigMac introduces descend-communicate-communicate-ascend (DCCA): tokens are first projected from $G$23 to $G$24, expert computation occurs in that lower dimension, and only then are outputs projected back. The resulting communication volume becomes

$G$25

so for $G$26 the token-movement volume is reduced by 75% (Jin et al., 24 Feb 2025). In the reported GPT3-XL comparison with $G$27, $G$28, and $G$29, fine-grained MoE uses 3.73B parameters, 3490.7T FLOPs, and 1488 GB of All-to-All per iteration, whereas BigMac uses 3.78B parameters, 3649.0T FLOPs, and 372 GB of All-to-All (Jin et al., 24 Feb 2025). Training and inference results report end-to-end gains up to $G$30 for training and $G$31 for inference while matching or exceeding fine-grained MoE quality on most downstream evaluations (Jin et al., 24 Feb 2025).

A second line of work treats fine granularity as a load-balancing problem over micro-batches. MicroMoE introduces MicroEP, which merges $G$32 expert-parallel groups into a larger MicroEP group and distributes expert loads across replicated experts by solving a per-micro-batch linear program minimizing the maximum GPU load (Zhao et al., 21 Nov 2025). The imbalance ratio is defined as

$G$33

with large $G$34 indicating a straggler GPU (Zhao et al., 21 Nov 2025). In the reported experiments on 4 nodes of 8 NVIDIA H100 80GB GPUs with 400 Gbps InfiniBand, MicroMoE improves end-to-end throughput by up to 47.6% over the state-of-the-art system, achieves an average improvement of 36.9%, and, in its adaptive form, maintains $G$35 across Zipf skewness $G$36 (Zhao et al., 21 Nov 2025). LP solve time is reported as at most 0.8 ms for 256 experts on 64 GPUs, and token routing at most 0.2 ms (Zhao et al., 21 Nov 2025).

A third line focuses on inference-time memory management for sparse experts. The serving system fMoE records per-iteration expert probability maps $G$37, matches them with historical maps using a blend of semantic and trajectory similarity, and prefetches only enough experts to exceed a dynamic threshold $G$38 while never fetching fewer than the top-$G$39 experts (Yu et al., 7 Feb 2025). On a six-GPU RTX 3090 testbed, the reported averages across Mixtral-8B, Qwen1.5-MoE, and Phi-3.5-MoE show TTFT reductions of 44% versus DeepSpeed, 35% versus Mixtral-Offload, and 30% versus MoE-Infinity; TPOT reductions of 70% versus DeepSpeed, 55% versus ProMoE, and 48% versus MoE-Infinity; and expert hit-rate improvements of 147% versus DeepSpeed and 63% versus MoE-Infinity (Yu et al., 7 Feb 2025). The abstract summarizes these results as a 47% inference-latency reduction and a 36% hit-rate improvement over state-of-the-art solutions (Yu et al., 7 Feb 2025).

5. Variants in multi-task learning, vision, and spatial modeling

Outside autoregressive language modeling, FGMoE appears in several domain-specific forms that preserve the same core idea: finer decomposition of capacity under a constrained parameter or compute envelope.

In LoRA-based multi-task learning, fine-graining is implemented by increasing the number of LoRA experts $G$40 while decreasing each expert’s rank $G$41 so that the total LoRA parameter budget remains roughly constant, using the heuristic constraint $G$42 (Yang et al., 1 Oct 2025). The reported configurations are Base $G$43, Medium $G$44, and High $G$45 for $G$46, where $G$47 denotes adaptive shared experts jointly normalized with sparse experts (Yang et al., 1 Oct 2025). On PASCAL-Context with a ViT-base backbone at $G$48 for 40 epochs, the baseline LoRA-MoE $G$49 achieves $G$50 over single-task ViT, ASE plus fine-grained $G$51 yields $G$52, and the medium fine-grained setting $G$53 gives the best overall result, $G$54 (Yang et al., 1 Oct 2025).

For dense prediction fine-tuning, FGMoE is defined through intra-task experts that partition along intermediate hidden dimensions of MLPs, shared experts that capture common information within a task, and a global expert layer that mediates cross-task transfer (Xu et al., 25 Jul 2025). The encoder is frozen and only the decoder is fine-tuned. On PASCAL-Context with a Swin-L backbone, the decoder-only model FGMoE+Swin-L reports 77.34 for mIoU, 64.16 for PartSeg, 64.13 for maxF, 16.40 for mErr, $G$55 Am, and 4.70M trainable parameters (Xu et al., 25 Jul 2025). On NYUD-v2 with Swin-T, FGMoE-tune reports 44.71 semantic-segmentation mIoU, 0.618 depth rmse, $G$56 Am, and 2.50M trainable parameters (Xu et al., 25 Jul 2025). The ablations further show that adding Deformable Mixer plus Global MoE, then Task-specific MoE, improves a multi-task baseline from 38.78/0.6312/21.05/75.6/−3.74 to 47.12/0.5616/20.17/77.1/+5.17 across SemSeg/Depth/Normal/Boundary/Am on NYUD-v2 (Xu et al., 25 Jul 2025).

In real-time open-vocabulary object detection, Dynamic-DINO decomposes each decoder FFN of hidden size $G$57 into $G$58 blocks of size $G$59 so that

$G$60

then replicates this split across $G$61 FFNs to obtain $G$62 experts (Lu et al., 23 Jul 2025). The method also initializes experts by slicing pretrained FFN weights and initializes routers so that groups of $G$63 experts reconstruct the original FFN at step zero, preventing an initial performance drop (Lu et al., 23 Jul 2025). With 16 experts and top-2 routing, zero-shot results at $G$64 improve COCO AP_box from 42.6 to 43.7 and LVIS AP_all from 31.1 to 33.6, while measured speed on A100 declines from 20.2 FPS to 17.1 FPS in PyTorch and from 108.9 FPS to 98.0 FPS in TensorRT FP32; on Jetson Orin NX, speed changes from 10.2 FPS to 9.4 FPS (Lu et al., 23 Jul 2025).

In face-forgery detection, the SFAM framework uses a facial-region mixture-of-experts (FaRMoE) in which patches are assigned to semantic regions such as eyes, nose, or mouth by landmark-based hard routing, and each region expert replaces CLIP’s shared key projection with a region-specific linear projection $G$65 (Luo et al., 23 Apr 2026). In the five-dataset ablation, Cross-AUC Avg improves from approximately 0.740 for baseline CLIP to approximately 0.885 for the full system, while Cross-AUC Min improves from approximately 0.601 to approximately 0.747 (Luo et al., 23 Apr 2026).

In spatial modeling, SMoE routes each spatial location independently by selecting the top-$G$66 experts per $G$67 location in a feature map, optionally using a learned tensor-routing gate $G$68 that is independent of the input (Dryden et al., 2022). The output is formed by gathering the selected expert outputs at each spatial site (Dryden et al., 2022). On a 64×64 location-dependent heat-diffusion task, SMoE with 3 experts and $G$69 reaches 100% of gridpoints within 1% error in 8 epochs, while a locally connected network also reaches 100% but uses 3× more parameters and requires 110 epochs (Dryden et al., 2022). On WeatherBench, replacing all 3×3 convolutions with SMoE reduces Z500 RMSE at 3 days from 316 to 278, then to 270 with a gate prior, 253 with CMIP6 pretraining, 232 with extra ERA5 back-extension, and 198 with higher-resolution data (Dryden et al., 2022).

6. Trade-offs, misconceptions, and research directions

A recurrent misconception is that “fine-grained” has a single canonical implementation. The literature instead spans soft token-level routing with top-$G$70 gating in LLMs (Krajewski et al., 3 Jun 2025), joint normalization of sparse and shared experts in LoRA-based MTL (Yang et al., 1 Oct 2025), deterministic one-hot routing by facial region (Luo et al., 23 Apr 2026), per-pixel top-$G$71 spatial routing (Dryden et al., 2022), and fine-grained micro-batch scheduling or expert offloading at the systems layer (Zhao et al., 21 Nov 2025, Yu et al., 7 Feb 2025). These are related by the granularity of specialization and dispatch, but not by a single router design.

A second misconception is that fine-graining necessarily increases the active compute budget. In the main LLM formalization, FGMoE preserves total non-router FLOPs and active parameters per token by shrinking expert size as expert count and the number of selected experts increase (Krajewski et al., 3 Jun 2025). BigMac similarly aims to preserve quality while shifting communication to a lower-dimensional representation, accepting a reported FLOP increase of 4.54% in exchange for a 75% communication reduction in its illustrative configuration (Jin et al., 24 Feb 2025).

The trade-offs are explicit across the literature. Fine-grained routing raises router cost and All-to-All pressure (Krajewski et al., 3 Jun 2025, Jin et al., 24 Feb 2025). Larger expert counts can create micro-batch load imbalance unless auxiliary losses, capacity factors, replica scheduling, or migration mechanisms are added (Krajewski et al., 3 Jun 2025, Zhao et al., 21 Nov 2025). In LoRA-based MTL, very fine settings such as $G$72 are reported as similar or slightly higher performing but with more router cost (Yang et al., 1 Oct 2025). In Dynamic-DINO, splitting each FFN into $G$73 is reported as optimal, while $G$74 can overfit on limited data (Lu et al., 23 Jul 2025). The scaling-law work likewise warns that very large $G$75 can become routing-bound even when it is statistically favorable (Krajewski et al., 2024).

Several directions are already named in the current literature. MicroMoE proposes heuristics in place of LP solving for low-latency inference scenarios, hierarchical scheduling for very large clusters, integration with FSDP or ZeRO-3, and more communication-aware objectives (Zhao et al., 21 Nov 2025). fMoE identifies cold-start behavior, prefetch-distance tuning, richer similarity metrics, and adaptive thresholding as open system-level issues (Yu et al., 7 Feb 2025). Theoretical and empirical work together suggest that the central unresolved design question is not whether granularity matters, but how to allocate it among expert size, number of active experts, routing normalization, communication topology, and training horizon for a fixed budget (Krajewski et al., 2024, Boix-Adsera et al., 11 May 2025, Krajewski et al., 3 Jun 2025).

FGMoE is therefore best understood as a design principle rather than a single architecture. Its common claim is that under sparse conditional computation, finer decomposition of expert capacity can improve approximation, convergence, specialization, or deployment efficiency, provided that routing, balance, and communication are co-designed with the expert layout.