Fine-Grained MoE-LLMs Overview

• Fine-grained MoE-LLMs are large-scale transformer architectures that partition feed-forward networks into many small, specialized experts, enabling conditional computation and parameter efficiency.
• They implement dynamic, sensitivity-aware routing and post-training optimizations to enhance performance, reduce redundancy, and improve scaling efficiency.
• These models support advanced fine-tuning and lifelong learning through expert specialization and hierarchical collaboration, offering robust adaptability for diverse tasks.

Fine-grained Mixture-of-Experts LLMs (Fine-grained MoE-LLMs) refer to a family of large-scale neural architectures and adaptation strategies that partition the feed-forward or adapter subnets of Transformer-based LLMs into numerous, typically small-capacity, experts. Each input token is routed through a carefully chosen, typically small subset of these experts, achieving conditional computation and improved specialization. Fine-grained MoE-LLMs have evolved rapidly, incorporating innovations in quantization, routing, partitioning, system design, and training methodology. The field is characterized by rapid scaling, efficient fine-tuning frameworks, hardware-aware deployment optimizations, and the systematic paper of sparsity, flexibility, and specialization in both parameter and computational dimensions.

1. Architectural Principles and Granularity

Fine-grained MoE-LLMs partition model capacity at a high resolution, employing hundreds of experts per layer (as opposed to early MoEs where each expert replaces an entire FFN). The "granularity" $G$ is introduced to express the ratio of a feed-forward layer’s hidden size to the per-expert size, $G = \mathrm{ff}/\mathrm{expert}$ (Krajewski et al., 12 Feb 2024). Higher values of $G$ correspond to finer-grained designs, where each expert covers a smaller dimensional subspace and a token is simultaneously routed to more experts (often $G$). This allows scaling to larger parameter counts while controlling per-token computational and memory costs.

The architectural pipeline of fine-grained MoE layers involves:

• Routing: Each token’s representation $x$ is scored via a gating network $G(x)$ (often softmax or sigmoid), with the top-$k$ or threshold-selected experts activated per token.
• Expert Specialization: The output for a token is typically computed as $y = \sum_{i \in \tau} G_i(x) E_i(x)$, where $E_i$ is expert $i$, $\tau$ is the selected set, and $G_i$ is the normalized routing probability.
• Parameter Efficiency: By activating only a sparse subset per token, the network’s effective (active) parameter count per forward step is vastly reduced compared to the model’s dense size.
• Expert Fusion/Collaboration: Recent approaches incorporate graph-based (Bai et al., 18 Dec 2024) or hierarchical (Zeng et al., 8 Apr 2025) expert composition, enabling further flexibility beyond simple independent routing.

Fine-grained MoE-LLMs often extend dense LLMs by decomposing or partitioning pre-trained weights into numerous experts, as in upcycling dense FFNs into MoE blocks (Liao et al., 24 Jul 2025). In parameter-efficient fine-tuning, fine-grained design is combined with LoRA or heterogeneous adapters for efficiency and robustness (Cao et al., 6 Jun 2025).

2. Routing, Specialization, and Post-Training Optimization

Token-to-expert routing underpins the specialization and efficiency of fine-grained MoE-LLMs. Classical MoEs enforce strict load balancing during pre-training, utilizing auxiliary losses to evenly assign tokens and avoid "expert collapse." However, this approach can underutilize particularly influential experts and enforce unnecessary redundancy.

Recent methods propose post-training routing optimization strategies, such as:

• Ban&Pick: After standard MoE LLM training, "Pick" identifies and reinforces key experts via measuring output divergence (e.g., Kullback-Leibler divergence of output distributions upon expert removal), while "Ban" dynamically prunes redundant experts by computing layer- and token-level sensitivity scores. This plug-and-play approach delivers both measurable performance uplift (e.g., up to ~4 points on AIME2024) and inference speedup (e.g., 1.25× on Qwen3-30B-A3B) (Chen et al., 8 Sep 2025).
• Dynamic Expert Skipping/Pruning: In deployment, aggressively reducing the number of activated experts per token (while keeping the total number constant) may yield significant throughput gains (≥10%) with minor or negligible accuracy drop when the number of per-layer activated experts $n_a$ remains above a minimal threshold (Yang et al., 6 May 2025).
• Sensitivity-Aware Routing: Both Ban&Pick and expert-specialized fine-tuning (ESFT) leverage token-wise and layer-wise statistics (e.g., routing scores, average gate values, token selection ratios) to determine which experts are most critical for individual tokens or downstream tasks (Wang et al., 2 Jul 2024).
• Heterogeneous Routing: Instead of homogeneous, competitive softmax gating, some systems employ sigmoid-based collaborative gating, or even graph neural network routers, to further leverage representation diversity and enable cooperative activation of experts (Cao et al., 6 Jun 2025, Bai et al., 18 Dec 2024).

Routing strategies have thus shifted towards dynamic, adaptive, and task- or sample-specific expert selection in both training and post-training phases, enabling tailored specialization and redundancy reduction.

3. Scaling Laws, Training, and Empirical Performance

Fine-grained MoE-LLMs are subject to extended scaling laws that incorporate granularity as a key hyperparameter. Let $$L(N, D, G) = c + \frac{g/G^\gamma + a}{N^\alpha} + \frac{b}{D^\beta}$$ denote the validation loss, where $N$ is the number of model parameters, $D$ is the dataset size in tokens, and $G$ is the granularity. These laws reveal:

• With optimal $G$ and $N$ for a compute budget, MoE models consistently outperform dense Transformers with equivalent compute (Krajewski et al., 12 Feb 2024).
• The efficiency gap (relative compute-per-loss) between dense and MoE models widens as model and dataset scales increase.
• Setting expert size equal to FFN block size (i.e., $G=1$) is suboptimal; finer granularity (larger $G$) almost always improves efficiency and quality.
• Empirical studies with up to 56B parameters show that fine-grained MoE-LLMs can match or exceed the convergence speed and validation performance of much larger active-parameter standard MoEs, offering step savings of 33-39% in training (Krajewski et al., 3 Jun 2025).

Fine-grained MoE-LLMs, when trained with recipes incorporating AdamW, load balancing losses, softmax-after-TopK routers, and careful parallelism, demonstrate improved generalization and downstream accuracy on a range of benchmarks.

4. Adaptation, Parameter Efficiency, and Lifelong Learning

Fine-grained MoE architectures serve as a foundation for advanced parameter-efficient fine-tuning (PEFT) frameworks:

• MoE-LoRA/PEFT with Graph Collaboration: Hybrid schemes leverage multiple LoRA or heterogeneous adapter experts, which are dynamically composed per token or task (Cao et al., 6 Jun 2025). Gating functions based on softmax, sigmoid, or GNN aggregation enable either cooperative or competitive specialization. Graph-based router functions further facilitate inter-expert collaboration and load balancing (Bai et al., 18 Dec 2024).
• Lifelong Learning: MoE-augmented LoRA (MoRAL) frameworks distribute adapters across experts and route task-specific queries such that catastrophic forgetting is reduced, and new domain knowledge is absorbed efficiently from question–answer pairs without loss of general proficiency (Yang et al., 17 Feb 2024).
• Expert-Specialized Fine-Tuning (ESFT): Only the experts most relevant to the target task—determined via average gate scores or token selection ratios—are selectively trained, substantially reducing parameter updates while preserving or matching full fine-tuning performance. This is particularly effective in architectures with high expert granularity (Wang et al., 2 Jul 2024).
• Heterogeneous MoE: By using structurally diverse adapters within the same layer, MoA-style systems address representation collapse and load imbalance, increasing task-conditional specialization and pruning redundancy (Cao et al., 6 Jun 2025).

These developments link fine-grained MoE-LLMs directly to modular, task-adaptive LLMs and continual learning systems.

5. Inference, Serving, and System-Level Optimizations

The sparsity and irregularity of fine-grained MoE-LLMs present unique challenges and opportunities in inference and deployment:

• Expert Offloading: fMoE employs per-iteration expert pattern tracking ("expert maps") and semantic/trajectory-based matching to dynamically prefetch and evict experts between CPU and GPU memory. This fine-grained offloading reduces end-to-end latency by up to 47% and boosts expert hit rate by 36% over prior solutions (Yu et al., 7 Feb 2025).
• Dynamic Computation Dropping: The DualSparse-MoE system uses post-training expert partition and static neuron importance profiling to coordinate tensor-level and neuron-level sparsity with dynamic token-expert computation dropping ("2T-Drop"), yielding proportional computational speedups versus a minimal accuracy loss (0.08–0.28%) and 1.41× speedup under expert parallelism (Cai et al., 25 Aug 2025).
• Hardware Co-Design: A3D-MoE introduces 3D heterogeneous stacking and unified 3D dataflow to accommodate variable GEMV/GEMM ratios and exploit on-die memory for expert weight reuse. Hardware resource-aware fusion schedulers and selective FP-8 loading for low-gate-score experts further reduce DRAM access, total latency (up to 2×), and energy (up to 4×), while increasing throughput to 1.8× that of prior baselines (Huang et al., 25 Jul 2025).
• Inference-Pruning Strategies: Dynamic reduction of number of routed experts per token, guided by empirical sensitivity analysis, provides throughput benefits—with minimum expert count per-token enforced to avoid collapse (Yang et al., 6 May 2025). Plug-and-play routing modifications such as Ban&Pick offer additional efficiency gains and robustness without the need for retraining (Chen et al., 8 Sep 2025).

These system-level advances ensure that the practical advantages of fine-grained MoE-LLMs in speed and memory cost are realized without compromising model quality in real-time or at scale.

6. Specialization, Flexibility, and Collaboration

Fine-grained MoE frameworks enable granular task and domain specialization:

• Hierarchical and Collaborative Routing: S′MoRE employs hierarchical, tree-structured routing and low-rank expert decomposition, formulated as inter-layer message-passing analogous to a GNN. The model achieves higher "structural flexibility"—the number of distinct routing pathways and output embeddings grows exponentially with layer count—improving adaptation to diverse or complex tasks (Zeng et al., 8 Apr 2025).
• Multi-Head Expert Models: By splitting token representations into multiple "heads" and assigning these to different experts (MH-MoE model), feature diversity and parameter utilization gains are realized. This multiscale specialization surpasses flat and conventional fine-grained MoE designs at matched FLOPs and parameter counts, and transfers robustly to 1-bit quantized LLMs (Huang et al., 25 Nov 2024).
• Collaboration and Load Balancing: Systems such as GMoE enforce both specialization (via a Poisson-distinction loss) and balanced usage (via a Normal-distribution loss) in the expert router dynamics. These strategies drive both higher specialization and stability, minimizing overtraining of particular experts or collapse of routing diversity (Bai et al., 18 Dec 2024).
• Cross-Domain and Scientific Task Adaptation: By combining science-aware routing, per-discipline expert specialization, and a preserved generalist expert, frameworks such as Innovator achieve both 25% domain-specific improvement and 99% general capability retention in multi-disciplinary LLMs (Liao et al., 24 Jul 2025).

The modular, compositional nature of fine-grained MoE-LLMs positions them as foundational architectures for multi-task, multi-domain, and lifelong learning settings.

7. Challenges and Future Directions

The field continues to address open questions:

• Expert Redundancy and Specialization: Balancing expert utility and model capacity is nontrivial; routers optimized for stability during pre-training often suppress meaningful specialization and underutilize high-impact experts. Post-training strategies such as Ban&Pick attempt to mitigate this.
• System Bottlenecks: Irregular workloads during inference, sub-optimal hardware utilization, and data movement remain technical bottlenecks. Continued advances in 3D integration, adaptive scheduling, and expert management are promising.
• Route Adaptation and AutoML: The search for more adaptive, robust, and automatic routing policies—potentially incorporating reinforcement learning or data-driven expert profiling—remains critical for scaling and deployment.
• Continual and Lifelong Learning: Integrating fine-grained expert adaptation with online learning, knowledge retention, and domain transfer is a rich area of ongoing research.

Further theoretical and empirical work on optimal granularity, expert heterogeneity, specialization metrics, and even hybrid architectures (e.g., adapters, residual experts, and graph routers) will underpin the next phase of progress in fine-grained MoE-LLMs.

---

Fine-grained MoE-LLMs constitute a rapidly advancing paradigm in scalable, adaptive, and efficient LLM design, combining architectural, algorithmic, and system-level innovations to deliver domain specialization, resource efficiency, and adaptability at unprecedented scale and resolution.