Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation

Abstract: Mixture-of-Experts (MoE) decouples model capacity from per-token computation, yet their scalability remains limited by the physical dimensions of depth and width. To overcome this, we propose Mixture of Universal Experts (MOUE),a MoE generalization introducing a novel scaling dimension: Virtual Width. In general, MoUE aims to reuse a universal layer-agnostic expert pool across layers, converting depth into virtual width under a fixed per-token activation budget. However, two challenges remain: a routing path explosion from recursive expert reuse, and a mismatch between the exposure induced by reuse and the conventional load-balancing objectives. We address these with three core components: a Staggered Rotational Topology for structured expert sharing, a Universal Expert Load Balance for depth-aware exposure correction, and a Universal Router with lightweight trajectory state for coherent multi-step routing. Empirically, MoUE consistently outperforms matched MoE baselines by up to 1.3% across scaling regimes, enables progressive conversion of existing MoE checkpoints with up to 4.2% gains, and reveals a new scaling dimension for MoE architectures.

• The paper introduces a novel MoUE framework that reuses universal experts across layers to convert depth into increased virtual width with minimal compute overhead.
• It employs a staggered rotational topology and connectivity-normalized load balancing to manage expert routing, enhancing training stability and parameter efficiency.
• Empirical results reveal consistent accuracy gains and improved efficiency, with up to a 4.2% performance boost during progressive pretraining.

Mixture of Universal Experts: Depth-Width Transformation for Maximum Model Capacity

Architectural Motivation and Conceptual Overview

The paper "Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation" (2603.04971) addresses key limitations in traditional Mixture-of-Experts (MoE) architectures that partition expert parameters by layer, leading to linear memory scaling and restricted parameter reuse. Standard MoE designs expand width by increasing the number of experts per layer but are fundamentally limited by memory and system constraints. The authors introduce the Mixture of Universal Experts (MoUE) framework, which generalizes MoE by enabling the recursive reuse of a shared pool of Universal Experts (UEs) across multiple layers. This transformation defines a new scaling dimension—Virtual Width—by turning additional depth into increased effective width at fixed per-token activation budgets.

Figure 1: Overview of MoUE architecture, where a pool of Universal Experts is accessible from multiple layers for recursive reuse with activation constraints.

Redundancy Analysis and Parameter Sharing Rationale

The foundation for cross-layer expert reuse rests on the observation of functional redundancy in deep MoE networks. Empirical analysis of trained MoE models using Centered Kernel Alignment (CKA) demonstrates high similarity between experts in adjacent and even distant layers. Specific operators reappear across large depth gaps, supporting the hypothesis that strict layer-wise partitioning is suboptimal and motivating efficient global expert reuse.

Figure 2: Heatmap of expert similarity across shallow and deep MoE layers, illustrating strong local and non-local redundancy.

MoUE Framework: Recursive Expert Pooling and Topology

Generalized Expert Connectivity

MoUE decomposes the expert pool into layer-local experts and a global set of UEs, governed by a connectivity mapping $\mathcal{C}$ that prescribes expert reachability across layers. This architecture allows exponential growth in the compositional space: for $L$ layers and $N_u$ universal experts with Top-$k$ routing, the path space scales as $(\binom{N_u}{k})^L$. The challenge lies in optimizing within this combinatorial manifold without increasing storage or compute overhead.

Staggered Rotational Topology

To enable tractable training and maximize path diversity, MoUE constrains expert reuse with a hierarchical ring-based topology. Layers are grouped, each group sharing a sliding window of UEs with progressive rotation. This ensures local specialization, controlled cross-layer reuse, and avoids repetitive expert cycles. Connectivity at each layer is realized as an allow-list over global expert indices, shrinking the search space and supporting efficient optimization.

Figure 3: MoUE expert routing overview, combining layer-local and cross-layer universal experts under structured topology.

Connectivity-Normalized Load Balancing

Conventional load balancing in MoE fails under MoUE’s heterogeneous expert exposure, penalizing universal experts for architectural reachability rather than actual overutilization. To resolve this, the paper introduces Universal Expert Load Balance (UELB), which normalizes expert utilization penalties by their exposure count across layers. UELB operates group-wise, rescaling loss terms and incorporating a warmup regimen to stabilize early expert exploration. This principled approach incentivizes depth-wise path coverage, actively converting combinatorial complexity to functional capacity.

Figure 4: Comparative illustration of standard Load Balancing Loss (LBL) vs. Universal Expert Load Balance (UELB) mechanisms.

Universal Router: Context-Aware Routing Dynamics

MoUE’s Universal Router is designed to maintain coherent trajectory state across recursive expert calls. Routing logits combine a semantic affine pathway and a contextual pathway parameterized with fast-weights updated online, biasing selection toward context-consistent experts. This dual routing ensures robust multi-step compositions and mitigates collapse into recurrent loops.

Empirical Results: Scaling, Training Dynamics, and Ablations

Performance Scaling

MoUE is evaluated with matched MoE baselines across width and depth expansion regimes. Width scaling (increasing UEs) improves efficacy at constant compute, e.g., increasing average accuracy from 41.0 to 42.3 with no increase in activated/total parameters. Depth scaling, sharing UEs while augmenting depth, offers higher gains with only modest parameter growth, e.g., a 3.0 point increase for MoUE L36 at TP comparable to baseline.

Figure 5: Average accuracy scaling vs. total, virtual, and activated parameters under different expansion strategies.

Stability and Training Dynamics

Validation metrics, loss convergence, and routing skew are summarized across training scenarios. MoUE exhibits improved validation loss at matched training FLOPs and stable load balance indicators throughout optimization, demonstrating enhanced compute efficiency and effective parameter utilization.

Figure 6: LM training and validation loss; load-balance dynamics for routing stability across training steps.

Progressive Warm-Start and Component Ablations

MoUE retains compatibility with existing MoE checkpoints via a progressive curriculum-based warm-start. Empirically, monotonic accuracy improvements are observed as universal expert pool size increases, achieving up to 4.2% gain during continued pretraining. Ablation studies confirm necessity of the staggered topology, UELB, warmup, and Universal Router’s stateful mechanisms for realizing virtual width’s benefits.

Figure 7: Performance impact of universal expert pool size in progressive warm-start experiments.

Expert Utilization and Functional Analysis

Domain Specialization and Layer Usage

Activation analysis demonstrates that UEs develop domain preferences and are selectively routed in deeper layers, while local experts dominate early stages. Heatmaps confirm effective cross-layer parameter sharing and path diversity.

Figure 8: Heatmap of expert activation frequency in MoUE, contrasting local and universal experts.

Figure 9: Layer-wise token routing proportion to Universal Experts after warm-start training.

Figure 10: Activation profile of universal experts across multiple domains, indicating functional specialization.

Implications and Future Directions

MoUE reframes sparse model scaling by introducing a novel depth–width transformation, leveraging parameter redundancy and recursive reuse to expand capacity without linear memory or compute costs. The connectivity-normalized routing and context-aware gating mechanisms promise further advances in efficient scaling, path compositionality, and continual expert specialization. MoUE architectures have practical implications for large-scale language modeling, especially in environments constrained by hardware and memory but demanding maximal effective capacity. Theoretical extensions could explore more adaptive, task-aware topologies and dynamic expert pool regulation.

Conclusion

The Mixture of Universal Experts architecture formalizes a new model scaling axis—Virtual Width—by exploiting redundancies and recursive reuse of universal expert parameters across depth. Empirically, MoUE delivers consistent accuracy and efficiency gains over standard MoE models without increased activation or compute, validating the depth–width exchange hypothesis. Connectivity-normalized load balancing and stateful routing are essential for unlocking and stabilizing this capacity. As AI models scale toward increasingly demanding applications, MoUE offers a principled pathway to robust, efficient conditional computation and flexible expert representation.