Cluster-Aware Upcycling in MoE Models

• Cluster-aware Upcycling is a technique that transforms pre-trained dense models into sparse Mixture-of-Experts by aligning expert initialization with semantic clusters.
• It employs activation-based clustering and fine-grained slicing, using methods like spherical k-means and truncated SVD to ensure effective expert specialization and routing.
• Empirical evaluations in vision and multilingual tasks demonstrate performance gains, increased expert diversity, and robust transfer, validating its structural benefits.

Cluster-aware Upcycling refers to a family of initialization and specialization techniques for Mixture-of-Experts (MoE) neural architectures, where pre-trained dense models are converted into sparse MoE structures with expert modules specialized according to the clustering structure of data or activations. This approach seeks to address expert symmetry, accelerate specialization, and improve transfer efficiency by aligning expert initialization and router dynamics with semantic or domain-specific clusters present in the underlying data. Cluster-aware upcycling has been investigated in both vision and multilingual language modeling, with distinct methodologies but common principles of leveraging cluster structure for expert allocation, initialization, and training stabilization (Chu et al., 15 Apr 2026, Jiang et al., 28 Apr 2026).

1. Methodological Frameworks for Cluster-Aware Upcycling

Two primary instantiations of cluster-aware upcycling have been proposed:

• Activation-based cluster upcycling for vision transformers (Chu et al., 15 Apr 2026), and
• Language cluster–aware upcycling for multilingual MoEs (Jiang et al., 28 Apr 2026).

Activation-Based Semantic Clustering and Expert Initialization

Cluster-aware upcycling for vision models begins with collecting $\ell_2$-normalized activation vectors $X = \{x_j\}_{j=1}^M$ from a pre-trained feedforward network (FFN) block on a calibration corpus. Spherical $k$-means partitions $X$ into $K = N_e$ clusters by solving

$$\{c_k\}_{k=1}^K = \arg\max_{\substack{\mu_1,\dots,\mu_K\|\mu_i\|_2=1}} \sum_{j=1}^M \max_{1\le i\le K} \mu_i^T x_j,$$

yielding semantic centroids $\{c_k\}$ and cluster blocks $X_i = \{x_j: i_j = i\}$. For each cluster, expert initialization proceeds via truncated SVD of a whitened projection $W S_i$, ensuring each expert $W_i$ best reconstructs the subspace of its cluster.

Dense-to-MoE Upcycling with Fine-Grained Slicing and Drop-Upcycling

In large-scale multilingual MoEs, pre-trained dense FFN weights are partitioned into sub-matrices. For each layer, the weight matrices $X = \{x_j\}_{j=1}^M$0 are split along the hidden axis into $X = \{x_j\}_{j=1}^M$1 slices, each forming the initialization for one expert. A scalar rescaling $X = \{x_j\}_{j=1}^M$2 preserves the expected forward-pass magnitude. Router weights are zero-initialized. To break initialization symmetry, "Drop-Upcycling" zeroes out random rows/columns in each expert's parameters and reinitializes them with noise matching the original mean and variance (Jiang et al., 28 Apr 2026).

2. Cluster-Aware Routing and Expert Specialization Strategies

Semantic Router Initialization

For vision MoEs, the router parameters $X = \{x_j\}_{j=1}^M$3 are initialized row-wise with the $X = \{x_j\}_{j=1}^M$4-normalized cluster centroids:

$X = \{x_j\}_{j=1}^M$5

This ensures that, initially, inputs close to centroid $X = \{x_j\}_{j=1}^M$6 are routed preferentially to expert $X = \{x_j\}_{j=1}^M$7, catalyzing semantic alignment between router and expert.

Data-Driven Expert Allocation and Emergent Clusters

In Marco-MoE, cluster-aware routing emerges from data-driven language-expert co-activation. Given a language $X = \{x_j\}_{j=1}^M$8, expert activation profiles are recorded (number of tokens routed to expert $X = \{x_j\}_{j=1}^M$9 for each layer). Pearson correlations of these activation vectors between language pairs define a language clustering metric $k$0. Hierarchical clustering on these metrics empirically recovers linguistic families and reveals that related languages exhibit similar expert utilization (Jiang et al., 28 Apr 2026).

3. Training Protocols and Cluster-Aware Losses

Expert-Ensemble Self-Distillation

Cluster-aware upcycling for vision models introduces an expert-ensemble self-distillation (EESD) loss. An exponential moving average (EMA) teacher of all MoE parameters produces a dense ensemble output

$k$1

against which the student's sparse output

$k$2

is regularized using either KL divergence on softmax predictions or squared error on hidden outputs:

$k$3

Router Regularization and Load-Balancing

In Marco-MoE, router training employs an auxiliary load-balancing loss ($k$4) and a router Z-loss ($k$5), with coefficients set to $k$6 and $k$7, respectively. This prevents expert collapse and encourages uniform expert utilization.

4. Empirical Evaluation and Comparative Performance

Quantitative Outcomes

Cluster-aware upcycling for vision transformers (CLIP ViT-B/32 and ViT-B/16 with 8-expert top-2 MoEs) consistently outperforms sparse upcycling baselines:

Model/Metric	Sparse Upcycling	Cluster-aware Upcycling	Gain
ViT-B/32 Retrieval	(30.8, 48.0, 39.4)	(31.0, 48.2, 39.6)	+0.2 avg.
ViT-B/32 ImageNet-1k	63.0%	63.2%	+0.2%
ViT-B/16 Retrieval	(34.9, 50.9, 42.9)	(35.4, 51.6, 43.5)	+0.6 avg.
VTAB-Natural	58.0%	59.1%	+1.1%

Few-shot and full fine-tuning further confirm consistent gains (e.g., 5-shot: 50.9 → 51.5, +0.6). In Marco-MoE, Nano and Mini models (0.6B–0.86B active params) match or exceed models with $k$8–$k$9B active parameters in both English and multilingual benchmarks, attaining best-in-class performance-to-compute ratios and competitive cultural/regional benchmark scores (Chu et al., 15 Apr 2026, Jiang et al., 28 Apr 2026).

Analysis of Expert Diversity and Routing

Cluster-aware initialization reduces inter-expert cosine similarity by $X$0–$X$1, lowers relative compactness (trace of within- vs. between-expert covariance) throughout training, and yields more orthogonal expert subspaces. Routing entropy per token is $X$2 lower than baseline, reflecting greater confidence in assignment, yet expert utilization remains balanced and robust against collapse (Chu et al., 15 Apr 2026).

5. Clustering Dynamics, Scaling, and Domain Transfer

Language clustering analysis in Marco-MoE demonstrates that expert utilization signatures naturally encode phylogenetic language relationships. For languages within a detected cluster $X$3, the most used experts (peak $X$4) are shared, while typologically distant languages activate distinct expert subsets. Hierarchical clustering accurately reconstructs known language family trees without explicit supervision. When expanding from 29 to 64 languages, expert activation signatures for pre-existing languages remain stable—indicating immunity to catastrophic interference during multilingual scaling (Jiang et al., 28 Apr 2026).

A plausible implication is that cluster-aware upcycling methodologies can generalize to additional domains (e.g., programming languages, visual modalities), enabling new forms of structured specialization as long as domain clusters are present in the data.

6. Limitations, Open Questions, and Future Perspectives

Current cluster-aware upcycling approaches require full-model retraining for new domain or language expansion. Efficient modular expert addition without retraining, as well as improved router architectures (e.g., hierarchical or adaptive $X$5) for sharper separation—especially for low-resource or micro-clusters—remain open challenges. Global scaling is limited by data availability for many low-resource languages or domains, suggesting that robust zero-shot adaptation or alternative specialization mechanisms will be necessary for comprehensive coverage (Jiang et al., 28 Apr 2026).

In summary, cluster-aware upcycling provides an efficient blueprint for MoE initialization and specialization by structurally aligning experts and routers with data clusters. This enables greater expert diversity, early specialization, robust transfer, and efficiency in both vision and massive-scale multilingual MoE architectures (Chu et al., 15 Apr 2026, Jiang et al., 28 Apr 2026).