Does sparse routing reduce superposition in large-scale MoE LLMs?

Establish whether the observed decrease in superposition with increased routing sparsity in toy Mixture-of-Experts models extends to large-scale Mixture-of-Experts language models, where the router must manage many experts and diverse data distributions.

Background

Toy-model studies have reported that increasing routing sparsity in MoE architectures reduces superposition, suggesting a potential mechanism for cleaner, more modular representations. Whether this relationship persists at the scale of production LLMs remained unverified.

Large-scale settings introduce additional complexities—such as millions of parameters and heterogeneous data—that could alter routing dynamics and representational structures, making the generalization of toy-model findings nontrivial.