Scaling CoRM to Multi-Billion-Parameter Models

Scale the Contrastive Routing Mechanism from models of up to 469 million parameters to multi-billion-parameter language models to determine whether its routing specialization and performance improvements persist at substantially larger scales.

Background

The experiments evaluate the Contrastive Routing Mechanism on models with at most 469 million parameters. The paper explicitly identifies scaling to multi-billion-parameter models as unresolved, leaving open whether the reported benefits generalize to substantially larger architectures and training regimes.

References

While CoRM demonstrates consistent gains over the standard token-choice router, several questions remain open. Our experiments are conducted on models up to 469M parameters, and scaling to multi-billion parameter models is left to future work.

— Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts  (2609.01100 - Xiros et al., 1 Sep 2026) in Limitations section