Lightweight expert–router coupling in Mixture-of-Experts remains open

Develop a lightweight and effective method to tightly couple router decisions with the true capabilities of experts in Mixture-of-Experts models without incurring prohibitive computational or memory costs.

Background

Mixture-of-Experts LLMs route tokens to a small subset of specialized feed-forward "experts" using a router. In standard MoE training, routers learn routing strategies indirectly from gradients, without explicit access to expert capabilities, which can lead to misrouting and hinder specialization. Prior approaches that couple routers and experts rely on dense activation or token-dependent computations, substantially increasing training cost. The paper highlights that finding a lightweight, effective coupling mechanism had remained an open challenge; the authors propose the ERC auxiliary loss as a candidate solution and evaluate it extensively.