Mechanistic Interpretation of the EMA Reference State

Characterize mechanistically what the Exponential Moving Average reference state encodes and what residual signal each expert processes after background subtraction, to establish an interpretable account of Contrastive Routing Mechanism behavior.

Background

The Contrastive Routing Mechanism uses a per-layer Exponential Moving Average of hidden states to represent shared background structure and subtracts this reference during routing. Although the paper analyzes the effect geometrically through singular-value decomposition, it does not determine the semantic content of the reference state or the precise residual information used by individual experts. The authors explicitly leave this mechanistic account unresolved as a direction for interpretability research.

References

Second, our SVD analysis characterizes the effect of subtraction geometrically but not semantically. A mechanistic account of what the reference state encodes, and what residual signal each expert acts on, remains an open and promising direction for interpretability research.

— Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts  (2609.01100 - Xiros et al., 1 Sep 2026) in Section Conclusions, final paragraph