Adaptation of decoder-side diagnostics to mixture-of-experts architectures

Determine how decoder-side memorization diagnostics should be adapted for mixture-of-experts architectures, accounting for the possibility that expert-routing decisions interact with memorization patterns differently than in dense language models.

Background

The study evaluates dense architectures because their total parameter count more directly represents the capacity available during inference. Mixture-of-experts models activate only a fraction of their parameters for each token, making nominal parameter count an ambiguous measure of effective capacity.

The paper notes that expert routing could interact with memorization in ways not observed in dense models. Consequently, it leaves unresolved how decoder-side probes such as CoDeC should be redesigned or interpreted for mixture-of-experts systems. This question matters for extending scale-aware memorization analysis to a major class of contemporary language-model architectures.