Stability mechanism of non-oscillatory Mamba-3 models

Determine how Mamba-3 models trained without oscillatory dynamics achieve stable length extrapolation beyond their training context, despite the absence of learned phase rotations.

Background

The paper compares pretrained Mamba-3 models with oscillatory and non-oscillatory dynamics. Although oscillation improves long-context performance in the reported experiments, models trained from scratch without oscillation also maintain stable perplexity beyond the 2K training context and do not exhibit the severe degradation observed when phase is removed from pretrained oscillatory models. The authors explicitly identify the mechanism underlying this stability as unresolved.