Impact of Absent Attention Mechanism on Mamba’s In-Context Learning

Ascertain whether the absence of an attention mechanism in pure Mamba state-space models causes difficulty in learning in-context.

Background

Based on empirical observations, the authors report that pure Mamba often fails to follow expected output formats in few-shot settings (e.g., IMDB), suggesting challenges with in-context learning. In contrast, the hybrid Attention–Mamba model adheres to formats and exhibits induction-like behavior.

They explicitly conjecture that the lack of attention in the pure Mamba architecture may be responsible for the observed difficulties in in-context learning, motivating further theoretical and empirical validation of this claim.