Emergent Mind

Is Mamba Capable of In-Context Learning?

Published Feb 5, 2024 in cs.LG


This work provides empirical evidence that Mamba, a newly proposed selective structured state space model, has similar in-context learning (ICL) capabilities as transformers. We evaluated Mamba on tasks involving simple function approximation as well as more complex natural language processing problems. Our results demonstrate that across both categories of tasks, Mamba matches the performance of transformer models for ICL. Further analysis reveals that like transformers, Mamba appears to solve ICL problems by incrementally optimizing its internal representations. Overall, our work suggests that Mamba can be an efficient alternative to transformers for ICL tasks involving longer input sequences.


  • The paper explores the in-context learning ability of Mamba, a state space model, as an effective alternative to transformer-based models.

  • Mamba demonstrates comparable or superior in-context learning performance to transformer and other models, such as S4 and RWKV, on various tasks.

  • A probing strategy unravels how Mamba incrementally refines its internal state for task-solving, hinting at similarities to transformers.

  • Mamba shows potential in natural language processing tasks, with scaling advantages suggesting its suitability for high-complexity language models.


In the field of AI, in-context learning (ICL) stands as a transformative facility exhibited by large neural networks, especially those with transformer architectures, negating the need for explicit retraining or fine-tuning to accommodate new tasks. Recently, there's been a burgeoning interest in models like Mamba—a selective structured state space model—primarily due to its potential advantages in handling longer sequences over transformers. The study under discussion contributes significantly to the current understanding of Mamba's ICL abilities, affirmation of which could present Mamba as a powerful and efficient alternative to transformers for ICL tasks.

In-Context Learning Performance Analysis

One central finding is that Mamba matches or exceeds the performance of (self-supervised) pre-trained transformer models in ICL tasks, overcoming the limitations posed by transformers in processing longer inputs. This result asserts the robustness of the Mamba architecture, as it performs comparably with transformers in tasks ranging from regression to complex language processing. The analysis extends to show Mamba's superiority to its predecessor S4 and other baseline models such as RWKV for these tasks. Importantly, results indicate that Mamba maintains its ICL capabilities across both in-distribution and out-of-distribution examples.

Mechanisms of In-Context Learning

Delving deeper into Mamba's ICL methodology, the study employs a probing strategy to elucidate the model's iterative optimization process for task solving. By examining intermediate representations layer by layer, the analysis suggests that Mamba refines its internal state incrementally to solve ICL tasks. Here, it exhibits an approach somewhat akin to transformers. Yet, some ambiguity remains in different cases such as ReLU networks and decision trees, pointing to areas for future scrutiny.

Application on Natural Language Processing Tasks

Further empirical evidence reinforces the efficacy of Mamba when selectively fine-tuned and pre-trained on large datasets for NLP tasks, showing that it compares favorably against contemporary models like RWKV, LLama, Pythia, and even GPT-J at similar or fewer parameters. In this domain, Mamba's scalability with in-context examples and parameter count is particularly noteworthy. The study indicates that as the model size increases, Mamba's ICL accuracy improves substantively, demonstrating its potential for high-complexity NLP.

Concluding Remarks

The paper crystallizes the contention that Mamba is not only capable of ICL but does so with a proficiency that puts it on an even keel with transformer models. Crucially, this capability extends to longer sequence inputs, situating Mamba as a compelling alternative to the transformer paradigm. In essence, for ICL tasks—whether they are function approximations or dense, intricate language modeling—Mamba's architecture represents a promising innovation. This work lays a strong foundation for deepening our understanding of state-of-the-art machine learning architectures and their inherent learning strategies.

Get summaries of trending AI papers delivered straight to your inbox

Unsubscribe anytime.

  1. Transformers learn to implement preconditioned gradient descent for in-context learning
  2. What learning algorithm is in-context learning? Investigations with linear models. In The Eleventh International Conference on Learning Representations
  3. In-Context Language Learning: Architectures and Algorithms
  4. Transformers as Statisticians: Provable in-context learning with in-context algorithm selection. Advances in neural information processing systems
  5. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp.  2397–2430. PMLR
  6. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901
  7. Xgboost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
  8. The Pile: An 800GB Dataset of Diverse Text for Language Modeling
  9. What can transformers learn in-context? A case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583–30598
  10. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.  5484–5495
  11. Mamba: Linear-Time Sequence Modeling with Selective State Spaces
  12. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2021a.
  13. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34:572–585, 2021b.
  14. In-context learning creates task vectors. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp.  9318–9333
  15. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(9):5149–5169
  16. Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. Transactions of the ASME–Journal of Basic Engineering, 82(Series D):35–45
  17. Exploring the Relationship Between Model Architecture and In-Context Learning Ability
  18. U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation
  19. Transformers can do bayesian inference. In International Conference on Learning Representations
  20. RWKV: Reinventing RNNs for the Transformer Era
  21. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations
  22. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9
  23. Do pretrained Transformers Really Learn In-context by Gradient Descent?
  24. Long Range Arena: A Benchmark for Efficient Transformers. In International Conference on Learning Representations
  25. LLaMA: Open and Efficient Foundation Language Models
  26. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30
  27. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pp.  35151–35174. PMLR
  28. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax, May 2021.

  29. Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

Show All 29

Test Your Knowledge

You answered out of questions correctly.

Well done!