---
title: 'Mamba-CL: Continual Learning with S6 Models'
url: https://www.emergentmind.com/topics/mamba-cl-algorithm
type: topic
---

# Mamba-CL: Continual Learning with S6 Models

The Mamba-CL algorithm denotes a family of approaches that utilize Selective State-Space Models (S6) within the Mamba architecture to address challenges in continual learning (CL), meta-continual learning (MCL), and scalable algorithm distillation, including in complex continuous-control and vision domains. Mamba-CL methods fundamentally exploit the S6 model’s linear-scaling, attention-free sequence modeling and introduce CL-specific mechanisms such as selectivity regularization and null-space consistency constraints to achieve high sample efficiency, fixed-memory operation, and robustness against catastrophic forgetting, outperforming conventional Transformer-based and rehearsal-free CL systems across benchmarks [2506.13892, 2412.00776, 2411.15469].

## 1. Foundations and Model Architecture

Mamba-CL adopts the S6 (Selective Structured State-Space) model, a discrete-time recurrent architecture parameterized as:
\[
h_t = S_t \odot h_{t-1} + G_t \odot z_t, \quad u_t = C_t h_t + D z_t
\]
where $z_t \in \mathbb{R}^M$ is the input embedding, $S_t, G_t \in \mathbb{R}^M$ are input-dependent, per-channel gates produced by small convolutional and linear projections (followed by a sigmoid), and $C_t, D$ are readout parameters. The model eschews explicit attention in favor of rich, tokenwise dynamics, supporting long-sequence modeling at linear compute and memory cost. Each transition tuple, e.g., $c_t = (s_t, a_t, r_t, s_{t+1})$ for RL or $(x_t, y_t)$ for classification, is projected via a learned embedding to $d_\mathrm{model}$ and fed to stacked S6 blocks with LayerNorm, feed-forward MLP (GELU activations), and residual connections [2506.13892, 2412.00776].

## 2. Continual and Meta-Continual Learning with Mamba-CL

Mamba-CL addresses the CL problem by processing a non-stationary data stream as a sequential prediction task. For each new sample or task, the recurrent hidden state $h_t$ is updated; new predictions $\hat{y}_t = f_\theta(h_{t-1}, x_t)$ are emitted with no explicit storage or recomputation of past samples [2412.00776]. Unlike Transformers, which require a linearly growing memory for historical key-value pairs, the fixed-size hidden state of SSMs enables strict memory control and efficient adaptation.

For MCL, meta-learning is performed episodically: a train stream $E^{\mathrm{train}}$ and test set $E^{\mathrm{test}}$ are sampled per episode. The meta-objective is:
\[
\min_\theta \;\mathbb{E}_{(E^{\mathrm{train}}, E^{\mathrm{test}})}\Bigl[ \sum_{(x,y)\in E^{\mathrm{test}}} \ell(f_\theta(E^{\mathrm{train}}, x), y) + \lambda \sum_{(x, y)\in E^{\mathrm{train}}} \ell_{\mathrm{slct}}((x, y)) \Bigr]
\]
where $\ell$ is the main loss (e.g., cross-entropy, MSE) and $\ell_{\mathrm{slct}}$ is the selectivity regularizer enforcing content-based gating analogous to attention [2412.00776].

## 3. Selectivity Regularization and Null-Space Consistency

The selectivity regularizer is motivated by the soft-attention weights in Transformers, which encode relevance of all prior representations to the current query. For Mamba-CL, this property is emulated by associating the model's gating patterns $(S_t, G_t)$ with a pseudo-attention matrix; the KL divergence between their softmaxed values and “ground-truth” association vectors (e.g., label equality matches) guides the network to focus on semantically relevant past tokens without explicit caching:
\[
\ell_{\mathrm{slct}}((x_t, y_t)) = \mathrm{KL}\bigl( \mathrm{softmax}(\alpha_t^{\mathrm{Model}}) \;\|\; \mathrm{softmax}(b_t) \bigr)
\]
[2412.00776].

In another Mamba-CL instance, updates to the core SSM parameters $(A, W^B, W^C, W^\delta)$ are restricted to the orthogonal complement (null space) of the feature subspaces of previous tasks, theoretically guaranteeing output consistency and preventing catastrophic forgetting. These constraints are:
\[
\begin{cases}
\delta_t \Delta A = 0 \\
X_t \Delta W^\delta = 0 \\
\delta_t X_t \Delta W^B = 0 \\
X_t \Delta W^C = 0
\end{cases}
\]
Gradient projections enforce these null-space updates, requiring only the storage of low-rank projectors per SSM block [2411.15469].

## 4. Training Procedure and Computational Efficiency

The Mamba-CL training loop proceeds by sampling meta-train episodes or tasks; for each, the hidden state is initialized, and the sequence is processed one token at a time. Training minimizes the primary loss and selectivity regularizer (if present). In the orthogonality-based CL version, after each task, new projectors are constructed via SVD of old-task features, and future gradients are projected into their null spaces [2411.15469].

Key efficiency gains are as follows:

| Model            | Time Complexity | Memory Complexity | Maximum Context Length (single RTX 3060) |
|------------------|----------------|------------------|------------------------------------------|
| Transformer      | O(L²·d_model)   | O(L·d_model)      | ∼400 tokens                              |
| Mamba-CL (S6)    | O(L·d_model)    | O(d_model)        | 2,000+ tokens                            |

Linear scaling in sequence length enables Mamba-CL to operate on long, multi-episode RL histories and long CL task streams that are infeasible for quadratic-cost Transformers [2506.13892, 2412.00776].

## 5. Applications: Algorithm Distillation, Reinforcement Learning, and Vision CL

In in-context RL, Mamba-CL utilizes large-scale offline trajectories from SOTA RL agents (e.g., PPO, SAC, DroQ) across multiple continuous-control tasks. The network autoregressively predicts next actions given all past transitions, allowing test-time adaptation to unseen tasks through in-sequence contextualization alone—no online gradient steps are taken during evaluation [2506.13892]. In vision CL, a pretrained SSM backbone is fine-tuned on sequential class-incremental tasks with feature-consistency constraints, providing state-of-the-art anti-forgetting on benchmarks such as 10-/20-split ImageNet-R, CIFAR-100, and DomainNet [2411.15469].

Empirical performance improvements over Decision Transformer and rehearsal-free CL baselines are consistent, with Mamba-CL showing better asymptotic adaptation, less catastrophic forgetting (e.g., accuracy up from 63.7% to 81.7% on 10-split ImageNet-R), and higher sample efficiency across both RL and vision domains [2506.13892, 2412.00776, 2411.15469].

## 6. Hyperparameters, Ablations, and Robustness

Critical hyperparameters include the S6 state size ($d_\mathrm{model}=384/512$ for RL, up to 2048 for vision), convolutional kernel size for gates, meta-batch size, selectivity regularization weight ($\lambda$), learning rates (Adam optimizer), and balance factor $\eta$ for null-space projection. Experiments confirm that S6 state sizes above 128 yield diminishing returns, while selectivity regularization is stable over a broad $\lambda$ range [2412.00776]. Mamba-CL is robust to input noise—unlike Transformers, which fail under high-noise streams—and performs well under large domain shifts and long or short meta-test episodes.

Ablation studies show that shorter context lengths (L ≪ trajectory length) sharply degrade performance in complex RL tasks, confirming the need for long in-context horizons. Replacing S6 blocks with standard self-attention degrades both performance and feasible sequence length, underscoring the advantage of SSMs for these regimes [2506.13892].

## 7. Limitations and Future Prospects

Mamba-CL requires access to extended, high-quality learning trajectories for pretraining, which may be prohibitively expensive in resource-constrained or real-world systems. While substantially advancing context length in RL and vision CL, most benchmarks still involve modest sequence horizons (≤200 for RL tasks). Understanding performance on extreme long-horizon tasks (≫10,000 steps), investigating data mix strategies (e.g., expert versus diverse rollouts), and fusing SSMs with architectures such as diffusion-based predictors, prompt-tuned modules, or MoE (Mixture-of-Experts) S6 variants represent active directions for future research [2506.13892, 2412.00776].

Overall, the Mamba-CL family demonstrates that SSM-based architectures, when equipped with CL-oriented regularization and update constraints, can deliver state-of-the-art continual adaptation accuracy, robust anti-forgetting, linear sequence scalability, and practical efficiency, advancing the state of the art in continual, meta-continual, and in-context learning across multiple domains.

Source: https://www.emergentmind.com/topics/mamba-cl-algorithm