Tiny Recursive Reasoning with Mamba-2 Attention Hybrid

Abstract: Recent work on recursive reasoning models like TRM demonstrates that tiny networks (7M parameters) can achieve strong performance on abstract reasoning tasks through latent recursion -- iterative refinement in hidden representation space without emitting intermediate tokens. This raises a natural question about operator choice: Mamba-2's state space recurrence is itself a form of iterative refinement, making it a natural candidate for recursive reasoning -- but does introducing Mamba-2 into the recursive scaffold preserve reasoning capability? We investigate this by replacing the Transformer blocks in TRM with Mamba-2 hybrid operators while maintaining parameter parity (6.83M vs 6.86M parameters). On ARC-AGI-1, we find that the hybrid improves pass@2 (the official metric) by +2.0\% (45.88\% vs 43.88\%) and consistently outperforms at higher K values (+4.75\% at pass@100), whilst maintaining pass@1 parity. This suggests improved candidate coverage -- the model generates correct solutions more reliably -- with similar top-1 selection. Our results validate that Mamba-2 hybrid operators preserve reasoning capability within the recursive scaffold, establishing SSM-based operators as viable candidates in the recursive operator design space and taking a first step towards understanding the best mixing strategies for recursive reasoning.

• The paper introduces a hybrid model (TR-mamba2attn) that integrates Mamba-2's sequential processing with attention for cross-position mixing, enhancing recursive reasoning.
• The paper demonstrates measurable improvements on ARC-AGI benchmarks, with up to +4.75% gain at pass@100 and increased candidate diversity.
• The paper outlines practical implications for scalable, efficient reasoning systems and sets a foundation for further exploration of hybrid SSM-attention architectures.

Mamba-2 Hybrid Operators for Tiny Recursive Reasoning: An Analysis

Overview and Motivation

This paper addresses the operator design for recursive latent reasoning, specifically examining whether Mamba-2-based hybrid architectures can supplant standard attention modules in the Tiny Recursive Model (TRM) paradigm without sacrificing, and potentially improving, abstract reasoning performance. The motivation arises from the observation that in recursive reasoning, iterative refinement—particularly in latent (hidden) representation space—is possibly more central to performance than sheer model scale. Through this lens, the work investigates whether Mamba-2's state space recurrence can serve not just as an efficient sequence modeler but as an effective core operator within TRM's recursive scaffold.

A key challenge is that while attention-based operators provide explicit, bidirectional context mixing, SSMs such as Mamba-2 process sequences causally and may not, a priori, capture all spatial dependencies needed for abstract reasoning tasks like those found in the ARC-AGI benchmark. The authors propose hybrid operators that integrate Mamba-2 for sequential processing with attention (or dense MLP-t mixing) for cross-position communication.

Methodological Contributions

The core technical contribution is the instantiation of a hybrid recursive model—termed TR-mamba2attn—that substitutes TRM's Transformer blocks with a stack of Mamba-2 $\rightarrow$ Mamba-2 $\rightarrow$ Attention $\rightarrow$ MLP modules. The architecture is rigorously parameter-matched to the Transformer-only baseline, facilitating a fair comparison. Two variants are evaluated: one using attention for cross-position mixing (TR-mamba2attn) and another with transposed-sequence MLP layers (TR-mamba2mlpt).

Another notable engineering insight is the rigorous use of post-norm (as opposed to pre-norm) for residual connections, which is demonstrated to be essential for stabilizing recurrent computation in unrolled recursive architectures.

Quantitative Findings and Empirical Insights

On the ARC-AGI-1 abstract reasoning benchmark, the TR-mamba2attn model achieves a +2.0% absolute gain on pass@2 (45.88% vs 43.88%) over the baseline attention-based TRM, with the performance gap increasing as the pass@K metric is relaxed—rising to +4.75% at pass@100. Crucially, this improvement in candidate coverage does not come at the cost of top-1 selection quality, as pass@1 parity is preserved.

Figure 2: Training curves showing that TR-mamba2attn consistently outperforms the attention-only baseline at pass@2 and higher K metrics, with gains emerging early and remaining stable throughout training.

A detailed analysis of prediction statistics on ARC-AGI-1 reveals that the hybrid model generates more unique candidates per puzzle (+27%), achieves higher vote entropy, and exhibits more diverse solution exploration. Conversely, the attention baseline concentrates more decisively on the leading candidate, yielding a higher vote share for the top prediction. This coverage vs. selection trade-off is evident both quantitatively and in stratification analyses: the hybrid performs better on harder puzzles where candidate diversity increases the chance of capturing ambiguous ground-truth answers, while the attention baseline has a slight edge in cases where the correct answer does not require such diversity.

Figure 1: TR-mamba2attn produces more unique candidates and exhibits higher exploration entropy, while TRM-attn demonstrates more decisive selection through higher vote concentration and margin.

On secondary tasks, such as Sudoku-Extreme and Maze-30x30-Hard, results are contingent on input dimensionality and the communication mechanism. Dense mixing (via MLP-t) outperforms all other models on 9x9 Sudoku, indicating that constraint satisfaction benefits from full connectivity. However, these variants fail to scale to larger spatial reasoning tasks, where the hybrid's efficiency (and likely inductive biases) confer a substantial advantage; TR-mamba2attn achieves 80.6% accuracy on the hardest Maze tasks, compared to 60.8% for the attention baseline, with all dense-mixing models failing completely.

Theoretical and Practical Implications

This work demonstrates that SSM-based mixers—specifically Mamba-2—are viable and in some regimes superior choices for recursive operator design. The result directly challenges the necessity of attention-centric architectures for high-performance latent recursion, expanding the operator design space to include SSM/attention hybrids.

On a practical level, these findings have implications for the construction of efficient, high-performing reasoning agents leveraging recursion with a minimal parameter budget. The demonstrated pass@K improvements for ARC-style abstract reasoning and the hybrid's scalability to large-instance tasks suggest that such architectures may be favored for real-world, computationally-constrained AGI systems.

Theoretically, the results inform ongoing debates about explicit versus latent chain-of-thought reasoning and the nature of reasoning in neural architectures—supporting the argument that "more thinking time" via recursion may unlock reasoning advances, independent of model scale, provided that the recursive operator is suitably expressive.

Future Directions

Several lines of further research are indicated:

• Deeper integration: Can the recursive loop be absorbed into SSM state updates, leveraging the inner recurrence of Mamba-2?
• Operator mixing optimization: What is the optimal ratio and arrangement of SSM blocks, attention heads, and dense mixers within the recursive scaffold?
• Compute-adaptive inference: How does this hybrid approach interplay with test-time compute scaling (e.g., dynamic recursive depth), especially in domains like program synthesis or combinatorial reasoning?
• Scaling laws: How does hybrid operator performance evolve with model size, recursion depth, or input dimensionality?

Conclusion

This paper rigorously establishes that Mamba-2 hybrid operators preserve—and on key metrics, enhance—the reasoning abilities of tiny recursive architectures on challenging abstract reasoning benchmarks. The results show a clear coverage advantage for the hybrid operator, with growing returns as the candidate set size increases, all without sacrificing decisive top-1 selection. These findings extend the recursive operator design space into the SSM regime and provide foundational evidence for the composability and expressivity of hybrid reasoning architectures. This work sets the stage for further exploration of optimal operator design and integrated recurrence in future reasoning models.