Solve the Loop: Attractor Models for Language and Reasoning

Abstract: Looped Transformers offer a promising alternative to purely feed-forward computation by iteratively refining latent representations, improving language modeling and reasoning. Yet recurrent architectures remain unstable to train, costly to optimize and deploy, and constrained to small, fixed recurrence depths. We introduce Attractor Models, in which a backbone module first proposes output embeddings, then an attractor module refines them by solving for the fixed point, with gradients obtained through implicit differentiation. Thus, training memory remains constant in effective depth, and iterations are chosen adaptively by convergence. Empirically, Attractor Models outperform existing models across two regimes, large-scale language-model pretraining and reasoning with tiny models. In language modeling, Attractor Models deliver a Pareto improvement over standard Transformers and stable looped models across sizes, improving perplexity by up to 46.6% and downstream accuracy by up to 19.7% while reducing training cost. Notably, a 770M Attractor Model outperforms a 1.3B Transformer trained on twice as many tokens. On challenging reasoning tasks, we show that our model with only 27M parameters and approximately 1000 examples achieves 91.4% accuracy on Sudoku-Extreme and 93.1% on Maze-Hard, scaling favorably where frontier models like Claude and GPT o3, fail completely, and specialized recursive reasoners collapse at larger sizes. Lastly, we show that Attractor Models exhibit a novel phenomenon, which we call equilibrium internalization: fixed-point training places the model's initial output embedding near equilibrium, allowing the solver to be removed at inference time with little degradation. Together, these results suggest that Attractor Models make iterative refinement scalable by turning recurrence into a computation the model can learn to internalize.

• The paper introduces an implicit fixed-point architecture using attractor models for scalable iterative refinement in language modeling and hard reasoning tasks.
• It employs a backbone causal transformer and a shared-weight attractor module with implicit differentiation to obtain stable, constant-memory training.
• Empirical results show notable perplexity reductions and high accuracy on reasoning tasks even with modest model sizes, emphasizing resource efficiency.

Summary of Attractor Models for Language and Reasoning

"Solve the Loop: Attractor Models for Language and Reasoning" (2605.12466) introduces Attractor Models, an implicit fixed-point architecture designed for scalable iterative refinement in language modeling and hard reasoning tasks. This architecture addresses key limitations of explicit recurrence in looped Transformers by leveraging fixed-point solvers, enabling adaptive, stable, and efficient training and inference across model regimes.

Motivation: Drawbacks of Looped Recurrence in Transformers

Looped and recurrent architectures, including Universal Transformers and looped LMs, enable latent iterative computation beyond purely feed-forward models. Such mechanisms are theoretically promising for algorithmic and reasoning-centric tasks, as they support iterative refinement of internal representations. Empirically, looped models have achieved improvements over standard Transformers in both large-scale language modeling and algorithmic reasoning. However, these advances come at significant practical cost:

• Training instability and memory inefficiency: Standard looped models require explicit unrolling, leading to memory and compute growth linear in loop depth.
• Test-train mismatch: Quality degrades if inference uses different recurrence depths than those observed during training.
• Fragile scaling: Some recursive architectures, especially in the small-model regime, experience catastrophic performance drops with increased capacity.

Empirical studies have revealed that for the majority of input tokens, the recurrent trajectory in looped LMs tends toward a fixed point, implying that their recursive map approximates an implicit function.

Attractor Model Architecture

Attractor Models generalize the concept of iterative reasoning by transforming the output embedding refinement into a fixed-point computation. The architecture comprises:

• Backbone module: Typically a causal Transformer that generates an initial output embedding $\mathbf{y}_0$ as a semantically meaningful proposal.
• Attractor module: A separate, typically smaller, shared-weight Transformer that refines $\mathbf{y}_0$ through recurrent application, seeking the fixed point $\mathbf{y}^*$ of the refinement map.

The fixed point $\mathbf{y}^*$ is obtained by solving

$$\mathbf{y}^* = \mathcal{T}_{\mathrm{a}}(\mathbf{y}^*, \mathbf{y}_0)$$

where persistent proposal injection ensures the refinement dynamics are input-dependent and do not degenerate into trivial attractors.

Implicit Differentiation (IFT): Gradients are computed through the fixed point using implicit differentiation, circumventing the need to backpropagate through all solver iterations. This results in memory requirements that are constant with respect to the number of solver steps, in contrast to standard explicit-loop architectures.

Adaptive Iteration: During inference (and optionally training), the root-finding solver (Anderson acceleration in implementation) adaptively iterates until the residual norm falls below a threshold, rather than running for a fixed number of loops.

Equilibrium Internalization

A notable empirical phenomenon is equilibrium internalization: over training, the backbone proposal $\mathbf{y}_0$ is increasingly optimized to be close to the fixed point $\mathbf{y}^*$. Consequently, fewer refinement iterations are required during inference, and in many cases the backbone proposal alone yields near-optimal performance. This self-distillation effect results in inference-time compute that approaches that of feed-forward models, while retaining the training-time regularization and generalization benefits of implicit recurrence.

Empirical Results

Large-Scale Language Modeling

Attractor Models demonstrate improved LLM performance over parameter-matched Transformers and state-of-the-art looped LMs (e.g., Parcae), across all tested scales (140M, 370M, 770M parameters). Key results include:

• Up to 46.6% improvement in perplexity on the Lambada out-of-distribution benchmark.
• Up to 19.7% gains in downstream benchmark accuracy.
• Attractor Models achieve Pareto efficiency: superior language modeling metrics at lower training compute and memory overhead.
• A 770M Attractor Model outperforms a 1.3B standard Transformer trained on double the amount of data, with 25–31% reduction in training FLOPs compared to explicit-loop baselines, owing to adaptive solver convergence.

Hard Reasoning with Tiny Models

On tasks such as Sudoku-Extreme and Maze-Hard—where most non-recurrent Transformers and even frontier LLMs (DeepSeek R1, Claude, o3-mini) achieve 0%—Attractor Models with only 27M parameters and ~1000 training examples achieve:

• 91.4% accuracy on Sudoku-Extreme
• 93.1% accuracy on Maze-Hard

Crucially, while existing recursive architectures (e.g., TRM, HRM) fail to scale (performance collapses to 0% at 27M parameters), Attractor Models scale favorably, with increased capacity yielding stronger results. This is attributed to the convergence-driven regularization of the fixed-point objective.

Theoretical Insights

The Attractor Model framework provides a principled implicit regularization effect, driving the recurrent map toward contractive dynamics and stable equilibria. In contrast, fixed-depth looped models can "cheat" by only approximating the desired refinement at fixed iteration counts—leading to brittle extrapolation and inferior generalization when the number of inference-time iterations is altered.

Root-finding initialization from the backbone proposal and persistent proposal injection during refinement are numerically critical for stability and convergence speed. Ablation studies confirm that naive initializations (e.g., zero or random noise) degrade convergence rates and final model quality.

Practical Implications and Future Directions

Practical Implications:

• Resource-Efficient Training and Inference: Constant-memory training in the recurrent module and adaptive inference make these models amenable to large-scale deployment and efficient scaling.
• Unified Architecture for Multiple Scales: The same attractor principle confers benefits in both large LLMs and small-data, small-model algorithmic reasoning tasks.
• Seamless Interpolation Between Feed-Forward and Recurrent Models: As equilibrium internalization strengthens, inference cost shrinks to that of pure feed-forward, but the model still learns with iterative refinement pressure.

Future Developments may include:

• Deeper mechanistic analyses of equilibrium internalization and its role as a regularizer.
• Exploring alternative attractor module parameterizations and solvers.
• Integration with vision, multimodal, or reinforcement learning domains leveraging implicit iterative reasoning.
• Theoretical tightening of necessary contractivity conditions and convergence guarantees for arbitrary Transformer blocks in this framework.

Conclusion

Attractor Models (2605.12466) offer a theoretically justified, empirically validated mechanism for scalable, stable, and efficient iterative refinement in language modeling and reasoning. Through implicit fixed-point computation and equilibrium internalization, they deliver strong performance and resource efficiency across model sizes—outperforming established recurrent and feed-forward baselines. This work opens new avenues for implicit architectures that blend the strengths of recurrent computation and feed-forward efficiency, suggesting iterative refinement as a central principle for future models in language and reasoning AI.