On Subquadratic Architectures: From Applications to Principles

Abstract: Transformers dominate modern sequence modeling, but their quadratic attention incurs substantial computational cost. Subquadratic architectures offer a scalable alternative. However, it remains unclear which designs yield the most effective sequence models. We compare three leading approaches: xLSTM, Mamba-2, and Gated DeltaNet. We evaluate these models on tasks with complex dependencies: (1) code-model pre-training, (2) distillation of code models from LLMs, and (3) pre-training of time-series foundation models. Across these settings, xLSTM delivers the strongest overall performance. To explain xLSTM's advantage, we present a unified formulation and analyze the underlying architectural mechanisms, focusing on state tracking and memory dynamics. Our results show that xLSTM enables more flexible and stable memory correction via its gating scheme. We corroborate these findings on controlled synthetic length-generalization tasks. Overall, our findings indicate that xLSTM's gains on complex tasks stem from robust state tracking and accumulation.

• The paper demonstrates that xLSTM outperforms its counterparts in code generation and time series forecasting by effectively balancing accumulation and state tracking.
• It employs rigorous empirical evaluations, including HumanEval metrics and synthetic tasks, to benchmark the performance of subquadratic architectures.
• The unified framework clarifies how explicit gating mechanisms enhance long-range dependency modeling and robust state management.

Comparative Analysis of Subquadratic Sequence Models: xLSTM, Mamba-2, and Gated DeltaNet

Introduction

This paper provides a systematic comparison of three prominent subquadratic architectures—xLSTM, Mamba-2, and Gated DeltaNet—across sequence modeling tasks characterized by complex, long-range dependencies. The focus is on applications beyond standard language modeling benchmarks, specifically on code modeling and time series forecasting, domains where structured dependencies and nontrivial memorization requirements distinguish practical utility. The work further introduces a unified architectural framework explaining empirical differences between these models, particularly emphasizing the primitives of accumulation and state tracking, and substantiates the theoretical framework via targeted synthetic experiments.

Empirical Evaluation on Application Domains

Code-focused LLM Pre-training

The authors pretrain 400M-parameter hybrid LLMs, interleaving a small number of self-attention layers with each tested subquadratic operator. Code generation is the principal evaluation target, with broad reasoning and commonsense tasks as secondary benchmarks.

xLSTM consistently outperforms Mamba-2 and Gated DeltaNet on HumanEval pass@k metrics at all pretraining data scales and configurations, including both code-only and code+natural-language mixtures. Notably, at the 20B token scale for code, xLSTM[7:1] surpasses the next-best backbone by up to 1.43 absolute points under pass@64. The margin narrows at 100B tokens across all subquadratic operators, yet the superiority of xLSTM persists.

Figure 1: HumanEval pass@k after code-focused pre-training for hybrid LLMs with 400M parameters under different data regimes; xLSTM maintains a measurable edge, especially at lower data volumes.

Secondary benchmarks (HellaSwag, PIQA, ARC, WinoGrande) reinforce the aggregate advantage of xLSTM, though with reduced margins, indicating that the differentiation among architectures sharpens as the complexity of structured dependencies increases.

Distillation of Transformer Knowledge

The study probes whether the advantages of xLSTM extend to the knowledge distillation regime, where subquadratic students—xLSTM[1:0] and Gated DeltaNet variants—are trained to match the outputs of a strong attention-based teacher (Qwen3-4B-Instruct). The distillation pipeline employs both output alignment and sparse knowledge distillation.

xLSTM[1:0] matches or exceeds Gated DeltaNet on three out of four code benchmarks (HumanEval, HumanEval+, MBPP, MBPP+) and secures the highest average performance. The negative-eigenvalue variant of Gated DeltaNet improves state tracking but does not close the gap to xLSTM[1:0]. These results extend to mathematical benchmarks including GSM8K and MATH-500, showing xLSTM[1:0] as the generally preferred matrix-state plug-in.

Time-series Foundation Model Pre-training

Extending the architecture comparison outside the NLP domain, the paper evaluates time-series forecasting using a unified training protocol. Models are assessed zero-shot on GIFT-Eval, considering scales from 1M to 80M parameters.

xLSTM[3:1] outperforms both Mamba-2 and Gated DeltaNet at all scales up to 40M parameters. For instance, at 10M parameters, xLSTM achieves a MASE of 0.733 and CRPS of 0.508, compared to 0.767 and 0.525 for Mamba-2, respectively. Only at 80M parameters do Mamba-2 and xLSTM perform nearly identically, suggesting that architectural disparities are more consequential at moderate model sizes.

Figure 2: GIFT-Eval performance for TSFMs across a range of parameter scales; xLSTM is consistently strongest for both MASE and CRPS until 80M parameters, where the margin tightens.

Unified Architectural Principle: State Tracking and Accumulation

The empirical results prompt a theoretical inquiry into the architectural primitives behind performance differences. The authors recast xLSTM, Mamba-2, and Gated DeltaNet within a unified framework based on explicit gating and memory update operations.

xLSTM combines a matrix-state “linear attention” block (mLSTM, xLSTM[1:0]) providing robust accumulation (akin to softmax-like summation over sequence history), with a nonlinear recurrent block (sLSTM, xLSTM[0:1]) supporting finite-state tracking. This duality enables explicit supervision over both long-range summing operations and structured state machines.

Mamba-2 features coupled input and forget gates (tied through a learned projection, as in GRUs), resulting in restricted expressiveness, particularly for unbounded accumulation tasks (such as $A^nB^nC^n$ counting).

Gated DeltaNet introduces explicit overwriting in memory cells via an orthogonal projection operator, enhancing retrieval tasks but potentially undermining stable accumulation. When parameterized to permit negative eigenvalues (Gated DeltaNet[-1,1]), state tracking on tasks such as parity improves, but at the expense of general accumulation fidelity.

Controlled Synthetic Experiments: Separating Model Primitives

To directly link architecture to computational primitive, the authors design synthetic tasks:

• Counting: $A^nB^n$, $A^nB^nC^n$, Majority
• State Tracking: Parity, Modular Arithmetic over $\mathbb{Z}_5$, and permutation composition in $S_3$

Models are trained at length 128 and evaluated with $4\times$ and $16\times$ extrapolation (lengths up to 2048).

xLSTM[1:0] generalizes robustly on counting, but fails at state tracking; pure recurrent layers enable state tracking but not unbounded accumulation. The hybrid xLSTM[1:1] smoothly combines both abilities, achieving perfect generalization on all state tracking tasks plus strong, though not maximal, counting performance. Mamba-2 and Gated DeltaNet show architecture-predicted limitations—collapse on counting or incomplete length generalization on state tracking.

Figure 3: Length generalization under controlled accumulation and state-tracking tasks; xLSTM[1:1] uniquely maintains near-perfect accuracy on both counting and parity even on extrapolated sequence lengths.

Implications and Future Directions

The investigation confirms that achieving both accumulation over arbitrary sequence lengths and robust finite-state tracking is essential for high-fidelity modeling of complex sequence dependencies. xLSTM's architecture, by decoupling matrix-state and recurrent updates, stands out in this respect. The study’s comprehensive protocol—spanning code, reasoning, and time series—demonstrates the broad relevance of these primitives.

Practically, these findings endorse xLSTM-heavy hybrids as competitive subquadratic backbones for foundation models and suggest task-dependent tuning of linear/recurrent layer ratios. Theoretically, the work sharpens our understanding of the interplay between parallelizable sequence operators and the automata-theoretic properties needed for complex language and program induction.

While tested here on moderate model sizes, open avenues include scaling to larger architectures, further refining kernel initializations for SSM-based models, and extending the operator comparison to yet broader data domains and operator families. Addressing practical deployment constraints, such as chunk-wise hardware acceleration and efficient recurrent computation in distributed settings, also remains a promising line of research.

Conclusion

Through matched empirical evaluation and a rigorous architectural analysis, this paper establishes that xLSTM outperforms Mamba-2 and Gated DeltaNet on tasks requiring both accumulation and state tracking. These results are consistent across pre-training, distillation, and foundation model settings in code and time series domains. The unified framework introduced herein not only elucidates the empirical patterns but also guides the principled design of future subquadratic sequence models. Further exploration at scale and across tasks will refine the landscape of operator selection for scalable, efficient foundation model architectures.