Effective Distillation to Hybrid xLSTM Architectures

Abstract: There have been numerous attempts to distill quadratic attention-based LLMs into sub-quadratic linearized architectures. However, despite extensive research, such distilled models often fail to match the performance of their teacher LLMs on various downstream tasks. We set out the goal of lossless distillation, which we define in terms of tolerance-corrected Win-and-Tie rates between student and teacher on sets of tasks. To this end, we introduce an effective distillation pipeline for xLSTM-based students. We propose an additional merging stage, where individually linearized experts are combined into a single model. We show the effectiveness of this pipeline by distilling base and instruction-tuned models from the Llama, Qwen, and Olmo families. In many settings, our xLSTM-based students recover most of the teacher's performance, and even exceed it on some downstream tasks. Our contributions are an important step towards more energy-efficient and cost-effective replacements for transformer-based LLMs.

• The paper demonstrates a novel distillation technique using a hybrid mLSTM–SWA architecture that achieves lossless teacher parity on various benchmarks.
• It outlines a three-stage pipeline—layer-wise alignment, sparse knowledge distillation, and expert merging—that fine-tunes student models with targeted domain capabilities.
• Empirical results indicate significant inference efficiency gains, with reduced latency and constant memory footprint, while reliably matching or exceeding teacher performance.

Effective Distillation to Hybrid xLSTM Architectures: Technical Analysis

Motivation and Context

Transformer-based LLMs attain state-of-the-art results across diverse domains due to quadratic attention mechanisms, but this comes at substantial inference and deployment cost associated with quadratic scaling in context length. Numerous prior works have explored post-training linearization—replacing full attention with sub-quadratic operators such as linear attention, SSMs, or gated RNNs—to achieve energy-efficient and cost-effective alternatives. However, existing distilled models reliably match teachers on language understanding tasks but reliably underperform on generative tasks, notably mathematical reasoning and code synthesis. The present work sets the explicit goal of achieving lossless distillation, using rigorous Win-and-Tie rate criteria to capture reliability as drop-in replacements.

Architecture and Distillation Pipeline

The core student architecture deploys a hybrid attention block that unifies a matrix-LSTM (mLSTM) path with sparse sliding-window attention and sink tokens, fused via learned, data-dependent gates. This intra-layer hybridization allows every attention head to exploit complementary modeling regimes: mLSTM's global memory capacity and SWA's exact local recall.

The distillation pipeline comprises three stages:

1. Layer-wise hidden-state alignment: Student mLSTM–SWA hybrids are initialized with teacher weights. Only mixer/gating parameters are optimized for MSE matching to teacher attention outputs.
2. Sparse knowledge distillation (full fine-tuning): All parameters are unfrozen and student logits are aligned with the teacher via a mixed CE/KL objective on precomputed top-$k$ teacher targets ($k=256$), a protocol maximizing token efficiency.
3. Expert merging: Specialists for math, code, STEM, and instruction-following domains are trained independently and consolidated via linear weight-space merging, introducing modularity and decoupling domain development.

Layer-wise normalization, rotary position embeddings, and head-wise feature maps are leveraged for robust adaptation, but careful ablations show that over-normalization degrades teacher alignment.

Empirical Evaluations

Benchmarking Protocol

Rigorous evaluation leverages Win-and-Tie rate $C_\alpha$—the fraction of benchmarks where student performance matches or exceeds that of the teacher within tolerance $\alpha$—and critical tolerance $\alpha^*$. Recovery rate (student/teacher score ratio) characterizes domain-specific parity, but $C_\alpha$ is required for reliable model selection and Pareto front comparisons across heterogeneous domains.

Results

• Language understanding tasks: xLSTM-based students achieve near-complete teacher parity ($C_\alpha=1.0$ or $>0.99$ across MMLU, ARC, PIQA), outperforming LoLCATs and QRWKV baselines with comparable parameter counts.
• Generative tasks (math, code, reasoning): Previous methods suffer pronounced performance gaps (recovery rates $\sim$0.4–0.8), whereas hybrid students recover virtually all teacher performance and exhibit positive transfer on several metrics, oftentimes exceeding teacher scores.
• Instruction-following/chat: Evaluated by GPT-5.1, merged xLSTM students are preferred over their teachers on MT-Bench and instruction benchmarks.
• Model merging: Weight-space fusion of domain specialists yields robust improvement in multi-domain generalization, particularly instruction-following, though STEM capabilities present residual negative interference.
• Ablations: mLSTM gating (vs. linear attention), SWA, and sink tokens all contribute significant gains, with full fine-tuning (FFT) far superior to LoRA-based PEFT or mixer-only adaptation.

Long-context Retrieval and Memory Limitations

On synthetic long-context retrieval (Needle-in-a-Haystack), vanilla xLSTM students degrade rapidly at longer context lengths relative to transformers, implicating limitations of fixed-size memory or lack of exposure to long-context traces during distillation; unresolved questions remain regarding scaling memory state or hybridization efficacy.

Inference Efficiency

Hybrid xLSTM students consistently halve latency and memory consumption during prompt prefill and autoregressive generation. Memory footprint remains constant with sequence length and batch size, unlike attention-based teachers that rapidly hit OOM as batch/context increases. Throughput scales up to 4$\times$ relative to transformer teachers for large generation budgets.

Practical and Theoretical Implications

This distillation pipeline demonstrates that mLSTM–SWA hybrids can serve as efficient, reliable drop-in replacements for Transformer LLMs. The modular expert-merge protocol enables parallel capability development and targeted updates, reflecting broader systems trends in scalable post-training (Branch-Train-Merge, weight-soups, TIES-merging).

For production, serving heterogeneous architectures requires fundamental advances in runtime scheduling, cache allocation and memory abstractions (e.g. vLLM/SGLang, Jenga), as hybrid model adoption accelerates in open and commercial settings.

From a theoretical standpoint, the results reinforce the expressivity and stability of recurrent state-space alternatives such as xLSTM over pure linear attention, especially via intra-layer hybridization and gating, and underscore the necessity of rigorous model comparison criteria beyond recovery rate.

Limitations and Future Directions

Residual deficits are most prominent on synthetic long-context evaluations and STEM reasoning, where domain interference and memory scaling limit performance. Future research should probe scalable memory capacity, more nuanced domain consolidation protocols (e.g., TIES-merging), richer hybrid memory designs, and on-policy or RL-based distillation for further expert refinement. Extension to large sparse-MoE teachers and consolidation of more diverse capabilities remains an open direction.

Conclusion

This work establishes a robust linearization pipeline utilizing mLSTM–SWA hybrid blocks and expert merging, advancing the distillation of transformer teachers to efficient, linear-complexity LLMs. Hybrid xLSTM students reliably match or exceed teacher performance across the majority of language understanding and generative benchmarks, with pronounced inference efficiency gains. The protocol enables modular capability development and scalable consolidation, charting a practical path for the next generation of efficient LLMs in both open and commercial environments (2603.15590).