Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts

Abstract: Hybrid Transformer architectures, which combine softmax attention blocks and recurrent neural networks (RNNs), have shown a desirable performance-throughput tradeoff for long-context modeling, but their adoption and studies are hindered by the prohibitive cost of large-scale pre-training from scratch. Some recent studies have shown that pre-trained softmax attention blocks can be converted into RNN blocks through parameter transfer and knowledge distillation. However, these transfer methods require substantial amounts of training data (more than 10B tokens), and the resulting hybrid models also exhibit poor long-context performance, which is the scenario where hybrid models enjoy significant inference speedups over Transformer-based models. In this paper, we present HALO (Hybrid Attention via Layer Optimization), a pipeline for distilling Transformer models into RNN-attention hybrid models. We then present HypeNet, a hybrid architecture with superior length generalization enabled by a novel position encoding scheme (named HyPE) and various architectural modifications. We convert the Qwen3 series into HypeNet using HALO, achieving performance comparable to the original Transformer models while enjoying superior long-context performance and efficiency. The conversion requires just 2.3B tokens, less than 0.01% of their pre-training data

• The paper introduces HALO, a data-efficient distillation method that reduces token usage by 100x compared to state-of-the-art approaches while preserving long-context recall.
• The paper details HypeNet’s architectural innovations, including hybrid position encoding, QK-Normalization, and output gating, which enhance both recall and computational efficiency.
• The study empirically validates the hybrid model’s performance on extended benchmarks, achieving near-teacher accuracy and significant speedups in decoding and prefilling.

Hybrid Linear Attention Done Right: Efficient Distillation and Architectures for Extreme Contexts

Introduction

"Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts" (2601.22156) introduces a highly data-efficient architecture and distillation pipeline for constructing hybrid long-context LLMs that combine softmax attention and RNN-based (linear) attention layers. The work specifically addresses limitations in contemporary Transformer-to-hybrid model distillation, proposing HALO—a procedure that dramatically reduces data requirements—and HypeNet, a hybrid model leveraging novel position encoding and architectural modifications for superior length generalization and efficiency.

Context and Motivation

Transformers, while flexible and powerful, scale quadratically with the input context, becoming computationally prohibitive at million-token contexts. Pure RNN-based or linear attention models scale linearly, but empirically degrade on recall-demanding tasks relative to softmax attention architectures, particularly for long-term dependencies and tasks like "needle-in-a-haystack". Hybrid models interleaving attention and RNN layers are a promising solution, yet systematic study and deployment are hampered by the cost and resource demands of large-scale pretraining. Most competitive hybrids are only open-sourced as weights and not easily distillable from existing models. Prior art on distilling pure attention models into hybrids still requires tens to hundreds of billions of tokens, with resulting hybrids showing poor long-context performance.

HALO: Data-Efficient Hybrid Distillation

HALO (Hybrid Attention via Layer Optimization) resolves the inefficiency and performance gaps of previous methods:

• Attention Weight Transfer: Pretrained attention layers’ weights are used to initialize corresponding RNN layers (e.g., Lightning Attention, GLA, GDN, RWKV-7).
• Hidden State Alignment (Stage 1): RNN layers are independently aligned to replicate their originating attention layer’s hidden states (via MSE loss).
• Layer Selection: Critically, HALO introduces an importance score based on the empirical recall and CSR decline when swapping each attention layer for RNN. Attention layers with significant recall drops but limited CSR impact are retained. This is substantially more efficient than exhaustive, KL-guided searches used in previous work.
• Distillation (Stage 2): End-to-end KL divergence-based knowledge distillation is performed from the teacher to the hybrid.
• Long-context Finetuning (Stage 3): The hybrid is finetuned for extended contexts with further reduced LR.

Notably, the complete conversion process consumes only 2.3B tokens—two orders of magnitude less than state-of-the-art distillers such as Jet-Nemotron (400B) or KL-LS (25B)—while preserving long-context capability.

HypeNet: Architectural Innovations for Hybrid Models

HypeNet is the hybrid architecture instantiated via HALO, distinguished by several critical technical advances:

HyPE: Hybrid Position Encoding

HyPE deploys a hybrid positional encoding strategy:

• RoPE (Rotary Position Encoding) is applied within RNN layers to provide strong short-range positional awareness.
• NoPE (absence of RoPE) is used in attention layers, supporting improved train-free length generalization.

This hybridization resolves positional encoding mismatch in existing hybrids and is empirically shown to enhance both short- and extremely long-context performance, a central requirement for million-token contexts.

Architectural Modifications

• QK-Normalization in RNN layers: Analogous to successful practices in attention heads, normalizing Q and K in the RNN boosts overall model robustness.
• Conversion of GQA to MHA for RNN: As RNNs lack a KV cache, the grouped query design of many attention models is replaced with multi-head, increasing RNN expressivity.
• Output Gating Mechanisms: Addition of output gates via element-wise gating, shown to improve both length generalization and recall, is standardized in both attention and RNN layers.
• Attention Logit Scaling: Position-dependent scaling (softmax logits scaled by $\log_\alpha(t + \alpha)$) in the attention mechanism is crucial for maintaining entropy and recall at extensive context lengths.

These modifications result in HypeNet models that, despite a roughly 10% increase in parameter count, are substantially more efficient due to reduced KV cache costs.

Modular Mixer Support

HALO allows near-arbitrary RNN-based mixers (Lightning, Mamba, GLA, GDN, etc.), with empirical results demonstrating that Lightning Attention provides the best balance of recall and length generalization for the tested hybrids.

Empirical Results

HALO and HypeNet are validated via rigorous experiments against Qwen3-series (as teacher) and several distilled baselines, including Jet-Nemotron and KL-LS.

• Long-Context Recall: On NIAH benchmarks up to 256K context, HypeNet+HALO matches or exceeds the teacher Qwen3, Jet-Nemotron, and KL-LS while being trained with astronomically fewer tokens. HypeNet achieves up to 99.8% accuracy at 256K context on NIAH, outperforming all baselines and nearly matching teacher performance beyond 128K.
• CSR Performance: On zero-shot tasks (ARC, HellaSwag, PIQA, MMLU), HypeNet's performance remains competitive with Qwen3 and other state-of-the-art baselines.
• Efficiency: HypeNet achieves up to 3.0x decoding and 3.4x prefilling speedups compared to Qwen3 with BFloat16, at million-token contexts, with drastically reduced memory usage.
• Ablation Studies: Each architectural modification (HyPE, QK-Norm, GQA→MHA, output gate) provides quantifiable CSR and recall benefits. The layer selection method outperforms both naive (uniform) and SOTA (KL-guided, empirical drop-based) approaches for the attention–RNN split.
• Training from Scratch: When trained from scratch, HypeNet with HyPE and Lightning Attention achieves strong NIAH accuracy on up to 64x the training context length, demonstrating superior train-free length extrapolation.

Theoretical and Practical Implications

The combination of HALO and HypeNet directly addresses several barriers in the study and deployment of efficient, long-context LLMs:

• Theoretical Implication: The superior recall and length generalization suggest that hybrid architectures with appropriate positional encoding and selective attention persistence are sufficient for near-Transformer-level performance with linear scaling, resolving perceived limitations of RNN layers.
• Practical Implication: The two-order-of-magnitude reduction in distillation tokens democratizes long-context hybrid LLM research, enabling academic labs to empirically validate hybrid innovations at scale without requiring industry-scale compute.

HALO’s explicit selection of recall-critical attention layers creates a new principled path for hybridization, while HyPE’s decoupling of position encoding between attention and RNN sublayers offers a generalizable solution likely to benefit broader classes of sequence models.

Limitations and Future Directions

While HALO and HypeNet make hybrid distillation practical and efficient, several limitations and open problems persist:

• Instruction/Alignment Loss: The distillation and finetuning process may attenuate instruction-following or task-aligned behaviors embedded via post-training in the pretrained teacher; efficient protocols for preserving or rapidly recovering alignment remain open.
• Generality Beyond Transformers: The conversion method is specific to Transformer-based architectures; its efficacy for non-Transformer or SSM-based models is untested.
• Mature RNN Designs: While Lightning Attention is currently optimal within HypeNet, advances in RNN-based sequence models may further improve hybrid performance and require updated benchmarks.

Conclusion

The HALO procedure and HypeNet architecture collectively establish a new efficiency-performance frontier in hybrid long-context LLMs, mitigating the quadratic bottleneck of attention and the recall deficiencies of RNNs. This work fundamentally broadens accessibility to hybrid model pretraining and innovation, while empirically demonstrating that careful architectural tuning and layer selection robustly close the gap to pure Transformer models for extreme context lengths. Further progress in hybrid LLMs will likely build on these advances in positional encoding, selective attention retention, and streamlined cross-architecture distillation (2601.22156).