Efficient Context Scaling with LongCat ZigZag Attention (2512.23966v1)

Abstract: We introduce LongCat ZigZag Attention (LoZA), which is a sparse attention scheme designed to transform any existing full-attention models into sparse versions with rather limited compute budget. In long-context scenarios, LoZA can achieve significant speed-ups both for prefill-intensive (e.g., retrieval-augmented generation) and decode-intensive (e.g., tool-integrated reasoning) cases. Specifically, by applying LoZA to LongCat-Flash during mid-training, we serve LongCat-Flash-Exp as a long-context foundation model that can swiftly process up to 1 million tokens, enabling efficient long-term reasoning and long-horizon agentic capabilities.

• The paper introduces LongCat ZigZag Attention (LoZA), a sparse attention mechanism that transforms full-attention modules into computationally efficient sparse variants.
• It employs a two-phase approach with mid-training calibration and selective sparsification, achieving up to 90% lower decode costs for long contexts.
• Empirical results validate that LoZA improves long-context generalization and energy efficiency in both foundation and chat models, enabling 1M-token contexts.

Efficient Context Scaling in LLMs via LongCat ZigZag Attention

Introduction

Scaling transformer-based LLMs to support extensive context windows, without incurring prohibitive computational costs, remains a significant technical barrier for both research and deployment. The "Efficient Context Scaling with LongCat ZigZag Attention" paper (2512.23966) addresses this by proposing LongCat ZigZag Attention (LoZA)—an efficient sparse attention mechanism designed to operate within the Mixture-of-Local-Attentions (MLA) framework, enabling seamless transformation of full-attention modules to sparse ones with minimal performance degradation. Applying LoZA in LongCat-Flash-Exp enables efficient inference and training for context windows of up to 1 million tokens, supporting advanced long-horizon reasoning and agentic workflows.

Algorithmic Design and Methodology

The LoZA construction proceeds in two distinctive phases: calibration and selective sparsification during mid-training, followed by additional fine-tuning to close residual performance gaps. The pipeline adapts MLA-based architectures (e.g., LongCat-Flash), replacing a subset of full-attention layers with streaming sparse attention (SSA).

Figure 1: The LoZA pipeline alternates between full and streaming sparse attention at the layer level, guided by calibration and subsequent training to realize computational sparsity.

Calibration attaches a learnable parameter $a_i$ to each MLA. These gating factors enable the network to interpolate between the outputs of full and sparse attention for each layer:

$$\hat{\mathbf{O}}_i = \alpha_i \cdot \mathbf{O}_i + (1 - \alpha_i) \cdot \mathbf{O}'_i,$$

where $\mathbf{O}_i$ is the dense output and $\mathbf{O}'_i$ is the SSA alternative. Layer importance is inferred by optimizing $a_i$ with the core model weights frozen, after which the least contributive 50% of MLAs (i.e., those with lowest $a_i$) are replaced permanently by SSA. Final mid-training phases allow the model to adapt to the sparsified topology, successfully minimizing adverse effects on generalization and reasoning benchmarks even at extreme sequence lengths.

This layer-level sparsity, in contrast to head-level sparsity (as in Duo-Attention), mitigates both kernel control flow complexity and heterogeneity in model parallelism scheduling. The convergence with the lottery ticket hypothesis framework—iterative pruning and fine-tuning—further frames LoZA as a theoretically grounded sparsification schema for LLMs.

Training and Data Composition

LoZA is integrated into LongCat-Flash-Exp during a dedicated mid-training phase, extending context from 32K to 256K tokens, and is extrapolated to 1M tokens via YaRN. The mid-training leverages 500B tokens across reasoning-rich, agentic, long-form, and codebase-centric data, ensuring robust adaptation to both short and ultra-long context tasks.

Post-training involves supervised fine-tuning (SFT) and Direct Preference Optimization (DPO), with a carefully curated dataset balanced across core instruction, mathematics, coding, and agentic tasks. This strategy avoids overfitting to particular context lengths and maintains competitive real-world and synthetic benchmark performance.

Empirical Performance

The practical utility of LoZA is validated in two key scenarios: LongCat-Flash-Exp-Base (foundation model) and LongCat-Flash-Exp-Chat (instruct/chat model). Results demonstrate that conversion to SSA—even at high sparsity ratios—does not degrade critical metrics (MMLU-Pro, GSM8K, HumanEval+, LongEval) in the base model, and enables strong long-context reasoning while preserving or enhancing computational efficiency in the chat/instruct model.

Notably, for benchmarks assessing long-context generalization (MRCR; LongBenchV2), LongCat-Flash-Exp-Chat not only matches but frequently exceeds other state-of-the-art models capable of 1M-token contexts (e.g., Qwen-3, GLM-4.6), as measured by area under the curve (AUC) and task completion accuracy.

Figure 2: Accuracy of LongCat-Flash-Exp-Chat compared with Qwen-3 on MRCR (2-needle)—demonstrating LoZA's effectiveness across context scales to 1M tokens.

LoZA further delivers considerable practical speedups. Streaming sparse attention kernels can yield up to 90% lower decode costs compared to dense attention for long contexts (128K+ tokens). End-to-end latency and energy use are similarly improved, with over 50% faster prefill and 30% reduced decode cost for large contexts, as verified on modern H20 hardware.

Figure 3: LoZA’s decode kernel achieves major reductions in compute cost relative to full attention, especially at 128K–1M context lengths.

Implementation Considerations

The transition to layer-wise sparsity eliminates common engineering pitfalls in heterogeneous head-level sparsity, notably kernel warp divergence and load imbalance. Uniform scheduling and simplified kernel design streamline inference, while model parallelism efficiency is maximized due to symmetric computational loads per device. Parameters such as block size ($b=128$), sink blocks ($s=1$), local blocks ($l=7$), and design choices for calibration/training are critical for optimal deployment.

Implications and Future Directions

The LoZA methodology demonstrates that properly calibrated, layer-level streaming sparse attention can unlock high context capacity (1M tokens and beyond) without a proportional increase in training or inference costs. This renders context-native agentic and retrieval-augmented applications viable at unprecedented sequence scales, and points to a future in which the quadratic bottleneck of dense attention is no longer a practical concern for foundation models.

The LoZA design philosophy—calibrate, sparsify, and adapt—offers a template for broader adoption in other LLM backbones, including those with mixture-of-experts (MoE) or multi-modal contexts. Its engineering is broadly compatible with leading inference accelerators and distributed settings.

Conclusion

LongCat ZigZag Attention (LoZA) provides a theoretically-motivated, empirically-validated approach to sparse attention via mid-training calibration and layer-level pruning, preserving model quality while dramatically increasing context length and decreasing compute cost (2512.23966). Its adoption permits scaling context windows to 1 million tokens and beyond, with strong implications for LLM-powered reasoning and agentic tasks. The methodology can potentially generalize to other structured sparse attention scenarios, facilitating future innovations in ultra-long-context models.