RTPurbo: Efficient Post-Hoc Sparsification

• RTPurbo is a post-hoc sparsification framework for LLMs that leverages intrinsic head- and token-level redundancies to convert full attention into efficient sparse attention.
• It partitions attention heads by ranking retrieval capacity and employs lightweight token indexers with dynamic top‑p routing to minimize redundant computation.
• The framework achieves substantial resource improvements, with up to 9.36× speedup and 85% reduction in KV memory while retaining near-lossless accuracy on long-context benchmarks.

RTPurbo is a post-hoc sparsification framework for transforming full-attention LLMs into highly efficient sparse-attention models using only a few hundred training steps. RTPurbo exploits the intrinsic head- and token-level sparsity of pretrained LLMs, and implements sparsification by selecting a small subset of “retrieval heads” for long-range token access, equipping those heads with lightweight low-dimensional token indexers, and applying a dynamic per-query top-$p$ token selection to minimize redundant computation and memory usage. Its adaptation procedure operates entirely post hoc and preserves near-lossless accuracy across long-context and reasoning benchmarks, delivering significant speed and memory reductions without requiring expensive native sparse pretraining (Zhou et al., 16 May 2026).

1. Foundational Observations: Intrinsic Sparsity in Attention

Analysis of full-attention LLMs reveals that attention redundancy is pervasive at both the head and token levels:

• Head-level sparsity: Only a small fraction of attention heads perform meaningful long-range retrieval; most heads restrict their attention to a local window and fixed “sink” tokens.
• Token-level sparsity: Even for remote-retrieving heads, each query accrues >90% of attention mass on a small subset of tokens, with the vast majority of potential keys receiving negligible weight.

These properties establish that full-attention models are amenable to aggressive sparsification without altering the overall distribution of attention mass or incurring significant capacity loss.

2. Sparsification Architecture and Algorithms

RTPurbo’s architecture introduces a hardware- and software-efficient partitioning of the attention mechanism:

• Head partitioning: A one-time offline calibration ranks heads by their normalized remote retrieval score $R_h$ on needle-in-haystack sequences. The top 15% (default) are designated as retrieval heads $\mathcal{H}_\mathrm{ret}$ and retain full key–value (KV) access; the remainder ($\mathcal{H}_\mathrm{loc}$) are assigned to local windowed attention plus sink tokens.
• Token indexer for retrieval heads: For each $h\in\mathcal{H}_\mathrm{ret}$, lightweight projectors $W^Q_h, W^K_h\in\mathbb{R}^{16\times d_h}$ are trained to map queries and keys into 16-dimensional space. At decode, projected similarities $s_h(m,n)$ are used to rank and softmax-normalize the N available keys.
• Dynamic top-$p$ routing: For each query in a retrieval head, projected softmax scores are used to select the minimal active key set $\mathcal{S}_h(m)$ such that $\sum_{n\in S}\beta_{m,n}\geq p$ (e.g., $R_h$0). The final full-dimensional attention computation (including rotary embeddings) is performed only on this active set. Local heads attend strictly within a sliding window (e.g., 8k tokens).

This hybridization ensures full retrieval capacity is reserved for critical heads/queries, maximizing both efficiency and representational integrity.

3. Adaptation and Training Regimen

The sparsification procedure is a two-stage, post hoc adaptation lasting a few hundred steps:

• Stage 1: Train the $R_h$1 projectors for $R_h$2 to align the projected top-$R_h$3 softmax with the original dense softmax (frozen backbone, $R_h$4600 steps, KL divergence loss).
• Stage 2: End-to-end self-distillation is performed with the sparse attention mechanism activated, distilling only the top-10 logits from the dense teacher to sparse student (again for $R_h$5600 steps).

Approximately 1M input/output label tokens are sufficient, and no backbone weights are modified apart from the token indexer layers. This ensures rapid and cost-effective adaptation (Zhou et al., 16 May 2026).

4. Dynamic Top-p: Mathematical and Empirical Rationale

The active key set for sparse attention in retrieval heads is given by

$R_h$6

where $R_h$7 are softmax-normalized projected similarities and $R_h$8 is the cumulative fraction of attention mass to retain (e.g., $R_h$9).

• Advantage over fixed top-$\mathcal{H}_\mathrm{ret}$0: Dynamic top-$\mathcal{H}_\mathrm{ret}$1 adaptively adjusts the number of attended tokens per query according to the dispersion of attention mass, avoiding under- or over-provisioning that would result from a static $\mathcal{H}_\mathrm{ret}$2. This is particularly effective for mixed workloads containing both “needle” and “diffuse” queries.

Top-$\mathcal{H}_\mathrm{ret}$3 thresholds in the range $\mathcal{H}_\mathrm{ret}$4 provide an optimal trade-off: $\mathcal{H}_\mathrm{ret}$5 recovers >93% true attention mass with $\mathcal{H}_\mathrm{ret}$62k keys on 64k-context, while lower values degrade accuracy (Zhou et al., 16 May 2026).

5. Computational Efficiency and Memory Impact

RTPurbo yields substantial improvements in resource utilization at both the prefill and decode stages:

• Theoretical complexity:
  • Full attention: $\mathcal{H}_\mathrm{ret}$7 per head per query
  • RTPurbo (decode): $\mathcal{H}_\mathrm{ret}$8 for projected indexer plus $\mathcal{H}_\mathrm{ret}$9 for the top-$\mathcal{H}_\mathrm{loc}$0 key set, with $\mathcal{H}_\mathrm{loc}$1
  • Overall runtime scales with the sum of retrieval and local head costs:

  $\mathcal{H}_\mathrm{loc}$2

• Empirical speed and memory results:

  • Prefill (1M-token context): Up to $\mathcal{H}_\mathrm{loc}$3 speedup over FlashAttention-2.
  • Decode (1M-token context): $\mathcal{H}_\mathrm{loc}$4 speedup.
  • KV cache memory: On average, $\mathcal{H}_\mathrm{loc}$5 reduction by discarding remote KVs for local heads.

Accuracy drops are negligible across tested tasks and models (≤0.5% for top-$\mathcal{H}_\mathrm{loc}$6) (Zhou et al., 16 May 2026).

6. Experimental Results and Benchmark Performance

Benchmarks on Qwen3-Coder-30B-A3B and Qwen3-30B-A3B-Think demonstrate near-lossless accuracy retention:

Task	FullAttn (%)	Best Sparse (%)	RTPurbo (%)
LongBench avg	53.80	50.99	54.24
RULER 64k avg	86.23	85.11	85.49
AIME24/25	86.67	46.67–84.4	86.67
MMLU-PRO avg	86.3	81.2	87.1

On multi-hop tasks at 128k–512k context, RTPurbo maintains >90% accuracy at >97% sparsity, whereas baseline systems exhibit significant breakdown (Zhou et al., 16 May 2026).

7. Deployment Considerations and Limitations

Deployment follows these steps:

1. Offline head calibration to partition $\mathcal{H}_\mathrm{loc}$7 and $\mathcal{H}_\mathrm{loc}$8.
2. Injection of 16-dimensional projection indexers for retrieval heads.
3. Implementation of runtime sparse kernels (sliding window for local, blockwise top-$\mathcal{H}_\mathrm{loc}$9 for retrieval).
4. Two-stage adaptation (projector training, self-distillation) with minimal computational overhead.
5. Activation of learned sparse attention at inference.

Limitations include the need for dense prefill for retrieval heads, possible dependence of head specialization on model family or data distribution, and open questions regarding portability to encoder–decoder or vision–language architectures.

RTPurbo thus establishes an efficient, adaptation-minimal pathway for post hoc, high-performance sparse inference in long-context LLMs (Zhou et al., 16 May 2026).