HySparse2: Two-Level KV Sharing for Million-Token Inference
HySparse2 addresses the computational and memory challenges of long-context agentic inference by combining two forms of KV cache sharing. The architecture splits the model into a self-decoder and cross-decoder, allowing prefill computation to terminate early while maintaining strong retrieval quality. By pairing cross-layer KV reuse with token-level sparse selection, HySparse2 reduces prefill FLOPs by up to 5× and shrinks the KV cache to 2.69 GB at one million tokens, while improving long-context retrieval scores by nearly 20 percentage points over prior sparse-attention baselines.Script
Agentic language models spend most of their time re-reading the past. Tool responses, retrieved documents, and conversation history must be prefetched before the model can generate a single new token, and that prefill cost grows with every interaction round.
HySparse2 solves this by dividing the model into two decoders. The self-decoder processes the long context once during prefill, then hands off compact KV caches to the cross-decoder, which handles all subsequent token generation without revisiting the full input.
The architecture uses two levels of sharing. At the outer level, cross-decoder layers project their keys and values from self-decoder hidden states instead of recomputing them. At the inner level, sparse layers reuse the KV cache and token selections from preceding full-attention layers, selecting 1,024 global tokens plus the most recent 128.
Token-level selection is the key to retrieval precision. HySparse used 64-token blocks, but agentic traces contain isolated evidence like role markers or tool outputs surrounded by irrelevant text. By selecting individual tokens, HySparse2 gains over 6 points on RULER version 2 and over 8 points on multi-needle retrieval without increasing the attention budget.
At one million tokens, HySparse2 cuts prefill computation by 5× compared to a hybrid sliding-window baseline and reduces the KV cache to 2.69 gigabytes. The self-decoder exits after just 25 layers during prefill, so a disaggregated serving system can deploy only half the model weights on the prefill node.
HySparse2 demonstrates that prefill and decode can be architecturally separated without sacrificing long-context quality. If you want to dive deeper into two-level KV sharing or explore other architectures for million-token inference, visit EmergentMind.com to create your own video explanations.
