Papers
Topics
Authors
Recent
Search
2000 character limit reached

HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing

Published 22 Sep 2026 in cs.CL | (2609.26368v1)

Abstract: Long-horizon and multi-turn agents typically generate short actions and process long observations from tools and environments. This growing context demands efficient prefill, compact KV-cache storage, and accurate long-context retrieval. To meet these demands, we introduce HySparse2, a hybrid sparse attention architecture with two-level KV sharing. At the outer level, KV Bridging adopts a YOCO-style self-decoder and cross-decoder structure, but bridges only full-attention layers. The self-decoder uses hybrid sliding-window attention (SWA), while the cross-decoder uses hybrid sparse attention. The KV caches for full-attention layers in the cross-decoder are generated from the hidden states of full-attention layers in the self-decoder. At the inner level, HySparse2 retains HySparse's core KV Reuse design with two refinements. First, it replaces block-level sparsity with token-level sparsity for finer long-context retrieval. Second, it removes the separate SWA branch from sparse layers and instead forces a sliding window of recent tokens into the sparse selection. This two-level KV sharing allows all cross-decoder KV caches to be constructed from self-decoder hidden states. Prefill can therefore exit after the self-decoder, skipping all cross-decoder layers. On an 80B-A3B MoE model, HySparse2 outperforms HySparse and Hybrid SWA on long-context retrieval and multi-turn agentic tasks, while substantially reducing prefill computation and KV-cache storage.

Summary

  • The paper introduces HySparse2, a model that uses two-level key-value (KV) sharing to improve long-horizon agentic inference, reducing prefill FLOPs by 2.92 times over HySparse and 5.02 times over Hybrid SWA, while also cutting KV-cache storage by 2.69 GB.
  • Token-level sparse selection in HySparse2 enhances retrieval accuracy by 19.81 percentage points on RULER-v2 and 11.30 percentage points on MRCR-v2 after post-training, demonstrating the effectiveness of finer granularity in selection over block-level selection.
  • HySparse2's use of a forced full-attention local window optimizes for efficiency, reducing the need for extra projection parameters and enabling the cross-decoder to be skipped during prefill, though resulting in a slight performance trade-off.

Problem setting and contribution

Long-horizon agentic inference is dominated by expanding input contexts rather than by generated actions. Tool responses, execution traces, retrieved documents, and prior conversational turns must be repeatedly prefetched before decoding can resume. This creates three coupled systems constraints: prefill computation grows with sequence length, the KV cache scales with the number of layers and cached tokens, and sparse retrieval must preserve evidence distributed across multiple interaction rounds.

“HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing” (2609.26368) addresses these constraints by combining two forms of KV sharing. The architecture extends HySparse (Gao et al., 3 Feb 2026) with a YOCO-style self-decoder/cross-decoder decomposition [yoco] and retains HySparse’s cross-layer reuse of KV representations and sparse-selection indices. Its central claim is that prefill KV-cache construction can terminate after the self-decoder, while the cross-decoder is reserved for autoregressive decoding.

The proposed model differs from its baselines in three principal respects:

  • Outer-level KV Bridging connects full-attention layers in the self-decoder to full-attention layers in the cross-decoder. Cross-decoder keys and values are projected from self-decoder hidden states rather than recomputed from cross-decoder states.
  • Inner-level KV Reuse allows sparse-attention layers to reuse the KV cache and selection indices produced by preceding full-attention layers.
  • Token-level sparse selection with a forced local window replaces HySparse’s block-level selection and separate sliding-window branch in sparse layers.

The resulting design targets a specific operating point: a small number of full-attention indexer layers provide global selection, sliding-window attention supplies local modeling in the self-decoder, and sparse cross-decoder layers retrieve a limited set of global and recent tokens.

Architecture and two-level KV sharing

HySparse2 divides the 49-layer backbone into a self-decoder and a cross-decoder. The self-decoder alternates full attention and sliding-window attention (SWA), whereas the cross-decoder alternates full attention and sparse attention (SA). HySparse2 uses five full-attention layers overall, with only one full-attention layer required during prefill-cache construction in the reported configuration.

At the outer level, KV Bridging pairs self-decoder and cross-decoder full-attention layers. For a cross-decoder full-attention layer, the key and value projections operate on the input hidden states of a corresponding self-decoder full-attention layer. The cross-decoder layer retains its own query projection, so the shared source hidden state does not imply identical attention behavior or identical KV tensors across layers. Each cross-decoder layer still constructs a distinct cache through its own layer-specific K/V projections.

This distinction is important. HySparse2 does not simply copy one KV cache across all cross-decoder layers; it shares the hidden-state source while preserving layer-specific representations. The method therefore attempts to obtain the computational benefit of cross-layer sharing without forcing all layers to use a common key and value basis.

At the inner level, each full-attention layer acts as both an exact-attention computation and an indexer for subsequent sparse layers. The top-scoring tokens are selected from the full-attention scores, and the following sparse layers reuse the corresponding entries of the full-attention KV cache. This removes the need for a separately trained indexer or an auxiliary distillation objective.

Figure 1

Figure 1: Two-level KV sharing connects self-decoder and cross-decoder full-attention layers while sparse layers reuse full-attention KV entries and token-selection indices.

The local-attention modification is structurally consequential. HySparse uses a separate SWA branch inside sparse layers, combining local and selected global attention through gated fusion. HySparse2 instead forces the most recent 128 tokens into the sparse selection and fills the remaining budget with the highest-scoring global tokens. Local and global tokens consequently use one shared KV cache.

The paper argues that this removes a cascading dependency. A separate cross-decoder SWA branch would require hidden states generated within the cross-decoder, preventing complete early exit during prefill. By forcing recent tokens into the sparse set, HySparse2 eliminates the separate branch and makes all cross-decoder KV caches derivable from self-decoder states. Under prefill–decode disaggregation, the prefill node therefore hosts the self-decoder and bridging projections, while decode nodes host the complete model.

Token-level sparsity and retrieval precision

HySparse2 replaces HySparse’s 64-token block selection with token-level selection. Both mechanisms select 1,024 global tokens in the reported model, but token-level selection can distribute that budget across arbitrary positions. The difference matters in multi-turn trajectories, where relevant evidence may consist of isolated role delimiters, identifiers, tool outputs, or answer-bearing tokens separated by irrelevant neighboring positions.

The ablation holds the backbone, global selection budget, and 128-token local window fixed. Token-level selection improves long-context metrics despite only modest changes on general reasoning benchmarks:

Metric Block-level Token-level Difference
RULER-v2 49.56 56.13 +6.57
MRCR-v2, two needles 12.94 21.08 +8.14
GraphWalks 29.38 34.92 +5.55
MMLU-Pro 35.74 36.97 +1.23
NoLiMa 40.27 38.43 -1.84

These results support the paper’s claim that block granularity creates an accuracy penalty for agentic retrieval under a fixed attention budget. The improvement is not attributable to a larger selected-token budget; it follows from allocating the same budget at finer granularity. The result also qualifies the efficiency argument for block sparsity: regular blocks may simplify kernels, but they can waste capacity on neighboring tokens that are not themselves relevant.

The paper relies on recent sparse-kernel implementations to make token-level selection practical, including TileLang-based kernels (Wang et al., 24 Apr 2025). Consequently, the reported quality advantage is not a purely architectural result independent of systems support; its end-to-end value depends on whether token-level kernels achieve favorable throughput and memory behavior on the target hardware.

Local attention and the forced window

The local-window ablation compares a gated SWA branch, no local branch, and the forced-window mechanism. Removing local attention generally degrades performance, confirming that sparse global retrieval alone is insufficient for modeling short-range dependencies. Forced SWA remains competitive with the separate gated branch:

Metric Gated SWA No SWA Forced SWA
RULER 88.19 84.55 89.84
RULER-v2 53.66 54.62 55.98
MRCR-v2 27.66 20.73 22.67
GraphWalks 35.39 36.48 37.13
LongPPL 6.8807 7.1307 6.9838

Forced SWA obtains the best RULER, RULER-v2, and GraphWalks scores, but it is not uniformly superior. Relative to the gated branch, it loses 5.08 points on GSM8K and 4.99 points on MRCR-v2, while LongPPL is 1.50% higher. The paper therefore characterizes the forced-window design as an efficiency–quality trade-off rather than as a universal improvement.

Its principal advantage is systems-level: it removes extra projection parameters, avoids a separate local KV cache, and enables the cross-decoder to be skipped during prefill. The empirical evidence supports this trade-off for the evaluated workloads, but the degradation on MRCR-v2 indicates that forced recency is not equivalent to a learned local-attention branch for every retrieval distribution.

Long-context quality

The models are 80B-A3B MoE systems with 49 Transformer layers and hidden size 2,048. HySparse2 uses five full-attention layers and MQA, while HySparse uses five full-attention layers and GQA; Hybrid SWA uses nine full-attention layers. The three models are trained with the same data and schedules within each training stage, although the attention configurations differ in head structure and sparse mechanism.

After approximately 500B pretraining tokens at 32k context, HySparse2 is broadly comparable to the baselines on general capabilities and strongest on the reported long-context metrics. It reaches RULER 90.77 and NoLiMa 49.76, compared with 84.89 and 40.27 for HySparse. Its Repo Code PPL is also marginally lower at 1.1570, versus 1.1588 for HySparse and 1.1578 for Hybrid SWA.

The general-capability results are mixed rather than uniformly improved. HySparse2 leads HySparse on BBH and MMLU-Pro but trails it on DROP, GSM8K, ARC-C, and WinoGrande. This matters because the architecture’s efficiency and long-context improvements do not establish dominance across all task families.

After light post-training with approximately 100B additional tokens and context extension to 256k, HySparse2 exhibits its largest advantages on agentic retrieval. Its mean MRCR-v2 score exceeds HySparse by 11.30 percentage points and Hybrid SWA by 6.44 points. Its mean RULER-v2 score exceeds HySparse by 19.81 points and Hybrid SWA by 18.65 points. At 256k tokens, HySparse2 reaches 58.45 on RULER-v2, compared with 32.61 for HySparse and 35.74 for Hybrid SWA.

Figure 2

Figure 2: After light post-training, HySparse2 maintains higher retrieval scores and lower AgentPPL and LongPPL across the evaluated context lengths.

HySparse2 also achieves lower AgentPPL and LongPPL across the evaluated lengths. The paper distinguishes the trends: AgentPPL increases with context length because additional turns and tool outputs introduce retrieval interference, whereas LongPPL decreases because its selected tokens benefit from the availability of more long-range evidence. This distinction suggests that perplexity on long-context examples is sensitive to benchmark construction and should not be interpreted as a single measure of retrieval competence.

Prefill computation and KV-cache storage

The main systems result is obtained at one million tokens. With FP8 KV-cache storage, HySparse2 reduces prefill FLOPs by 2.92×2.92\times relative to HySparse and by 5.02×5.02\times relative to Hybrid SWA. Its KV cache occupies 2.69 GB, compared with 6.72 GB for HySparse and 12.09 GB for Hybrid SWA.

Model Prefill FLOPs at 1M tokens, relative scale KV cache
Hybrid SWA 5.02×5.02\times HySparse2 12.09 GB
HySparse 2.92×2.92\times HySparse2 6.72 GB
HySparse2 1.00 2.69 GB

Figure 3

Figure 3: HySparse2 lowers both prefill computation and FP8 KV-cache storage as context length increases.

The reduction comes from several interacting mechanisms rather than from sparsity alone. The self-decoder produces the source states required for all bridged KV caches; cross-decoder layers do not need to process the entire prefixed sequence during prefill. Sparse layers attend only to selected tokens, while MQA reduces KV-head storage relative to the GQA baselines. The forced local window further ensures that local attention does not require a separate cache.

The paper also reports an operational consequence under prefill–decode disaggregation: for the 49-layer configuration, the prefill node deploys only the first 25 layers plus bridging projections, reducing its model-weight memory requirement by nearly half. The projected cross-decoder KV caches are transferred to decode nodes because they are smaller than the source hidden states. This benefit assumes that the system can tolerate the communication and scheduling overhead of cache transfer; the paper reports cache size and FLOPs but does not provide a complete distributed-serving latency or bandwidth analysis.

KV Bridging quality and comparison with KV Mirror

KV Bridging is evaluated at a larger 290B-A8B scale using approximately 1.8T pretraining tokens. Relative to a model without bridging, the bridged model has broadly comparable results: MMLU increases from 72.68 to 72.80, TriviaQA from 73.32 to 74.10, and LongPPL improves from 3.6053 to 3.4202. RULER changes by only 0.31 points, while BBH, GSM8K, and especially DROP decline. DROP decreases from 71.37 to 68.17, so the claim that bridging preserves quality must be interpreted as an aggregate characterization rather than as task-wise invariance.

The paper further compares KV Bridging with KV Mirror (Liu et al., 9 Jul 2026). KV Mirror uses U-shaped early-to-late layer connections, whereas KV Bridging connects corresponding full-attention layers. Under the reported 80B-A3B setup, KV Bridging reaches a final RULER score of 87.65, compared with 81.28 for KV Mirror.

Figure 4

Figure 4: KV Bridging produces higher RULER scores than KV Mirror during pretraining.

The proposed explanation is that full-attention source states receive denser global supervision through attention scores, while SWA source states directly connect only to local tokens. This interpretation is plausible, but the experiment does not isolate all possible differences between the connection schemes, and the paper presents the explanation as a hypothesis rather than a demonstrated causal mechanism.

Limitations and open questions

The evaluation establishes strong results for the specified 80B-A3B and 290B-A8B configurations, but several assumptions constrain interpretation. First, the comparisons use different attention layouts and KV-head configurations: HySparse2 uses MQA with 256-dimensional heads, whereas the baselines use GQA with 192/128-dimensional query/key-value head dimensions. Therefore, the reported cache and kernel advantages cannot be attributed exclusively to two-level KV sharing.

Second, the principal long-context comparison follows light post-training with agentic data. The results demonstrate the combined effect of architecture and training, but they do not fully separate token-level sparsity, KV Bridging, MQA, context extension, and post-training mixture effects. The ablations address some components, although they are primarily reported after pretraining and at shorter context lengths.

Third, the architecture retains full-attention layers because they provide exact token selection and are considered important for quality. At least one full-attention layer remains necessary during prefill in the reported design. The paper suggests replacing such layers with lightweight indexers, but does not evaluate that alternative. It also does not report end-to-end serving latency, accelerator utilization, kernel-level throughput, communication cost under disaggregation, or the impact of dynamic sequence lengths.

Finally, token-level sparsity improves retrieval under a fixed budget but may impose implementation complexity and irregular memory access. The reported FLOP reductions therefore do not by themselves establish proportional wall-clock speedups. The open systems question is whether token-level kernels and cache-transfer mechanisms preserve the theoretical advantage under production serving conditions at million-token contexts.

Conclusion

HySparse2 combines YOCO-style decoder separation with HySparse-style cross-layer KV reuse. KV Bridging allows cross-decoder KV caches to be constructed from self-decoder states, while token-level selection and a forced recent-token window improve retrieval precision and remove the cross-decoder’s prefill dependency. On the reported 80B-A3B models, the architecture improves long-context retrieval substantially, reduces one-million-token prefill FLOPs by 2.92×2.92\times relative to HySparse and 5.02×5.02\times relative to Hybrid SWA, and reduces KV-cache storage to 2.69 GB. Its quality is not uniformly superior on general reasoning tasks, and its systems advantages depend on token-level kernel efficiency and disaggregated-serving costs. Within those qualifications, the paper presents two-level KV sharing as a coherent architectural mechanism for simultaneously reducing prefill work, cache storage, and long-context retrieval loss.

Whiteboard

Explain it Like I'm 14

1. What is this paper about?

This paper introduces HySparse2, a new design for LLMs that helps them work with very long conversations and documents more quickly and cheaply.

This is especially useful for AI agents. An AI agent might:

  1. Read a user request.
  2. Use a tool, such as a search engine or calculator.
  3. Receive a long result.
  4. Remember earlier steps and continue working.

As the conversation gets longer, the AI has more information to read and remember. This makes it slower and requires more computer memory. HySparse2 tries to solve this problem while still helping the model find important information accurately.

2. What questions did the researchers study?

The researchers wanted to know whether they could build an AI model that:

  • Handles very long contexts, sometimes up to one million tokens.
  • Finds important information hidden inside long conversations.
  • Uses less computing power before generating an answer.
  • Stores less information in memory.
  • Performs well on multi-step tasks involving tools and many conversation turns.
  • Keeps normal language, reasoning, and coding abilities.

In simple terms, they asked:

Can an AI read less of a huge conversation, remember it more efficiently, and still find the right details?

3. How does HySparse2 work?

Attention: deciding what to look at

LLMs use a mechanism called attention. It helps the model decide which earlier words are important when understanding or generating a new word.

Imagine reading a very long book while answering a question. You probably would not reread every page equally. Instead, you would:

  • Pay close attention to the most relevant pages.
  • Keep the latest pages nearby because they may contain useful details.
  • Occasionally check the whole book when necessary.

HySparse2 uses a similar strategy.

Sparse attention

Normal attention looks at almost every previous token, or small piece of text. This can be expensive for long inputs.

Sparse attention looks at only a selected part of the text. HySparse2 chooses:

  • The most relevant individual tokens from the distant past.
  • A fixed number of recent tokens, called a local window.

This is like using bookmarks to jump to important pages while also keeping the most recently read pages open.

The paper improves on an earlier method that selected whole blocks of tokens. HySparse2 selects individual tokens instead. This is more precise: if one word in a block is useful, the model does not have to spend memory on all the neighboring words.

Two kinds of attention

HySparse2 uses two main types of attention:

  • Full attention: The model can examine the entire available context. This is accurate but expensive.
  • Sparse attention: The model examines only selected tokens. This is faster and uses less memory.

Only a small number of layers use full attention. These layers also help decide which tokens are important for the sparse layers.

The KV cache

While processing text, a LLM stores information about earlier tokens in a memory structure called the KV cache. “KV” means key and value, two pieces of information used by attention.

The KV cache is like a set of notes the model keeps while reading. For a very long conversation, these notes can become extremely large.

HySparse2 uses two-level KV sharing:

  1. KV Bridging: Some information created in the first part of the model is reused by later parts.
  2. KV Reuse: Nearby layers reuse the same selected information instead of creating a completely new copy.

Because of this sharing, the model can prepare much of its memory after processing only the first half of the model. The paper calls this an early exit during prefill.

Prefill and decoding

When an AI receives a long input, it first processes the whole input. This is called prefill. It then generates the answer one piece at a time, called decoding.

HySparse2 makes prefill faster by allowing it to stop after the self-decoder, which is roughly the first half of the model. The remaining layers do not need to process the entire long input during this stage.

4. How did the researchers test it?

The researchers built and compared three types of models:

Model Main idea
Hybrid SWA Uses full attention and a recent-token window
HySparse Uses full attention plus block-based sparse attention
HySparse2 Uses token-based sparse attention and two-level KV sharing

They trained models with about 80 billion total parameters, using a type of model called a Mixture of Experts, or MoE. An MoE model is like a large team of specialists: only some parts of the model work on each input, which can save computation.

The models were tested on:

  • General knowledge
  • Mathematics and reasoning
  • Coding
  • Long-document understanding
  • Finding information in long conversations
  • Multi-turn AI-agent tasks
  • Computing cost
  • Memory used by the KV cache

The researchers also performed smaller experiments called ablation studies. In an ablation study, researchers remove or change one feature at a time to see how much that feature matters.

5. What did they find?

Better long-context retrieval

HySparse2 was especially good at finding important information in long contexts.

After additional training for agent tasks, HySparse2 improved over HySparse by:

  • 11.30 percentage points on the MRCR-v2 retrieval test.
  • 19.81 percentage points on the RULER-v2 retrieval test.

At a context length of 256,000 tokens, HySparse2 scored 58.45 on RULER-v2, compared with:

  • 32.61 for HySparse
  • 35.74 for Hybrid SWA

This suggests that HySparse2 is better at locating useful details spread across many turns of a conversation.

Better long-context language understanding

HySparse2 also had lower scores on AgentPPL and LongPPL. These measures estimate how surprised the model is by the correct next text; lower is better.

This means HySparse2 was generally better at predicting text that depended on information from far earlier in the conversation.

Much less computation

At one million tokens, HySparse2 used:

  • 2.92 times fewer prefill operations than HySparse
  • 5.02 times fewer prefill operations than Hybrid SWA

In other words, it required much less mathematical work to prepare a long input.

Smaller memory requirements

At one million tokens, the KV cache required approximately:

Model KV-cache size
Hybrid SWA 12.09 GB
HySparse 6.72 GB
HySparse2 2.69 GB

This is important because AI systems need enough computer memory to store their caches. A smaller cache can reduce hardware costs and make long-context models easier to run.

Selecting individual tokens helped

The researchers compared selecting whole blocks of tokens with selecting individual tokens.

Token-level selection improved several long-context tasks:

  • RULER-v2 increased by 6.57 points.
  • Two-needle MRCR-v2 increased by 8.14 points.
  • GraphWalks increased by 5.55 points.

This supports the idea that choosing exact tokens is better than selecting large chunks that may contain lots of irrelevant information.

Recent tokens still matter

The experiments also showed that the model needs access to recent information. When the researchers forced the most recent 128 tokens into the selected set, performance was generally better than when the model ignored local context.

This makes sense because the latest part of a conversation often contains the current question, instructions, or tool result.

KV Bridging mostly preserved quality

KV Bridging made prefill cheaper without causing a large overall quality loss. Most test results stayed similar, although some reasoning tasks became slightly worse.

The researchers therefore concluded that KV Bridging is a useful trade-off: it greatly reduces computation while keeping the model’s abilities broadly intact.

6. Why are these results important?

Long-context AI is becoming increasingly useful, but long inputs are expensive. An AI agent may need to remember:

  • Earlier user instructions
  • Search results
  • Computer actions
  • Tool outputs
  • Previous decisions
  • Documents and code

If the model examines everything in the same detailed way, it becomes slow and needs a large amount of memory.

HySparse2 offers a way to make this process more efficient:

  • It looks carefully at only a small number of important tokens.
  • It keeps recent information available.
  • It shares stored information between layers.
  • It can prepare the long input using only about half of the model.

7. Possible impact and limitations

If the results hold up in real-world systems, HySparse2 could help create AI agents that are:

  • Faster when reading long histories
  • Cheaper to operate
  • Able to remember more information
  • Better at finding details from earlier conversations
  • Easier to run on hardware with limited memory

For example, an AI assistant working on a long research project might be able to remember months of notes and tool results without becoming too slow.

However, the method is not perfect. Some ordinary reasoning and mathematics scores were slightly lower than those of competing systems. The experiments were also performed on particular model sizes, training data, and benchmarks, so further testing would be needed.

Simple conclusion

HySparse2 is a method for helping LLMs handle extremely long conversations more efficiently. It combines careful selection of important words, attention to recent information, and sharing of memory between model layers.

The main result is that HySparse2 used much less computation and memory while becoming better at finding information in long, complicated contexts. This could make future AI agents more practical for tasks that involve many conversation turns, documents, searches, and tool calls.

View Paper Prompt  View All Prompts 

Knowledge Gaps

The paper leaves the following knowledge gaps, limitations, and open questions unresolved:

  • Hardware-level speedups are not demonstrated. The reported efficiency gains are based primarily on FLOP counts and estimated KV-cache sizes; end-to-end latency, throughput, kernel utilization, memory bandwidth, communication overhead, and energy consumption on real inference systems are not measured.
  • Prefill–decode disaggregation is not evaluated in practice. The paper proposes deploying only the self-decoder on the prefill node and transferring projected KV caches, but does not quantify transfer latency, network bandwidth requirements, synchronization costs, or performance under concurrent requests.
  • The comparison is confounded by architectural differences. HySparse2 uses MQA and different head dimensions, whereas HySparse and Hybrid SWA use GQA and different query/key-value dimensions. Consequently, the observed gains cannot be attributed solely to two-level KV sharing or token-level sparsity.
  • The contribution of each architectural change is not fully isolated. HySparse2 simultaneously changes decoder structure, KV Bridging, selection granularity, local-window handling, MQA/GQA configuration, and attention projections. The experiments do not provide a complete factorial ablation separating these effects.
  • The benefit of token-level sparsity under equal practical latency is unknown. The ablation matches the nominal number of selected tokens but does not establish whether token-level selection remains superior when block-level and token-level implementations are compared at equal wall-clock latency, memory traffic, or kernel efficiency.
  • The efficiency and quality trade-off across sparsity budgets is unexplored. Only a fixed configuration of 128 forced local tokens and 1,024 global tokens is reported. The paper does not characterize performance as a function of the local-window size, global-token budget, number of full-attention layers, or sparse-layer placement.
  • The sparse-selection mechanism is evaluated using oracle full-attention scores, but deployment costs are unclear. The retained full-attention layers act as exact indexers during inference, yet the paper does not quantify how much of the claimed efficiency remains when selection-score computation, top-kk selection, token gathering, and dynamic memory access are included.
  • The feasibility of replacing full-attention indexers with lightweight learned indexers is unresolved. The paper suggests this as future work but does not measure the resulting retrieval degradation, training stability, calibration, or efficiency gains.
  • The mechanism underlying KV Bridging remains insufficiently explained. The paper hypothesizes that full-attention hidden states provide better globally informed representations, but does not analyze representation similarity, information loss, attention-score distortion, or why particular source-to-target layer mappings work best.
  • Alternative KV Bridging mappings are not systematically explored. The evaluation compares the proposed mapping with KV Mirror, but does not test learned mappings, many-to-one mappings, one-to-one versus many-to-many connections, different source-layer depths, or adaptive layer assignment.
  • KV Bridging is not evaluated at the largest reported context lengths. Its quality ablation is conducted at a 32k training context, leaving uncertainty about whether bridged KV representations remain reliable at 256k or 1M tokens.
  • The model’s quality outside the selected benchmark suite is uncertain. Evaluation focuses mainly on standard academic benchmarks, synthetic retrieval tests, internal perplexity sets, and agent trajectories. Robustness on real production tools, web browsing, code execution, planning, dialogue, multimodal observations, and adversarial tool outputs is not established.
  • The representativeness of the agentic datasets is unclear. AgentPPL uses 1,000 internal trajectories and LongPPL uses 349 examples, but the paper does not describe their domains, duplication controls, difficulty distribution, contamination checks, or coverage of different agent architectures and tool-use patterns.
  • Task-level agent performance is not directly measured. Lower AgentPPL and higher retrieval scores do not establish improvements in completed task success, tool-call accuracy, planning quality, execution reliability, or recovery from tool errors in end-to-end agents.
  • Robustness to adversarial or misleading context is unexplored. The paper does not test whether sparse selection fails when irrelevant tokens are designed to attract attention, when evidence is distributed across many turns, or when distractors resemble the relevant content.
  • Failure cases in multi-hop retrieval are not analyzed. GraphWalks and related benchmarks provide aggregate scores, but the paper does not identify whether errors arise from missed intermediate nodes, incorrect local-window selection, layer-level information loss, or KV Bridging distortions.
  • Long-context extrapolation beyond the 256k evaluation range is unverified. Although costs are projected to 1M tokens, quality results are not reported at 512k, 1M, or longer contexts, so the claimed suitability for million-token inference remains primarily computational rather than empirical.
  • Training-context and inference-context mismatch is not fully characterized. Models are pretrained at 32k and post-trained at 256k, but the effects of training directly at longer contexts, curriculum design, and context-length distribution on sparse retrieval quality are not studied.
  • The impact of token-level dynamic sparsity on batching and serving is unknown. Variable token selection can complicate batching, memory layouts, caching, and accelerator scheduling; these system-level effects are not evaluated.
  • The interaction with KV-cache quantization is unexplored. All reported cache-size comparisons use FP8, while the paper does not test whether HySparse2 is compatible with 2-bit, 4-bit, mixed-precision, or per-token KV quantization and whether quantization disproportionately harms sparse retrieval.
  • The interaction with other cache-compression methods is not established. The paper discusses head-, sequence-, precision-, and layer-level compression, but does not evaluate combinations with latent KV representations, token merging, cache eviction, or cross-layer attention beyond its own design.
  • Decode-time memory and compute costs are incompletely reported. The paper emphasizes prefill FLOPs and KV storage, but provides limited analysis of per-token decode latency, query-dependent sparse-access overhead, cache read amplification, and performance as generation length increases.
  • The effect of multi-turn cache updates is not quantified. The architecture is motivated by accumulating agent histories, yet the paper does not measure incremental append cost, cache rebuild frequency, behavior after very many turns, or the effect of selectively truncating or modifying previous tool outputs.
  • The forced recent-token window may be unsuitable for long-range local dependencies. Always retaining the most recent 128 tokens may over-prioritize the current suffix and miss locally relevant information from earlier turns; adaptive or query-dependent local windows are not investigated.
  • The fixed local-window policy is not tested across different interaction patterns. Its effectiveness for short dialogue turns, long tool outputs, code traces, structured documents, and rapidly alternating tool calls remains unknown.
  • The quality cost of removing the separate SWA branch is not fully understood. The ablation shows notable degradation on some tasks, including GSM8K and MRCR-v2, but the paper does not investigate which capabilities depend on the removed branch or whether a cheaper gated/local mechanism could recover them.
  • Statistical reliability is not reported. Results generally appear as single scores without confidence intervals, multiple random seeds, significance tests, or variance across training runs, making it difficult to assess whether small differences are reproducible.
  • The scaling behavior of the architecture is only partially tested. Most experiments use an 80B-A3B model, with one KV Bridging ablation at 290B-A8B. The effects of model size, expert count, dense versus MoE architectures, and different layer widths remain uncertain.
  • Generalization across languages and domains is limited. Although Chinese and English benchmarks are included, the paper does not evaluate multilingual long-context retrieval systematically or determine whether token-level selection and KV Bridging behave differently across scripts, tokenization schemes, and low-resource languages.
  • The effect of positional encoding choices is not isolated. HySparse2 uses partial RoPE for SWA layers and NoPE for full and sparse attention, but the paper does not compare alternative positional encodings or determine whether this design contributes to long-context performance independently of sparsity.
  • Training stability and optimization dynamics are underreported. The paper does not provide loss curves, convergence comparisons, gradient analyses, or sensitivity to learning rates and initialization for the two-level KV-sharing architecture.
  • The MTP and speculative-decoding claims remain prospective. The paper mentions replacing the MTP layer with a larger drafter and enabling asynchronous speculative decoding, but provides no experiments showing acceptance rates, speedups, quality effects, or compatibility with sparse attention.
  • The impact of KV Bridging on calibration and uncertainty is unknown. Aggregate accuracy and perplexity do not reveal whether bridged caches alter confidence, likelihood calibration, abstention behavior, or reliability under distribution shift.
  • Parameter and training-cost overheads are not comprehensively accounted for. The paper discusses reduced inference cost and some eliminated projections but does not report total parameter counts, training FLOPs, activation memory, additional projection costs, or the break-even point between training overhead and serving savings.
  • Reproducibility is constrained by missing implementation details. The paper does not fully specify sparse-kernel implementations, top-kk selection algorithms, layer pairings, routing details, hardware configurations, batching policies, or exact dataset composition, limiting independent verification of the reported results.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

The paper’s main practical contribution is an inference architecture for long-context LLMs that improves retrieval quality while reducing prefill computation and KV-cache memory. The following uses are plausible with an already trained or retrained HySparse2-compatible model and suitable inference infrastructure.

  • Long-horizon software engineering agents — Software development
    • Deploy HySparse2 in coding agents that repeatedly inspect repositories, issue tool calls, execute tests, and revise patches.
    • Token-level sparse selection can preserve individual relevant lines across files instead of retaining entire fixed-size blocks, while the forced recent-token window keeps the latest compiler errors, test results, and user instructions available.
    • Potential products include repository-scale code assistants, autonomous debugging systems, and pull-request agents that maintain context across many tool interactions.
    • Dependencies: The model must be trained or post-trained for code and agent trajectories; token-level sparse-attention kernels must be efficiently implemented; retrieval quality should be validated on proprietary repositories rather than assumed from RULER or MRCR-v2 scores.
  • Multi-turn customer-support and enterprise-service agents — Business software
    • Use the architecture for agents that accumulate long histories of user messages, CRM records, search results, policy documents, and tool outputs.
    • Lower KV-cache storage can support more simultaneous sessions, while stronger long-context retrieval can reduce failures caused by overlooking an earlier customer constraint or policy detail.
    • A practical workflow is: run the self-decoder during request prefill, construct bridged caches, and send the smaller cache to decode servers for interactive response generation.
    • Dependencies: Sensitive data still requires access control, encryption, retention policies, and auditability. Sparse retrieval does not guarantee factuality or compliance, so high-risk responses need verification and escalation.
  • High-throughput document question answering — Legal, finance, healthcare, and research
    • Deploy HySparse2 for querying large collections or individual long documents such as contracts, litigation records, clinical guidelines, financial filings, technical manuals, or scientific papers.
    • The improved token-level selection is particularly useful when relevant evidence is scattered across distant sections and does not align with fixed blocks.
    • Potential tools include long-document review assistants, evidence-finding interfaces, and systems that answer questions while exposing the selected passages for human inspection.
    • Dependencies: The reported gains concern retrieval-oriented benchmarks and agent trajectories, not all forms of document reasoning. Production systems should retain citations, perform independent retrieval, and evaluate recall for domain-specific terminology and tables.
  • Cost reduction for hosted LLM APIs — Cloud infrastructure
    • Integrate two-level KV sharing into inference servers to reduce prefill FLOPs and cache memory for long prompts.
    • The paper reports, at one million tokens, approximately 2.92×2.92\times lower prefill FLOPs than HySparse and 5.02×5.02\times lower than Hybrid SWA, with a reported FP8 KV cache of 2.69 GB versus 6.72 GB and 12.09 GB, respectively.
    • Providers could use the savings to increase session concurrency, reduce GPU memory requirements, or offer longer context windows at similar prices.
    • Dependencies: These measurements are for an 80B-A3B MoE configuration and may not transfer directly to other model sizes, hardware, batch sizes, precisions, or serving stacks. Kernel overhead and data-transfer costs must be benchmarked end to end.
  • Prefill–decode disaggregated serving — Data-center systems
    • Operate a relatively small prefill cluster containing the self-decoder and KV-bridging projections, while decode nodes run the full architecture.
    • Because prefill can exit after the self-decoder, the paper estimates that a 49-layer configuration needs only roughly the first 25 layers on the prefill node, reducing its model-memory requirement by nearly half.
    • This can support workloads with large retrieved inputs but comparatively short generated answers, such as search-augmented assistants and tool-using agents.
    • Dependencies: Network bandwidth and latency must be sufficient to transfer the projected KV cache. Scheduling, cache ownership, fault recovery, and compatibility between prefill and decode implementations require production engineering.
  • Long-context retrieval and reasoning evaluation — Academia and industrial model testing
    • Use the architecture and its design principles as a controlled testbed for studying token-level versus block-level sparsity, local-window policies, KV sharing, and prefill optimization.
    • The paper provides actionable evaluation dimensions: multi-needle retrieval, graph traversal, long-context perplexity, agent trajectories, prefill FLOPs, and KV-cache size.
    • Research teams can reproduce ablations to determine whether an efficiency optimization harms retrieval, mathematical reasoning, code understanding, or general capabilities.
    • Dependencies: Comparisons should control training data, context length, parameter count, attention budget, and quantization. Benchmark improvements do not by themselves establish reliability in real deployments.
  • Interactive personal assistants — Daily life
    • A local or cloud assistant could maintain longer histories of conversations, schedules, messages, travel plans, household tasks, and tool results with lower memory pressure.
    • The forced recent window helps preserve the current conversation, while global sparse selection can retrieve older preferences or commitments.
    • Lower KV-cache requirements could make persistent sessions more feasible on consumer GPUs or edge devices.
    • Dependencies: The paper does not demonstrate on-device deployment, small-model performance, privacy protections, or energy consumption. Personal assistants should allow users to inspect, delete, and correct stored context.
  • Real-time interactive education and tutoring — Education
    • Apply the model to tutoring sessions that combine prior dialogue, worked examples, uploaded course materials, quizzes, and teacher feedback.
    • Efficient long-context retrieval can help the system connect a current question to earlier misconceptions or relevant sections of a textbook.
    • Potential workflows include persistent study assistants and teacher-facing summaries of long student interactions.
    • Dependencies: Educational deployment requires age-appropriate safeguards, pedagogical validation, privacy protections, and mechanisms to prevent confident but incorrect explanations. The reported general-capability results are mixed, so domain-specific evaluation is necessary.

Long-Term Applications

These applications depend on additional research, larger-scale validation, specialized training, or reliable integration with external systems.

  • Million-token autonomous research and operations agents — Cross-sector AI systems
    • Extend HySparse2-style agents to maintain months of interaction history, large evidence collections, execution traces, and evolving plans.
    • Applications could include scientific literature synthesis, enterprise operations, incident management, and complex procurement or logistics workflows.
    • Two-level KV sharing and token-level retrieval could make persistent, multi-stage reasoning more affordable than dense full attention.
    • Dependencies: Performance at one million tokens is reported primarily through cost analysis, not a complete end-to-end autonomous-agent evaluation. Long-term memory management, evidence provenance, temporal updates, contradiction handling, and catastrophic retrieval failures require further study.
  • Clinical decision-support agents — Healthcare
    • A future system could integrate longitudinal patient records, laboratory results, imaging reports, medication history, clinical guidelines, and clinician conversations while retrieving only the most relevant tokens at each reasoning stage.
    • The reduced cache footprint could make long patient histories more practical in hospital-scale systems.
    • Dependencies: Clinical deployment requires prospective trials, calibrated uncertainty, robust retrieval of safety-critical contraindications, privacy compliance, interoperability with electronic health-record systems, and human approval. The paper does not establish medical accuracy or safety.
  • Regulatory, legal, and financial compliance monitoring — Policy and regulated industry
    • Build agents that compare current transactions, contracts, communications, or policies against large and frequently changing regulatory corpora.
    • Sparse attention could lower the cost of monitoring long audit trails while token-level selection preserves precise clauses, dates, exceptions, and transaction details.
    • Potential products include continuous compliance review, automated case preparation, and regulator-facing evidence packages.
    • Dependencies: Every selected-token mechanism must be auditable and reproducible. Approximate retrieval can omit a legally decisive passage, so systems need dense-retrieval fallbacks, deterministic logging, human review, and jurisdiction-specific validation.
  • Embodied robotics with persistent task memory — Robotics
    • Integrate the architecture into robots that accumulate natural-language instructions, sensor summaries, navigation history, demonstrations, and tool or actuator outcomes over extended tasks.
    • Recent-token forcing can preserve the current control context, while sparse global retrieval can recover earlier instructions or environmental facts.
    • Potential systems include household robots, warehouse agents, and maintenance robots operating across multiple shifts.
    • Dependencies: The paper addresses language-model attention rather than real-time control, multimodal tokens, safety guarantees, or latency deadlines. Further work must test worst-case retrieval latency, sensor grounding, action verification, and behavior under missing or misleading observations.
  • Energy-efficient and carbon-aware LLM serving — Energy and data centers
    • Combine reduced prefill computation with low-precision KV storage and workload-aware scheduling to reduce energy per long-context request.
    • Providers could allocate HySparse2-style models to input-dominated workloads and dense models to tasks where exact global attention is more important.
    • Dependencies: FLOP reductions do not automatically imply proportional energy or carbon reductions; memory movement, communication, GPU utilization, and cooling may dominate. Direct power measurements across hardware are required.
  • Adaptive attention systems with learned compute allocation — Model architecture research
    • Extend the architecture so that the model dynamically chooses the number of full-attention layers, local-window size, or global tokens based on query difficulty and context structure.
    • Lightweight indexers could eventually replace some retained full-attention indexer layers, as suggested in the paper, further reducing prefill cost.
    • Potential outcomes include task-adaptive LLM serving and quality-of-service modes that trade latency for retrieval accuracy.
    • Dependencies: Learned indexers must avoid systematic retrieval blind spots, preserve end-to-end training stability, and provide predictable worst-case costs. The paper explicitly treats reducing full attention as future work.
  • Asynchronous speculative decoding for long-context agents — Inference acceleration
    • Replace the current single MTP layer with a larger drafter conditioned on hidden states available at the self-decoder/cross-decoder boundary.
    • A separate drafting process could generate candidate responses while the full model verifies them, reducing interactive latency after expensive long-context prefill.
    • Dependencies: The paper identifies this as future work. Benefits depend on draft acceptance rates, synchronization overhead, verification cost, and whether speculative errors are concentrated in tool calls or safety-critical actions.
  • Long-context foundation models for public-sector knowledge services — Policy and public information
    • Governments and public institutions could use such models to search large archives, legislation, administrative records, and public consultations while lowering infrastructure costs.
    • Systems could support policy analysts, benefits navigation, archival discovery, and public-facing question answering.
    • Dependencies: Public-sector use requires transparent citations, accessibility, multilingual and jurisdiction-specific evaluation, records-management compliance, and protections against exclusion caused by sparse retrieval errors. The paper’s Chinese and English benchmark coverage does not establish broad multilingual robustness.
  • Personalized lifelong assistants with persistent memory — Daily life
    • A mature system could maintain years of user-approved preferences, projects, health routines, correspondence, and household history while selectively retrieving relevant events.
    • KV-cache sharing could make frequent multi-session interaction less costly than repeatedly recomputing complete histories.
    • Dependencies: This requires durable memory storage beyond a transient KV cache, explicit consent and deletion controls, secure personalization, identity management, and robust defenses against prompt injection through old documents or tool outputs.
View Paper Prompt  View All Prompts 

Glossary

  • Agentic inference: Model inference involving autonomous or semi-autonomous agents that interact with tools, environments, and multi-turn contexts. “Agentic inference combines long contexts with multi-turn interaction.”
  • Attention sink: A token or position that attracts disproportionate attention and can affect attention distributions. “learnable per-head sink biases”
  • Autoregressive decoding: Generating a sequence one token at a time, conditioning each token on previously generated tokens. “Prefill–decode disaggregation”
  • Backpropagation: Computing gradients through a neural network to update its parameters during training. “full attention backpropagates through scores across the entire visible context”
  • Bounded replay: An approximation technique that recomputes only a limited portion of prior model computation. “approximated by methods such as bounded replay”
  • Byte perplexity: A language-model uncertainty measure calculated at the byte rather than token level. “reports byte perplexity over long code repositories”
  • Cascading dependency: A dependency chain in which later computations require progressively larger sets of earlier intermediate states. “This cascading SWA dependency therefore requires the cross-decoder to process a token suffix far longer than the window size.”
  • Context extension: Techniques for enabling a model to process sequences longer than those used during training. “HySparse2 therefore requires no RoPE adjustment during long-context extension.”
  • Cross-decoder: The decoder component that attends to representations produced by a separate self-decoder. “the backbone is divided into a self-decoder and a cross-decoder.”
  • Cross-layer KV sharing: Reusing key–value representations across different Transformer layers. “Cross-layer KV-cache sharing can reduce both KV storage and prefill computation.”
  • Cross-layer KV reuse: Sharing key–value caches and selection information between successive attention layers. “KV Reuse lets sparse layers reuse the full-attention layer's KV cache and selection indices within each hybrid block.”
  • Disaggregation: Separating computational stages or services across different devices or nodes. “Under prefill–decode disaggregation, the HySparse2 prefill node hosts only the self-decoder and the KV Bridging projections.”
  • Distillation: Training one model or module to reproduce the behavior of another model or target system. “without distilling an auxiliary indexer module”
  • Early exit: Terminating computation before all network layers have been executed. “This design also enables a complete early exit after the self-decoder during prefill.”
  • End-to-end training: Training all components of a system jointly using a single overall objective. “This design supports native end-to-end training without a separate indexer”
  • FP8: An 8-bit floating-point numerical format used to reduce memory usage and computational cost. “with FP8 KV-cache storage.”
  • Forced local window: A fixed set of recent tokens that are always included in sparse attention. “a forced window of recent tokens replaces the separate SWA branch in sparse layers”
  • Full attention: Attention in which each query can attend to every permitted key–value position in the sequence. “Both HySparse2 and HySparse retain a small number of full-attention layers”
  • Gated fusion: Combining multiple computational branches using learned gating values. “combines sparse attention with a separate 128-token SWA branch through gated fusion.”
  • Global token: A token selected to provide attention-based access to information from distant portions of the context. “HySparse2 uses token-level selection with 128 forced local tokens and 1,024 global tokens.”
  • Grouped-query attention (GQA): An attention design in which multiple query heads share a smaller number of key–value heads. “Hybrid SWA and HySparse use GQA”
  • Hidden state: An intermediate vector representation produced by a neural-network layer. “The KV caches for full-attention layers in the cross-decoder are generated from the hidden states of full-attention layers in the self-decoder.”
  • Hybrid attention: An architecture that combines multiple attention mechanisms, such as full, sliding-window, and sparse attention. “Both decoders use hybrid attention.”
  • Hybrid sparse attention: An attention mechanism that combines dense or local attention with attention restricted to selected positions. “we introduce HySparse2, a hybrid sparse attention architecture with two-level KV sharing.”
  • Indexing layer / indexer: A layer or module that scores context elements for later selective retrieval. “These full-attention layers also serve as indexers”
  • Key–value (KV) cache: Stored key and value vectors reused during autoregressive inference to avoid recomputing past tokens. “Agentic workloads require more compact KV caches.”
  • KV Bridging: The mechanism that constructs cross-decoder key–value caches from self-decoder hidden states. “At the outer level, KV Bridging connects full-attention layers across the two decoders.”
  • KV Mirror: A cross-layer connection scheme that pairs early and late layers in reverse order to transfer key–value representations. “KV Mirror uses U-shaped connections that pair early and late layers in reverse order”
  • Latent state: A compressed intermediate representation that encodes information from a larger representation. “compress KV representations into latent states”
  • Long-context retrieval: Selecting relevant information from a very large input sequence. “This growing context demands efficient prefill, compact KV-cache storage, and accurate long-context retrieval.”
  • Mixture-of-Experts (MoE): A neural architecture that routes each input through a subset of specialized expert subnetworks. “On an 80B-A3B MoE model”
  • Multi-head attention: Attention using multiple parallel query, key, and value subspaces. “Future architectures could also allocate less computation to full attention and more to sparse attention, for example by reducing the number of query heads”
  • Multi-query attention (MQA): An attention variant in which all query heads share one key head and one value head. “whereas HySparse2 uses MQA”
  • Oracle token selection: Token selection based on exact attention scores from a reference or full-attention computation. “HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing”
  • Partial rotary positional embedding (RoPE): Applying rotary positional encoding to only part of each representation. “SWA layers in the self-decoder use partial rotary positional embeddings (RoPE)”
  • Perplexity: A measure of how uncertain a LLM is when predicting a sequence; lower values generally indicate better predictive performance. “LongPPL measures perplexity on selected tokens that depend on long-range context”
  • Prefill: The initial processing of all input tokens before autoregressive generation begins. “Prefill can therefore exit after the self-decoder”
  • Prefill FLOPs: The number of floating-point operations required to process the input during prefill. “At 1M tokens, HySparse2 reduces prefill FLOPs by 2.92×2.92\times”
  • Quantization: Representing numerical values with reduced precision to save memory and computation. “Precision-level methods reduce the numerical precision of cached keys and values”
  • Query, key, and value projections: Learned transformations that produce the three representations used by an attention operation. “Each cross-decoder full-attention layer has its own K/V projections”
  • Rotary positional embedding (RoPE): A positional encoding method that rotates query and key representations according to token position. “HySparse2 therefore requires no RoPE adjustment during long-context extension.”
  • Self-decoder: The decoder component that constructs representations and key–value caches shared with later decoder layers. “YOCO uses a self-decoder to construct a global KV cache shared across cross-decoder layers”
  • Sigmoid output gate: A learned gating function based on the sigmoid activation that scales an attention output. “all sparse and SWA layers use sigmoid output gates”
  • Sliding-window attention (SWA): Attention restricted to a fixed-size neighborhood of recent tokens. “The self-decoder uses hybrid sliding-window attention (SWA)”
  • Sparse attention: Attention that restricts each query to a selected subset of key–value entries. “Sparse attention reduces long-context attention costs by restricting each query to a subset of KV entries”
  • Speculative decoding: A generation method in which a smaller draft model proposes tokens that a larger model verifies. “During pretraining, HySparse2 uses a single MTP layer”
  • Token-level sparsity: Selecting individual tokens rather than contiguous blocks for sparse attention. “First, it replaces block-level sparsity with token-level sparsity for finer long-context retrieval.”
  • Top-kk selection: Selecting the kk highest-scoring elements according to a ranking criterion. “HySparse2 therefore applies top-kk selection to individual tokens in full-attention layers”
  • Transformer layer: A neural-network layer built around self-attention and position-wise feed-forward transformations. “Each model has 49 Transformer layers”
  • Two-level KV sharing: Sharing key–value representations both between decoders and within hybrid attention blocks. “HySparse2 uses two-level KV sharing.”
View Paper Prompt  View All Prompts 

Open Problems

We're still in the process of identifying open problems mentioned in this paper. Please check back in a few minutes.

Tweets

Sign up for free to view the 6 tweets with 1617 likes about this paper.

Sign Up for Free