H2EAL: Hybrid Sparse Attention for LLM Inference

• H2EAL is an HB-based accelerator that integrates hybrid sparse attention with memory-compute co-placement to mitigate the KV cache bottleneck in long-context LLM inference.
• It employs a dual mechanism with streaming and retrieval heads, combining static and dynamic sparsity to balance workload and reduce inter-bank communication in distributed HB memory banks.
• Evaluations demonstrate speedups up to 48.21× and energy efficiency improvements up to 73.48× with minimal accuracy degradation, proving its efficacy for edge deployments.

H2EAL, short for Hybrid-Bonding Architecture with Hybrid Sparse Attention for Efficient Long-Context LLM Inference, is an HB-based accelerator for long-context LLM inference at the edge that couples sparse-attention algorithms with accelerator design rather than treating them as separable layers of optimization. Its central target is the KV cache bottleneck in autoregressive decoding: memory capacity grows linearly with sequence length, every new token must attend to all previous tokens, and the attention step becomes memory-bound because there is little reuse, especially at the low batch sizes typical on edge devices. H2EAL addresses this by combining hybrid sparse attention with memory-compute co-placement, interleaved KV cache storage, and a load-balancing scheduler with parallel tiled attention. In the reported evaluation, it achieves 5.20~48.21x speedup and 6.22~73.48x energy efficiency improvement over a baseline HB implementation, with a 0.87% average accuracy drop on multiple benchmarks (Fu et al., 20 Aug 2025).

1. Problem setting and architectural rationale

Long-context inference stresses both memory capacity and memory bandwidth because the KV cache must retain the full prefix and be repeatedly read during decoding. The cited work emphasizes that this is particularly acute in edge deployment, where the computation is highly memory dependent and attention is dominated by cache movement rather than arithmetic intensity (Fu et al., 20 Aug 2025).

The paper positions hybrid bonding (HB) as an alternative to conventional near-memory processing (NMP) architectures with in-die DRAM integration. Traditional NMP is described as suffering from limited logic density, limited storage capacity, insufficient throughput for long-context attention, and poor KV cache management as contexts become large. By contrast, HB stacks memory dies on a logic die using Cu-Cu direct bonding, yielding very high bandwidth, lower power than HBM-like approaches, and more room to implement compute on the logic die. The paper cites HB examples with 34 GB/s per 1 Gb at 0.88 pJ/bit (Fu et al., 20 Aug 2025).

A key constraint, however, is that HB is not a unified memory system. Its banks are physically distributed, are not directly interconnected like GPU unified memory, and each has relatively limited local storage. That distributed-memory property makes conventional sparse-attention methods difficult to map efficiently. H2EAL is therefore defined not merely as a sparse-attention scheme, but as an algorithm-hardware co-design specifically for the mismatch between long-context attention and HB’s distributed banks (Fu et al., 20 Aug 2025).

2. Hybrid sparse attention and head specialization

The algorithmic core of H2EAL is hybrid sparse attention, which assigns different sparse regimes to different heads instead of imposing a uniform pattern. The paper distinguishes two head types: a streaming head, which uses a fixed pattern centered on sink tokens and local tokens, and a retrieval head, which uses a query-dependent pattern that dynamically selects relevant pages or tokens (Fu et al., 20 Aug 2025).

Head identification follows an optimization-based formulation with a gating parameter $\alpha$. For layer $i$, head $j$, the attention is written as

$$\mathrm{Attn}_{i,j} = \alpha_{i,j}\cdot \mathrm{Full\_Attn} + (1-\alpha_{i,j})\cdot \mathrm{Streaming\_Attn},$$

where $\alpha_{i,j} \in [0,1]$, $\alpha$ is initialized to 1, and retrieval heads are treated as full attention during training. This formulation lets the model learn whether a head should behave more like full attention or like a streaming head (Fu et al., 20 Aug 2025).

For streaming heads, H2EAL applies static sparsity: attention is restricted to sink tokens and local tokens, both pre-determined and fixed during inference. The stated effect is reduced KV-cache use, lower bandwidth demand, and simpler hardware support. For retrieval heads, H2EAL applies dynamic sparsity using a page-based organization of the KV cache. Instead of selecting individual tokens directly, contiguous tokens are grouped into pages. For a page $p$, the metadata is

$$\tau_{\max} = \max_{k\in K_p} k,\qquad \tau_{\min} = \min_{k\in K_p} k,$$

where $K_p$ denotes all keys in the page. For a query $q$, page relevance is approximated by

$i$0

after which the system selects the top-$i$1 most relevant pages and performs attention only on those pages (Fu et al., 20 Aug 2025).

The retrieval path is also coupled to cache management. The paper states that accumulated attention scores are maintained for pages, and when the KV cache reaches budget, the page with the lowest accumulated score is evicted. In this sense, retrieval heads are sparse in two ways: by page-level top-$i$2 selection and by page eviction under memory pressure. The reported sparse-attention parameters are static sparsity = 0.5, retrieval selection length = 4k, and page size = 32 (Fu et al., 20 Aug 2025).

3. Memory-compute co-placement and the HB execution model

The principal hardware mechanism in H2EAL is memory-compute co-placement, introduced to address the distributed-memory bottleneck of HB. The stated principle is to map both token KV cache storage and the corresponding computation to the same bank whenever possible. This localizes KV operations and reduces inter-bank communication (Fu et al., 20 Aug 2025).

The design places sink tokens, a subset of local tokens, and importance scores in the logic die, because these structures are frequently accessed and relatively compact. By contrast, paged tokens and page metadata are placed in the memory banks. The paper describes a pipeline in which popped local tokens are handled by FIFO behavior, streaming heads send popped tokens to memory, retrieval heads compute page metadata using min/max units, and the least important page is discarded when capacity is full (Fu et al., 20 Aug 2025).

This placement strategy is complemented by interleaved KV cache storage and adaptive heterogeneous mapping, both named as hardware-side design elements. The paper’s rationale is that dynamic sparsity would otherwise cause repeated movement of selected data across distant banks, making sparse attention itself a source of communication overhead. H2EAL instead aligns the sparse-attention structure with the physical bank organization. The work also notes that Softmax cross-bank communication is handled efficiently using FlashAttention-style IO-aware techniques (Fu et al., 20 Aug 2025).

A recurrent misconception addressed by the design is that sparse attention alone is sufficient for efficient long-context inference on HB. The architecture is built around the opposite claim: because HB is distributed rather than unified, the sparsity pattern, storage layout, and compute placement must be jointly designed if bank traffic and locality are to be controlled (Fu et al., 20 Aug 2025).

4. Load balancing, tiled attention, and bank mapping

Hybrid sparse attention introduces a second systems problem beyond locality: workload imbalance. Retrieval heads are substantially heavier than streaming heads, and if heads are assigned naively, some banks stall while waiting for the slowest head. The paper identifies this synchronization effect as especially problematic on HB because the banks are physically separate (Fu et al., 20 Aug 2025).

H2EAL addresses this through a load-balancing scheduler with parallel tiled attention. Heads are grouped into tiles, each tile containing a mix of retrieval and streaming heads. Let $i$3 denote the number of retrieval heads, $i$4 the number of streaming heads, $i$5 the number of tiles, and $i$6 the tile set. The paper formulates the tiling objective as

$i$7

subject to

$i$8

$i$9

Here, $j$0 and $j$1 are the physical positions of retrieval and streaming heads, and $j$2 is the maximum distance between paired heads in a tile. The stated goal is to keep paired heads physically close so as to reduce communication overhead (Fu et al., 20 Aug 2025).

The mapping procedure then handles three bank/head cardinality cases. When $j$3 or $j$4 is divisible by $j$5, each head is assigned $j$6 banks. When $j$7, heads are split into $j$8 disjoint subsets and executed sequentially in a pipeline,

$j$9

When $$\mathrm{Attn}_{i,j} = \alpha_{i,j}\cdot \mathrm{Full\_Attn} + (1-\alpha_{i,j})\cdot \mathrm{Streaming\_Attn},$$0 and $$\mathrm{Attn}_{i,j} = \alpha_{i,j}\cdot \mathrm{Full\_Attn} + (1-\alpha_{i,j})\cdot \mathrm{Streaming\_Attn},$$1 is not divisible by $$\mathrm{Attn}_{i,j} = \alpha_{i,j}\cdot \mathrm{Full\_Attn} + (1-\alpha_{i,j})\cdot \mathrm{Streaming\_Attn},$$2, the paper uses a greedy decomposition

$$\mathrm{Attn}_{i,j} = \alpha_{i,j}\cdot \mathrm{Full\_Attn} + (1-\alpha_{i,j})\cdot \mathrm{Streaming\_Attn},$$3

choosing larger divisors first to minimize the number of pipeline stages (Fu et al., 20 Aug 2025).

The scheduler is therefore balancing compute across heterogeneous heads while simultaneously matching model structure to HB bank structure. In the reported ablation, this balancing removed 3613 idle cycles on banks 0–4 in one example and yielded 2.01× speedup, which the paper uses to argue that sparsity without scheduling is insufficient on HB (Fu et al., 20 Aug 2025).

5. Implementation parameters and evaluation protocol

The evaluation uses a cycle-level simulation framework for the HB accelerator. The memory subsystem is specified as 4 stacked layers, 256-bit macro bandwidth, 0.88 pJ/bit access energy, and 32 GB total capacity, organized as 32 MB/macro, 16 macros/bank, 16 banks/layer, and 4 layers. The logic die is implemented at 22 nm with 25 mm × 28 mm area, a GEMM accelerator composed of 16 DCIM macros, throughput of 900 GOPS × 16 /bank @ int8, 24 TOPS/W @ int8 energy efficiency, 8 × 128 KB/bank SRAM cache, a 4 × 4 2-D mesh NoC with 256-bit bandwidth, and 0.7 V voltage. The memory die operates at 400 MHz and 1.2 V, with refresh periods of 32 ms below 85°C, 16 ms below 95°C, 8 ms below 105°C, and 4 ms below 115°C (Fu et al., 20 Aug 2025).

The evaluated models are Mistral-7B, LLaMA2-7B, and LLaMA3-8B. All are quantized to 8-bit weights, 8-bit activations, and 8-bit KV cache. Accuracy is evaluated on LongBench, comprising 21 datasets with lengths up to 22k words, and on Needle-in-a-Haystack (NIAH). Baselines include full attention, sparse attention without tile balancing, and, for accuracy comparison, StreamingLLM and H2O. The paper reports speedup / latency reduction, throughput in tokens/s, energy efficiency in tokens/J, accuracy on LongBench and NIAH, attention-only microbenchmarks, and end-to-end inference performance (Fu et al., 20 Aug 2025).

6. Quantitative results, accuracy retention, and trade-offs

At 256k context length, the attention microbenchmark shows large gains over full attention. For Mistral-7B, H2EAL reports 28.09× speedup and 69.20× energy-efficiency improvement; for LLaMA2-7B, 48.21× and 73.48×; and for LLaMA3-8B, 28.20× and 70.45×. Compared to sparse attention without balance, the gains are more modest but still material: 1.221× for Mistral-7B, 1.605× for LLaMA2-7B, and 1.195× for LLaMA3-8B. This is consistent with the paper’s claim that sparse attention alone does not resolve the HB execution bottleneck (Fu et al., 20 Aug 2025).

For end-to-end inference at 256k, LLaMA2-7B increases from 40.8 tokens/s with full attention to 430.8 tokens/s with H2EAL, while energy efficiency rises from 1.90 tokens/J to 23.20 tokens/J. LLaMA3-8B increases from 113.1 tokens/s to 469.7 tokens/s, and from 6.05 tokens/J to 25.83 tokens/J. The paper states that these correspond to 10.56× and 4.15× latency speedup compared with full attention at 256k, and 12.21× and 4.27× energy-efficiency improvement, respectively. At 64k, throughput improves from 127.9 to 459.5 for LLaMA2-7B and from 253.4 to 482.1 for LLaMA3-8B (Fu et al., 20 Aug 2025).

Accuracy degradation is reported as small. On LongBench for LLaMA3-8B over 21 datasets, the average score changes from 39.48 under full attention to 38.88 under H2EAL, a drop of 0.60 points, described in the abstract as about 0.87% average accuracy degradation. The paper gives examples in both directions: 2WikiMQA improves from 29.19 to 31.13, LCC from 34.81 to 37.89, while TriviaQA declines from 87.59 to 84.88 and GovReport from 34.52 to 32.35. On NIAH, static sparsity 0.5 is reported to provide a favorable balance between accuracy and efficiency, and the system remains robust under varying context lengths and needle positions (Fu et al., 20 Aug 2025).

The paper identifies several trade-offs. H2EAL is not a plug-and-play sparse-attention method; it requires HB-aware placement, tiling, and scheduling. Dynamic sparsity incurs metadata loading, min/max computation, top-$$\mathrm{Attn}_{i,j} = \alpha_{i,j}\cdot \mathrm{Full\_Attn} + (1-\alpha_{i,j})\cdot \mathrm{Streaming\_Attn},$$4 selection, and page management. Distributed-memory complexity makes communication optimization necessary, and more aggressive compression can reduce accuracy if important tokens are removed. The reported explanation for the small accuracy loss is the combination of head-wise specialization, preservation of sink + local tokens for streaming heads, query-aware selection for retrieval heads, page-level selection, importance-score-based eviction, and the empirically selected static sparsity 0.5. The paper also notes that sparse attention can outperform full attention on some long-context tasks by reducing attention noise; this suggests that, within the evaluated regime, the method is not solely a performance optimization but also a controlled approximation of long-context attention (Fu et al., 20 Aug 2025).