---
title: 'H2EAL: Hybrid Sparse Attention for LLM Inference'
url: https://www.emergentmind.com/topics/h2eal
type: topic
---

# H2EAL: Hybrid Sparse Attention for LLM Inference

H2EAL, short for **Hybrid-Bonding Architecture with Hybrid Sparse Attention for Efficient Long-Context LLM Inference**, is an HB-based accelerator for long-context LLM inference at the edge that couples sparse-attention algorithms with accelerator design rather than treating them as separable layers of optimization. Its central target is the **KV cache** bottleneck in autoregressive decoding: memory capacity grows linearly with sequence length, every new token must attend to all previous tokens, and the attention step becomes memory-bound because there is little reuse, especially at the low batch sizes typical on edge devices. H2EAL addresses this by combining **hybrid sparse attention** with **memory-compute co-placement**, **interleaved KV cache storage**, and a **load-balancing scheduler with parallel tiled attention**. In the reported evaluation, it achieves **5.20~48.21x speedup** and **6.22~73.48x energy efficiency improvement** over a baseline HB implementation, with a **0.87%** average accuracy drop on multiple benchmarks [2508.16653].

## 1. Problem setting and architectural rationale

Long-context inference stresses both memory capacity and memory bandwidth because the KV cache must retain the full prefix and be repeatedly read during decoding. The cited work emphasizes that this is particularly acute in edge deployment, where the computation is highly memory dependent and attention is dominated by cache movement rather than arithmetic intensity [2508.16653].

The paper positions **hybrid bonding (HB)** as an alternative to conventional **near-memory processing (NMP)** architectures with in-die DRAM integration. Traditional NMP is described as suffering from **limited logic density**, **limited storage capacity**, insufficient throughput for long-context attention, and poor KV cache management as contexts become large. By contrast, HB stacks memory dies on a logic die using **Cu-Cu direct bonding**, yielding very high bandwidth, lower power than HBM-like approaches, and more room to implement compute on the logic die. The paper cites HB examples with **34 GB/s per 1 Gb at 0.88 pJ/bit** [2508.16653].

A key constraint, however, is that HB is not a unified memory system. Its banks are physically distributed, are not directly interconnected like GPU unified memory, and each has relatively limited local storage. That distributed-memory property makes conventional sparse-attention methods difficult to map efficiently. H2EAL is therefore defined not merely as a sparse-attention scheme, but as an **algorithm-hardware co-design** specifically for the mismatch between long-context attention and HB’s distributed banks [2508.16653].

## 2. Hybrid sparse attention and head specialization

The algorithmic core of H2EAL is **hybrid sparse attention**, which assigns different sparse regimes to different heads instead of imposing a uniform pattern. The paper distinguishes two head types: a **streaming head**, which uses a fixed pattern centered on **sink tokens** and **local tokens**, and a **retrieval head**, which uses a query-dependent pattern that dynamically selects relevant pages or tokens [2508.16653].

Head identification follows an optimization-based formulation with a gating parameter $\alpha$. For layer $i$, head $j$, the attention is written as
\[
\mathrm{Attn}_{i,j} = \alpha_{i,j}\cdot \mathrm{Full\_Attn} + (1-\alpha_{i,j})\cdot \mathrm{Streaming\_Attn},
\]
where $\alpha_{i,j} \in [0,1]$, $\alpha$ is initialized to 1, and retrieval heads are treated as full attention during training. This formulation lets the model learn whether a head should behave more like full attention or like a streaming head [2508.16653].

For **streaming heads**, H2EAL applies **static sparsity**: attention is restricted to sink tokens and local tokens, both pre-determined and fixed during inference. The stated effect is reduced KV-cache use, lower bandwidth demand, and simpler hardware support. For **retrieval heads**, H2EAL applies **dynamic sparsity** using a page-based organization of the KV cache. Instead of selecting individual tokens directly, contiguous tokens are grouped into **pages**. For a page $p$, the metadata is
\[
\tau_{\max} = \max_{k\in K_p} k,\qquad \tau_{\min} = \min_{k\in K_p} k,
\]
where $K_p$ denotes all keys in the page. For a query $q$, page relevance is approximated by
\[
\max(q\cdot \tau_{\min}, q\cdot \tau_{\max}),
\]
after which the system selects the **top-$k$** most relevant pages and performs attention only on those pages [2508.16653].

The retrieval path is also coupled to cache management. The paper states that accumulated attention scores are maintained for pages, and when the KV cache reaches budget, the page with the **lowest accumulated score** is evicted. In this sense, retrieval heads are sparse in two ways: by **page-level top-$k$ selection** and by **page eviction under memory pressure**. The reported sparse-attention parameters are **static sparsity = 0.5**, **retrieval selection length = 4k**, and **page size = 32** [2508.16653].

## 3. Memory-compute co-placement and the HB execution model

The principal hardware mechanism in H2EAL is **memory-compute co-placement**, introduced to address the distributed-memory bottleneck of HB. The stated principle is to map both **token KV cache storage** and the corresponding **computation** to the **same bank** whenever possible. This localizes KV operations and reduces inter-bank communication [2508.16653].

The design places **sink tokens**, a subset of **local tokens**, and **importance scores** in the **logic die**, because these structures are frequently accessed and relatively compact. By contrast, **paged tokens** and **page metadata** are placed in the memory banks. The paper describes a pipeline in which popped local tokens are handled by FIFO behavior, streaming heads send popped tokens to memory, retrieval heads compute page metadata using min/max units, and the least important page is discarded when capacity is full [2508.16653].

This placement strategy is complemented by **interleaved KV cache storage** and **adaptive heterogeneous mapping**, both named as hardware-side design elements. The paper’s rationale is that dynamic sparsity would otherwise cause repeated movement of selected data across distant banks, making sparse attention itself a source of communication overhead. H2EAL instead aligns the sparse-attention structure with the physical bank organization. The work also notes that **Softmax cross-bank communication** is handled efficiently using **FlashAttention-style IO-aware techniques** [2508.16653].

A recurrent misconception addressed by the design is that sparse attention alone is sufficient for efficient long-context inference on HB. The architecture is built around the opposite claim: because HB is distributed rather than unified, the sparsity pattern, storage layout, and compute placement must be jointly designed if bank traffic and locality are to be controlled [2508.16653].

## 4. Load balancing, tiled attention, and bank mapping

Hybrid sparse attention introduces a second systems problem beyond locality: **workload imbalance**. Retrieval heads are substantially heavier than streaming heads, and if heads are assigned naively, some banks stall while waiting for the slowest head. The paper identifies this synchronization effect as especially problematic on HB because the banks are physically separate [2508.16653].

H2EAL addresses this through a **load-balancing scheduler with parallel tiled attention**. Heads are grouped into **tiles**, each tile containing a mix of retrieval and streaming heads. Let $n_r$ denote the number of retrieval heads, $n_s$ the number of streaming heads, $t$ the number of tiles, and $T=\{T_0,\dots,T_{t-1}\}$ the tile set. The paper formulates the tiling objective as
\[
\min~\mathrm{MaxDist}(\mathrm{Loc}_r, \mathrm{Loc}_s)
\]
subject to
\[
\mathrm{Loc}_r, \mathrm{Loc}_s \in T_i~(i=0, \dots, t-1),
\]
\[
t = \min(n_r, n_s), \quad |T_i| \leq \left\lceil \frac{n_s+n_r}{t} \right\rceil.
\]
Here, $\mathrm{Loc}_r$ and $\mathrm{Loc}_s$ are the physical positions of retrieval and streaming heads, and $\mathrm{MaxDist}$ is the maximum distance between paired heads in a tile. The stated goal is to keep paired heads physically close so as to reduce communication overhead [2508.16653].

The mapping procedure then handles three bank/head cardinality cases. When **$n_h = n_b$ or $n_b$ is divisible by $n_h$**, each head is assigned $n_b/n_h$ banks. When **$n_h > n_b$**, heads are split into $\lceil n_h/n_b \rceil$ disjoint subsets and executed sequentially in a pipeline,
\[
n_{h1} \rightarrow n_{h2} \rightarrow \cdots \rightarrow n_{hk}.
\]
When **$n_h < n_b$** and **$n_b$ is not divisible by $n_h$**, the paper uses a greedy decomposition
\[
n_h=\sum_{i=1}^l n_{hi}'\quad\mathrm{with}~ n_b~\mathrm{mod}~ n_{hi}' = 0,\ \forall i,
\]
choosing larger divisors first to minimize the number of pipeline stages [2508.16653].

The scheduler is therefore balancing compute across heterogeneous heads while simultaneously matching model structure to HB bank structure. In the reported ablation, this balancing removed **3613 idle cycles** on banks 0–4 in one example and yielded **2.01× speedup**, which the paper uses to argue that sparsity without scheduling is insufficient on HB [2508.16653].

## 5. Implementation parameters and evaluation protocol

The evaluation uses a **cycle-level simulation framework** for the HB accelerator. The **memory subsystem** is specified as **4 stacked layers**, **256-bit macro bandwidth**, **0.88 pJ/bit access energy**, and **32 GB total capacity**, organized as **32 MB/macro**, **16 macros/bank**, **16 banks/layer**, and **4 layers**. The **logic die** is implemented at **22 nm** with **25 mm × 28 mm** area, a GEMM accelerator composed of **16 DCIM macros**, throughput of **900 GOPS × 16 /bank @ int8**, **24 TOPS/W @ int8** energy efficiency, **8 × 128 KB/bank SRAM cache**, a **4 × 4 2-D mesh** NoC with **256-bit bandwidth**, and **0.7 V** voltage. The **memory die** operates at **400 MHz** and **1.2 V**, with refresh periods of **32 ms** below **85°C**, **16 ms** below **95°C**, **8 ms** below **105°C**, and **4 ms** below **115°C** [2508.16653].

The evaluated models are **Mistral-7B**, **LLaMA2-7B**, and **LLaMA3-8B**. All are quantized to **8-bit weights**, **8-bit activations**, and **8-bit KV cache**. Accuracy is evaluated on **LongBench**, comprising **21 datasets** with lengths up to **22k words**, and on **Needle-in-a-Haystack (NIAH)**. Baselines include **full attention**, **sparse attention without tile balancing**, and, for accuracy comparison, **StreamingLLM** and **H2O**. The paper reports **speedup / latency reduction**, **throughput in tokens/s**, **energy efficiency in tokens/J**, **accuracy on LongBench and NIAH**, **attention-only microbenchmarks**, and **end-to-end inference performance** [2508.16653].

## 6. Quantitative results, accuracy retention, and trade-offs

At **256k context length**, the attention microbenchmark shows large gains over full attention. For **Mistral-7B**, H2EAL reports **28.09× speedup** and **69.20× energy-efficiency improvement**; for **LLaMA2-7B**, **48.21×** and **73.48×**; and for **LLaMA3-8B**, **28.20×** and **70.45×**. Compared to **sparse attention without balance**, the gains are more modest but still material: **1.221×** for Mistral-7B, **1.605×** for LLaMA2-7B, and **1.195×** for LLaMA3-8B. This is consistent with the paper’s claim that sparse attention alone does not resolve the HB execution bottleneck [2508.16653].

For end-to-end inference at **256k**, **LLaMA2-7B** increases from **40.8 tokens/s** with full attention to **430.8 tokens/s** with H2EAL, while energy efficiency rises from **1.90 tokens/J** to **23.20 tokens/J**. **LLaMA3-8B** increases from **113.1 tokens/s** to **469.7 tokens/s**, and from **6.05 tokens/J** to **25.83 tokens/J**. The paper states that these correspond to **10.56×** and **4.15×** latency speedup compared with full attention at 256k, and **12.21×** and **4.27×** energy-efficiency improvement, respectively. At **64k**, throughput improves from **127.9** to **459.5** for LLaMA2-7B and from **253.4** to **482.1** for LLaMA3-8B [2508.16653].

Accuracy degradation is reported as small. On **LongBench** for **LLaMA3-8B** over **21 datasets**, the average score changes from **39.48** under full attention to **38.88** under H2EAL, a drop of **0.60 points**, described in the abstract as about **0.87%** average accuracy degradation. The paper gives examples in both directions: **2WikiMQA** improves from **29.19** to **31.13**, **LCC** from **34.81** to **37.89**, while **TriviaQA** declines from **87.59** to **84.88** and **GovReport** from **34.52** to **32.35**. On **NIAH**, **static sparsity 0.5** is reported to provide a favorable balance between accuracy and efficiency, and the system remains robust under varying context lengths and needle positions [2508.16653].

The paper identifies several trade-offs. H2EAL is **not a plug-and-play sparse-attention method**; it requires HB-aware placement, tiling, and scheduling. **Dynamic sparsity** incurs metadata loading, min/max computation, top-$k$ selection, and page management. **Distributed-memory complexity** makes communication optimization necessary, and more aggressive compression can reduce accuracy if important tokens are removed. The reported explanation for the small accuracy loss is the combination of **head-wise specialization**, preservation of **sink + local tokens** for streaming heads, **query-aware selection** for retrieval heads, **page-level selection**, **importance-score-based eviction**, and the empirically selected **static sparsity 0.5**. The paper also notes that sparse attention can outperform full attention on some long-context tasks by reducing attention noise; this suggests that, within the evaluated regime, the method is not solely a performance optimization but also a controlled approximation of long-context attention [2508.16653].

Source: https://www.emergentmind.com/topics/h2eal