FlashAccel: Leveraging High-Bandwidth Flash for High-Throughput LLM Inference

Abstract: LLM inference is increasingly limited by the capacity of High-Bandwidth Memory (HBM) in GPUs, as model weights and KV cache grow rapidly. High-Bandwidth Flash (HBF) provides higher capacity than HBM while retaining comparable bandwidth, making it a promising substrate for capacity-constrained LLM inference. However, its inherently high access latency, low bandwidth utilization, and lack of support for heterogeneous resource management make it difficult to integrate HBF into GPUs for LLM inference. We present FlashAccel, a co-designed system that enables efficient LLM inference using HBF. FlashAccel integrates HBF into HBM-based GPUs, providing architectural support to mitigate access latency. It improves bandwidth utilization through specialized data layouts for both model weights and KV cache, and introduces an HBF-aware storage management layer together with a programming model to organize persistent data in HBF and coordinate heterogeneous memory resources at the system level. Experimental results demonstrate that integrating six HBF stacks into the GPU enables FlashAccel to deliver an average improvement of 2.54$\times$ and 1.93$\times$ in throughput per GPU and energy efficiency over the HBM-only GPU under 100ms latency constraint, respectively.

• The paper introduces a hardware–software co-designed accelerator that uses high-bandwidth flash for model weights and KV cache, combining execution-aware data layouts, SRAM prefetching, and FTL-free memory management to overcome flash latency and bandwidth imbalance.
• FlashAccel achieves an average 2.54× throughput per GPU and 1.93× higher energy efficiency than an HBM-only baseline under a 100ms decode SLO, while matching HBM-based latency within approximately 4% in simulation.
• The design enables capacity-driven batching and sustained KV-cache reuse, but its benefits depend on simulated HBF performance, compressed-attention methods such as GQA or MLA, sufficient write endurance, and acceptable flash write bandwidth.

FlashAccel (2607.10186) is a hardware–software co-designed accelerator that integrates High-Bandwidth Flash (HBF) into HBM-based GPUs to address the memory capacity bottleneck of high-throughput LLM inference. The paper argues that HBF—Flash memory engineered for HBM-comparable bandwidth via 3D stacking, TSVs, and massively increased plane-level parallelism—can serve as a substrate for model weights and KV cache, but only if three obstacles are resolved: Flash's high access latency, low effective bandwidth utilization under LLM access patterns, and the absence of system support for heterogeneous memory management. The proposed design delivers an average 2.54× throughput-per-GPU and 1.93× energy-efficiency improvement over an HBM-only baseline under a 100ms decode SLO.

Motivation: HBM capacity as the serving bottleneck

The paper grounds the capacity argument in concrete serving scenarios. Running Qwen3-32B on an H100 leaves roughly 16GB of HBM for KV cache after weights, capping batch size at 8 for 8K contexts; larger batch sizes improve decode throughput substantially with only marginal latency impact under the Roofline model. Capacity also constrains KV cache reuse in multi-turn interactions—prior characterization work suggests high reuse efficiency may require up to 4× more KV cache capacity than is typically provisioned. Scaling out with more GPUs is presented as an expensive alternative: collective communication can account for up to 20% of total latency, and reliability degrades at scale, with failures in a 1024-GPU cluster occurring on average every 7.9 hours.

Prior Flash-based inference designs (AiF, Lincoln, Cambricon-LLM) target edge deployment with in-storage compute; their hundreds of GB/s of bandwidth and few TFLOPS of compute are inadequate for datacenter throughput targets, and none store KV cache in Flash. FlashAccel's positioning is therefore distinct: keep the powerful GPU as compute engine and attach HBF as a directly addressable, high-bandwidth capacity tier.

A notable and somewhat bold claim in the paper is that KV cache endurance in Flash is no longer a disqualifying concern. Using published DeepSeek inference statistics, the authors compute that each GPU writes roughly 22.7TB (prefill) or 26.9TB (decode) of KV cache per day, while a 1TB SLC HBF sustains about 55TB of writes per day over five years; relaxed retention can extend endurance by up to 50×. This argument implicitly depends on the continued prevalence of GQA/MLA-style attention, which compresses KV writes relative to MHA—the paper's own write analysis confirms this dependency.

Challenges

Three challenges are quantified. First, access latency: HBF read latency ($t_R \approx 4\mu s$) exceeds HBM's (~100ns) by 40×; for a 4k×4k FP8 GEMV that takes 4µs on an H200, HBF access alone would add 50% to end-to-end latency. Second, bandwidth utilization: reaching H200-class bandwidth (4.8TB/s) requires ~4,916 planes reading concurrently, so any load imbalance across planes directly reduces effective bandwidth. The paper models uneven KV cache distribution as a balls-into-bins problem and estimates that uniformly assigning 100GB of Qwen3-235B KV cache across 4,916 planes yields a maximum plane load 52% above average—i.e., a 52% bandwidth loss. Weight access suffers analogously because individual weight tensors (e.g., 12MB in Qwen3-235B) are smaller than the ~19MB hyper page, and framework weight placement follows source-code declaration order rather than execution order. Third, resource management: neither HBM-style paging (no persistence) nor conventional FTL-based Flash stacks (GB-scale mapping tables, translation overhead) fit HBF's role as a directly accessible persistent tier.

Hardware and data layout design

FlashAccel combines HBF, HBM, and GPU cores into one accelerator, realizable via either co-located integration (CLI, replacing most HBM stacks) or cascaded integration (CSI, attaching HBF through the HBM base die). Each HBF stack contains 8 Flash dies with 96 planes per die, SRAM caches placed in the otherwise-unused circuit-die area plus an 8MB base-die SRAM, sized to at least $$2 \times \text{Peak\_Bandwidth} \times \text{Read\_Latency}$$ to sustain a double-buffered prefetch pipeline. The design targets SLC for lower latency and endurance.

The data layout work is the technical core. Weights are partitioned into page-sized units and striped across all planes in execution order, determined offline from the fixed inference computation graph, so sequential access approaches peak bandwidth. KV cache handling addresses a dynamically changing active request set: nonresident blocks are aggregated into full hyper pages before writing, residuals are placed randomly, and after the write phase a selective offloading step migrates excess pages from overloaded planes to HBM until all planes are at average load. Attention then reads HBF at hyper-page granularity and computes per-block partial attention (FlashAttention-style) reduced per request, avoiding plane conflicts from request-parallel kernels. Increasing KV block size further smooths plane load, but the paper concedes a trade-off: larger blocks buffer more partially-filled data in HBM (512KB blocks with batch 256 require at least 8GB HBM for Qwen3-235B) and increase fragmentation; 256KB blocks are chosen as the balance point.

System software

The storage layer eliminates the FTL entirely by exploiting the append-only write pattern of both weights and KV cache: physical addresses are recorded directly in per-object indexing metadata stored in HBF for persistence. An isolated block allocation policy places each request's KV cache in separate blocks so that request-granularity eviction avoids write amplification. The programming model exposes a unified virtual address space over HBM, HBF, and SRAM through a small interface set: NandMmap for static weights, GroupMmap/GroupWrite/GroupArrange for grouped KV cache access with plane-balanced offloading, and SramPrefetch/SramRelease for latency hiding. Prefetch-aware scheduling extracts prefetch operations into a queue following the compute graph's topological order and executes them asynchronously, overlapping Flash latency with computation; correctness is preserved when kernels outrun prefetch because unmigrated pages retain their Flash mappings.

Evaluation

The evaluation is simulation-based, built on LLMCompass with an added NAND simulator modeling page access at plane granularity. The baseline is a DGX-H200 node (8×H200, 141GB and 4.8TB/s per GPU); the modeled HBF stack provides 192GB at 768GB/s per stack with $t_R = 4\mu s$ and $t_{Prog} = 75\mu s$, with $t_R$/$t_{Prog}$ conservatively held at published XL-Flash values despite a 4× capacity increase per plane. Four frontier-scale models (Qwen3-235B, Qwen3-Coder-480B, LLaMA3.1-405B, DeepSeek-V3-671B) spanning GQA/MLA and dense/MoE configurations are evaluated under long-context (8.11K/2.53K) and agentic (15K/6K) sequence configurations.

Key results:

Configuration	Throughput per GPU vs. 8×H200 (100ms SLO)	Energy efficiency vs. 8×H200
8×CSI	2.54× (2.15× avg across SLOs)	1.93×
8×CLI	2.04×	1.66×

The mechanism is capacity-driven batching: FlashAccel shifts the throughput limiter from HBM capacity to latency budget, enabling much larger batch sizes (e.g., batch 110 vs. 30 for LLaMA3.1-405B at 15K/6K under a 50ms SLO). The ablation is informative: disabling prefetching, weight layout, and KV cache layout degrades throughput by 55%, 7%, and 15% respectively; unoptimized "plain HBF" loses 65% and falls below the HBM-only GPU. Full FlashAccel matches HBM-based latency within ~4%, including 2% offloading overhead—evidence that the co-design closes the latency and bandwidth gaps rather than merely tolerating them.

Two secondary results strengthen the case. On KV cache reuse, 8×CSI/CLI achieve the ideal hit rate by retaining all session KV caches, whereas a 16×H200 system remains 50% lower, reducing computed tokens by up to 89% versus 8×H200. On scaling, moving to 16 GPUs forces RDMA-class interconnects (~9× lower bandwidth than NVLink), so the HBM baseline gains little and per-GPU throughput sometimes degrades; an 8×HBF system outperforms a 16×HBM system in per-GPU throughput, a cost-efficiency result that holds despite HBF read energy of 8 pJ/bit (2.7× HBM3e) raising TDP by 1.23–1.31×.

Limitations and open questions

Several constraints deserve emphasis. The evaluation is entirely simulation-based, with HBF parameters extrapolated from a 96-wordline-layer XL-Flash die; the authors note that the assumed 149mm² HBF area (1.23× an HBM3e stack) would shrink toward parity only with the 300+-layer devices demonstrated in recent ISSCC work. Write bandwidth is a real constraint—peak write bandwidth is 245.8GB/s versus 4.6TB/s read—and the endurance argument relies on the conservative assumption of only 10× endurance gain from retention relaxation, plus the assumption that GQA/MLA-style compressed attention remains dominant. Under a strict 50ms SLO, CLI underperforms the baseline on DeepSeek-V3 and Qwen3-235B, and 4-GPU HBF configurations cannot serve LLaMA3.1-405B or Qwen3-Coder-480B at all because weight loading alone approaches the 50ms budget. The KV cache layout also assumes KV blocks can be treated as uniform units for load balancing, an approximation that may weaken under heterogeneous context lengths. Open questions include how HBF-based designs interact with speculative decoding or disaggregated architectures that change the read/write ratio, and whether the FTL-free append-only storage model survives workloads with frequent KV cache edits (e.g., KV cache compression or token eviction within a request).

Conclusion

FlashAccel demonstrates that HBF's capacity advantage translates into serving throughput only through deliberate co-design: distributed SRAM prefetching to absorb the 40× latency gap, execution-order and hyper-page-aligned layouts to approach plane-level peak bandwidth, and an FTL-free, append-only storage layer with a unified virtual address space. The reported 2.54× throughput and 1.93× energy-efficiency gains over HBM-only GPUs, achieved with fewer GPUs than equivalent HBM deployments, make a quantitative case for HBF as a capacity tier for datacenter LLM inference, contingent on the endurance, write-bandwidth, and device-density assumptions the paper makes explicit.