Determine HBF suitability and data-placement policies for LLM inference

Determine whether frequently used and rarely used mixture-of-experts experts can be prefetched from High-Bandwidth Flash (HBF), which access patterns incur the greatest HBF latency cost, how large language model weights and key-value cache should be divided between HBF and high-bandwidth memory (HBM), and what sustained HBF load costs in heat and device lifetime.

Background

High-Bandwidth Flash is proposed as a NAND-based memory tier below HBM, but its suitability for LLM inference depends on workload access patterns, data placement, prefetching opportunities, thermal behavior, and flash reliability. These issues are especially significant for mixture-of-experts models, whose router selects the experts needed for each token only during execution, potentially leaving no time to fetch the selected weights in advance.

The paper identifies these questions as unresolved because HBF devices were not yet available for direct measurement. Existing storage simulators replay recorded access sequences and therefore cannot capture feedback from latency into subsequent program behavior, while GPU and cycle-accurate simulators do not provide the same combination of real inference execution and practical runtime. HBFSim is introduced as an evaluation platform intended to study such decisions before HBF hardware becomes available.

References

The choices a system designer faces cannot wait for HBF silicon to arrive. The first HBF devices for inference are expected to reach customers as samples in early 2027, and several questions are open now: whether the frequently used and the rarely used experts of a mixture-of-experts model can be prefetched from HBF, which access patterns pay most for HBF latency, how an LLM's model weights and key-value cache are split between the two tiers, what sustained load costs in heat and device lifetime, and more.

— HBFSim: Fast and Faithful Simulation of High-Bandwidth Flash Under Real GPU Execution  (2609.09800 - Hu et al., 9 Sep 2026) in Section 1, Introduction

How HBF should be used, and whether HBF is worth building at all, are both open in public today, and no published method settles either question.

— HBFSim: Fast and Faithful Simulation of High-Bandwidth Flash Under Real GPU Execution  (2609.09800 - Hu et al., 9 Sep 2026) in Section 2, Motivation