Beyond Capacity: Scalable MoE LLM Inference via High-Bandwidth Flash with Direct GPU and HBM Paths
Abstract: Modern mixture-of-experts (MoE) LLMs increasingly strain the capacity and cost efficiency of high-bandwidth memory (HBM), as rapidly growing expert weights must be provisioned close to GPUs. High-bandwidth flash (HBF) offers substantially greater capacity, but conventional designs typically deliver HBF-resident expert weights to the GPU through HBM, leaving an additional direct GPU-HBF connection underutilized. We explore an HBF organization that simultaneously exploits two independent expert-delivery routes: a direct path that transfers expert weights from HBF to the GPU and a relay path that transfers them from HBF through the HBM base die to the GPU. Whole experts are assigned to one of the two routes, and transfers over both routes proceed concurrently, increasing aggregate expert-delivery bandwidth without replicating expert weights or introducing a shared relay bottleneck. Early expert determination identifies upcoming experts ahead of their conventional execution point, allowing HBF read latency to overlap with preceding computation, while separate management of immutable expert weights and mutable KV-cache data reduces interference between the two traffic classes. We evaluate the architecture using an event-driven continuous-batching LLM serving simulator with empirically measured GPU compute latencies. Across representative MoE workloads, concurrently utilizing the direct GPU-HBF and HBF-HBM-GPU routes consistently improves expert-delivery efficiency over designs restricted to either route alone. For a representative workload, the proposed architecture can achieve 1.94 higher throughput and 1.90 end-to-end speedup over a design that delivers all HBF-resident expert weights to the GPU through the HBM base die.
Summary
- The paper introduces DASH, a dual-path architecture that combines direct GPU-to-flash and HBM-relay routes to deliver expert weights concurrently, achieving up to 6.4 TB/s modeled GPU-facing bandwidth.
- The paper shows that DASH improves geometric-mean throughput by 1.90× over RelayOnly and 1.84× over DirectOnly, while reducing end-to-end latency by up to 44.8% across evaluated MoE workloads.
- The paper combines exact early expert selection with HBF-aware placement and phase-sensitive KV-cache writes, but identifies hardware prototyping, flash endurance, workload skew, and router compatibility as key open challenges.
Problem setting and central thesis
“Beyond Capacity: Scalable MoE LLM Inference via High-Bandwidth Flash with Direct GPU and HBM Paths” (2608.14333) addresses a specific systems problem created by the divergence between MoE model capacity and GPU-local memory capacity. MoE architectures reduce per-token computation by activating only a small subset of experts, but they retain the full expert-weight set as addressable model state. Consequently, sparsity reduces computation without proportionally reducing storage requirements. The paper reports surveyed MoE checkpoints with total weight footprints between 281 GB and 1.5 TB, with expert weights constituting 94.1–98.8% of those footprints. These sizes exceed the 80 GB HBM capacity of an NVIDIA H100 before accounting for KV caches, activations, and serving-system workspace.
Long-context inference intensifies the problem through a second, mutable memory population: the KV cache. Continuous batching causes KV states from requests with different prompt and generation lengths to coexist, while each decode step also performs input-dependent expert accesses. The resulting system must simultaneously support large, read-mostly, sparsely accessed expert weights and a growing, frequently updated KV working set.
The paper’s central claim is that HBF should not be treated solely as a backing tier behind HBM. Instead, HBF can serve as a first-class GPU-attached memory tier if its independent GPU-facing bandwidth is combined with a second route through the HBM base die. The proposed architecture, DASH, therefore exposes two concurrent delivery paths for HBF-resident data:
- a Direct path from HBF to the GPU; and
- a Relay path from HBF through the HBM base die to the GPU.
Experts are assigned wholly to one route, and transfers across the two routes proceed concurrently without replicating expert weights or funneling all traffic through a shared relay bottleneck. This architectural mechanism is coupled with early expert selection, HBF-aware placement, and phase-sensitive KV-cache write scheduling.
DASH architecture
DASH connects the GPU I/O die, HBM base dies, and HBF base dies using independent UCIe links. The design provides three physical data paths: GPU–HBM, GPU–HBF, and HBM–HBF. The Relay path uses the HBM base-die router and GPU-facing link but does not access HBM cells or the HBM controller’s DRAM interface. Thus, the HBM component of the Relay path is a routing and buffering element rather than a capacity or storage destination.
Each modeled UCIe-A link provides 1.6 TB/s of usable streaming bandwidth after accounting for link and implementation overheads. The evaluated full configuration contains two 512 GB HBF stacks and two 24 GB HBM stacks. With two direct GPU–HBF links and two HBM-mediated relay routes, DASH can expose up to 6.4 TB/s of aggregate GPU-facing bandwidth for SRAM-ready HBF data, compared with 3.2 TB/s for the single-route baselines.
Figure 1: DASH’s independent GPU–HBM, GPU–HBF, and HBM–HBF links, together with the modeled HBF and HBM base-die buffering structures.
The HBF base die contains an HBF controller, a local scheduler, and 18 MiB of physical banked SRAM per stack, of which 16 MiB is usable data storage. The HBM base die contributes 9 MiB of physical SRAM per stack, with 8 MiB usable for relay buffering. These buffers are double-buffered so that one region can drain toward the GPU while another receives data from HBF. The design consequently overlaps NAND page sensing, SRAM filling, and D2D transmission, subject to bank, path, and buffer availability.
The paper emphasizes that the Relay path does not automatically provide additional memory bandwidth merely by adding a logical route. Its benefit depends on avoiding shared resources that serialize Direct and Relay traffic. DASH assigns independent SRAM banks and schedules different ready chunks over the two routes. This makes the effective completion time for expert delivery approximately the maximum of the Direct-path and Relay-path completion times, rather than their sum.
The paper also evaluates Compact-DASH, which halves the number of HBM/HBF stack pairs while retaining both route types. Compact-DASH performs comparably to the single-path full-stack baselines, suggesting that route diversity can partially compensate for reduced memory-stack count. This result is important because it separates the benefit of dual-path connectivity from simply increasing the number of HBF or HBM stacks.
Data placement and HBF-specific memory management
DASH adopts a placement policy based on capacity, mutability, and reuse. Frequently updated data and intermediate activations remain in HBM. Large read-mostly data, including expert weights and write-once/read-many prefill KV state, reside in HBF. Small attention weights, such as QKV and output-projection weights, are replicated across HBM and HBF to enable parallel access through either memory path.
Expert weights are partitioned across HBF dies and planes rather than mapping each expert to a small fixed subset of planes. A selected expert can therefore be streamed through multiple NAND-access resources in parallel, preserving aggregate HBF bandwidth despite MoE sparsity.
Figure 2: Distribution of expert-weight chunks across HBF planes and separation of model-weight and KV-cache erase-block ownership.
The placement policy also distinguishes immutable model weights from mutable KV-cache data at erase-block granularity. This is necessary because NAND updates are out of place and garbage collection can relocate live pages. If model weights and KV pages share erase blocks, KV reclamation can induce unnecessary inspection and movement of model-weight pages. DASH instead maintains weight-owned and KV-owned blocks. KV garbage collection can then erase an empty KV block directly without examining or relocating immutable model-weight pages.
Decode KV updates are initially accumulated in HBM because individual token-level updates are poorly matched to HBF’s page-program granularity. Once sufficient data accumulate, DASH writes page-aligned waves from HBM to HBF. In contrast, prefill generates large KV bursts and can write them directly to HBF. This distinction avoids exposing HBF’s long program latency on every decode step while preserving HBM capacity for the active working set.
The design’s memory policy therefore treats HBF reads and writes asymmetrically. Expert weights are optimized for parallel read delivery, whereas KV writes are optimized for coalescing, page utilization, and temporal overlap with computation. This is a necessary condition for using flash-like memory in an inference path: read bandwidth alone is insufficient if write scheduling causes program operations to interfere with latency-critical expert reads.
Exact early expert determination
The most technically distinctive software–hardware mechanism is Lookahead Expert Execution. HBF reads incur NAND sensing latency , modeled nominally as 3 s and swept up to 32 s. If expert addresses are issued only after conventional attention, residual, RMSNorm, and routing operations complete, this latency appears directly on the MoE critical path.
DASH exploits the structure of a bias-free, scale-invariant router. The conventional router input can be decomposed into a term dependent only on the pre-attention input and a term dependent on the attention output. The first term is computed before attention; the second can be computed immediately when the attention output becomes available, without waiting for the output projection and RMSNorm sequence to finish. The resulting logits differ from the conventional logits only by a positive scalar shared across experts. Since positive scaling preserves rank, the top- expert set is exactly unchanged.
This is not speculative expert prediction. It is an algebraic reformulation that produces the exact top- decision earlier, under explicit router assumptions. DASH computes the early selection in FP32 and uses it to initiate HBF reads. The subsequent RMSNorm remains part of the ordinary computation path, but the selected expert weights are already being sensed and transferred.
Figure 3: Earlier exact top- determination by decomposing router computation and eliminating dependence on the shared positive RMS scaling factor.
The assumption is material. Early selection is applicable only when routing is scale-invariant and lacks expert-specific additive bias. The paper explicitly excludes routers that violate these conditions, including the router used by DeepSeek-V3. For such models, DASH retains the placement and dual-path delivery mechanisms but performs conventional late expert selection. Therefore, the strongest latency benefits of Lookahead Expert Execution do not apply uniformly across contemporary MoE architectures.
The isolated evaluation quantifies the effect. At s, early expert determination reduces E2E latency by 3.33% for Qwen3 and 1.99% for DeepSeek-V2, with TPOT reductions of 3.86% and 2.45%, respectively. At s, E2E reductions increase to 9.50% and 8.69%, while TPOT reductions reach 10.88% and 10.53%. The implication is direct: the mechanism becomes more valuable as NAND sensing latency increases, but only to the extent that the interval between early selection and expert consumption is long enough to absorb that latency.
Execution modes and continuous serving
DASH defines four principal execution modes: parallel HBM/HBF reads, dual-path HBF reads, direct prefill writes, and HBM-to-HBF KV writeback.
Figure 4: Concurrent HBM/HBF execution, dual-path HBF delivery, and phase-aware HBF write scheduling.
Parallel HBM/HBF reads are useful for replicated attention weights and partitioned KV ranges. The GPU can process partial attention contributions from HBM and HBF concurrently before combining them. Dual-path HBF reads are more central to MoE execution: different chunks of selected experts are sent simultaneously through Direct and Relay routes. Direct prefill writes exploit the large, naturally page-compatible KV bursts generated during prompt processing. Decode writeback accumulates smaller updates in HBM and transfers them to HBF only when a full page wave can be formed.
The serving simulator models continuous batching with chunked prefills and decode reservations at iteration boundaries. This matters because the memory demand is not a static batch-level property. Active requests, selected experts, KV occupancy, and the mixture of prefill and decode work evolve over time. The simulator uses empirically measured H100 operator latencies and reports a median relative error of 0.51% across 1,107 validation measurements, with 90% of errors below 3.52%. This gives the GPU-compute component of the simulation a stronger empirical basis than a purely analytical model, although the HBF and interconnect components remain modeled rather than measured in silicon.
Under continuous batching for Qwen3-235B-A22B, DASH reduces P90 E2E latency by 61.2% at 50% of RelayOnly’s saturation rate, 53.5% at 75%, and 50.5% at 90%. P90 TPOT reductions are 62.2%, 53.9%, and 50.3%, respectively. The result persists under mixed prefill/decode scheduling: Mixed increases peak throughput over a serial policy by 10.3% for DASH and 12.6% for the single-route baselines, while DASH maintains 34.1–37.1% higher peak throughput than those baselines.
These results show that the architecture is not limited to fixed-batch microbenchmarks. Its advantage survives request-level interference and dynamic admission, although the evaluation uses controlled Poisson traces and balanced routing rather than production traces with potentially skewed expert demand.
End-to-end performance
The principal evaluation compares DASH with RelayOnly, DirectOnly, and Compact-DASH across six MoE configurations, including Qwen3-235B-A22B, Mixtral-8x22B, Grok-1, Llama 4 Maverick, DeepSeek-V3, and DeepSeek-V2. Five models participate in the main batch-size and sequence-length sweeps; DeepSeek-V2 is used for the lookahead study.
For batch sizes from 1 to 64 with 1K-token prefills and 128 generated tokens, DASH consistently outperforms both single-route baselines. Across the evaluated models and batch sizes, it achieves a geometric-mean throughput speedup of 1.90x over RelayOnly and 1.84x over DirectOnly. E2E latency decreases by 42.2% and 40.8%, respectively.
Figure 5: Normalized throughput and E2E latency under batch-size scaling and varying input/output sequence lengths, including cases where Llama 4 Maverick’s KV cache overflows HBM.
For the 20 model–workload combinations in the sequence-length sweep, DASH achieves geometric-mean throughput speedups of 1.79x over RelayOnly and 1.63x over DirectOnly. E2E latency reductions are 40.1% and 35.6%. Improvements are larger in long-decode workloads than in prefill-dominated workloads because repeated decode iterations create more opportunities for concurrent expert delivery, KV access, and scheduled writeback.
The most demanding reported case is Llama 4 Maverick with a 197.413 GB KV cache generated by a long decode sequence. Even after HBM-to-HBF KV spill begins, DASH provides 1.92x higher throughput and 48.0% lower E2E latency than RelayOnly. This result supports the paper’s claim that HBF can accommodate not only model weights but also long-lived KV state, provided that decode updates are buffered and migrated in page-aligned batches.
The paper also reports a representative five-model geometric-mean speedup of 1.90x and an E2E latency reduction of 44.8% over RelayOnly at batch size 4, a 1K-token prefill, and 128 decode tokens. These headline results are consistent with the broader sweeps rather than being isolated to one model.
Comparison with CPU–GPU offloading
The authors compare DASH with weight streaming and CPU-execution alternatives using a shape-equivalent Qwen3 BF16 expert layer on an Intel Xeon Platinum 8452Y. They construct a favorable Hybrid oracle that independently selects CPU/GPU splits and reports the best median TPOT over the evaluated split space. Even with this advantage, Hybrid remains 8.22–12.32x slower than DASH.
The comparison attributes the gap to two avoided costs in DASH: CPU execution of expert layers and host-to-GPU staging of expert weights. The result does not establish that every CPU-offloading system would exhibit the same gap. It is specific to the evaluated CPU, NUMA placement, PCIe-connected configuration, measured expert shapes, and the oracle’s explored split space. Nevertheless, within those conditions, near-GPU HBF delivery is substantially more effective than host-side alternatives.
Sensitivity, scalability, and endurance
Sensitivity analysis shows that DASH benefits from larger HBF page sizes, lower read latency, and higher D2D bandwidth. Larger page waves improve effective aggregate read bandwidth by increasing the amount of data transferred per HBF access. Reducing 0 similarly increases the number of reads that can complete within a fixed interval. Increasing per-route D2D bandwidth beyond the nominal 1.6 TB/s continues to reduce E2E latency, indicating that the interconnect can become the bottleneck after NAND data become ready.
HBF program latency is largely hidden when it remains within the available overlap window. In the reported Llama 4 Maverick sensitivity study, program latencies up to 500 1s introduce zero E2E-exposed stall. At 5 ms, however, programming contributes 19.94 seconds of cumulative exposed stall over the execution. This establishes an important qualification: write scheduling converts program latency into hidden work only while sufficient independent computation and data movement exist.
The cost analysis is explicitly parametric because public HBF disclosures do not provide per-stack pricing or package-integration costs. With two 512 GB HBF stacks and two 24 GB HBM stacks, DASH provides 1,072 GB of nominal capacity, compared with 96 GB for four 24 GB HBM stacks. Under the illustrative assumption that one HBF stack costs the same as one HBM stack and that DASH adds no incremental integration cost, the nominal capacity-per-cost gain is 11.17x. The paper does not present this as a measured economic result. It instead derives the condition under which the gain exceeds 1x: the normalized HBF cost and incremental integration cost must satisfy the stated parametric bound. Actual package yield, PHY area, cooling, controller complexity, and HBF pricing remain unresolved.
The endurance analysis is similarly qualified. Assuming 100,000 program/erase cycles, a write rate of 1,015.13 MB/s, a writable KV capacity of 206.58 GB, unit write amplification, uniform wear, and full-time utilization, the projected lifetime is 0.645 years. This is a stress-case projection, not a device qualification. The authors correctly note that deployed lifetime requires measured WAF, erase-count distributions, bad-block growth, and workload-specific duty cycles. The result nevertheless demonstrates that endurance can be a first-order deployment constraint under sustained long-prefill workloads, even when decode writes remain in HBM.
Limitations and open questions
The evaluation is simulation-based. GPU operator timing is empirically profiled, but no physical DASH prototype, HBF device, UCIe package, or measured base-die implementation is reported. The HBF configuration relies on public targets and assumptions about subarray-level parallelism, page-wave organization, NAND timing, SRAM buffering, and link provisioning. Consequently, the reported throughput and latency values establish modeled system behavior rather than demonstrated hardware performance.
The exact early-routing transformation is limited to routers with the required algebraic structure. DeepSeek-V3, one of the evaluated models, does not qualify for early selection because its router violates the paper’s assumptions. The paper shows that dual-path delivery remains applicable in this case, but it does not quantify a complete alternative mechanism for eliminating late-selection latency.
The serving evaluation uses balanced per-expert rows and controlled request traces. Real deployments may exhibit expert popularity skew, correlated routing, bursty arrivals, heterogeneous prompt distributions, preemption, and admission policies that alter route contention and HBF wear. The paper’s route assignment strategy assigns each active expert wholly to Direct or Relay; the effectiveness of more granular tile-level routing, replication, or congestion-aware migration is left open.
Finally, the endurance projection exposes a substantial unresolved systems issue. Under the specified stress assumptions, the projected continuous-activity lifetime is less than one year. Since the estimate assumes unit WAF and idealized wear distribution, actual behavior could be better or worse. The paper leaves open how HBF firmware, overprovisioning, wear leveling, KV eviction, request admission, and workload shaping should be co-designed to satisfy a target service lifetime.
Conclusion
DASH proposes a coherent architecture–runtime solution for MoE inference beyond HBM capacity. Its main architectural contribution is the concurrent use of Direct GPU–HBF and HBF–HBM–GPU Relay routes, while its main runtime contributions are exact early expert determination and phase-aware KV-cache write scheduling. Across modeled workloads, the combination produces approximately 1.8–1.9x throughput improvements and 35.6–44.8% E2E latency reductions over single-route designs, with larger gains in several long-context and continuous-batching scenarios.
The paper’s strongest conclusion is conditional rather than universal: HBF can function as a high-performance main memory tier for MoE inference when independent delivery paths, parallel expert placement, NAND-aware writes, and router-aware lookahead are jointly implemented. Whether these modeled benefits survive the physical constraints of HBF endurance, package integration, controller design, and workload skew remains the principal question left open by the work.
Paper Prompts
Sign up for free to create and run prompts on this paper.
Top Community Prompts
Explain it Like I'm 14
1. What is this paper about?
This paper presents DASH, a new way to help very LLMs run faster and more cheaply.
Some modern LLMs use a design called a Mixture of Experts, or MoE. These models contain many smaller “expert” networks, but each word or token uses only a few of them. This saves computing work, but the computer still needs to store all the experts.
The problem is that these expert weights can be much larger than the memory built into a GPU. The paper suggests using a new type of memory called High-Bandwidth Flash (HBF). HBF has much more storage than GPU memory, while still being able to read data quickly.
DASH improves HBF by giving the GPU two separate ways to receive data instead of only one.
2. What questions does the research ask?
The researchers mainly wanted to find out:
- Can HBF store the huge expert weights that do not fit in GPU memory?
- Can HBF send expert weights to the GPU quickly enough for language-model generation?
- Can two data paths be used at the same time to increase total speed?
- Can the system hide HBF’s startup delays while the GPU is doing other work?
- Can DASH handle both:
- Expert weights, which are large and mostly read-only, and
- The KV cache, which grows and changes while the model processes long conversations?
- Does DASH improve speed and reduce waiting time compared with systems using only one HBF-to-GPU path?
Here, the KV cache is like the model’s notebook. It stores information from earlier words so the model does not need to calculate everything again for every new word.
3. How did the researchers study the idea?
Understanding the memory problem
The paper focuses on three kinds of memory:
- HBM: Very fast memory placed close to the GPU, but with limited capacity.
- HBF: Much larger memory that can store huge amounts of data, but has longer delays before reading or writing.
- GPU memory access paths: The connections that carry data between the GPU and the memories.
A simple analogy is a kitchen:
- The GPU is the chef.
- HBM is a small counter next to the chef. It is fast, but has little space.
- HBF is a large pantry. It holds much more food, but it takes longer to fetch something.
- The data paths are the routes from the pantry and counter to the chef.
DASH’s two paths
DASH connects HBF to the GPU in two ways:
- Direct path: HBF sends data straight to the GPU.
- Relay path: HBF sends data through the HBM connection before reaching the GPU.
These paths can work at the same time. Different pieces of an expert can travel over different paths, like sending packages by two roads instead of one.
The system does not need to make a second copy of every expert. Instead, it divides expert weights into chunks and sends the chunks through both routes.
Placing different data in different memories
DASH uses different locations depending on how data behaves:
- Large, mostly unchanged expert weights are stored in HBF.
- Frequently changing information is kept in HBM when possible.
- Some smaller, frequently reused model weights are copied into both HBM and HBF.
- KV-cache data is moved from HBM to HBF in large groups instead of many tiny pieces.
This is useful because HBF is better at large reads and writes than at many very small updates.
Starting expert reads early
Normally, the model decides which experts it needs fairly late in the process. This could force the GPU to wait for HBF to start reading the expert weights.
DASH uses lookahead expert execution. It calculates which experts will probably be needed earlier than usual, so HBF can begin reading their weights while the GPU is still finishing other calculations.
This is similar to a student opening the next textbook page before finishing the current problem. When the student is ready, the page is already available.
Simulation rather than a physical prototype
The researchers tested DASH using an event-driven simulator. A simulator is a computer program that imitates how a real system would behave over time.
They modeled:
- Six MoE LLMs, including Qwen3, Mixtral, Grok, Llama 4, and DeepSeek.
- Different batch sizes and prompt lengths.
- Short and very long text generation.
- HBF reading and writing delays.
- The two DASH data paths.
- The growth of the KV cache.
- Continuous batching, where new requests join while other requests are still being processed.
They also measured GPU operation times on an NVIDIA H100 GPU and used those measurements in the simulator.
The main comparisons were:
- RelayOnly: Uses only the HBF–HBM–GPU route.
- DirectOnly: Uses only the direct HBF–GPU route.
- DASH: Uses both routes at the same time.
- Compact-DASH: A smaller version with fewer memory stacks.
4. What did the researchers find?
DASH was faster than using only one path
Across the main tests, DASH consistently performed better than both single-path designs.
For workloads with a batch of requests, DASH achieved approximately:
- 1.90× higher throughput than RelayOnly.
- 1.84× higher throughput than DirectOnly.
- About 42% lower end-to-end latency than RelayOnly.
- About 41% lower end-to-end latency than DirectOnly.
Throughput means how much work the system completes in a certain amount of time. Latency means how long a user waits for a result.
For different prompt and generation lengths, DASH still achieved strong results:
- About 1.79× higher throughput than RelayOnly.
- About 1.63× higher throughput than DirectOnly.
- Around 40% lower latency than RelayOnly.
DASH handled very large KV caches
In one test with Llama 4, the KV cache grew to about 197 GB. This is far too large for the available HBM alone.
DASH moved older KV-cache information to HBF while keeping newer, smaller updates in HBM. Even in this difficult case, DASH had:
- 1.92× higher throughput than RelayOnly.
- 48% lower end-to-end latency.
This shows that DASH is not only useful for storing expert weights. It can also manage the growing memory needed for long conversations.
Early expert selection reduced delays
The early expert-selection method helped the system avoid waiting for HBF reads.
When the assumed HBF read delay was small, the improvement was modest. When the delay was increased to 32 microseconds, early selection reduced:
- End-to-end latency by about 9% for Qwen3.
- Time per generated token by about 11% for Qwen3.
- Similar amounts for DeepSeek-V2.
The improvement comes from overlapping memory work with computation instead of performing the tasks one after another.
DASH was much faster than CPU offloading
Another possible solution is to store expert weights in regular computer memory and send them to the GPU, or even run some experts on the CPU.
The paper found that these CPU-based approaches were 8.22 to 12.32 times slower than DASH in the tested cases. The main reasons are:
- The CPU is slower at this work.
- Data must travel between the CPU and GPU.
- These transfers create extra waiting.
The smaller version also worked well
Compact-DASH used fewer HBM and HBF memory stacks. Even so, it performed similarly to the single-path systems.
This suggests that DASH might improve performance without requiring the largest possible hardware setup.
5. Why are these results important?
Large MoE LLMs can have hundreds of gigabytes or even more than a terabyte of weights. A single GPU’s HBM cannot store all of that, especially when space is also needed for the KV cache and temporary calculations.
DASH offers a way to:
- Store large expert models without adding many extra GPUs.
- Use HBF’s large capacity while reducing its delay problems.
- Make better use of the connections between memory and the GPU.
- Keep the GPU busy instead of making it wait for expert weights.
- Support long prompts and long conversations.
- Reduce the need to move data through slower CPUs or storage devices.
The key idea is that HBF should not be treated merely as a slow backup storage area. With the right connections and scheduling, it can become an important part of the main memory system.
Conclusion and possible impact
The paper argues that using two independent HBF-to-GPU routes at the same time can make large MoE LLMs considerably faster. DASH also improves performance by reading experts early, organizing data carefully, and grouping small KV-cache updates into larger writes.
The results are based on simulation and measured GPU timings, not on a complete physical DASH chip. Therefore, real hardware may perform somewhat differently. Still, the study suggests a promising direction for future AI systems.
If built in practice, DASH could help companies run very LLMs with fewer GPUs, lower memory costs, and better support for long-context applications such as extended conversations, document analysis, and software assistants.
Knowledge Gaps
Knowledge Gaps, Limitations, and Open Questions
The paper leaves the following issues unresolved:
- Lack of hardware prototype validation: DASH is evaluated exclusively through simulation; the feasibility of integrating GPU, HBM, and HBF base dies with three independent UCIe links remains undemonstrated.
- Unverified HBF assumptions: Key parameters—including the assumed $1.6$ TB/s usable link bandwidth, read latency, program latency, four subarrays per plane, and SRAM organization—are based on specifications, patents, or assumptions rather than measurements from a production HBF device.
- Incomplete physical-design analysis: The paper does not quantify the area, routing complexity, thermal impact, signal integrity, power-delivery requirements, or packaging constraints of attaching multiple high-bandwidth UCIe links to the GPU and memory base dies.
- No end-to-end energy evaluation: Although link power is estimated, the study does not report total system energy, energy per generated token, HBF controller power, NAND sensing/program power, GPU power changes, or the energy cost of dual-path transfers.
- Cost claims are not substantiated: The paper claims capacity-per-cost and area-efficiency benefits but does not provide a complete cost model covering HBF stacks, HBM stacks, UCIe PHYs, base-die SRAM, packaging, cooling, controllers, and manufacturing yield.
- Limited workload diversity: Evaluation focuses on six MoE models and a small set of fixed input/output lengths; broader workloads involving varying prompt distributions, generation lengths, context lengths, batch sizes, sparsity patterns, and request arrival processes remain unexplored.
- Restricted continuous-batching evaluation: The continuous-batching study appears to focus primarily on Qwen3 with one arrival configuration, leaving the behavior of DASH under diverse traffic distributions, bursty arrivals, heterogeneous service-level objectives, and other MoE architectures uncertain.
- Idealized expert-routing distribution: The continuous-batching experiments use analytically balanced per-expert rows, which may understate contention, load imbalance, hot-expert effects, and path skew produced by real user inputs and router behavior.
- Unclear impact of expert-weight reuse and caching: The evaluation does not fully characterize how repeated expert selections, inter-request locality, expert caching policies, or partial expert residency in HBM affect DASH’s relative advantage.
- Limited baseline coverage: Comparisons omit potentially competitive alternatives such as multi-GPU systems with optimized expert parallelism, CPU/NVMe offloading with asynchronous prefetching, compressed or quantized expert caching, expert replication, and other near-memory or CXL-based designs.
- Potentially favorable baseline configuration: All configurations use the same placement policy and HBF-resident experts, but the paper does not establish whether RelayOnly, DirectOnly, or CPU-based baselines were independently optimized for placement, prefetching, buffering, routing, and scheduling.
- No comparison with state-of-the-art serving engines: The simulator-based results are not validated against optimized implementations such as production continuous-batching runtimes, expert-parallel serving systems, or GPU kernels designed specifically for MoE inference.
- Unvalidated GPU performance modeling under contention: GPU operator latencies are profiled in isolation on an H100 and interpolated over validated ranges, but the study does not establish how accurate these timings remain when simultaneous HBM/HBF transfers, DMA activity, synchronization, and memory-system contention occur.
- Router transformation applicability is limited: Lookahead Expert Execution applies only to scale-invariant routers without expert-specific additive bias; the paper does not quantify how often modern MoE models satisfy these conditions or develop a general method for routers that do not.
- No accuracy or numerical-equivalence validation: The early-routing transformation is argued to preserve top- ranking, but the paper does not report numerical comparisons across precisions, quantization schemes, ties, near-ties, router numerical errors, or full-model output equivalence.
- Additional routing overhead is not fully assessed: Computing and storing the transformed routing weights and performing FP32 early selection may introduce extra memory traffic, storage overhead, compute cost, and implementation complexity that are not separately quantified.
- Speculative or incorrect expert reads are not addressed: The paper does not explain how the system handles early decisions that become unavailable, invalid, or changed under routers with stochasticity, numerical nondeterminism, dynamic policies, or model variants requiring late routing.
- Scheduling policy details are insufficient: The criteria for assigning expert chunks to the direct versus relay path, resolving path contention, prioritizing KV traffic versus expert weights, and preventing starvation are not specified sufficiently to reproduce or analyze the scheduler.
- Buffer-sizing sensitivity is incomplete: DASH relies on two interleaved HBF transfer regions and two HBM relay regions, but the paper does not evaluate performance, backpressure, or latency sensitivity as SRAM capacity and buffering depth vary.
- HBF write endurance is not quantified: KV-cache write frequency, write amplification from out-of-place updates and garbage collection, block wear distribution, expected device lifetime, and refresh or wear-leveling overhead are not modeled in detail.
- Garbage-collection behavior is simplified: Separating weights and KV cache by erase block reduces interference, but the paper does not evaluate fragmentation, free-block exhaustion, garbage-collection pauses, migration bandwidth, or workload-dependent write amplification over long-running service periods.
- Failure and reliability mechanisms are absent: The design does not discuss NAND bad blocks, ECC failures, link errors, die failures, SRAM failures, data loss, degraded-path operation, or recovery mechanisms for persistent model weights and KV cache.
- Model loading and initialization costs are omitted: The time and bandwidth required to load hundreds of gigabytes of model weights into HBF, initialize metadata, construct placement mappings, and reach a service-ready state are not reported.
- Dynamic model management is unexplored: The architecture is evaluated with a fixed model, while multi-tenant serving, model swapping, model updates, adapter/LoRA loading, and concurrent models may substantially alter capacity and traffic demands.
- KV-cache management is not evaluated across more policies: The proposed HBM accumulation and HBF writeback strategy is not compared with paging, eviction, compression, recomputation, prefix sharing, or token-level KV-cache policies.
- No analysis of long-term online operation: Results use finite inference traces, so sustained throughput degradation from accumulated KV data, garbage collection, wear leveling, fragmentation, and changing request populations remains unknown.
- Scalability beyond the evaluated configuration is uncertain: The paper evaluates two HBF and two HBM stacks, but does not establish how performance scales with larger stack counts, multiple GPUs, multiple GPU I/O dies, heterogeneous link widths, or shared HBF resources.
- Interference among multiple concurrent GPUs is unexamined: The proposed topology and schedulers are not evaluated under multi-GPU access to shared HBF stacks, leaving contention, isolation, fairness, and routing scalability unresolved.
- Memory-coherence and software integration are unspecified: The paper does not explain how address translation, page metadata, cache coherence, synchronization, memory protection, and GPU runtime support would be implemented for HBF-resident weights and KV cache.
- Quality-of-service behavior is incomplete: Average throughput and selected P90 results are reported, but fairness across requests, P99 and worst-case latency, deadline violations, admission control, and isolation between latency-sensitive and throughput-oriented workloads are not fully analyzed.
- Sensitivity analysis is not comprehensive: The study varies read latency and selected workload parameters, but does not jointly evaluate bandwidth asymmetry, program bandwidth, SRAM size, link failures, HBF capacity, ECC overhead, expert size, routing sparsity, and GPU/HBM bandwidth.
- Benefits under compute-bound workloads are unclear: Since DASH primarily improves expert-data delivery, its advantage when inference is compute-bound, when expert weights are highly quantized, or when GPU compute latency dominates memory transfers remains uncertain.
Practical Applications
Immediate Applications
- Large-scale MoE LLM serving with reduced GPU count — Sector: cloud computing, AI infrastructure, software. Deploy DASH-like systems to host expert weights in high-bandwidth flash while retaining frequently updated activations and hot data in HBM. This can support models whose weights exceed a single GPU’s HBM capacity without adding GPUs solely for memory capacity. The reported results indicate approximately 1.8–1.9× throughput improvements and 35–45% lower end-to-end latency relative to single-path HBF designs. Potential products/workflows: inference servers, accelerator cards with HBM/HBF memory, and runtime integrations for frameworks such as vLLM, TensorRT-LLM, or custom continuous-batching engines. Dependencies: commercially available HBF devices, GPU support for independent memory paths, UCIe-compatible packaging, sufficiently high HBF read bandwidth, and model-specific expert placement.
- Capacity expansion for long-context inference — Sector: generative AI, enterprise search, coding assistants. Use HBF as a persistent tier for large KV caches generated by long prompts and extended decoding. HBM can absorb fine-grained decode updates, while page-aligned batches are written back to HBF when HBM approaches capacity. This is particularly applicable to document analysis, long-context question answering, repository-scale code generation, and multi-turn assistants. Potential tools: KV-cache managers that classify data by update frequency, perform page-level writeback, and maintain HBM/HBF address mappings. Dependencies: workload locality, acceptable flash endurance, effective garbage-collection isolation, and application tolerance for the latency of cache migration.
- Dual-path expert-weight streaming — Sector: AI accelerators and inference software. Partition each expert across HBF dies and planes, then schedule different chunks over the direct GPU–HBF path and the HBF–HBM–GPU relay path. This can be implemented as a memory-controller or runtime scheduling policy without replicating all expert weights. Potential products: expert-aware DMA engines, HBF memory controllers, and runtime schedulers that dynamically assign transfers according to path availability and buffer occupancy. Dependencies: independent link bandwidth, sufficient SRAM buffering, balanced expert striping, and a scheduler capable of avoiding contention between expert and KV traffic.
- Continuous-batching optimization for online inference — Sector: cloud services and model serving. Integrate DASH’s memory-aware scheduling into servers that mix prefill and decode requests. The scheduler can reserve resources for latency-sensitive decode tokens, admit prefill chunks opportunistically, and coordinate expert reads with KV-cache writes. This is useful for chat APIs and other workloads with asynchronous arrivals and heterogeneous prompt/output lengths. Potential workflow: jointly optimize batch admission, expert-path assignment, KV writeback, and memory credits at each iteration boundary. Dependencies: accurate GPU operator timing, request-arrival prediction or feedback control, and mechanisms to prevent large prefills from degrading P90/P99 decode latency.
- Early expert selection for compatible MoE routers — Sector: model runtime optimization. For routers whose top- ranking is invariant to the shared RMS scaling factor and that have no expert-specific additive bias, compute the input-dependent and attention-dependent routing terms earlier than conventional execution. Expert reads can then begin before output projection and final normalization complete. Potential tools: compiler passes or model-conversion utilities that precompute transformed routing weights such as and insert lookahead routing kernels. Dependencies: mathematical equivalence of the transformed router, FP32 or otherwise sufficiently accurate routing computation, and validation that early selection does not alter top- choices. The paper explicitly indicates that this method is not directly applicable to routers such as DeepSeek-V3’s when their routing rules violate these assumptions.
- Memory-aware placement of model tensors — Sector: AI systems and compiler/runtime design. Apply the paper’s placement policy to inference runtimes: keep mutable activations and hot state in HBM, store large read-mostly expert weights in HBF, and replicate smaller universally reused attention weights across both memories. Potential tools: automated placement planners based on tensor size, reuse distance, mutability, and access granularity. Dependencies: reliable profiling of model access patterns, sufficient HBM space for working data, and support for replicated-weight consistency during model loading.
- Flash-aware KV-cache management — Sector: databases, operating systems, and AI serving. Separate model-weight blocks from KV-cache blocks at erase-block granularity so that KV garbage collection does not relocate immutable expert weights. This can reduce write amplification and simplify metadata management. Potential products: flash translation layers specialized for AI inference, KV allocators with block ownership, and endurance-aware cache eviction systems. Dependencies: firmware-level control over block allocation and garbage collection, page-aligned allocation, and careful monitoring of program/erase wear.
- Infrastructure planning and cost modeling for AI data centers — Sector: cloud economics and policy. Use the reported comparisons against multi-GPU and CPU-offload approaches to evaluate whether memory capacity should be expanded with HBF rather than additional GPUs. HBF-based designs may be attractive when inference is memory-bandwidth bound and additional GPU compute would remain underutilized. Potential workflow: capacity-per-cost analysis incorporating GPU utilization, UCIe link power, HBF endurance, packaging, and cooling. Dependencies: actual HBF prices and power characteristics were not reported; the paper’s link and bandwidth figures are partly specification-based or modeled rather than measured in a complete prototype.
- Academic simulation and systems research platform — Sector: academia. Extend the described event-driven simulator to study alternative HBF organizations, routing algorithms, link allocations, KV policies, and model architectures. Researchers can reproduce sensitivity studies over read latency, program latency, batch size, context length, and arrival rates. Potential outputs: open-source simulators, benchmark suites, and design-space exploration tools for heterogeneous GPU memory. Dependencies: access to the simulator implementation, validated HBF device parameters, and real hardware measurements to replace assumptions such as four subarrays per plane and nominal s read/s program latencies.
Long-Term Applications
- Commercial GPU or accelerator packages with integrated HBM–HBF memory — Sector: semiconductor hardware and data-center infrastructure. Develop accelerator packages in which HBM and HBF are first-class, independently addressable memory tiers connected through UCIe or a similar die-to-die fabric. Such products could target trillion-parameter or multi-hundred-billion-parameter MoE models while reducing reliance on multi-GPU model parallelism. Dependencies: HBF commercialization, thermal and power feasibility, packaging yield, signal integrity, ECC, memory coherence, and production-grade controller firmware. The paper evaluates the architecture through simulation rather than a fabricated system.
- General-purpose heterogeneous memory operating systems — Sector: operating systems and runtime infrastructure. Build memory managers that automatically classify tensors, KV entries, optimizer state, embeddings, and intermediate data across HBM, HBF, CPU memory, and storage. The same principles could extend beyond inference to training checkpoints, retrieval indexes, and multimodal model state. Dependencies: hardware support for transparent addressing and migration, robust telemetry, predictable access-pattern analysis, and policies that account for flash wear as well as latency.
- Low-cost deployment of private and sovereign AI services — Sector: public policy, regulated industries, and enterprise IT. HBF-enabled inference appliances could make large MoE models deployable in hospitals, universities, government agencies, and businesses that cannot justify clusters of high-end GPUs. Keeping expert weights locally resident may also reduce dependence on external cloud APIs and support data-sovereignty requirements. Potential products: on-premises AI appliances for medical documentation, legal search, government archives, and enterprise coding. Dependencies: validated total cost of ownership, security of persistent model storage, encryption and secure boot, sufficient throughput under real multi-tenant workloads, and regulatory certification where applicable.
- Long-context healthcare and scientific assistants — Sector: healthcare, life sciences, and academia. Persistent HBF-backed KV caches could support analysis of large longitudinal patient records, genomic documents, scientific corpora, or instrument logs. The architecture is especially relevant when sessions require very long contexts or sustained generation. Dependencies: privacy-preserving storage, access control, clinical validation, predictable latency under concurrent users, and safeguards against stale or improperly retained conversational state. This is an infrastructure opportunity, not evidence that the paper’s results directly improve medical accuracy.
- Robotics and embodied AI with large expert libraries — Sector: robotics and autonomous systems. Future robots could maintain large collections of task-, environment-, or modality-specific experts in HBF and stream only the selected experts to an onboard accelerator. Dual-path delivery could reduce the need to keep every expert in scarce local HBM. Potential tools: expert libraries for navigation, manipulation, perception, and language control, managed by a real-time expert prefetcher. Dependencies: much stricter worst-case latency than the workloads evaluated in the paper, reliable routing predictions, ruggedized flash, low-power packaging, and validation under sensor-driven distribution shifts.
- Energy-efficient inference at scale — Sector: energy and sustainability. If HBF allows fewer GPUs to satisfy model-capacity requirements, data centers may reduce embodied hardware costs and potentially energy per generated token. The architecture could also support power-aware scheduling that shifts traffic between direct and relay paths according to link utilization. Dependencies: complete system-level power measurements are needed. The paper estimates approximately 6.4–7.7 W per fully utilized 1.6 TB/s UCIe direction, but this excludes all accelerator, HBF, cooling, controller, and packaging costs.
- Hardware–software co-design for routing-aware models — Sector: machine learning research and compiler design. MoE architectures could be designed with routing functions that support safe lookahead selection, predictable expert access, and balanced expert placement. Training objectives might additionally penalize routing patterns that create severe path or die imbalance. Potential innovations: routers optimized jointly for mathematical accuracy, expert locality, HBF parallelism, and tail-latency behavior. Dependencies: changes must preserve model quality, routing fairness, and robustness; early selection is only exact for a restricted class of routers, so broader applicability requires new algorithms or approximate-selection guarantees.
- Persistent AI memory and session-resumption systems — Sector: consumer software and daily life. HBF could support long-lived assistant sessions, personal knowledge bases, and locally stored conversation or application context without consuming all HBM. A device could retain large context histories and load relevant portions as needed. Dependencies: consumer-grade HBF hardware, strong privacy controls, efficient relevance-based retrieval, flash endurance under frequent updates, and user controls for deletion and data retention. The paper’s results concern server-class inference and do not establish feasibility on phones or laptops.
- Policy and procurement standards for AI memory infrastructure — Sector: technology policy and public procurement. Organizations could develop evaluation standards that compare GPU scaling, CPU offload, SSD offload, and HBF-based systems using throughput, P90/P99 latency, energy per token, cost per token, endurance, and capacity utilization. This would help public-sector and enterprise buyers avoid selecting hardware solely by GPU count. Dependencies: independent measurements on real HBF hardware, standardized MoE and long-context workloads, transparent pricing, and lifecycle data covering replacement rates and e-waste.
Glossary
- Activations: Intermediate numerical representations produced during neural-network computation. “Each model's weights alone exceed the NVIDIA H100's 80\,GB HBM capacity~\cite{nvidia_h100}, even before accounting for KV caches or activations.”
- Autoregressive decoding: Sequential generation in which each newly generated token depends on previously generated tokens. “Because autoregressive decoding is often memory-bandwidth bound, GPUs added primarily to satisfy memory-capacity requirements can leave much of their compute underutilized~\cite{pope2023scaling,sarathi_serve}.”
- BF16 (bfloat16): A 16-bit floating-point format designed to retain a relatively wide numerical range for deep-learning workloads. “Qwen3-235B-A22B & GQA & 470.19 & 128+0 / 8 & BF16”
- Continuous batching: Dynamic batching technique that admits and removes requests while inference is ongoing. “To maintain GPU utilization, modern serving systems commonly use continuous batching, admitting and removing requests at token boundaries rather than waiting for a fixed batch to complete~\cite{yu2022orca,sarathi_serve}.”
- D2D (die-to-die): Communication between semiconductor dies, typically through a high-speed interconnect. “DASH equips the GPU I/O die and both memory base dies with UCIe 3.0 physical layers (PHYs) and die-to-die (D2D) adapters, providing a common transport across the three data paths~\cite{ucie_spec3,sharma2022universal,ucie_sip_2024}.”
- Double buffering: Technique that uses two buffers so one can be processed while the other is being filled. “NAND sensing and shared-TSV SRAM fills are accounted for separately from this D2D rate and overlapped through double buffering.”
- ECC (error-correcting code): Redundant information used to detect and correct errors in stored or transmitted data. “The additional 2\,MiB accounts for 12.5\% ECC check-bit storage, assuming eight check bits per 64 data bits.”
- Endurance: The amount of program/erase activity that nonvolatile memory can withstand before reliability degrades. “Finally, finite program/erase endurance makes sustained writes a lifetime concern~\cite{hong2022guardederase}.”
- Expert routing: The process of assigning tokens to selected subnetworks, or experts, in a mixture-of-experts model. “Because MoE routing is input-dependent, the selected experts are not known until routing completes~\cite{shazeer2017moe,fedus2022switch,jiang2024mixtral,deepseekv2,deepseekv3}.”
- FP8: An 8-bit floating-point representation used to reduce the storage and computation cost of neural-network operations. “DeepSeek-V3 & MLA & 688.59 & 256+1 / 8 & FP8-mixed”
- Garbage collection: Flash-memory process that relocates live pages and erases blocks containing obsolete data. “Because HBF erases at block granularity, garbage collection may relocate live pages~\cite{agrawal2008ssd}.”
- GQA (grouped-query attention): Attention mechanism in which multiple query heads share key and value heads. “The attention block~\cite{vaswani2017attention} performs QKV and output projections while reading and updating the KV cache, whose capacity grows with context length despite the reduced number of KV heads in GQA~\cite{gqa} and MQA~\cite{mqa}.”
- HBM (high-bandwidth memory): A vertically stacked memory technology providing high data bandwidth near a processor. “DASH integrates HBM and HBF.”
- HBF (high-bandwidth flash): Die-stacked NAND flash architecture designed to combine high storage capacity with substantial parallel read bandwidth. “High Bandwidth Flash (HBF)~\cite{sandisk_hbf} offers a promising approach to alleviating this capacity wall.”
- HBM-to-HBF writeback: Transfer of accumulated data from high-bandwidth memory to high-bandwidth flash. “Once HBM approaches its capacity limit and each HBM stack has accumulated its partition of a full page wave, DASH writes the wave back in parallel over the HBM--HBF paths, as shown in Figure~\ref{fig:fig_6}(c).”
- HBM base die: The foundational logic die that controls or routes communication to stacked HBM memory dies. “This path supports HBM-to-HBF writeback and relays HBF-resident reads through the HBM base die without accessing memory cells.”
- HBM-mediated relay path: A route that transfers data from HBF through HBM-side logic to the GPU without using HBM storage cells. “At each MoE layer, selected-expert transfers can use either the GPU--HBF Direct path or the HBM-mediated Relay path, enabling effective utilization of both transfer paths even within a single HBM/HBF pair.”
- KV cache: Stored key and value tensors used to avoid recomputing prior context during transformer decoding. “Long-context requests generate larger key--value (KV) caches that must be retained throughout decoding~\cite{deepseekv4,llama4_maverick}.”
- Lookahead expert execution: Early identification and fetching of experts before conventional routing finishes. “DASH extends HBF latency hiding to input-dependent MoE expert accesses.”
- MLA (multi-head latent attention): Attention mechanism that compresses key-value information into latent representations to reduce cache requirements. “DeepSeek-V3 & MLA & 688.59 & 256+1 / 8 & FP8-mixed”
- MoE (mixture of experts): Neural-network architecture that activates only a subset of specialized subnetworks for each input token. “MoE models increase total parameter capacity by incorporating many experts, while limiting per-token computation to a small subset of activated experts~\cite{MoE, shazeer2017moe,fedus2022switch}.”
- MQA (multi-query attention): Attention mechanism in which all query heads share a single key and value head. “The attention block~\cite{vaswani2017attention} performs QKV and output projections while reading and updating the KV cache, whose capacity grows with context length despite the reduced number of KV heads in GQA~\cite{gqa} and MQA~\cite{mqa}.”
- NAND flash: Nonvolatile memory technology that stores data in electrically programmable memory cells arranged in pages and blocks. “HBF is a die-stacked NAND memory that combines high capacity with aggregate read bandwidth from concurrent accesses across multiple dies and planes.”
- NUMA (non-uniform memory access): Computer architecture in which memory-access latency depends on the processor or memory location. “We measure a shape-equivalent Qwen3 BF16 expert layer on an Intel Xeon Platinum 8452Y in the H100-local NUMA node using physical cores~\cite{intel8452y}.”
- Out-of-place writes: Flash-storage updates performed by writing new data elsewhere rather than overwriting the original page. “HBF retains NAND semantics: reads and programs operate on pages, erasure occurs at block granularity, and updates require out-of-place writes followed by garbage collection~\cite{kioxia_tc58_2019,agrawal2008ssd}.”
- Page buffer: Temporary memory inside a flash device that holds data sensed from a NAND page before transfer. “First, each read must sense a page from the NAND array into a page buffer before data transfer begins, incurring a startup latency of ~\cite{kioxia_tc58_2019,sandisk_hbf_patent_2025}.”
- Page granularity: The minimum unit at which flash data is read or programmed. “HBF retains NAND semantics: reads and programs operate on pages, erasure occurs at block granularity, and updates require out-of-place writes followed by garbage collection~\cite{kioxia_tc58_2019,agrawal2008ssd}.”
- Page wave: Aggregate page-sized data volume programmed or transferred concurrently across multiple flash components. “A full HBF page wave is the aggregate data volume that can be programmed concurrently across all HBF stacks, dies, and planes.”
- Pipeline bubbles: Idle intervals in a pipelined computation caused by dependencies or insufficient work. “Tensor parallelism incurs communication and synchronization overhead from inter-GPU collectives, whereas pipeline parallelism adds inter-stage transfers and pipeline bubbles~\cite{pope2023scaling,narayanan2021megatron}.”
- Prefill: Transformer-inference phase that processes the input prompt and constructs its initial KV cache. “Prefill generates large KV writes that can be programmed directly, whereas decode produces small updates that must be buffered and coalesced into pages.”
- Program latency: Time required to program, or write, data into flash memory. “HBF programming has substantially higher latency and lower bandwidth than reading, with $t_{\mathrm{PROG}$ typically ranging from tens to hundreds of microseconds (e.g., 100\,s), so fine-grained writes can stall inference~\cite{kioxia_tc58_2019}.”
- Residual connection: Neural-network connection that adds a layer’s input to its output. “Figure~\ref{fig:fig_3} shows an MoE Transformer layer~\cite{vaswani2017attention,shazeer2017moe,fedus2022switch}, which consists of an attention block followed by an MoE block, with RMS normalization~\cite{rmsnorm2019} and residual connections~\cite{he2016deep}.”
- RMSNorm (root mean square normalization): Normalization method that scales activations using their root-mean-square magnitude without subtracting the mean. “The conventional routing logits are”
- SRAM (static random-access memory): Fast volatile memory used for temporary buffering and caching. “It contains an HBF controller, a local scheduler, and 18\,MiB of physical banked SRAM per stack.”
- Tail latency: Latency experienced by high-percentile requests, often measured using metrics such as P90 or P99. “We integrate DASH into a continuous-batching serving simulator that co-schedules decode tokens and chunked prefills at iteration boundaries, improving throughput and tail latency under dynamic arrivals.”
- Tensor parallelism: Model-parallel technique that partitions tensor operations across multiple processors or GPUs. “Tensor parallelism incurs communication and synchronization overhead from inter-GPU collectives, whereas pipeline parallelism adds inter-stage transfers and pipeline bubbles~\cite{pope2023scaling,narayanan2021megatron}.”
- Through-silicon vias (TSVs): Vertical electrical connections passing through stacked semiconductor dies. “HBF stack vertically integrates multiple flash core dies above a logic base die, with the dies interconnected by through-silicon vias (TSVs).”
- Top- selection: Choosing the highest-scoring experts for each input token. “DASH determines the top- experts without waiting for the RMS value.”
- TPOT (time per output token): Average time required to generate each output token during inference. “At 3\,s, expert early decision reduces E2E latency by 3.33\% and 1.99\%, and TPOT by 3.86\% and 2.45\% for Qwen3 and DeepSeek-V2, respectively.”
- UCIe (Universal Chiplet Interconnect Express): Standardized high-speed interconnect for communication among chiplets and dies. “DASH attaches both HBM and HBF to the GPU through independent UCIe links~\cite{sharma2022universal}, allowing each memory to serve GPU requests without relying on the other.”
- Write amplification: Increase in physical storage writes relative to the amount of logical data written, often caused by garbage collection and out-of-place updates. “During garbage collection, a KV-owned block is erased directly if it contains no live KV pages.”
