NPU Memory Virtualization

• NPU Memory Virtualization is a set of hardware and software techniques that decouple neural accelerator memory views from underlying DRAM/HBM, enabling address translation, multi-tenant isolation, and efficient resource sharing.
• It employs diverse methods such as page-based translation (NeuMMU) and range-based schemes (vChunk) to optimize DMA-driven DNN workloads by reducing translation latency and overhead.
• The approach balances performance, hardware cost, and security, playing a crucial role in scaling multi-core heterogeneous systems and supporting dynamic deep learning applications.

Neural Processing Unit (NPU) memory virtualization encompasses a set of hardware and software mechanisms that decouple a neural accelerator's memory view from underlying physical DRAM or HBM resources. Effective NPU memory virtualization enables address translation, multi-tenant isolation, oversubscription, and NUMA-style data sharing for deep neural network (DNN) workloads. Recent literature details diverse architectures for memory virtualization in NPUs, especially as the deployment scope shifts from monolithic, scratchpad-centric accelerators to inter-core connected and heterogeneous systems. The principal techniques include page-based virtual memory translation (as in NeuMMU), range-based schemes optimized for DMA-driven bulk transfers (as in vNPU's vChunk), as well as non-virtualized approaches that rely on flat physical address mapping (IANUS).

1. Architectural Challenges in NPU Memory Virtualization

The architectural context and memory access patterns of NPUs impose nontrivial challenges for virtualization. Unlike CPUs and GPUs, which interleave memory operations with computation and amortize translation through TLB locality, modern NPUs allocate almost all on-chip SRAM as software-managed scratchpads. These scratchpads are loaded via large DMA-driven tile fetches from off-chip DRAM/HBM, causing thousands of simultaneous translation requests per tile (Hyun et al., 2019). Conventional IOMMU approaches—modest TLBs and a handful of page-table walkers—become bottlenecked, resulting in up to 95% performance degradation relative to an ideal, oracle MMU.

For inter-core connected NPUs (Graphcore IPU, Tenstorrent, vNPU), granular DMA-driven workloads exhibit highly bursty, coarse-grained memory access patterns. Page-based translation introduces excessive overhead due to low TLB hit rates and frequent page-table walks during large tile fetch bursts (Feng et al., 13 Jun 2025).

IANUS, in contrast, eschews virtual memory entirely within the NPU, exposing a flat physical DRAM address space, thus bypassing translation challenges but also relinquishing OS-level protections and sharing (Seo et al., 2024).

2. Representative Microarchitectures

NeuMMU: Throughput-First Virtual Memory Support

NeuMMU introduces three principal mechanisms designed for tile-oriented NPUs (Hyun et al., 2019):

• Pending-Request Merging Buffer (PRMB): Each of the 128 massively parallel page-table walkers (PTWs) is paired with a 32-entry fully-associative merge buffer. When multiple outstanding tile burst requests map to the same virtual page, PRMB collapses them, ensuring only one walk per distinct VPN.
• Lightweight Translation-Path Registers (TPreg): Each PTW maintains a 16B register that caches top levels (L4→L2) of the x86-64 page-table. Matching entries bypass upper-level walks, reducing the average walk cost by approximately 2.5×.
• Shared Fully-Associative IOMMU TLB: 2K 4 KB-page entries, 5-cycle hit latency.

Combined, NeuMMU achieves a 0.06% slowdown versus an oracle MMU while reducing MMU energy by 16.3× compared to GPU-style baselines.

vNPU/vChunk: Range-Based Memory Virtualization

vNPU's vChunk component for inter-core connected NPUs departs from fine-grained page-based translation by employing coarse-grained, range-based mechanisms (Feng et al., 13 Jun 2025):

• Range Translation Table (RTT): Each core maintains a 128-entry RTT mapping virtual address base and size pairs directly to physical base addresses.
• Range-TLB (R-TLB): Small associative caches (4 entries) accelerating lookups for recently accessed ranges.
• Translation Path: On DMA launch, the translation unit checks the R-TLB, refilling from the RTT via a last-use hint and linear scan if necessary.
• Address Permissions: RTT supports per-range read/write flags, allowing enforcement of software-level isolation.

Translation stalls only occur on R-TLB misses. Patterned access within ML workloads keeps refill overhead low, maintaining 95–98% of physical memory bandwidth and avoiding the page-walk thrashing typical of page-based methods.

IANUS: Unified Physical Memory Addressing

IANUS forgoes in-hardware memory virtualization entirely. The NPU DMA engine and PIM computations access a unified 48-bit physical address space implemented atop PIM device DRAM; all logical mapping resides in the host OS (Seo et al., 2024). There are no TLBs, page-tables, or hardware translation, and protection/sharing must be software-enforced outside the NPU.

3. Quantitative Translation Models and Overhead

A variety of analytical models quantify translation cost, latency, and effective bandwidth. For NeuMMU, the average translation latency is:

$$L_{\text{avg}} = H_{\text{TLB}} \cdot L_{\text{TLB}} + (1-H_{\text{TLB}})[H_{\text{TP}} \cdot L_{\text{L1}} + (1-H_{\text{TP}}) \cdot L_{\text{walk}}]$$

where $L_{\text{walk}}=400$ cycles, $L_{\text{TLB}}=5$ cycles, $H_{\text{TLB}}$, $H_{\text{TP}}$ are measured hit rates (Hyun et al., 2019).

In vChunk, the average translation latency for a DMA burst is:

$$\langle L \rangle = t_{\text{mem}} + M \cdot t_{\text{ref}}$$

where $M$ is the R-TLB miss rate and $t_{\text{ref}}$ is the refill cost, typically much less than a page-table walk (Feng et al., 13 Jun 2025).

End-to-end experimental benchmarks report:

System	Slowdown (4 TLB)	Slowdown (32 TLB)
Page-based UVM	1.20×	1.09×
Range-based vChunk	1.043×	1.043×

vChunk sustains 95–98% effective bandwidth, while UVM-based NPUs achieve only 60–75% due to translation stalls during DMA bursts.

4. Hardware Cost, Area, and Scalability

Both NeuMMU and vChunk designs emphasize minimal area, power, and logic overhead.

• NeuMMU: 32 KB SRAM for PRMBs, 2 KB for TPregs, 0.10 mm² in 32 nm, <0.01% of FPGA resources, and 13.7 mW leakage (Hyun et al., 2019).
• vChunk: <0.1 mm² in 7 nm per NPU die, under 2% overall FPGA LUT/FF cost, a few mW of dynamic power; RTT and R-TLB collectively occupy under 2 KB per core (Feng et al., 13 Jun 2025).

These costs are orders of magnitude smaller than the on-chip SRAM scratchpads and are negligible at the full-die scale, enabling adoption without meaningful reduction in compute or memory capacity.

5. Security, Sharing, and Multi-Tenancy

Full address translation (NeuMMU, vChunk) enables shared global virtual address spaces, secure per-page (page-table) or per-range (RTT) permissions, and support for OS-level protection domains (Hyun et al., 2019, Feng et al., 13 Jun 2025). Large DNN working sets—common in modern embedding-based workloads—can be demand-paged or migrated on a fine granular basis. NeuMMU's small-page support (4 KB) avoids the fragmentation characteristic of coarse-grained mappings.

vChunk further ensures that each tenant’s DMA accesses are isolated by meta-zone-protected translation metadata, with Table 1 in (Feng et al., 13 Jun 2025) demonstrating <1% inter-tenant slowdown due to translation interference. In contrast, static partitioning (MIG) or shared page-based schemes experience much larger cross-tenant stalls (up to 40% slowdown).

IANUS, lacking in-hardware translation, does not provide isolation or protection at the NPU level; these must be handled at the host or application runtime (Seo et al., 2024).

6. Comparative Summary and Current Limitations

A comparison of key approaches appears below:

Feature	NeuMMU	vChunk	IANUS
Translation Type	Page-based (4 KB)	Range-based	None (physical only)
Virtualization	Full VA->PA, OS	Range-based, per-core	Host-only mmap
Energy Penalty	1/16.3× baseline	Negligible	N/A
Perf. Overhead	0.06% (oracle)	<4.3% (DMA workloads)	N/A
Area/Power	~0.10 mm², 13.7 mW	<0.1 mm², few mW/core	N/A
Isolation	OS page-table	RTT + meta-zone	None (NPU)

Page-based translation offers generality but can be too slow for burst-heavy workloads; range-based virtualization matches DMA-oriented access granularity, providing low-latency, high-bandwidth translation that scales with inter-core NoC architectures. Current limitations include RTT linear scan for vChunk under large range counts, R-TLB sensitivity to highly irregular dynamic workloads, and lack of direct support for OS-level paging in physical-only schemes (IANUS) (Feng et al., 13 Jun 2025, Seo et al., 2024).

7. Outlook and Future Directions

The evolving landscape of NPU memory virtualization is characterized by a trend toward heavier hardware specialization for both translation efficiency and hardware topology awareness. Hybrid models—combining range and page-based translation—may address irregular workloads (e.g., GNNs with less predictable memory reuse) for future systems (Feng et al., 13 Jun 2025). Dynamic resizing of on-chip translation tables and prefetching of vulnerable ranges are suggested optimizations.

As workloads grow toward multi-hundred GB-scale and demand more agile resource sharing, the ability to combine OS-level virtual address spaces, efficient fine-grained translation, and low energy overheads within highly parallel accelerator fabrics will remain central. Standardization of memory metadata, active profiling for translation hot spots, and topology-aware memory mapping are likely research frontiers in the virtualization of next-generation NPUs (Hyun et al., 2019, Feng et al., 13 Jun 2025).