NeuMMU: Optimized MMU for Neural Processors

• NeuMMU is a specialized memory management unit designed for NPUs, enabling rapid virtual-to-physical address translation for high-throughput deep neural network workloads.
• It employs a fully-associative IOTLB, a vast bank of parallel page-table walkers, and request merging buffers to drastically reduce redundant page walks and minimize translation overhead.
• Supporting dual page sizes and unified virtual address spaces, NeuMMU improves demand paging efficiency and cuts energy consumption by over 16× compared to conventional designs.

NeuMMU is a memory management unit (MMU) architecture specifically tailored for high-throughput neural processing units (NPUs) that accelerates virtual-to-physical address translation during deep neural network (DNN) workloads. Unlike GPU-centric designs, NeuMMU accommodates the bursty, large-scale translation demands originating from scratchpad-based NPU dataflows and enables unified virtual address spaces, efficient demand paging, and minimal overhead for deep learning acceleration (Hyun et al., 2019).

1. Architectural Positioning and Components

NeuMMU targets the memory phase of modern NPUs, in which the direct memory access (DMA) engine issues virtual address (VA) translation requests to fetch tiles into the on-chip scratchpad memory (SPM) prior to computation. The translation pipeline operates as follows: the CPU issues kernel calls with VAs, the DMA divides tensors into tiles, and for each tile, translation requests are sent to NeuMMU. NeuMMU performs VA-to-physical address (PA) resolution primarily using a fully-associative single-level I/O translation lookaside buffer (IOTLB) backed by a large bank of parallel page-table walkers (PTWs).

The main NeuMMU architectural blocks are:

• IOTLB: single-level, fully-associative buffer (2K entries, 5-cycle hit latency, 4KB page granularity).
• PTWs: bank of N (e.g., 128) hardware walkers able to perform four-level x86-64 radix memory walks.
• Pending Translation Scoreboard (PTS): a global 128-entry scoreboard for tracking inflight translation virtual page numbers (VPNs).
• Pending Request Merging Buffer (PRMB): per-PTW buffer (32 merge slots) to absorb redundant requests for the same page.
• Translation-Path Registers (TPreg): single-entry per PTW, storing concatenated L4/L3/L2 entries (≈16 bytes) for recent translation-path caching.
• Dual page-size support: seamless operation on 4KB (default) and 2MB OS-provided large pages.

2. Data-Driven Motivation for NeuMMU

Profiling across six canonical DNN models (e.g., AlexNet, GoogLeNet, ResNet, DeepBench RNNs) with 4KB pages showed that each 5MB tile fetch from DRAM typically touches ~1.2K–several thousand distinct pages and generates translation bursts reaching hundreds per microsecond. Translation lookaside buffer (TLB) reuse in such scenarios remained very low (miss rates of 90–99%), and conventional GPU-style IOMMUs (2K TLB, 8 PTWs) could service only ~5% of ideal throughput (≈95% slowdown), primarily due to saturation of PTWs rather than TLB inefficiency.

Critical bottlenecks identified were:

• Translation-burst intensity: demand for rapid translation resolution.
• Insufficient TLB reuse: leading to high miss rates in bursts.
• Page-walk bandwidth: number of parallel PTWs determined system throughput.

3. Component Microarchitecture

3.1 IOTLB Design

• Size: 2K entries (sizable up to 128K in studies).
• Associativity: fully associative for optimized burst performance.
• Format: $\langle$VA-page# → PA-page#, valid, protection bits$\rangle$; 5 cycle hit latency.

3.2 PTW with PRMB and PTS

• PTW count: N=128 in NeuMMU, N=8 baseline.
• PRMB per PTW: ≈32 entries, tagged by VPN.
• PTS: 128-entry global, fully-associative.
• Upon IOTLB miss:
  1. PTS checked; if VPN inflight, merged to PRMB.
  2. If VPN not present, assigned to a free walker for 4-level walk.
  3. On completion, PRMB drains to DMA at 1/cycle.

3.3 Translation-Path Registers (TPreg)

• Single entry per PTW (≈16 bytes): holds L4/L3/L2 table entries.

• TPreg hit enables skipping of up to three memory accesses; observed ~99.5% hit rate at L4/L3, ~63% at L2, delivering a 2.7× reduction in walk memory traffic.

3.4 Page Size Flexibility

• 4KB pages: optimal for dense DNNs and sparse embeddings.
• 2MB pages: substantially reduce IOTLB misses for convolutional/recurrent models (512× reduction), but less effective for random-access workloads.
• Mixed page allocations supported without ISA changes; transparently managed by the OS.

3.5 OS/ISA Requirements

• Standard IOMMU-compliant interface (e.g., ATS over PCIe/H-link).
• No new instruction set or page-table format needed; uses standard x86-64 4-level radix trees and native PMDs for large pages.

4. Impact on Performance and Implementation Cost

4.1 Modeling Effective Access Time (EAT)

Effective access time for translation: $$\text{EAT} = H_\mathrm{TLB} \cdot T_\mathrm{hit} + (1 - H_\mathrm{TLB})\cdot (T_\mathrm{hit} + T_\mathrm{walk})$$ where $H_\mathrm{TLB}$: IOTLB hit rate, $T_\mathrm{hit}$: 5 cycles, $T_\mathrm{walk}$: 4-levels × 100 cycles + memory queuing. Jointly increasing $H_\mathrm{TLB}$ and parallel walker provision reduces the weighted walk cost to near zero.

4.2 Throughput, Latency, and Efficiency

• Baseline IOMMU (2K TLB, 8 PTWs): achieves only 5.1% of oracle throughput (≈95% performance loss).
• NeuMMU (2K TLB, 128 PTWs, 32×PRMB, TPreg): attains 99.94% of oracle throughput (0.06% slowdown).
• Sparse embedding accesses: NeuMMU reduces remote-access latency by 31–71% against CPU copy approaches.

4.3 Resource Consumption

• PRMB: 32KB SRAM (128×32×8B).
• TPreg: 2KB SRAM (128×16B).
• PTS: ≈768B SRAM.
• Total area ≈0.10 mm², leakage ≈13.7 mW (32nm CACTI6.5).
• Energy: 16.3× reduction vs. baseline IOMMU for translation, due to 18.8× fewer page walks.

5. Design Rationales and Trade-offs

5.1 Limitations of GPU-inspired MMUs

• GPUs: moderate translation rates, instruction-level memory access coalescing, sufficient with multi-level TLBs and 8 PTWs.
• NPUs: thousands of discrete DRAM requests per tile in aggressive bursts; TLBs are quickly overwhelmed and insufficient walker bandwidth leads to pipeline stalls.
• NPU SPMs address on-chip data with VAs, isolating translation demand to memory phase only.

5.2 Throughput-Driven NeuMMU Strategy

• Burst absorption via PRMB, minimizing redundant walks.
• Wide PTW bank (128 vs. 8) for high translation throughput.
• TPreg supporting partial page-table caching for minimized memory traffic.

5.3 Pipeline Integration

NeuMMU is inserted between the DMA's VA request queue and DRAM's request queue, intercepting memory phase traffic only and remaining outside the computation datapath. Parallel pipelined state machines manage the staged radix-tree walks. In high-translation-demand scenarios, even with large numbers of incoming VA requests (e.g., 1000 VA requests over 1000 cycles), walk merging ensures only ≈100 unique PTWs proceed simultaneously.

6. Broader Implications for Heterogeneous Accelerators

NeuMMU facilitates unified virtual address spaces between CPU and NPU, simplifying programming models. It supports demand paging, memory oversubscription, and fine-grained remote (NUMA) data access. These features position NPUs as first-class processors in heterogeneous systems, bridging the previous I/O-centric isolation. As DNN workloads increase in size and structural irregularity (graph neural networks, memory-augmented models), NeuMMU’s merge-and-cache strategies offer a blueprint for next-generation accelerator-targeted MMUs (Hyun et al., 2019).