3D-Stacked Near-Memory Processing

• 3D-stacked near-memory processing is an architecture that integrates specialized compute units with stacked DRAM using TSVs for high-bandwidth, low-latency data access.
• It eliminates the memory wall by enabling efficient offloading and resource management for applications like graph analytics, deep learning, and scientific computing.
• Advanced designs feature dynamic scheduling, thermal management, and adaptive techniques that yield significant speedup and energy savings in data-intensive workloads.

3D-stacked near-memory processing (NMP) denotes an architecture in which computation engines are integrated immediately adjacent to or beneath stacked DRAM layers via high-density vertical wiring—most commonly through-silicon vias (TSVs)—to exploit the vast internal bandwidth of the memory stack. This paradigm targets the elimination of the classic “memory wall” by reducing long-distance data transfers over narrow, power-hungry off-chip channels, addressing the core bottlenecks in large-scale, data-intensive applications. NMP systems are distinct from in-DRAM analog compute (where computation occurs within the memory arrays themselves) and instead use a dedicated “logic layer” at the base of a 3D stack, hosting programmable or specialized compute units with tightly coupled access to multiple memory banks. Numerous architectural, resource management, and system-level techniques have been developed to realize the potential of 3D-stacked NMP in domains such as graph analytics, deep learning, and scientific computing.

1. Architectural Fundamentals and System Organization

A canonical 3D-stacked NMP system consists of multiple DRAM dies (often 4–8, each further subdivided into banks or “vaults”) bonded atop a CMOS logic die. TSVs are employed as dense, low-latency vertical interconnects, providing each vault or bank with a high-bandwidth, sub-nanosecond path to the logic layer. Each vault typically features its own vault controller and can be paired directly with one or more processing engines—these are commonly small RISC cores, SIMD/vector units, coarse-grain reconfigurable arrays, or fixed-function accelerators for kernel-specific workloads (Khan et al., 2020, Mutlu et al., 2020, Hassanpour et al., 2021).

This structure enables data residing within a DRAM vault to be accessed and processed by the collocated logic, often at bandwidths of hundreds of GB/s per vault. Collectively, the internal peak bandwidth of a full stack is given by:

$$B_{\text{stacked}} = N_{\text{TSV}} \times f_{\text{clk}} \times W_{\text{TSV}}$$

where $N_{\text{TSV}}$ is the number of TSV lanes per vault, $f_{\text{clk}}$ is TSV frequency, and $W_{\text{TSV}}$ is the data width per TSV (Khan et al., 2020, Mutlu et al., 2020).

Integration methods include TSV-based stacks (Hybrid Memory Cube, HMC; High-Bandwidth Memory, HBM), monolithic 3D die fabrication, and future hybrid-bonded DRAM-logic structures for even denser vertical wiring (Singh et al., 2019, Pan et al., 6 Oct 2025).

2. Resource Management: Computation Offload, Scheduling, and Data Mapping

To extract performance, NMP architectures require sophisticated resource management along several dimensions (Khan et al., 2020):

• Computation offloading: Program kernels or code blocks with high memory intensity, poor cache locality, or large working sets are identified for offload to NMP engines. Strategies include static compiler-based mapping (e.g., the TOM framework for CUDA blocks), dynamic runtime heuristics, or hybrid schemes. Data-intensive primitives such as graph operations (e.g., PageRank), deep learning layers, and non-temporal data scans are typical offload targets.
• Data placement and partitioning: Data is partitioned to maximize vault locality. Vault-aware placement aligns working sets with the NMP engine serving a given memory region, minimizing cross-vault network traffic and contention. For graph analytics, vertexes and adjacency data are co-located in vaults to enable high local bandwidth (Mutlu et al., 2019).
• Memory scheduling and contention resolution: Vault controllers within the logic layer apply round-robin, QoS, or workload-adaptive scheduling across banks. Global schedulers re-order memory requests to optimize load balance and minimize coherence or synchronization penalties.
• Coherence and consistency: Most 3D NMP systems opt for selective cache bypass—offloaded regions are marked uncacheable to ensure the logic layer operates on DRAM-resident data. Bulk invalidate and DMA write-back mechanisms restore host caches post-computation. More complex directory-based or lazy protocols (e.g., LazyPIM, CONDA, MRCN) manage fine-grained shared-memory coherence, exploiting speculative execution and batched conflict detection to minimize off-chip coherence traffic (Kabat et al., 2023, Ghose et al., 2018).

Recent advances in resource management leverage machine learning and reinforcement learning for adaptive data and computation mapping (Majumder et al., 2021). AIMM, for example, employs a deep Q-network to continuously optimize data page placement and kernel scheduling across large NMP mesh fabrics, yielding up to 70% speedup over static policies.

3. Performance, Energy, and Scaling Models

The main advantage of NMP lies in exploiting memory-stack bandwidth and proximity to data. Key models are:

• Bandwidth: Peak attainable BW is set by the number of TSVs and their signaling rate. Sustained bandwidth is limited by DRAM protocol constraints and actual working set locality (Khan et al., 2020, Hassanpour et al., 2021).
• Latency: The critical path for a near-memory compute request comprises TSV traversal, logic-layer pipeline, and any vault/bank-level queueing:

$$L_{\text{total}} = L_{\text{TSV}} + L_{\text{logic}} + L_{\text{queue}}$$

• Energy: NMP substantially lowers the energy per bit moved and overall computation energy:

$E_{\text{bit}} = C_{\text{switch}} V_{dd}^2$

for TSV transfer, and

$E_{\text{compute}} = C_{\text{logic}} V_{dd}^2$

for on-die compute (Khan et al., 2020).

• Speedup/Energy reduction: Empirical studies report up to 13.8× speedup and 87% energy savings for graph analytics, 25× throughput gains and 10× energy reduction for genomics, and bulk bitwise operation throughput/energy improvements of 44×/35× over CPU baselines (Mutlu et al., 2019, Khan et al., 2020).

A roofline-style analytical model for offloaded kernels is:

$$S = \frac{R + 1}{R + B_{\text{ext}} / B_{\text{int}}}$$

with $R$ the compute/memory ratio, $B_{\text{ext}}$ off-chip, and $N_{\text{TSV}}$0 in-stack bandwidth (Singh et al., 2019).

4. Power and Thermal Management

Thermal constraints are critical due to the high power density of stacked logic under thermally sensitive DRAM. The logic layer’s activity can drive local DRAM temperature above 85 °C, incurring penalties in refresh rates and possibly triggering thermal throttle or shutdown (Khan et al., 2020).

Mitigation mechanisms include:

• Dynamic voltage/frequency scaling (DVFS): Adapts compute throughput to stay within safe thermal margins.
• Dynamic throttling: Runtime mechanisms (e.g., token-based, warp-throttling) cap the number of simultaneous kernel offloads based on on-chip sensor feedback.
• Spatial duty-cycling: Rotationally deactivate a subset of logic cores/vaults to provide cooldown opportunity.
• Thermal- and power-aware scheduling: Schedulers monitor instantaneous power (e.g., via MSRs) and adapt offload policies to respect split power budgets (e.g., 30 W logic, 10 W DRAM refresh) (Khan et al., 2020).

5. Advanced Case Studies and Comparative Results

Notable studies demonstrate the practical benefits and design trade-offs across domains:

Architecture / Strategy	Speedup	Energy Reduction	Key Domain
CAIRO (HMC Atomic offload)	2×	20× ED²P	Graph kernels, GPGPU
UPMEM (2D NMP)	25×	10×	Genomics scanning
Tesseract (vault-partitioned graph)	13.8×	87%	Graph analytics
QeiHaN (3D DNN NMP)	4.3×	3.5×	DNN inference (AlexNet/BERT)
Mensa (edge DNN)	3.1×	3.0×	Edge neural nets
DL-PIM (data-locality, HMC/HBM)	6–15%	–	General big/HPC kernels

NMP is also prevalent in heterogeneous mapping for sparsely activated models (e.g., MoE LLMs (Huang et al., 11 Sep 2025, Pan et al., 6 Oct 2025)) and in configurable architectures for LLM decoding (Ai et al., 5 Apr 2026), where compute substrate design and operator-aware scheduling are primary levers for maximizing performance under logic-die area constraints.

6. Open Challenges and Research Directions

Despite demonstrated benefits, key challenges persist (Khan et al., 2020, Singh et al., 2019, Mutlu et al., 2020):

• Programmability: Existing approaches rely on explicit library calls, pragma annotations, or runtime APIs for offload. Compiler frameworks that transparently identify, partition, and deploy NMP kernels across diverse stacks are under active development.
• Coherence and virtual memory: Achieving transparent, scalable coherence with minimal off-chip traffic remains open—especially for shared-memory and virtualized environments. Recent advances (MRCN, LazyPIM) employ batched, speculative protocols with analytical modeling of conflict dynamics (Kabat et al., 2023, Ghose et al., 2018).
• Thermal and reliability constraints: High-density stacking amplifies hotspot risks and reliability/aging effects (TSV electromigration, DRAM retention, RowHammer). There is ongoing research in cross-layer design for proactive thermal throttling, yield optimization, and security/privacy enforcement.
• Resource management automation: Multi-objective, learning-based resource managers (AIMM, MAB-UCB) are being investigated for runtime adaptation across dynamic NMP system states (Majumder et al., 2021, Pandey et al., 2023).
• Scalability: Systems spanning many stacks require distributed resource mapping, efficient interconnect/fabric design, and bandwidth/power-aware offload strategies (Singh et al., 2019, Mutlu, 2023).
• Emerging integration technologies: Monolithic 3D, hybrid bonding, and combination with other memory technogies (PCM, ReRAM) promise further increases in bandwidth density and stack capacity, but raise new manufacturing and system design complexities (Pan et al., 6 Oct 2025, Mutlu, 2023).

Widespread adoption of 3D-stacked NMP hinges on continued advancements in programming models, standardized system interfaces, unified virtual memory and coherence, robust resource management, and cross-discliplinary design spanning memory technology, architecture, and systems software (Khan et al., 2020, Mutlu et al., 2020, Singh et al., 2019).