Memory Architectures: Trends & Innovations

• Memory architectures are organizational frameworks that define how systems store, access, and compute on state across conventional, distributed, and non-von Neumann substrates.
• They optimize key parameters such as latency, bandwidth, capacity, and energy efficiency, overcoming physical and economic scaling constraints.
• They integrate near-memory processing with heterogeneous hierarchies and distributed control to enhance performance in complex computing environments.

Memory architectures are the organizational schemes by which a system stores, places, accesses, moves, protects, and sometimes computes on state. In contemporary usage, the term spans conventional cache–main-memory hierarchies, distributed and disaggregated memories, processing-in-memory substrates, persistent memory nodes, domain-specific accelerator memories, and non-von-Neumann formulations in which memory is the active substrate of computation. Across these settings, the central design variables are latency, bandwidth, capacity, energy, physical distance, signaling cost, reliability, and software visibility rather than capacity alone (Mutlu et al., 2018, Khan et al., 2020, Liu et al., 28 Aug 2025).

1. Architectural objectives and scaling constraints

Modern memory architecture is shaped by a convergence of physical and economic limits. Recent system-level analyses state that memory latency, bandwidth, capacity, and energy increasingly limit performance, while SRAM scaling has stalled and DRAM cost/GB has been flat for over a decade. In accelerator-centric systems, the imbalance is explicit: one analysis cites NVIDIA GPUs whose compute throughput rose by roughly 30× in four years, while memory capacity and bandwidth improved by only 2.5× and 2.1×, respectively, recasting the classical memory wall as a mismatch between arithmetic growth and data supply (Liu et al., 28 Aug 2025, Qureshi et al., 2020).

Distance and signaling have therefore become first-order architectural parameters. One comparison gives On-die (5nm, e.g. SRAM) at 0.028 µm pitch, 5 fJ per bit, and 131 TB/s bandwidth per chip; Hybrid bonding (e.g. V-Cache) at 9 µm, ~600 fJ, and 2.5 TB/s; Microbump (e.g. HBM) at 36 µm, ~2,000 fJ, and 1.2 TB/s; and C4 solder bump (e.g. DDR) at 730 µm, ~10,000 fJ, and 0.1 TB/s. The same work argues that closer integration yields denser pins, lower energy per bit, and higher bandwidth, and that a giant pool of far memory is inefficient for performance-critical access (Liu et al., 28 Aug 2025).

At the technology level, the dominant problem is no longer density alone. SAFARI’s retrospective on DRAM and NAND flash emphasizes latency stagnation, bandwidth bottlenecks from limited pin counts, energy overheads from data movement, and reliability/endurance limits such as retention loss, read disturb, and programming vulnerabilities. The paper’s general conclusion is that memory and storage increasingly have to be treated as heterogeneous physical substrates whose internal asymmetries can be measured, modeled, and exploited rather than as uniform black boxes (Mutlu et al., 2018).

2. Locality, distribution, and physically composable memory

A recurring architectural response to these limits is to distribute state near the logic that consumes it. In re-configurable logic, a direct example is the replacement of a centralized global settings register map by a distributed configuration memory architecture in which each subsystem stores a local copy of the needed settings and a common configuration bus writes local memories directly. The paper distinguishes three forms—Global memory map, Global memory with local copies, and Distributed memory architecture—and reports that, for 226 target registers per slave, the distributed design uses 2556.0 ALMs, 7499 registers, and 1887 ALUTs, corresponding to 25% of the ALMs and 20% of the registers of the global design with maximum routing register stages; approximate f_max rises from just under 140 MHz to about 210 MHz. The same architecture uses a subsystem “Ready” signal to gate safe updates across clock domains and supports dynamic partial reconfiguration through a uniform configuration interface (Beasley, 2020).

Distributed organization appears again at manycore scale. Self-aware Memory (SaM) divides memory into autonomous self-managing modules and distributes memory management across enriched MMU-like core-side components and memory-side management components. Its self-optimization loop extends the MAPE cycle with Consensus building, and the evaluation reports that there is no single universally best parameter setting, but that the overhead of decentralized optimization can be amortized by runtime improvement. For large shared on-chip memories, the Distributed Shared Memory Controller (DSMC) replaces a monolithic many-ported crossbar with a hierarchical, distributed, staged architecture using lower-radix switches, staged speed-up, and fractal randomization of memory-bank addressing; the reported result is 20% higher throughput, 20% lower latency, and 30% less interconnection area at approximately the same power consumption (Mattes et al., 2014, Luan et al., 2020).

At rack and package scale, the same principle is expressed as physically composable disaggregation. One recent proposal argues against huge, homogeneous shared memories and instead organizes the system as compute-memory nodes with Private local memory for node-exclusive data, In-package shared memory for shared state within a processor, and Off-package DRAM as a bulk capacity tier. The explicit thesis is that distance matters and that software should compose the hierarchy by placing data according to locality, bandwidth, and energy efficiency rather than treating memory as a flat shared resource (Liu et al., 28 Aug 2025).

3. Memory-centric computation and in-memory execution

A second major trajectory in memory architecture is the migration of computation toward or into memory. The survey of processing-in-memory (PIM) and near-memory processing (NMP) organizes the field into DRAM-based PIM, NVM-based PIM, Analog crossbar / resistive compute fabrics, 3D-stacked DRAM with logic layer, Programmable near-memory cores, Fixed-function accelerators near memory, and Reconfigurable logic. The motivation is uniform across these variants: reduce off-chip data movement, exploit the high internal bandwidth of memory arrays or stacks, and better match execution to application-specific access patterns, while recognizing that power, thermal limits, refresh, reliability, and locality determine whether offloading is actually beneficial (Khan et al., 2020).

The Erudite architecture makes the compute/memory co-design explicit by coupling Flash-based NVMe SSDs with programmable accelerators modeled on GPU streaming multiprocessors. Each Erudite Processing Unit (EPU) contains an Erudite Compute Unit (ECU), a few GBs of HBM, and an array of Flash SSDs acting as storage-class memory. The latency-hiding argument is quantitative: with 16 GB/s PCIe bandwidth, 64 μs SSD latency, and 512-byte transfers, sustaining bandwidth requires about 2000 simultaneous accesses in flight. Erudite therefore relies on massive GPU thread parallelism, direct user-space NVMe access, and an interconnect designed for fine-grained transfers and millions of outstanding requests rather than CPU-mediated storage access (Qureshi et al., 2020).

Algorithm–memory co-design is equally important in in-memory computing. MEMHD redesigns hyperdimensional associative memory so that its structure matches 128×128 SRAM-based IMC arrays: rows correspond to vector dimension $D$, columns to the number of centroids $C$, and inference becomes a single in-memory matrix-vector operation, $pred = \arg\max_{i, j} \delta_{\text{dot}(C^{bi}_{j}, H^{b}))$. The reported consequences are hardware-oriented: up to 13.69% higher accuracy with the same memory usage, or 13.25× more memory efficiency at the same accuracy level; on 128×128 arrays, up to 80× fewer cycles and 71× fewer arrays than baseline mapping methods, with 80× lower energy than BasicHDC (Kang et al., 11 Feb 2025).

At the circuit and device level, logic-in-memory can collapse storage, bit-stream generation, and arithmetic into a single substrate. An MTJ-based memory with logic-in-memory (LIM) and stochastic computing integrates binary storage, deterministic in-memory bit-stream generation, parallel stochastic arithmetic, and accumulation-based reconversion. The architecture reports $22\times$–$64\times$ speedup versus serial stochastic implementations, computational latency reduced by over three orders of magnitude compared to prior IMC approaches, and robustness under 30% injected noise. In a different but related line, SAFARI’s RowClone and LISA move bulk copy, initialization, and data movement inside DRAM, with RowClone reported to provide 1–2 orders of magnitude speedup and energy reduction for copy and initialization (Razi et al., 25 Apr 2026, Mutlu et al., 2018).

4. Heterogeneous hierarchies, specialization, and software-visible memory tiers

Heterogeneous memory systems differ not only in medium but in the extent to which heterogeneity is exposed to software. In sparse matrix–matrix multiplication on multilevel memory systems, the contrast between Intel KNL and NVIDIA Pascal illustrates this directly. KNL exposes 16 GB on-package MCDRAM and 96 GB DDR system memory, and the study finds that standard cache-reuse algorithms often perform as well as multi-memory-aware methods because the memory subsystems have similar latency characteristics. On Pascal GPUs, by contrast, the hierarchy of GPU global memory (HBM), Host pinned memory, and Unified Memory (UVM) has a major latency gap, so selective placement and chunking become crucial when the problem exceeds HBM capacity (Deveci et al., 2018).

Byte-addressable persistent memory extends the hierarchy across the traditional memory/storage boundary. The proposed HPC/HPDA architecture built around B-APM places persistent memory in compute-node memory channels, allowing ordinary load/store access to large persistent regions. The paper distinguishes Single-Level Memory (SLM), where DRAM and B-APM are separate addressable spaces, from Dual-Level Memory (DLM), where DRAM acts as a cache in front of B-APM. It also introduces a surrounding systemware layer—Job Scheduler, Data Scheduler, Object Store, Filesystem, and Programming Environment—to manage persistence, workflow reuse, and data movement at system scale (Jackson et al., 2018).

A more radical view is memory specialization by data lifetime and access asymmetry. The proposal for short-term RAM (StRAM) and long-term RAM (LtRAM) treats the familiar SRAM/DRAM pair as insufficient once cost-per-byte scaling stalls. StRAM is defined for transient, frequently accessed, short-lived data, with lifetimes on the order of sub-seconds and with an emphasis on symmetric read/write behavior and effectively unlimited endurance; LtRAM targets persistent or long-lived, read-heavy data, with lifetimes on the order of minutes or longer and with an acceptance of higher write latency, higher write energy, and lower write endurance in exchange for low read energy and high density. The paper explicitly argues for non-hierarchical optimization, meaning that placement should be based on access behavior and lifetime rather than a rigid “closer = faster = more expensive” ladder (Li et al., 5 Aug 2025).

The broader design methodology for such systems is domain-specific. The survey on domain-specific memory architectures defines them as memory systems tailored to a family of applications and emphasizes accelerators, customized local memories, DMA engines, prefetchers, buffers, and coherence models ranging from Non-coherent to LLC-coherent to Fully coherent. It also notes that Private Local Memories (PLMs) can consume more than 90% of accelerator area in some embedded designs and that multi-port memories scale roughly quadratically with the number of ports, making memory architecture a central part of accelerator synthesis rather than a back-end implementation detail (Soldavini et al., 2021).

5. Reliability, resilience, and coordination as architectural concerns

Memory architecture also includes the way faults, failures, and concurrent accesses are represented and controlled. A major cross-layer position is that reliability cannot be achieved economically by hiding faults from the architecture. The dissertation on reliable and scalable memory systems argues instead for exposing fault granularity and then matching protection strength to the common case. Its exemplars include ArchShield, which at DRAM BER $10^{-4}$ uses a Fault Map and Replication Area to achieve about 4% memory overhead with <2% performance degradation; XED, which uses On-Die ECC detection and Catch-Word signaling to provide 172x higher reliability than ECC-DIMM with <0.01% performance overhead; Citadel, which combines TSV-SWAP, Tri-Dimensional Parity (3DP), and Dynamic Dual-Granularity Sparing (DDS) to achieve about 700x higher resilience than strong symbol-based ECC at about 14% storage overhead; and SuDoku-Z, which protects STTRAM caches with per-line ECC-1, CRC, RAID-like parity, and skewed hashing, reaching about 0.1% slowdown with 32 bits per 512-bit cache line rather than 50 bits for ECC-5 (Nair, 2017).

Device-level characterization complements these architectural techniques. SAFARI’s work on Adaptive-Latency DRAM (AL-DRAM), Flexible-Latency DRAM (FLY-DRAM), Voltron, retention-error recovery, read-disturb mitigation, and SoftMC shows that timing margins, intra-module variation, voltage sensitivity, and flash error mechanisms are themselves architectural resources once they are measured precisely. In this view, reliable memory systems emerge from joint control of device behavior, controller policy, and data movement rather than from a single uniform ECC layer (Mutlu et al., 2018).

At system scale, resilience changes when memory survives compute failure. Memory-Oriented Distributed Computing (MODC) assumes a shared pool of byte-addressable disaggregated memory that remains available even if compute nodes fail. It then expresses work as idempotent tasks over named data items in disaggregated memory, uses lock-free per-worker queues stored in that memory, and recovers from worker failure by work stealing and task restart rather than by reconstructing state from checkpoints. In the reported PageRank case study, MODC with failure completes in 23.3 s, with degradation from failure-free execution of less than 1%, whereas MPI with checkpointing is 19% slower than MODC when failure happens just after a checkpoint and up to 51% slower when failure happens just before the next checkpoint (Keeton et al., 2021).

Even small-scale sharing problems reveal the same architectural logic. A two-client RAM arbiter for a single RAM module uses a fixed-priority policy in which Client1 has higher priority than Client2, plus explicit handling of the Address Clash Problem. When read and write target the same address in the same access window, the arbiter bypasses stale RAM output and forwards the updated value via TEMP_RD_DATA <= TEMP_WR_DATA, ensuring that the reader observes the newly written value. This is a microarchitectural instance of a general principle: memory architecture includes the legality, serialization, and observability rules governing concurrent access (Banerji, 2014).

6. Non-von-Neumann and emerging memory substrates

Emerging device architectures often redefine memory by the physics of readout rather than by a classical hierarchy. In 3D ferroelectric memory, the proposed taxonomy is organized by polarization sensing: capacitor-based memories such as 1T-1C FeRAM and 1T-nC FeRAM, transistor-based memories such as 1T FeFET, and hybrids such as 1T-1C FeMFET, 2T-1C FeRAM, and 2T-nC FeRAM. The paper’s central claim is that sensing determines the scaling law: capacitor-based sensing ties margin to capacitor area, while transistor-assisted sensing ties current margin to transistor gain and memory window. It therefore compares architectures by write voltage, retention, endurance, sense margin, scalability, and integration mode, and reports, for example, >10 years retention and >10$^ {12}$ write endurance for 1T-1C FeRAM, versus Si-based FeFET endurance typically < 10$^6$ cycles; it further distinguishes parallel stacking from sequential stacking as the two main 3D integration families (Duan et al., 13 Apr 2025).

In integrated photonic neural networks, memory appears as a hierarchy of physical state variables rather than as a single storage array. The review classifies memory into volatile Response-induced memory and Multistable-induced memory, and non-volatile Structural memory, Polarization-based memory, Ionic memory, and Charge-trapping memory. Delay lines and slow-light structures provide group-delay-based short-term memory, with $\tau_g(\omega) = -\dv{\omega} \arg \left[ H(\omega) \right]$, while silicon microring dynamics exploit nonlinear relaxation with characteristic times such as thermal relaxation $\tau_\mathrm{th} \sim 60$–$C$0 ns and free-carrier lifetime $C$1–$C$2 ns. Non-volatile photonic memory is then implemented through phase-change materials, ferroelectrics, ionic devices, or charge trapping, supporting feed-forward, reservoir, spiking, and hybrid optoelectronic recurrent architectures (Foradori et al., 24 Apr 2026).

Outside hardware-centric computing, some architectures treat memory as the active medium of cognition itself. Memory-centred cognitive architectures for socially interactive robots define memory as an active association substrate rather than a passive adjunct, with priming as the basic mechanism by which partial sensory stimulation reactivates associated multimodal patterns and supports prediction, contingent behavior, and multi-modal alignment. In a more formal construction, a self-organizing memory architecture for autonomous machines uses a continually updated snapshot $C$3, a derived weak poc set $C$4, and a cubical model space $C$5, with both space requirements and update-execute complexity bounded quadratically in the number of sensors. This suggests that, in non-von-Neumann settings, “memory architecture” can denote the organization of internal state spaces and activation dynamics as much as the arrangement of semiconductor arrays (Baxter, 2016, Guralnik et al., 2015).

Taken together, these lines of work describe memory architecture as a multi-scale discipline. At one end are bitlines, banks, DIMMs, vaults, TSVs, and package interfaces; at the other are disaggregated runtimes, persistent workflows, photonic state variables, associative cortical networks, and geometric model spaces. The unifying technical theme is that memory is no longer adequately specified by capacity and latency alone: its architecture now includes locality, signaling distance, update protocol, fault model, software visibility, and, increasingly, its role as a computational substrate in its own right.