---
title: Memory-Heterogeneous Accelerators (MemHA)
url: https://www.emergentmind.com/topics/memory-heterogeneous-accelerators-memha
type: topic
---

# Memory-Heterogeneous Accelerators (MemHA)

Searching arXiv for recent and foundational papers on memory-heterogeneous accelerators and related heterogeneous memory/coherence designs.
Memory-Heterogeneous Accelerators (MemHA) denote accelerator architectures and systems that exploit multiple memory organizations, technologies, or access modes within a single design. In the cited literature, this heterogeneity appears because accelerator-rich SoCs and AI systems exhibit sharply different memory behaviors: some accelerators stream long bursts, some access a few words irregularly, some benefit from private caches, and others push most traffic off-chip; likewise, some data are short-lived and bandwidth-sensitive, while other data are long-lived, capacity-dominated, or latency-critical [2109.06382][2504.14866]. MemHA therefore spans coherence specialization, accelerator-local MMUs, heterogeneous scratchpads, SRAM/GCRAM composition, local-plus-remote memory hierarchies, processing-in-memory substrates, and phase-aware split execution across distinct devices [1707.09450][1902.06468][2509.12993][2606.29986].

## 1. Conceptual scope and taxonomy

Across the literature, MemHA is not a single microarchitectural template but a family of design strategies organized around complementary memory properties. At least three recurring forms are visible. First, the same accelerator may expose several coherence or address-translation modes, with the mode selected statically or at runtime. Second, a single on-chip hierarchy may mix fast and dense memories according to lifetime, access order, or retention requirements. Third, whole systems may split workloads across physically distinct memory substrates, such as local HBM plus remote memory nodes, SRAM-PIM plus HBM-PIM, or GDDR-based prefill plus HBM-based decode [2109.06382][2602.21278][2510.02675].

| Axis of heterogeneity | Representative options | Example papers |
|---|---|---|
| Interface and control | non-coh DMA, LLC-coh DMA, coh DMA, fully-coh; accelerator-local MMUs | [2109.06382], [1707.09450] |
| On-chip memory technology | SRAM, Si-GCRAM, Hybrid-GCRAM; SHIFT plus RANDOM scratchpads | [2504.14866], [2602.21278], [2109.01269] |
| System and phase specialization | local HBM plus remote memory-nodes; SRAM-PIM plus HBM-PIM; GDDR prefill plus HBM decode | [1902.06468], [2509.12993], [2606.29986] |

This taxonomy suggests that MemHA is best understood as a cross-layer co-design problem rather than a narrow memory-device question. In some papers, heterogeneity is expressed through cache-coherence flexibility; in others, through retention-aware array composition, logic-layer compute placement, or cross-vendor serving pipelines. The unifying premise is that memory behavior is workload-dependent enough that fixed homogeneous provision is systematically suboptimal.

## 2. Coherence, address translation, and unified memory placement

A foundational MemHA problem is that there is no single universally optimal coherence interface for fixed-function, loosely coupled accelerators. Cohmeleon identifies four primary coherence modes: **Non-Coherent DMA ("direct off-chip")**, in which the accelerator has no private cache and coherence is handled in software via explicit CPU-side flushes and invalidations; **LLC-Coherent DMA**, in which DMA transactions are routed through the shared LLC; **Coherent DMA (I/O coherence)**, in which the interconnect implements full hardware coherence; and **Fully-Coherent (Private Cache)**, in which the accelerator tile includes a private cache implementing a full MESI/MOESI or custom protocol [2109.06382]. Because these modes trade off invocation latency, on-chip bandwidth, off-chip traffic, and software overhead, Cohmeleon reports that statically selecting one mode at design time leads to **20–50 % performance loss** and **40–70 % more off-chip accesses** compared to the best possible choice in each situation.

Fine-grain specialization can be pushed below the level of whole-accelerator mode selection. "A Case for Fine-grain Coherence Specialization in Heterogeneous Systems" proposes an architecture that enables low-complexity independent specialization of each individual coherence request by building on the Spandex coherence interface. The paper’s abstract reports that these techniques can reduce **execution time by up to 61%** or **network traffic by up to 99%** while adding minimal complexity to the protocol [2104.11678]. This places request-level specialization and invocation-level mode selection on a continuum rather than in opposition.

Address translation is a second control-plane dimension of MemHA. Kim et al. analyze CPU-managed MMUs, local PTW-only designs, and hierarchical TLB+PTW designs, and conclude that accelerators should not rely on the CPU MMU for any aspect of address translation, but instead must have a local, fully fledged MMU tailored to the application [1707.09450]. Their translation model expresses accelerator-side translation overhead as
$$
Acc\_MMU = N_{ref}\bigl[h\cdot T_{TLB} + (1-h)\cdot T_{PTWalk}\bigr].
$$
The same study states that even a **1 ns round-trip to the CPU’s L2 TLB kills >50 % of accelerator speedup**, and that workload-optimal designs differ markedly: streaming kernels can be well served by a **32-entry L1 only**, whereas random-pattern kernels benefit from larger TLB reach or **8–32 way** PTW parallelism.

Unified address spaces do not eliminate these optimization pressures. On the Grace Hopper Superchip, all processing units share a single **64 KB-page virtual address space** via a unified page table and an ATS-TBU, yet measured bandwidth and latency still depend strongly on placement, and the reported optimization rule is to place data as close as possible to the dominant consumer [2408.11556]. The paper applies the standard latency-bandwidth model
$$
T(S) = \alpha + S/\beta
$$
and shows that transparent fine-grained access shifts, rather than removes, the burden of memory optimization.

## 3. On-chip heterogeneous memories and profiling-guided composition

A major branch of MemHA research treats on-chip memory composition as a function of application behavior. GainSight is a profiling-driven framework with two components: retargetable hardware backends that emit cycle-accurate memory traces, and an application-agnostic analytical frontend that computes data lifetimes, read/write frequencies, and capacity utilization, then correlates them with SRAM, Si-GCRAM, and hybrid GCRAM device models [2504.14866]. Its key lifetime metric is
$$
\tau = t_2 - t_1,
$$
where $t_1$ is the first write or fetch into the on-chip memory and $t_2$ is the last read before overwrite, eviction, or invalidation. GainSight further models active energy as
$$
E_{\mathrm{active}} = E_r\,(N_r + R) + E_w\,(N_w + R).
$$
In its reported case studies, **64% of L1** and **18% of L2** GPU cache accesses, and **79% of systolic-array scratchpad accesses**, are short-lived and suitable for Si-GCRAM; heterogeneous arrays augmenting SRAM with GCRAM can reduce active energy by **up to 66.8%** [2504.14866].

OpenGCRAM extends this profiling perspective into macro generation and design-space exploration. The compiler supports **6T SRAM**, **2T Si-Si GCRAM**, and **2T OS-Si GCRAM** in **TSMC 40 nm**, generating DRC/LVS-clean macros, SPICE-characterized timing and power, and Pareto frontiers over area and leakage [2602.21278]. Reported cell-area ratios are **0.69× 6T-SRAM** for Si-Si GCRAM and **0.35× 6T-SRAM** for OS-Si GCRAM. The study also characterizes SRAM as the “speed peak,” Si-Si GCRAM as a “transient high-bandwidth cache” regime, and OS-based GCRAM as a long-term low-leakage store. Its design rules include matching retention to data lifetime and reserving SRAM for the fastest, hottest data. This suggests that workload-guided lifetime profiling and macro-level memory compilation can be coupled directly, rather than treated as separate stages.

Heterogeneous scratchpads offer a more tightly specialized on-chip embodiment. SMART, for SFQ-based systolic CNN accelerators, combines ultra-dense sequential **SHIFT** scratchpads with a large shared **RANDOM** scratchpad implemented as a pipelined multi-bank CMOS-SFQ array [2109.01269]. The SHIFT banks support sequential accesses in **0.02 ns per word**; the RANDOM path provides **0.11 ns/word** service for arbitrary indexing. An ILP-based compiler deploys CNN models across this hierarchy, and the reported results are **3.9×** throughput improvement and **86%** energy reduction for single-image inference, or **2.2×** throughput improvement and **71%** energy reduction for batch inference, at **+3% total chip area** over the latest SHIFT-based baseline. SMART illustrates a form of MemHA in which heterogeneity is driven by access order, not only by retention or bandwidth.

## 4. Memory-centric architectures, remote pools, and processing-in-memory

At the system level, MemHA often appears as an explicit partition across memory tiers with complementary latency, bandwidth, capacity, area, or power characteristics. The cited work spans capacity expansion, 3D-stacked in-memory engines, heterogeneous PIM, and 2.5D-integrated CiM/CiD systems [1902.06468][1812.11473][2509.12993][2510.02675].

| System | Heterogeneous organization | Reported outcome |
|---|---|---|
| MC-DLA | on-package HBM plus ring-attached remote memory-nodes | average **2.8×** speedup; capacity scales to **tens of TBs** [1902.06468] |
| ORIGAMI | logic-layer compute engines plus off-memory platform | **1.55×** avg speedup; **29×** avg EDP improvement [1812.11473] |
| HPIM | SRAM-PIM plus HBM-PIM | average **6.2×** lower latency than A100; peak **22.8×** speedup [2509.12993] |
| HALO | HBM-CiD plus analog CiM via 2.5D integration | **18×** speedup over AttAcc1; **2.4×** over CENT [2510.02675] |

The memory-centric deep learning system in "Beyond the Memory Wall" decouples capacity-optimized memory nodes from the host bus and embeds them into the same high-bandwidth device-side interconnect used for accelerator-to-accelerator communication [1902.06468]. Its first tier is local HBM; its second tier is a ring-attached remote pool addressed through `cudaMallocRemote` and `cudaFreeRemote`, with a BW_AWARE policy that stripes pages across adjacent memory nodes. In the reported 8-device evaluation on eight DNNs, this design achieves an average **2.8×** speedup and expands system-wide memory to **tens of TBs**.

ORIGAMI addresses the limited area and power budgets of the logic layer in 3D-stacked memory by extracting a small set of recurring ML compute patterns—MAC, comparator, optimization, and special nonlinear operations—into heterogeneous in-memory compute engines, then splitting the residual work to an off-memory FPGA/GPU/TPU-class platform [1812.11473]. The compiler exploits model-level, partial-level, and block-level parallelism, and the reported system operates within **1 %** of an ideal unlimited-logic reference while delivering **1.55×** average performance speedup and **29×** average EDP improvement over an FPGA-only baseline.

HPIM is explicitly framed as a memory-centric heterogeneous PIM accelerator for LLM inference. It couples an **SRAM-PIM** subsystem for latency-critical attention operations with an **HBM-PIM** subsystem for weight-intensive GEMV, coordinated by a controller and scheduler [2509.12993]. The HBM-PIM subsystem is built on **four HBM3 stacks** with **96 GB** capacity and **over 100 TB/s** internal bandwidth; the SRAM-PIM side provides **32** cores, each with near-array GEMV support, a **64×64** TCU, vector and scalar units, and local scratchpads. The paper models per-token latency as
$$
T_{total} = \max(T_{SRAM},T_{HBM}) - \delta_{overlap},
$$
and reports **6.2× lower latency** than an NVIDIA A100 on average across OPT models, with a peak **22.8×** speedup.

HALO reaches a related conclusion for low-batch LLM inference but with a different substrate. It combines HBM-based Compute-in-DRAM with on-chip analog Compute-in-Memory using **2.5D integration**, mapping compute-bound prefill GEMMs to CiM and memory-bound decode GEMVs to CiD [2510.02675]. Its reported geometric-mean results on LLaMA-2 7B and Qwen3 8B include **18×** end-to-end speedup over AttAcc1, **2.4×** over CENT, and energy reductions of **2.0×** and **1.8×**, respectively. Taken together, HPIM and HALO show that MemHA can mean pairing distinct PIM or in-memory substrates inside one accelerator, not only adding more tiers to a conventional hierarchy.

## 5. Runtime orchestration, phase-aware control, and heterogeneous mapping workflows

MemHA designs frequently require runtime policies because the best memory mode depends on workload size, footprint, and system contention. Cohmeleon formulates coherence-mode selection as a reinforcement-learning problem in which the state vector
$$
s_t = (n_{fc}, n_{nc\_dma}, n_{llc\_dma}, tile\_footprint, acc\_footprint)
$$
is discretized into **243** states, the action space consists of the four coherence modes, and the reward is a weighted sum
$$
R_t = x\cdot R_{exec} + y\cdot R_{comm} + z\cdot R_{mem}.
$$
The implementation uses a **243×4** Q-table, $\epsilon$-greedy exploration, and in practice a simpler on-policy update
$$
Q \leftarrow (1-\alpha)Q + \alpha R_t
$$
to emphasize immediate invocation performance [2109.06382]. Hardware support is intentionally small: a coherence-control register per accelerator, mode support in the interconnect/LLC, and counters exposed via a register block costing **<100 LUTs per tile** and read with **~100 ns overhead**. On FPGA prototypes with **11** accelerators, Cohmeleon converges in **≲10 application iterations** and achieves on average **38 % higher throughput** with **66 % fewer off-chip DRAM accesses** than state-of-the-art static policies.

Phase-aware control becomes even more explicit in cross-device LLM serving. HMA-Serve pairs **GDDR-based accelerators for prefill** with **HBM-based GPUs for decode**, arguing that HBM bandwidth sits almost entirely idle during prefill and is therefore economically mismatched to that phase [2606.29986]. Its three central mechanisms are phase-wise quantization, a layer-wise compute-transfer pipeline, and deferred dequantization. The pipeline changes the first-token latency from a naive
$$
TTFT = T_{comp} + T_{eg}
$$
to an overlapped form
$$
TTFT \simeq \max(T_{comp},T_{eg}) + T_{decode\_first\_token}.
$$
The system ships raw BFP8 KV bytes over the network and reconstructs them lazily on the decode GPU; the reported full-KV reconstruction cost is **≈1.5 ms per request** versus **~72 ms** for a naive element-wise path. Across four Qwen3 models and three production traces, HMA-Serve reports **up to 3.2× higher goodput** and **4.8× higher goodput-per-dollar** than memory-homogeneous baselines, with **no measurable loss** on generation-quality benchmarks.

For analog in-memory computing, runtime and compile-time mapping are linked by precision sensitivity. The unified AIMC workflow classifies heterogeneous mapping by granularity and optimization strategy, and applies a four-stage procedure: precision sensitivity profiling, partitioning-strategy selection, mapping granularity and allocation, and final optimization/deployment [2606.02672]. In its GPT-2 case study, sensitivity is dominated by **4 of 49 projections**, with the **first decoder block’s attention output** dominating by **an order of magnitude**. A threshold rule with $\tau=3$ maps **9 projections** to digital and **40** to analog, giving an analog MAC ratio of
$$
40/49 \approx 82\%
$$
with **<0.5%** increase over baseline perplexity. This result is notable because it rejects both chip-level “all analog” and “all digital” simplifications in favor of projection-level heterogeneity.

## 6. Misconceptions, limitations, and open research directions

A recurring misconception is that unified address spaces or full hardware coherence make memory decisions largely automatic. The GH200 measurements contradict this: transparent access exists, but bandwidth and latency remain NUMA-sensitive, and careful placement still matters materially for kernels, LLM inference, and collectives [2408.11556]. A related misconception is that MemHA is synonymous with accumulating more HBM. Several cited systems do the opposite: HMA-Serve argues explicitly that **HBM is not all you need** for disaggregated LLM serving, and GainSight/OpenGCRAM emphasize retention-aware composition with GCRAM rather than bandwidth-only scaling [2606.29986][2602.21278].

The literature also identifies several technical limits. Cohmeleon notes that a tabular Q-table scales poorly if more state attributes are tracked, and that RL overhead becomes problematic if decisions are made more frequently than once per accelerator task; its stated future directions are Deep Q-Networks, predictive workload models, and within-kernel mode switching [2109.06382]. GainSight provides a clear optimization formulation over subpartitions, area budgets, and retention constraints, but the paper explicitly does **not** present a full integer-programming solver [2504.14866]. SMART depends on mature cryo-CMOS SRAM at **4 K**, reports that current ILP is per-layer and near-optimal under a fixed prefetch horizon, and identifies **cooling overhead (×400 at 4 K)** as a practical system-level energy cost [2109.01269].

Several works converge on a common future direction: tighter coupling between profiling, hardware synthesis, and runtime control. OpenGCRAM positions macro generation, SPICE-accurate characterization, and Pareto filtering as a basis for plugging in new memory technologies and new PDKs [2602.21278]. GH200 points toward hardware counters and programmable migration thresholds for coherent layered memories [2408.11556]. Cohmeleon points toward finer-grain adaptation, while AIMC mapping points toward projection-level rather than layer-level control [2109.06382][2606.02672]. This suggests that future MemHA research will likely be defined less by any single memory device than by the sophistication with which systems observe, model, and exploit workload-specific memory behavior across coherence, translation, retention, capacity, bandwidth, and phase structure.

Source: https://www.emergentmind.com/topics/memory-heterogeneous-accelerators-memha