Decoupled Mixed-Precision Memory Hierarchies

Updated 8 March 2026

Decoupled mixed-precision memory hierarchies are architectures that separate computing and memory tasks, allowing each tier to operate at an optimal precision level for its function.
The system design integrates low-precision analog processing with high-precision digital accumulation to achieve significant energy savings and maintain near full-precision accuracy.
Mixed-precision assignment is optimized via methods like integer programming and dynamic scheduling, enabling improved throughput and reduced carbon emissions in large-scale workloads.

A decoupled mixed-precision memory hierarchy is a system architecture that physically, logically, or functionally separates processing elements and memory/storage hierarchies, while allowing each tier to operate at a different numerical precision. These hierarchies underpin state-of-the-art solutions in both hardware accelerators and software-hardware codesign, enabling significant gains in energy efficiency, memory utilization, compute throughput, and carbon sustainability, while preserving or closely matching high-precision performance in large-scale scientific and machine learning workloads.

1. Architectural Principles of Decoupling and Precision Hierarchy

Decoupled mixed-precision memory hierarchies separate computational and memory tasks across multiple subsystems, each operating at a purpose-optimized precision and bandwidth. Classical instances include:

Computational Memory Unit (CMU): Analog, in-place processing where weights or matrix entries are stored as device conductances (commonly in phase-change memory (PCM) or resistive RAM crossbars), supporting low-precision (e.g., 2-6 bits) matrix-vector or matrix-matrix multiplication via Ohmic or Kirchhoff’s laws. This tier performs bulk MAC operations with extreme area and energy efficiency but is subject to analog non-idealities, variability, and noise (Gallo et al., 2017, R. et al., 2017, Nandakumar et al., 2020).
High Precision Digital Unit (HPU/DPU): Standard digital (e.g., CPU/GPU or vector core) tier, maintaining full-precision (32/64-bit) accumulators and implementing all control, residual calculation, optimizer logic, and accuracy-critical updates.
Multi-Level Caching in Modern Systems: Hierarchies often extend to DRAM and SSD, as in “M2Cache,” where HBM, DRAM, and SSD serve as progressively larger/lower-bandwidth memory tiers, each storing model weights or activations at mixed precision and with intelligent caching policies (Peng et al., 2024).

The architectural decoupling allows each tier’s interface to be tuned according to its underlying device limits (precision, update granularity, endurance) without constraining system-level accuracy.

2. Mixed-Precision Assignment and Bit-Width Optimization

Bit-width assignment in decoupled systems is highly algorithm and workload dependent. Strategies include:

Device-Limited Analog Precision: Physical devices (e.g., PCM, RRAM) operate natively at 2–8 bits effective precision, with the CMU supporting only coarse or noisy updates and low-resolution MAC operations.
Full-Precision Digital Accumulation: Gradient or residual updates, control flow, and error corrections are always executed in 32/64-bit floating-point, preventing precision loss due to accumulation or rare updates (Nandakumar et al., 2020, R. et al., 2017).
Integer-Programming Bit-Width Allocation: In Post Training Quantization (PTQ) for generative models, bit-widths (2/4/8/16 bits) are auto-assigned to each layer or module using formal integer programs that maximize sensitivity-preserving metrics (e.g., SSIM for content, SQNR for quality) subject to global memory budgets. This creates genuine multi-tier memory stratification, with critical layers protected by higher precision (Zhao et al., 2024).
Dynamic/Heuristic Assignment: For LLM inference at scale, neuron or group importance is evaluated on the fly for each token, and tiered (FP16/INT8/INT4) precisions are dynamically applied per module or neuron, exploiting runtime activity patterns and uncertainty estimation to achieve optimal trade-offs (Peng et al., 2024).

The table below summarizes core mixed-precision assignment strategies:

System/Domain	Low-Precision Tier	High-Precision Tier	Assignment Method
PCM In-Memory Accelerators	PCM (analog, 2–6 bit)	Digital (32–64 bit)	Fixed, by device/algorithm role
FPGA M4BRAM	2/4/8-bit in-BRAM MAC	Full-precision memory access	Per-layer/port control, tile granularity
LLM Inference (M2Cache)	INT4/8 DRAM/SSD	FP16 HBM	Dynamic per-neuron, activity-aware
Diffusion (MixDQ)	INT2/4/8 tensors	FP16 for most sensitive	Integer programming per layer/group

3. Algorithmic Integration and Workflow

Mixed-precision decoupled hierarchies manifest distinct algorithmic flows depending on application:

In-memory Accelerators for Linear Algebra: Solvers such as iterative refinement use a high-precision outer loop in the digital tier, correcting the output of low-precision analog CMUs after each in-memory batch matrix operation. Matrix-vector operations $A v$ are analog, while residual computation and solution update are digital, with convergence governed by the precision of both tiers (Gallo et al., 2017).
Deep Neural Network Training/In‐Memory Deep Learning: The forward and backward passes (matrix-vector multiplies) are performed by CMUs, while gradient accumulation and optimizer logic remain purely digital. Accumulators $\chi_{ji}$ aggregate small, per-example updates in high precision and only commit to the resistive memory array when the quantized threshold (hardware granularity $\epsilon$ ) is exceeded. This separation handles both analog noise and digital optimization fidelity (Nandakumar et al., 2020, R. et al., 2017).
FPGA and NPU-based GEMM Kernels: In architectures such as M4BRAM, both compute-in-memory (CIM) and memory access are decoupled at the hardware port level, with dual-mode operation supporting mixed-precision matrix-matrix multiplications (e.g., W4A16) alongside regular memory transactions (Chen et al., 2023, He et al., 23 Jan 2026).
Large-Scale Language and Generative Models: Tiered quantization and caching orchestrate data transfers among HBM, DRAM, and SSD, exploiting multi-level data reuse and precision-awareness. Only the most active and important weights reside in HBM at the highest precision; lower-importance and cold weights are quantized and paged on demand (Peng et al., 2024, Zhao et al., 2024).

4. Hardware Implementations and System Architectures

Decoupled mixed-precision memory hierarchies are instantiated via multiple physical realizations:

Phase-Change Memory Crossbars and Digital Units: Phase-change memory arrays combined with standard CPUs, FPGAs, or ASICs deliver high-throughput, area/energy-efficient MACs, with high-precision digital units handling control, updates, and error correction (Gallo et al., 2017, Nandakumar et al., 2020, R. et al., 2017).
FPGA Block RAM Compute-in-Memory Blocks (M4BRAM): Bit-serial compute engines are embedded within standard dual-port BRAM blocks, enabling concurrent mixed-precision MAC and standard memory access. This architecture allows seamless DNN dataflow integration and supports layer-wise parallelism with fine-grained control over bit-width per activation and weight (Chen et al., 2023).
Ascend NPUs and W4A16 GEMM: On domain-specific NPUs, “cube” (matrix) and “vector” (elementwise/type conversion) cores are fully decoupled, sharing no buffers and communicating only via global memory and memory transfer engines (MTEs). On-the-fly dequantization is performed on vector cores, while cube cores execute FP16 tile GEMMs, with performance bottlenecked by inter-unit memory transfers (He et al., 23 Jan 2026).
Tiered GPU/DRAM/SSD Systems (M2Cache, MixDQ): Several levels of memory form a true physical and logical hierarchy, each with its own bit-width constraints and caching policies. For example, M2Cache implements a neuron-level cache in GPU HBM, a layer-wise cache in DRAM, and a full-model store in SSD, all coordinated through precision- and activity-aware data movement (Peng et al., 2024).

5. Performance, Energy, and Sustainability Impacts

Decoupled mixed-precision hierarchies realize several key advantages, but also reveal architectural bottlenecks:

Energy Efficiency: In-memory analog computation (PCM crossbars, compute-in-SRAM/BRAM) achieves up to $173\times$ reduction in energy when compared to standard 32-bit digital designs in neural network applications (Nandakumar et al., 2020). Dynamic sparsity and precision scaling yield a $7\times$ reduction in operational carbon emissions during LLM inference (Peng et al., 2024).
Throughput: Systems routinely achieve $2\times$ – $2.3\times$ speedup for DNN inference and training relative to same-precision digital accelerators. FPGA systems (M4BRAM) deliver $2.16\times$ average speedup with negligible accuracy loss ( $<$ 0.5%) (Chen et al., 2023). On commercial NPUs, theoretically perfect quantization (e.g., W4A16) is bottlenecked by extra GM traffic, capping actual speedup at $1.48\times$ (He et al., 23 Jan 2026).
Accuracy Preservation: By decoupling high-precision update/accumulation from low-precision compute, and through workflow such as iterative refinement or high-precision digital accumulators, systems match or nearly match floating-point software baselines: e.g., MNIST accuracy within $0.6\%$ of FP64; LLMs and few-step diffusion models within $0.5$ FID or $0.003$ CLIP score of FP16 (Nandakumar et al., 2020, Zhao et al., 2024).
Scaling, Limits, and Bottlenecks: Limits arise from analog device variability, nonlinearity, insufficient analog precision, peripheral (DAC/ADC) overhead, and, in multi-core NPUs, inter-unit memory bandwidth (Gallo et al., 2017, He et al., 23 Jan 2026). Redundant global memory handoffs can erode potential quantization gains, highlighting the need for tighter hardware-software co-design.

6. Design Trade-Offs, Algorithmic Flexibility, and Generalization

Key trade-offs include:

Device Error vs. Area/Energy: Increasing $K$ (PCM devices per weight) reduces analog error ( $\sigma_{\rm error}\sim K^{-1/2}$ ) but costs area/energy (Gallo et al., 2017).
Scheduling and Pipeline Flexibility: Many systems support complex optimizer logic, batch-norm, momentum, dropout, and sparse programming only in the high-precision digital tier, offloading only data-parallel MACs to analog/memory tiers (Nandakumar et al., 2020, Zhao et al., 2024).
Cache Granularity and Update Policy: In M2Cache, fine-grained cache updating at the neuron level, using an “Adjacent Token Update” (ATU) policy, leverages temporal locality to minimize cache evictions and transfer costs (Peng et al., 2024).
Bit-Width Allocation Algorithms: Integer programming, heuristic importance ranking, and dynamic at-runtime decisions allow matching each subblock/layer/neuron to its task significance, supporting extension to any model with multi-metric or multi-group structure (Zhao et al., 2024, Peng et al., 2024).

The principle of decoupling precision, compute, and storage tiers applies broadly. These hierarchies are now canonical for sustainable LLM deployment, DNN acceleration, and hybrid analog-digital scientific computing.

7. Future Directions and Open Challenges

Hardware Co-Design: Direct data paths between compute cores (e.g., vector “dequant” cores and GEMM “cube” cores in NPUs) to bypass global memory, fused dequant+MAC instructions, and improved peripheral (DAC/ADC) matching are critical for approaching theoretical quantization savings (He et al., 23 Jan 2026).
Dynamic Algorithmic Scheduling: Enhanced on-chip network control, adaptive cache prefetch, and uncertainty/risk-aware mixed-precision assignment are active areas of research.
Device-Level Improvements: Higher analog precision, improved endurance, and error correction in resistive-memory crossbars are required for deeper scientific workloads and ill-conditioned problems (Gallo et al., 2017).
Expansion to Broader Domains: Metric-decoupled sensitivity and auto-mixed-precision allocation are broadly applicable to transformer-based vision, speech, and fused retrieval-augmented models (Zhao et al., 2024).
Sustainability at Scale: As LLM sizes and demands for “green” inference and training increase, multi-level (e.g., HBM/DRAM/SSD) decoupled mixed-precision caching strategies (like M2Cache) are expected to become the standard for carbon- and energy-constrained deployment (Peng et al., 2024).

Open controversies and limitations remain around the complexity of software support, quantization-induced numerical instabilities, and device-specific non-idealities in analog accelerators. Ongoing work aims to conclusively close the gap between theoretical and achieved gains at hyperscale.

Selected Key References: