---
title: 'Memory Reuse Rate (MRR): Metrics & Implications'
url: https://www.emergentmind.com/topics/memory-reuse-rate-mrr
type: topic
---

# Memory Reuse Rate (MRR): Metrics & Implications

Memory Reuse Rate (MRR) quantifies the proportion of computation or memory operations in a system that are serviced by reusing previously stored results or data, rather than recomputing or fetching from primary memory. The metric provides a fundamental measure of temporal and spatial locality across diverse domains ranging from deep neural network (DNN) accelerators and static code analysis to transformer inference and large-reasoning models. Its computation and interpretation are strongly context-dependent but consistently play a critical role in optimizing latency, bandwidth, and overall system efficiency.

## 1. Formal Definitions Across Domains

The definition of MRR varies by domain but shares the core objective of quantifying effective reuse:

- **Transformer Inference (LLMCache):** For a transformer with $L$ layers and a dataset $D$, each input $X \in D$ performs one cache lookup per layer. Define the hit indicator
  $$
  I_l(X) = 
  \begin{cases}
    1 & \text{if the cache at layer } l \text{ is used for } X \\
    0 & \text{otherwise}
  \end{cases}
  $$
  Then the global MRR is
  $$
  \mathrm{MRR} = \frac{1}{|D|\cdot L} \sum_{X \in D} \sum_{l=1}^L I_l(X),
  $$
  with $ \mathrm{MRR}_l = \frac{1}{|D|} \sum_{X \in D} I_l(X)$ as the layerwise rate [2512.16843].

- **Large Reasoning Models (ENGRAM-R):** MRR is the proportion of tokens saved by reusing memory instead of recomputation:
  $$
  \mathrm{MRR} = \frac{T_\mathrm{FC} - T_\mathrm{ER}}{T_\mathrm{FC}},
  $$
  where $T_\mathrm{FC}$ and $T_\mathrm{ER}$ are the token counts for full-context and memory-augmented runs, respectively [2511.12987].

- **DNN Accelerator Architecture (Voltra):**
  $$
  \mathrm{MRR} = \frac{R}{F},
  $$
  where $R$ is the number of local operand reuses (on-chip), and $F$ is the count of expensive off-chip or shared-memory fetches [2602.11357].

- **LLVM Static Analysis:**
  After solving for a reuse distance profile $p_d$ (probability of a memory access having reuse distance $d$ for $d \geq 1$), MRR (for an LRU cache of capacity $C$) is computed as:
  $$
  \mathrm{MRR} = \sum_{d=1}^{C} p_d,
  $$
  representing the steady-state hit rate for the cache under the observed profile [2311.12883].

## 2. Measurement and Algorithmic Realizations

Each context requires distinct measurement strategies, typically leveraging either dynamic instrumentation or static analysis:

- **LLMCache:** Semantic fingerprinting (SimHash, PCA, or MinHash) generates fixed-length keys for each input. Cache banks per layer store activations indexed by these fingerprints. Cosine or Jaccard similarity governs matches (threshold $\tau$). Hits are tallied by PyTorch hooks, and MRR computed as hit fraction post-inference [2512.16843].

- **ENGRAM-R:** Instrumentation of the inference loop counts tokens in both the baseline and memory-reusing runs for both input and reasoning steps. Fact-card rendering and citation control guarantee that evidence is genuinely reused, not simply rephrased [2511.12987].

- **Voltra Accelerator:** Hardware counter arrays count both operand fetches from shared/off-chip memory and number of subsequent on-chip reuses. Analytical models relate the unrolling factors $(U_M, U_N, U_K)$ in 2D and 3D systolic dataflows to achievable MRR [2602.11357].

- **LLVM Static Analysis:** Construction of a bracketed static memory trace from control-flow and loop annotations enables recursive computation of the reuse distance histogram. Once the profile is obtained, the MRR is immediately calculated for any cache capacity using the definition above. Notably, this method executes in time independent of program input size [2311.12883].

## 3. Reported MRR Values and Correlations

Empirical measurements across systems reveal strong associations between high MRR, throughput gains, and resource savings:

| System/Task                | Reported MRR (or Proxy)           | Impact on Latency/Throughput | Impact on Fidelity        |
|----------------------------|-----------------------------------|-----------------------------|--------------------------|
| GPT-2, WikiText-103 [2512.16843] | 72% global; $>$90% low layers      | $2.5 \times$–$3.3 \times$ speedup    | $<$0.5% accuracy loss    |
| BERT-base, SQuAD [2512.16843]    | 78% global                        | $2.4 \times$ speedup                 | $<$0.2% F1 drop          |
| ENGRAM-R, LoCoMo [2511.12987]    | 88.4% input; 71.7% reasoning      | 68% latency reduction                | $+$2.5% for multi-hop    |
| Voltra, ResNet50 [2602.11357]    | 100% spatial utilization (proxy)  | Up to $2.0\times$ spatial, $2.36\times$ speedup | –               |
| LLVM Static Analysis [2311.12883]| Derived per cache profile         | –                                   | –                       |

Extensive ablations on similarity threshold $\tau$, cache budget, and eviction policy in LLMCache reveal distinctive trade-offs: higher $\tau$ (0.88) ensures $<$0.1% accuracy loss but reduces MRR ($\sim$65%), while lower $\tau$ boosts MRR at some cost in upper-layer accuracy [2512.16843]. In ENGRAM-R, input and reasoning MRRs above 85% consistently yield order-of-magnitude reductions in total token budget, especially for multi-hop tasks [2511.12987]. In hardware, the 3D-spatial scheme in Voltra directly multiplies spatial or temporal MRR by the unrolling factor, up to 2$\times$ that of traditional 2D arrays, translating into up to 50% savings in DRAM bandwidth [2602.11357].

## 4. Methodological Variants and Design Considerations

The parameterizations and system-level choices critically affect MRR outcomes and their downstream implications:

- **Fingerprinting and Matching Criteria (LLMCache):** Choice of hash/similarity function and threshold $\tau$ mediates the tension between hit rate (MRR) and output quality.
- **Eviction Strategies:** LRU, frequency-based, and divergence-aware policies adjust the window of reuse, with LRU/FRQ optimizing short-term MRR and divergence-aware methods sacrificing some MRR for longer-term output fidelity [2512.16843].
- **Dataflow Architecture (Voltra):** Size and aspect of the on-chip buffer tiles, FIFO depth, and streamer channel width together define maximal attainable spatial/temporal MRRs, but incur area/power costs and may exacerbate bank contention [2602.11357].
- **Static Trace Analysis (LLVM):** The granularity of loop brackets and block-level CFG determines the accuracy of the reuse-distance estimation, though the method is invariant to input data size [2311.12883].
- **ENGRAM-R Retrieval Budget (K):** Reducing $K$ increases MRR_input but may undercut recall of critical facts; a retrieval budget sweep balances reuse against answer accuracy [2511.12987].

## 5. Practical Effects and Trade-offs

High MRR consistently confers substantial reductions in wall-clock latency, compute, and off-chip bandwidth usage, with controllable or negligible degradation in accuracy. Key empirical insights include:

- LLMCache achieves between 2.2$\times$ and 3.1$\times$ speedups at global MRRs of 70–78%, with $<$0.5% drop in F1 or end-task accuracy, especially when restricting caching to lower transformer layers [2512.16843].
- In large-reasoning pipelines via ENGRAM-R, input and reasoning MRRs of $\sim$90% and $\sim$75% reduce context and reasoning tokens by factors of $>5\times$ and $>3\times$, with observed accuracy improvements in composition-heavy QA tasks [2511.12987].
- Hardware spatial and temporal utilization scales directly with achieved MRR; Voltra's 3D tiling and streaming yield measured 2.12–2.94$\times$ boosts in effective reuse [2602.11357].
- Memory-vs-hit-rate curves display strong diminishing returns beyond moderate cache or memory investments: doubling cache from 500MB to 1GB raises MRR by only $\sim$4% in LLMCache [2512.16843].
- Data compaction (e.g., PCA) for cached outputs effects minor ($\sim$1%) absolute reductions in MRR, often justified by memory savings [2512.16843].

## 6. Static Analysis and Reuse Profiling

Barai et al. demonstrate that accurate MRR estimation is attainable in constant time via LLVM-based static analysis, without dynamic trace generation. Constructed control-flow graphs, loop annotations, and recursive profile computation directly yield the probability distribution of reuse distances, from which MRR for any LRU cache is calculated as the cumulative probability up to the cache's capacity [2311.12883]. This method enables rapid, accurate forecasting of memory system performance for arbitrary input or workload scale.

## 7. Broader Implications and Optimization Guidelines

Maximizing MRR is a central goal in the design of high-throughput inference engines, efficient accelerators, and cache-optimized programs. Across domains, the following principles surface:

- **Locality-aware Dataflows:** Deep spatial and temporal reuse (3D unrolling, streaming, loop tiling) significantly augment MRR and system efficiency [2602.11357].
- **Semantic Indexing and Adaptive Eviction:** Tailoring cache lookup keys, thresholds, and eviction policies allows practitioners to tune the balance between speed, memory, and task fidelity [2512.16843].
- **Typed Memory and Citation Control:** For reasoning models, explicit fact-card abstractions and enforced memory citation prevent regeneration of redundant material, sharply elevating effective MRR [2511.12987].
- **Static Profiling:** Rapid, input-invariant MRR estimation empowers compiler and hardware designers with actionable intelligence for buffer sizing, tile partitioning, and prefetch allocation [2311.12883].

Empirically, elevated MRR is consistently associated with reduced token usage, lower inference latency, and increased hardware utilization, provided mechanisms for memory freshness and semantic matching are robust. Adaptive policies that govern retrieval, eviction, and memory update ensure that high MRR can be attained without compromising correctness or long-horizon compositional accuracy.

Source: https://www.emergentmind.com/topics/memory-reuse-rate-mrr