Miras Framework: Associative Memory in DL

• Miras Framework is a design paradigm that defines sequence models as associative memory mapping keys to values with attentional bias and retention control.
• It decomposes model design into memory architecture, attentional bias, retention gate, and learning algorithm, enabling tailored inductive biases for language modeling and reasoning.
• Empirical benchmarks show Miras variants outperform Transformers and linear RNNs, achieving state-of-the-art results in recall-intensive and long-context tasks.

The Miras Framework encompasses a general design paradigm for constructing deep learning architectures built on associative memory principles with an emphasis on test-time learning dynamics, attentional bias, and explicit retention control. It offers a systematic approach for unifying and extending a broad family of sequence models, including Transformers and modern linear RNNs, while enabling new models with tailored inductive biases for language modeling, reasoning, and recall-intensive settings (Behrouz et al., 17 Apr 2025).

1. Foundations and Conceptual Overview

At its core, the Miras framework regards any sequence model as instantiating an associative memory module $M: K \to V$, which learns, at each step, a mapping from key to value via an attentional bias objective. Each step alternates between learning from the newest key-value pair via a local attentional bias loss and retaining or forgetting past knowledge through a parametric retention gate.

The update at time $t$ is generally:

$$W_t = \mathrm{Arg\,min}_W \left[\tilde{\ell}_t(W; k_t, v_t) + \mathrm{Ret}_t(W, W_{t-1})\right]$$

which parallels FTRL-style (Follow-The-Regularized-Leader) updates common in online optimization. All modern sequence architectures are interpretable as special cases within this learn-retain formalism.

2. The Four Axes of Architectural Design

Miras decomposes model specification into four mutually orthogonal choices:

A. Memory Architecture:

• Can be vector-valued (as in RetNet), matrix-valued (Hebbian RNNs, DeltaNet), or deep parametric (MLP-based, as in Titans).
• Depth and form govern capacity for storing complex, long-range mappings.

B. Attentional Bias Objective ($\ell(\cdot)$):

• Standard options:
  • Dot-product similarity (classic self-attention)
  • $\ell_2$ regression (Delta rule)
• Novel proposals:
  • $\ell_p$ regression (for variable sensitivity/robustness):

  $\ell_p(W; k, v) = \|W k - v\|_p^p$ - Huber loss (robust to outliers): Piecewise $\ell_2/\ell_1$-style steps. - Robust regression (adversarial shifts in value).

C. Retention Gate:

• Standard:

  • Elementwise forget gate $\alpha_t \in [0,1]^d$
  • Global $\ell_2$ regularization
• Novel forms:
  • KL/f-divergence regularization on simplex-constrained weights
  • Elastic-net ($\ell_1 + \ell_2$ regularization)
  • General $\ell_q$-norms with mirror descent
  • Bregman divergences for advanced updates

D. Memory Learning Algorithm:

• Single-step SGD, GD with momentum, mirror descent, or closed-form (e.g., softmax attention).
• Multi-step inner-loop variants possible for richer memory updates.

This taxonomy permits a systematic exploration of model space, revealing that existing transformers, Titans, RetNet, and other architectures are recoverable as limiting choices along these four axes.

3. Miras Model Instantiations

Three principal model families demonstrate the flexibility and empirical power of the Miras approach:

Model	Memory	Attentional Bias	Retention Gate	Algorithm
Moneta	2-layer MLP	$\ell_p$ ($p=3$)	$\ell_q$ ($q=4$) norm	SGD
Yaad	2-layer MLP	Huber (adaptive)	Local $\ell_2$ + global $\ell_2$	SGD
Memora	2-layer MLP	$\ell_2$ regression	KL-divergence + entropy	SGD, softmax

• Moneta is specialized for high-fidelity recall, particularly on noisy/hard-retrieval tasks; Update:

$$W_t = \frac{A_t}{\|A_t\|_4^2},\quad A_t = \alpha_t A_{t-1} - \eta_t \nabla\ell_3$$

• Yaad targets robustness to outliers using Huber loss with input-dependent threshold $\delta_t$.
• Memora uses KL-divergence for retention, which allows weight resets and normalization via softmax.

All variants utilize a Llama-style macro-architecture, replacing canonical attention blocks with a Miras memory module.

4. Empirical Benchmarks and Performance

Miras variants have been benchmarked on language modeling (Wiki, lumi), commonsense reasoning (PIQA, HellaSwag, ARC, SIQA, BoolQ), needle-in-a-haystack synthetic tasks (S-NIAH 1–8K), and scaling tests up to 32K context tokens.

Key findings:

• Miras variants consistently outperform both Transformers and the latest linear RNNs (RetNet, Gated DeltaNet, Mamba2, TTT) at all model scales ≥340M.
• Moneta achieves $>$99% accuracy on S-NIAH-PK up to 8K contexts (vs. 90% for best RNN baselines).
• Each variant demonstrates domain-specific strengths: Moneta for recall, Yaad for outlier-sensitive tasks, Memora for balanced metrics.
• Hybrid stacking of Miras layers with sliding-window attention provides further improvements, though pure Miras models attain state-of-the-art results independently.

Ablations verify the criticality of advanced retention gates and attentional biases; varying $p$ (Moneta) and $q$ directly impacts scaling with input length and recall robustness.

5. Practical Guidelines for Practitioners

• Memory Architecture:

Use deep/MLP memory for long/complex contexts; vector/matrix modules suffice for short or resource-constrained applications.

• Attentional Bias:

$\ell_p$ (with $p\approx3$) for balance between noise-tolerance and sharpness; Huber for outlier resilience; dot-product or $\ell_2$ for standard settings.

• Retention Gate:

Channel-wise gates capture richer temporal structure. KL-based gating enables hard resets. Elastic-net promotes sparsity if desired.

• Learning Algorithm:

SGD suffices for most $\ell_p$, Huber configurations; mirror descent needed for non-Euclidean penalties.

Further recommended practices include chunk-wise block parallelization (block size 16–64), low-rank projections for scalar hyperparameters, gradient clipping in the memory update to control instabilities, and usage of advanced activation (GELU, SwiGLU) and normalization (RMSNorm, RoPE) schemes (Behrouz et al., 17 Apr 2025).

6. Extensions and Impact

Miras delineates the design space for sequence modeling via explicit enumeration of associative memory type, attentional bias, retention function, and update scheme. This modularity demystifies architectural innovations in the field, framing new models as systematic points within this four-dimensional taxonomy.

Potential extensions include dynamically learning bias/retention exponents per token, constructing mixture-of-bias-memory modules, integrating adaptive step sizes, and introducing hierarchical or multi-scale memory organization. A plausible implication is that Miras' structure offers a robust blueprint for next-generation recurrent and hybrid architectures, tuned for long-context, high-noise, or recall-intensive scenarios (Behrouz et al., 17 Apr 2025).