MQMTAR: Multi-Query Multi-Token Recall

• The paper introduces MQMTAR, a benchmark evaluating multi-query retrieval of key–value pairs across long distractor gaps and multi-token language phenomena.
• It formalizes a token-level task where repeated linguistic elements like bigrams and phrases are recalled, highlighting the limitation of single-query benchmarks.
• Empirical results show that attention mechanisms and Echo's SKA outperform gated convolutions, underlining the necessity for efficient long-context recall.

Multi-query Multi-token Associative Recall (MQMTAR) denotes a retrieval regime in which a model must preserve many key–value bindings across a sequence, survive distractor content and varying interaction distances, and answer multiple later queries from the same prefix. In the cited literature, the formal synthetic benchmark is Multi-Query Associative Recall (MQAR), introduced to correct the mismatch between earlier single-query associative-recall tests and the demands of natural language; the “multi-token” qualifier refers chiefly to the motivating linguistic phenomena—repeated bigrams, repeated phrases, names, and other multi-token expressions—rather than to a separate span-retrieval formalism (Arora et al., 2023, Sridhar et al., 7 May 2026).

1. Terminological scope and historical placement

Earlier associative-recall benchmarks typically presented one query at a fixed position, used a small vocabulary, and asked the model to retrieve a single associated value. MQAR was introduced because these conditions did not match the retrieval structure of real language, where a model may need to perform many recall operations in one forward pass, at different positions, over a large vocabulary, and sometimes in connection with repeated phrases or other multi-token material rather than isolated symbols (Arora et al., 2023).

Within this literature, MQAR is the formal task name. The later Echo work treats MQMTAR / Multi-Query Associative Recall as the failure mode exposed when a sequence contains many KV pairs, then a distractor gap, then several queries asking for associated values. In that usage, MQMTAR functions less as a distinct benchmark definition than as a descriptive emphasis on the same multi-query retrieval problem under long-context conditions (Sridhar et al., 7 May 2026).

This distinction matters conceptually. The benchmark formalism is token-level, but the motivating linguistic examples are often multi-token. As a result, the phrase MQMTAR is best understood as referring to multi-query associative recall in settings where the underlying language phenomena may involve multi-token spans, even though the core synthetic objective remains retrieval of tokenized key–value associations.

2. Formal task structure

MQAR is formulated over an input sequence of interleaved keys, values, and queries: $\bm{u}[0 \cdots N-1] \triangleq \left\{\paren{\bm{k}_0, \bm{v}_0, \bm{q}_0}, \ldots, \paren{\bm{k}_{\frac{N}{3}-1}, \bm{v}_{\frac{N}{3}-1}, \bm{q}_{\frac{N}{3}-1}}\right\}.$ Each element is drawn from a vocabulary $C$, and the goal is, for each query position, to determine whether there exists an earlier key matching that query and, if so, to output the corresponding value: $$\text{for each } 1 \le i \le \frac{N}{3}-1,\ \text{check whether there exists } 0 \le j < i \text{ such that } \bm{q}_i \equiv \bm{k}_j,\ \text{and if so, output } \bm{v}_j.$$

The role positions are fixed modulo 3: $\begin{aligned} \calK &= \{i \in \{0, \ldots, N-1\} \lvert\ i \equiv 0 \mod{3}\},\ \calV &= \{i \in \{0, \ldots, N-1\} \lvert\ i \equiv 1 \mod{3}\},\ \calQ &= \{i \in \{0, \ldots, N-1\} \lvert\ i \equiv 2 \mod{3}\}. \end{aligned}$ These are encoded as one-hot blocks: $\bm{u}[i,:] \equiv \begin{cases} [\bm{k}_i:\bm{0}^c:\bm{0}^c] &\text{ if } i \in \calK \ [\bm{0}^c:\bm{v}_i:\bm{0}^c] &\text{ if } i \in \calV \ [\bm{0}^c:\bm{0}^c:\bm{q}_i] &\text{ if } i \in \calQ \end{cases}$ The crucial shift from classical associative recall is that retrieval is required for each query position, not for a single designated query.

The Echo paper instantiates this family of tasks in a fine-tuning grid with the number of KV pairs $M \in \{4, 8, 16, 32\}$ and distractor gap length from 64 to 4,096 tokens. Each configuration is trained and evaluated independently, so the benchmark directly tests whether a model can retrieve bindings across long gaps and multiple stored pairs rather than generalizing from a different setting (Sridhar et al., 7 May 2026).

3. Why the benchmark was introduced

The empirical motivation for MQAR is a mismatch between what older synthetic tests measure and what LLMs appear to need in actual text. In Zoology, attention-free gated-convolution architectures could match attention on standard synthetic associative-recall tests, yet still underperformed on language modeling. Fine-grained analysis showed that 82% of the perplexity gap between attention and gated convolutions on the Pile was explained by AR Hits, defined as tokens that complete a repeated bigram seen in context but not heavily memorized during training (Arora et al., 2023).

The motivating language phenomena are notably richer than the synthetic benchmark itself. The paper uses examples such as “Hakuna Matata” / “no worries,” names like “Tim Rice,” and repeated phrases and technical terms in ArXiv, GitHub, and StackExchange. It also reports that repeated bigrams in real corpora often occur at different distances, with a power-law distribution of gaps. These observations explain why a benchmark with one query, one lookup, and a tiny symbol table can overstate the recall competence of efficient architectures.

MQAR therefore isolates four demands that earlier tests largely omitted: multiple recalls in one pass, different key-query distances, the need to keep many associations accessible simultaneously, and large-vocabulary token identities. The “multi-token” reading of MQMTAR enters here. MQAR is formally token-level, but it was motivated by natural-language recall patterns that frequently involve multi-token expressions. This suggests that MQAR is a proxy for a broader family of retrieval phenomena rather than an exhaustive description of them.

The same literature also marks a limitation. The repeated-bigram heuristic does not capture fuzzier or more abstract recall, such as synonyms or higher-level concepts. MQMTAR, in its current usage, is therefore narrower than “long-range memory” in the most general linguistic sense.

4. Empirical behavior across attention, gated convolutions, and sparse hybrids

On the Pile, after pretraining on 10B tokens, attention outperformed the strongest gated-convolution baselines by up to 2.1 perplexity points. At the 360M scale, the gap was especially concentrated on AR Hits, and attention maintained a marked advantage over Hyena, RWKV, H3, and Long Conv (Arora et al., 2023).

Model	Pile PPL at 360M	AR Hits log-PPL at 360M
Attention	9.44	1.98
Hyena	10.07	3.83
RWKV	9.79	3.82
H3	10.38	4.81
Long Conv	13.13	—

At larger scale, the gap persisted. At 1.4B parameters, attention achieved 8.19 PPL and Hyena 9.65 PPL on the Pile; on AR Hits specifically, the comparison was 1.91 versus 3.43. The paper also reports that a 70M attention model outperformed a 1.4B Hyena model on AR slice perplexity, and characterizes the smaller attention model as “a full perplexity point better” on AR. In sequence-length scaling, attention solved MQAR with model dimension essentially independent of sequence length, whereas gated convolutions required model dimension that grows at least linearly with sequence length to solve MQAR reliably. As the number of required recalls per example increased, RWKV degraded sharply while attention remained stable (Arora et al., 2023).

Zoology also evaluated simple convolution-attention hybrids. The best results came from input-dependent sparse attention, not from random sparsity. At 360M scale, the reported overall PPL values were 11.01 for pure BaseConv, 12.94 for random selection, 9.54 for programmatic selection, 9.59 for learned selection, and 8.59 for a full attention hybrid. On AR Hits, programmatic selection closed 85% of the gap between pure BaseConv and attention, while learned selection closed 72% using only $k=256$ attention positions per example. The abstract’s summary is that such hybrids can close 97.4% of the gap to attention while maintaining sub-quadratic scaling (Arora et al., 2023).

These results establish the central empirical lesson of the benchmark family: retrieval quality on multi-query recall, rather than generic sequence mixing alone, is a principal bottleneck for efficient attention-free LLMs.

5. Constant-memory retrieval and the Echo architecture

Echo addresses the same retrieval problem from a different direction. It is a KV-cache-free associative recall architecture built around Spectral Koopman Attention (SKA), used as a drop-in replacement for attention layers inside an otherwise Mamba-2-based model. The key claim is that content-based retrieval can be implemented from streaming sufficient statistics rather than from a KV cache that grows as $O(Td)$ with sequence length $T$ (Sridhar et al., 7 May 2026).

For each token, SKA projects keys, queries, and values as

$$z_t = W_k x_t \in \mathbb{R}^r,\quad z_t^q = W_q x_t \in \mathbb{R}^r,\quad v_t = W_v x_t \in \mathbb{R}^P.$$

It then applies sequence-max normalization,

$C$0

and accumulates additive statistics

$C$1

The value map is obtained by ridge regression,

$C$2

With Cholesky factorization $C$3, the whitened Koopman operator is

$C$4

and the query is whitened, power-filtered, and unwhitened: $C$5 Retrieval is then

$C$6

or equivalently

$C$7

When $C$8, SKA reduces to exact ridge retrieval.

The memory state per head is fixed: $C$9 plus the previous key and a scalar norm, for a total of

$$\text{for each } 1 \le i \le \frac{N}{3}-1,\ \text{check whether there exists } 0 \le j < i \text{ such that } \bm{q}_i \equiv \bm{k}_j,\ \text{and if so, output } \bm{v}_j.$$0

floats per head, independent of $$\text{for each } 1 \le i \le \frac{N}{3}-1,\ \text{check whether there exists } 0 \le j < i \text{ such that } \bm{q}_i \equiv \bm{k}_j,\ \text{and if so, output } \bm{v}_j.$$1. The paper states that the 50M model’s total SKA state is about 77 KB in FP32 (Sridhar et al., 7 May 2026).

On the 50M MQAR benchmark, the contrast is sharp. A pure Mamba-2 model remained near chance, around $$\text{for each } 1 \le i \le \frac{N}{3}-1,\ \text{check whether there exists } 0 \le j < i \text{ such that } \bm{q}_i \equiv \bm{k}_j,\ \text{and if so, output } \bm{v}_j.$$2, across every configuration. Mamba-2 + Attention reached 100% on all but one cell, degrading at the largest gap with $$\text{for each } 1 \le i \le \frac{N}{3}-1,\ \text{check whether there exists } 0 \le j < i \text{ such that } \bm{q}_i \equiv \bm{k}_j,\ \text{and if so, output } \bm{v}_j.$$3. Mamba-2 + SKA achieved 100% on every cell tested, including distractor gaps of 4,096 tokens with 32 KV pairs. In smaller synthetic transfer experiments, average accuracy was 54.4 for SSM, 76.2 for SSM+Attn, and 81.1 for SSM+SKA; the SSM+SKA model was reported as best on MQAR (82.2), System Prompt (85.3), ToolTrace (80.5), NIAH (84.2), MultiHop (76.7), and CommonWord (74.1). In length generalization for needle-in-a-haystack trained at length 64, SSM dropped to 2.0% at 2048 and 4096, SSM+Attn to 42.5% at 2048 and 5.0% at 4096, while SSM+SKA remained at 81.2% at 2048 and 65.1% at 4096 (Sridhar et al., 7 May 2026).

Ablations attribute the retrieval gain primarily to the spectral operator / SKA mechanism, not to the action mask. Additional reported choices include independent projections for SKA, preference for sequence-max normalization over per-token $$\text{for each } 1 \le i \le \frac{N}{3}-1,\ \text{check whether there exists } 0 \le j < i \text{ such that } \bm{q}_i \equiv \bm{k}_j,\ \text{and if so, output } \bm{v}_j.$$4 normalization, power filter order $$\text{for each } 1 \le i \le \frac{N}{3}-1,\ \text{check whether there exists } 0 \le j < i \text{ such that } \bm{q}_i \equiv \bm{k}_j,\ \text{and if so, output } \bm{v}_j.$$5 as a sweet spot, ungated additive injection over sigmoid gating, and spectral normalization with learned $$\text{for each } 1 \le i \le \frac{N}{3}-1,\ \text{check whether there exists } 0 \le j < i \text{ such that } \bm{q}_i \equiv \bm{k}_j,\ \text{and if so, output } \bm{v}_j.$$6 to stabilize the operator (Sridhar et al., 7 May 2026).

6. Interpretation, misconceptions, and research significance

A common misconception is that strong performance on classical synthetic associative recall implies strong long-context recall in LLMs. The Zoology results reject that equivalence. Gated convolutions could solve older synthetic tests but still exhibited a substantial recall deficit on language modeling, with 82% of the average quality gap explained by AR Hits. The benchmark was designed precisely to expose that mismatch (Arora et al., 2023).

A second misconception is that MQMTAR is already a general multi-token span-retrieval benchmark. The cited literature does not support that reading. MQAR is explicitly a token-level formal task. Multi-tokenity enters through the language phenomena used to motivate the task—repeated phrases, entities, and bigrams—and through the observation that real text often requires recall over such structures. This suggests that MQMTAR names an empirical problem class broader than the exact synthetic objective, not a completed formalism for arbitrary span retrieval.

A third misconception is that constant-memory recurrence and content-addressed retrieval are interchangeable. Echo’s results argue otherwise. Pure Mamba-2 maintains $$\text{for each } 1 \le i \le \frac{N}{3}-1,\ \text{check whether there exists } 0 \le j < i \text{ such that } \bm{q}_i \equiv \bm{k}_j,\ \text{and if so, output } \bm{v}_j.$$7 memory per step but exhibits a structural memory cliff, whereas attention solves retrieval by retaining a KV cache, and SKA attempts to recover content-based lookup from constant-size sufficient statistics. In this framing, the core issue is not merely model size or recurrence depth but the mechanism by which token interactions are represented and recovered (Sridhar et al., 7 May 2026).

The broader significance of MQMTAR lies in its role as a diagnostic and design target. In Zoology, it identifies recall as the bottleneck separating efficient convolutional LLMs from attention-based ones. In Echo, it motivates a cache-free retrieval mechanism intended for long chain-of-thought reasoning, agentic tool-calling, and related long-context settings. A plausible implication is that future work on “multi-token” associative recall will need to move beyond token-level proxies toward explicit span-level or compositional retrieval objectives while preserving the benchmark’s central pressure points: many bindings, long distractor gaps, variable distances, and multiple queries from the same prefix.