Recurrent Attention Hybrid Language Models

• Recurrence and attention are complementary in recurrent-attention hybrid language models, providing a balance between precise memory retrieval and efficient representation of sequential data.
• Recurrence-based approaches like Recurrent Attention Translation & AMSRN show good improvements in key tasks like neural machine translation, where solutions like Recurrent Attention improve retrieval by caching attention distributions, leading to competitive results (BLEU transmission gains of up to 2.9 in some cases).
• Recurrent-attention techniques boost low-resource settings’ efficiency in language modeling via recurrent steps, such as QRNNs in Efficient Language Modeling for Low-Resource Settings, achieving a significant performance gain on benchmarks compared to Transformer models, and the efficiency and modeling power of these techniques makes them strong candidates for use in practice.
• What are some of the key architectural categories in recurrent-attention hybrid models, and how do their efficiencies vary? How do these architectures balance to optimize workflows?

Recurrent-attention hybrid LLMs combine recurrent or state-space sequence processing with attention-based memory access. Their defining principle is that recurrence and attention need not be competing alternatives: recurrence can encode, compress, transport, or continuously update sequence information, while attention can provide precise content-addressable retrieval. Hybridization occurs at several levels, including attention-history modeling, recurrent memory augmented with attention, heterogeneous layer stacks, within-layer branch fusion, cross-layer alternation, inference procedures, and parameter-efficient adaptation. The resulting design space spans recurrent-attention mechanisms for neural machine translation and language modeling, low-resource RNN–Transformer architectures, post-training linearization of pretrained Transformers, hybrid state-space LLMs, memory-aware serving systems, and training objectives that explicitly coordinate distinct memory pathways.

1. Conceptual foundations and architectural taxonomy

A recurrent-attention hybrid contains at least two forms of temporal memory. Recurrent components update a fixed-size state sequentially or through parallelizable scans. Attention components compute content-dependent interactions over stored representations, often retaining token-indexed keys and values. In simplified form, recurrence follows

$h_t = F(h_{t-1},x_t),$

whereas causal attention computes a weighted combination of previous values,

$y_t=\sum_{i\leq t}\alpha_{t,i}v_i.$

The two mechanisms differ in memory semantics. A recurrent state compresses preceding tokens into a bounded representation and can provide linear sequence processing with constant-size inference state. Attention preserves token-position-specific representations and supports direct lookup, but its full form has quadratic interaction cost and a key–value cache that grows with context length. Sliding-window attention reduces the accessible history but retains explicit local retrieval.

The distinction is functional as well as computational. Controlled cache interventions on Qwen3.5 and Falcon-H1 found that exact retrieval survives through the attention cache but collapses through recurrent state, whereas output language and persona remain substantially available through recurrence (Afendulev et al., 3 Sep 2026). In the principal experiments, KV-only conditions retained approximately 64–98% of full exact-retrieval accuracy, while recurrent-only retrieval collapsed to zero. Conversely, recurrent-only language following retained approximately 70–80% of full performance, while KV-only language following fell to approximately 1%. These results support the characterization:

$$\text{attention} \rightarrow \text{addressable exact retrieval},$$

$$\text{recurrence} \rightarrow \text{compressed response prior and contextual control}.$$

This distinction should not be interpreted as an absolute separation. The recurrent state can retain semantic and morphological information, and the attention cache can contain stylistic information. The interventions measure causal contribution under specific generation procedures rather than static information content.

Several architectural categories recur in the literature:

1. Recurrent attention histories: attention distributions themselves receive recurrent state, as in recurrent-attention modeling for neural machine translation.
2. Attention over recurrent histories: an RNN or LSTM produces a sequence of hidden states that an attention module later revisits, as in AMSRN.
3. Heterogeneous layer stacks: recurrent or quasi-recurrent layers replace selected Transformer layers, while attention remains in deeper or later layers.
4. Parallel branch hybrids: recurrent and attention branches process the same representation independently and are fused afterward.
5. Within-layer hybrids: recurrent and local-attention branches coexist inside the same attention sublayer.
6. Procedural hybrids: a recurrent architecture is combined with chunk-wise reading, summary generation, or other inference-time memory-management procedures.
7. State-aware systems hybrids: speculative decoding, prefix caching, state quantization, and cross-model state transfer are redesigned around recurrent state semantics.

2. Recurrent modeling of attention and historical memory

The earliest recurrent-attention formulation considered here, “Neural Machine Translation with Recurrent Attention Modeling,” treats attention as a temporally structured object rather than a sequence of independent distributions (Yang et al., 2016). A conventional encoder–decoder model represents a source sentence $x=(x_1,\ldots,x_S)$ with bidirectional encoder annotations $h_i$ and uses a decoder state $s_j$ to compute attention scores

$e_{ij}=v_a^\top\tanh(W_a h_i+U_a s_j).$

The normalized attention distribution is

$$\alpha_{ij} = \frac{\exp(e_{ij})} {\sum_{i'=1}^{S}\exp(e_{i'j})},$$

and the source context is $c_j=\sum_i\alpha_{ij}h_i$.

The recurrent-attention model adds a dynamic memory $y_t=\sum_{i\leq t}\alpha_{t,i}v_i.$0 for every source position. Rather than receiving only the attention assigned to position $y_t=\sum_{i\leq t}\alpha_{t,i}v_i.$1, the recurrent unit receives a local attention-history vector,

$y_t=\sum_{i\leq t}\alpha_{t,i}v_i.$2

and updates

$y_t=\sum_{i\leq t}\alpha_{t,i}v_i.$3

The attention score then incorporates both static source content and dynamic attention history:

$y_t=\sum_{i\leq t}\alpha_{t,i}v_i.$4

The dynamic state affects addressing but is not included in the context read by the decoder. The context remains a weighted average of the static annotations $y_t=\sum_{i\leq t}\alpha_{t,i}v_i.$5. Thus, recurrence determines where the model looks, while the read operation returns source content from the static memory.

This construction was intended to represent fertility, coverage, revisitation, and relative distortion. A window of one tests primarily coverage-like information; a window of eleven exposes neighboring attention histories and local ordering. In English–German translation, the window-11 model improved BLEU over the baseline by $y_t=\sum_{i\leq t}\alpha_{t,i}v_i.$6 on newstest2014 and $y_t=\sum_{i\leq t}\alpha_{t,i}v_i.$7 on newstest2015, while the window-one model did not improve substantially. With UNK replacement, the best recurrent-attention system reached 22.1 and 25.0 BLEU on those tests. On Chinese–English, the window-one model improved from 27.3 to 27.9 BLEU, and the window-11 model reached 28.8, or 29.3 with UNK replacement.

AMSRN applies a different direction of hybridization to language modeling. It preserves an ordinary LSTM recurrence while storing all previous hidden outputs in an expandable memory,

$y_t=\sum_{i\leq t}\alpha_{t,i}v_i.$8

and uses the current state $y_t=\sum_{i\leq t}\alpha_{t,i}v_i.$9 to produce a key $$\text{attention} \rightarrow \text{addressable exact retrieval},$$0 and two dimension-selection masks $$\text{attention} \rightarrow \text{addressable exact retrieval},$$1 and $$\text{attention} \rightarrow \text{addressable exact retrieval},$$2 (Liu et al., 2016). The first mask determines which dimensions participate in similarity scoring:

$$\text{attention} \rightarrow \text{addressable exact retrieval},$$3

while the second determines which dimensions are retrieved:

$$\text{attention} \rightarrow \text{addressable exact retrieval},$$4

The output distribution uses both the current recurrent state and retrieved memory. Independent, tied, and complementary relationships between the masks were evaluated. On Penn Treebank, Switchboard, and Chinese Gigaword, memory selection improved over the LSTM baseline, reaching perplexities of 133.36, 92.49, and 86.85, respectively, compared with 143.31, 93.90, and 94.03 for the LSTM. Entropy regularization improved Penn Treebank to 131.43 but worsened Switchboard and Gigaword, indicating that concentrated attention is not universally appropriate.

AMSRN has $$\text{attention} \rightarrow \text{addressable exact retrieval},$$5 memory for a prefix of length $$\text{attention} \rightarrow \text{addressable exact retrieval},$$6 and straightforward sequence cost of $$\text{attention} \rightarrow \text{addressable exact retrieval},$$7, whereas recurrent-attention translation maintains $$\text{attention} \rightarrow \text{addressable exact retrieval},$$8 dynamic states and performs approximately $$\text{attention} \rightarrow \text{addressable exact retrieval},$$9 recurrent-memory updates. These models establish two complementary principles: attention histories can themselves be recurrently modeled, and recurrently generated representations can serve as an expandable attention-addressable memory.

3. Layer placement, branch interaction, and recurrent approximations

The efficiency-oriented RNN–Transformer work “Efficient Language Modeling for Low-Resource Settings with Hybrid RNN–Transformer Architectures” studies where recurrent processing should replace Transformer computation (Lindenmaier et al., 2 Feb 2025). Its blocks include feed-forward Transformer blocks, relative multi-head attention, AWD-QRNN layers, and RNN-dropout. The QRNN computes convolutional gate inputs in parallel and retains a recurrent pooling stage, giving linear sequence processing with a simpler sequential operation than a conventional LSTM.

Three principal layouts were evaluated:

• Attn-QRNN: predominantly recurrent processing with one late attention layer;
• PAR Transformer: reduced attention with one attention block followed by three feed-forward blocks;
• Hybrid Transformer: two lower AWD-QRNN layers followed by a reduced Transformer stack containing four attention stages.

The Hybrid Transformer layout is

$$\text{recurrence} \rightarrow \text{compressed response prior and contextual control}.$$0

where $$\text{recurrence} \rightarrow \text{compressed response prior and contextual control}.$$1 denotes AWD-QRNN, $$\text{recurrence} \rightarrow \text{compressed response prior and contextual control}.$$2 relative multi-head attention, $$\text{recurrence} \rightarrow \text{compressed response prior and contextual control}.$$3 a feed-forward block, and $$\text{recurrence} \rightarrow \text{compressed response prior and contextual control}.$$4 RNN-dropout. The architecture places recurrent processing in lower layers and attention in higher layers. On Enwik8, the approximately 41M-parameter Hybrid Transformer achieved 1.013 test BPC, compared with 1.047 for PAR and 1.058 for Transformer-XL. On Wikitext-103 it achieved 20.91 test perplexity with approximately 60M parameters, compared with 21.59 for PAR and 24.0 for Transformer-XL. The Hybrid–PAR difference on Wikitext-103 was not statistically distinguishable, whereas the Enwik8 improvement over PAR was reported as significant with $$\text{recurrence} \rightarrow \text{compressed response prior and contextual control}.$$5.

The results support selective rather than wholesale replacement of attention. A single late attention layer was insufficient to match the reduced Transformer, but recurrent lower layers improved performance when attention remained available at higher representational levels. The evidence does not establish that lower recurrent placement is universally optimal: the study did not exhaustively compare alternating layouts, upper recurrent layers, or all possible recurrent mechanisms.

Later work applies hybridization to pretrained Transformers. “Liger: Linearizing LLMs to Gated Recurrent Structures” converts pretrained Transformer attention into gated recurrent structures by reusing key projections to construct gates (Lan et al., 3 Mar 2025). The conversion replaces softmax attention with recurrent modeling, optionally retaining a sliding-window softmax branch. The recurrent state follows a gated update such as

$$\text{recurrence} \rightarrow \text{compressed response prior and contextual control}.$$6

where $$\text{recurrence} \rightarrow \text{compressed response prior and contextual control}.$$7 is derived from the pretrained key projection. Low-rank adaptation is then applied to query, key, and value matrices. The default LoRA rank is eight, and approximately $$\text{recurrence} \rightarrow \text{compressed response prior and contextual control}.$$8 of model parameters are trainable.

The principal hybrid, Liger Attention, combines a gated recurrent modeling branch with causal sliding-window attention:

$$\text{recurrence} \rightarrow \text{compressed response prior and contextual control}.$$9

The default local window is 64. The recurrent branch supplies compressed long-range memory, while the sliding-window branch preserves exact local softmax interactions. On Llama-3-8B, Liger-GLA achieved an average benchmark score of 67.0, compared with 71.7 for the original model, and recovered the reported 93% of original performance using approximately 0.02 billion adaptation tokens. Removing sliding-window attention reduced the average to 54.2, demonstrating that the recurrent state was not a complete substitute for local softmax attention. Liger Attention maintained approximately 16.37 GB of GPU memory across generation lengths from 1K to 32K in the reported experiment, while the Llama-3 baselines reached out-of-memory at 32K.

Distil-xLSTM takes a distillation rather than architectural-hybrid approach (Thiombiano et al., 24 Mar 2025). Its student contains only sLSTM and mLSTM blocks, alternating across six blocks, and is trained from a Qwen2.5-1.5B teacher. The mLSTM maintains a matrix-valued memory,

$x=(x_1,\ldots,x_S)$0

and retrieves through a query. The student is trained with cross-entropy, softened teacher distributions, annealing, and optional hidden-state alignment. It is therefore a recurrent alternative learned from attention behavior, not an explicit attention–recurrence hybrid. The reported evidence is limited to training stability and loss behavior; no numerical perplexity, downstream benchmark, latency, or memory comparison establishes Transformer-equivalent quality or practical speedup.

4. Modern hybrid language-model topologies and memory specialization

Modern hybrid LMs differ principally in whether recurrent and attention pathways are sequential, parallel, or fused within layers.

WuNeng retains ordinary multi-head attention and adds RWKV-7 state-driven heads (Xiao et al., 27 Apr 2025). Its recurrent state follows a generalized delta-rule update,

$x=(x_1,\ldots,x_S)$1

Ordinary attention outputs $x=(x_1,\ldots,x_S)$2, recurrent state outputs $x=(x_1,\ldots,x_S)$3, and intermediate middle heads $x=(x_1,\ldots,x_S)$4 interact through concatenation, additive modulation, or gated fusion. Multi-token state processing allows the continuous RWKV-7 state to modify attention queries across a sequence. WuNeng is explicitly expressivity-oriented: it retains full attention and does not eliminate the KV cache. Preliminary WuNeng-7B results exceeded Qwen2.5-7B-Instruct on MMLU and GSM8K, but the experiments were preliminary and lacked systematic ablations, latency measurements, and controlled long-context studies.

The Mamba–Transformer study “Understanding and Enhancing Mamba-Transformer Hybrids for Memory Recall and Language Modeling” compares sequential and parallel integration (Lee et al., 30 Oct 2025). Mamba supplies a fixed-size recurrent state, while sliding-window attention supplies explicit local retrieval. Sequential configurations include Mamba-first and attention-first arrangements. Parallel configurations process both branches from the same input and fuse them through averaging, projection, or merge attention.

At approximately 430M parameters, the parallel merge-attention model reached 51.4 commonsense accuracy and 19.7 recall accuracy. The best sequential MFSF model reached 51.5 and 19.3. At 1.3B parameters, parallel merge attention again led with 61.0 commonsense accuracy and 29.0 recall, while sequential MFSF reached 60.3 and 28.4. The study reports that sequential hybrids tend to perform better on shorter contexts, while parallel hybrids are more effective on longer contexts. Its interpretation is that sequential composition aligns branch representations but can reduce representational diversity; parallel composition preserves more independent recurrent and attention representations and relies on trainable fusion.

The topology-dependent adaptation behavior is examined directly in “Where Should LoRA Go?” (Borobia et al., 24 Apr 2026). Qwen3.5-0.8B contains 18 GatedDeltaNet layers and six attention layers in a sequential arrangement. Falcon-H1-0.5B contains parallel Mamba-2 and attention branches in every block. Attention-only LoRA used 1.08M parameters on Qwen and 2.21M on Falcon, compared with 10.82M and 11.47M for full eligible adaptation.

Attention-only adaptation was the strongest or most efficient condition in both models. On GSM8K-trained Qwen, softmax-only improved accuracy from .297 to .398, while GDN-only reduced it to .148, a decline of 14.8 percentage points. On Falcon, SSM-only improved GSM8K from .383 to .469, or 8.6 points, but attention-only reached .555. The results suggest that sequential recurrent-backbone adaptation can disrupt downstream representations, whereas parallel recurrent adaptation can remain constructive. Because only one model represents each topology, topology and model-family effects are not fully separable.

A related functional analysis, “The Key Handoff,” examines two-hop retrieval (Kale et al., 26 Sep 2026). In sequential hybrids, early computation forms an intermediate key, recurrent layers transport it, and a later attention layer converts it into an answer. The key’s causal effect peaks at the input to the resolving attention layer. Across 23 depth profiles, the resolving layer was aligned with the key peak, and in sequential hybrids attention accounted for 88–96% of the answer-conversion rise in the reported tasks.

The study also found that some post-attention recurrent layers act as addressable associative memories. Recurrent-memory interventions changed answers toward the value bound to the queried key rather than the value occupying the same position. This behavior was observed in four hybrids, with odds changes ranging from approximately 1.3 to 2.7, but was absent in several related models. Thus, recurrent state can function as a transporter, a compressed associative memory, or both, depending on architecture and training.

5. Training objectives, inference procedures, and adaptation surfaces

Architectural access to multiple memory pathways does not guarantee that a model will use them effectively. “Balancing Memory Pathways” finds that hybrid models rely substantially more on attention than recurrence (Lee et al., 5 Oct 2026). On Qwen3.5, cross-segment QA accuracy was 30.2% with full access, 20.2% with attention-only cross-segment access, and 5.1% with recurrent-only access. Standard SFT improved full accuracy from 30.2% to 34.6% but left recurrent-only accuracy essentially unchanged, from 5.1% to 5.0%.

The proposed auxiliary objective masks attention across past and current segments while allowing recurrent state propagation to remain uninterrupted:

$x=(x_1,\ldots,x_S)$5

The auxiliary pass forces information from earlier segments to reach the prediction through recurrence. With $x=(x_1,\ldots,x_S)$6, Qwen3.5 recurrent-only accuracy increased from 5.0% to 22.8%, while attention-only accuracy remained close to the SFT condition. Average QA performance increased from 34.6% to 38.5% for Qwen3.5 and from 31.0% to 36.2% for Nemotron-H. Gains were especially strong on Qasper, NarrativeQA, MINTEval, and long-horizon agentic tasks. Across the agent benchmarks, average improvements were 3.5 points on Quest, 9.3 on Treasure, 22.0 on PutNextLocal, and 13.5 on GoToObjMaze. The method can slightly harm retrieval-dominated tasks, demonstrating a trade-off between recurrent aggregation and exact attention retrieval.

Distillation offers another route to train recurrent substitutes. Distil-xLSTM uses teacher logits, annealed temperature, and hidden-state alignment, but its student remains recurrent-only. By contrast, Liger uses lightweight LoRA to adapt a converted recurrent or hybrid operator. These approaches differ from S0 tuning, which learns only the initial recurrent state of each recurrent layer while freezing all model weights (Young, 1 Apr 2026). On Qwen3.5-4B, S0 tuning with approximately 48 execution-verified HumanEval solutions improved greedy pass@1 from 48.8% to 72.2%, exceeding the primary LoRA result of 61.5% by 10.8 percentage points. The tuned initial state adds no per-token inference computation and can be selected as a task-specific state file. On FalconH1-7B, S0 reached 71.8% and LoRA 71.4%, a statistically indistinguishable result under three seeds. Transfer gains were reported on MATH-500 and GSM8K, but not Spider.

Component-aware LoRA placement further shows that adaptation surfaces are not interchangeable. Attention is a minority parameter pathway but often the most effective PEFT target. Recurrent-state initialization and recurrent-backbone LoRA address different mechanisms: S0 changes the launch condition of the recurrent trajectory, whereas LoRA changes the transformation at every token. State offsets applied at every recurrent step achieved 27.1 percentage points of improvement on Qwen3.5, above S0 but with per-step inference cost.

Inference procedures can also compensate for recurrent memory limitations. Smooth Reading divides long contexts into chunks, preserves the recurrent state between chunks, and generates task-oriented summaries containing target, clues, reason, and continuation control (Liu et al., 25 Jul 2025). It is not an architectural attention–recurrence hybrid, but a recurrent-only method with a procedural hybrid structure. On LongBench, SWA-3B-4k improved from 41.70 under one-step inference to 50.99 under Smooth Reading, exceeding the Qwen2.5-3B one-step score of 47.38. On NIAH, SWA-Smooth Reading achieved approximately 99.8% average accuracy from 64K through 256K contexts, whereas one-step SWA averaged 3.40% over the same range. The method preserves linear computation and constant-size recurrent state but introduces summary-generation overhead and substantial chunk-size sensitivity.

6. Systems, state management, and open limitations

Hybrid models require systems mechanisms that understand both token-indexed attention state and path-dependent recurrent state.

Tree speculative decoding illustrates the issue. In an attention-only Transformer, a candidate tree can be verified using an ancestry mask. In a recurrent hybrid, each candidate must also receive the recurrent state produced by its own root-to-node path. Bole converts Gated DeltaNet recurrence into an exact tree-structured closed form, computes proposal nodes in parallel, stores token-level factors rather than full state snapshots, and reconstructs only the accepted path (Wang et al., 3 Aug 2026). The recurrent state satisfies a tree-linear system whose ancestor matrix is nilpotent, enabling a finite Neumann expansion. Bole achieved 3.4–7.7× speedups for linear-attention tree verification and reduced transient state memory by 82–99×. Across four Qwen3.5 models, it achieved up to 4.72× offline decode throughput over autoregressive decoding and up to 2.03× over the strongest tree-speculative baseline.

GDN Tree-Scan addresses the same problem through branch-local state selection and replay in vLLM (Ma, 20 Sep 2026). Attention uses an additive tree-bias mask, while each recurrent node explicitly reads the state of its tree parent. Only the accepted chain is published as durable recurrent state. On Qwen3.6-27B-FP8 at batch size one, a six-node root-branch tree increased committed tokens per event by 17.2% and achieved 23.88 token-weighted decode tokens per second versus 18.80 for native five-step MTP. The reported 27.0% throughput gain is token-weighted and does not imply equivalent end-to-end task speedup because the SWE/Codex workload was prefill-heavy.

Prefix caching faces a related mismatch. Full-attention KV caches are token-addressable, whereas recurrent states generally cannot be rolled back to arbitrary prefix boundaries. Tail-Replay avoids recurrent checkpoints by replaying a recent suffix of a matched prefix to reconstruct an approximate Gated DeltaNet state (Liu et al., 31 Aug 2026). With 5–10% replay budgets, it retained 92.8–99.9% of full-prefill quality across LongBench and RULER. At 32K matched prefixes, it achieved 9.1–14.3× TTFT speedups over full prefill on the evaluated models. The method depends on fading-memory behavior and is approximate; it can fail when distant information remains important.

LatentPort extends state management across model sizes (Villani, 7 Sep 2026). A Qwen3.5-4B source and Qwen3.5-9B target share recurrent and convolutional state geometries but differ in hidden width and parameter count. Translating attention KV while directly copying GDN recurrent and convolutional state produced substantially better continuation loss than translated KV alone. On 64 PG19 documents with 4K prefixes, adding the recurrent package reduced NLL by 0.7473 nats per token relative to KV-only, improving all documents. A 434,176-parameter behavioral correction produced excess NLL of 0.076, JS divergence of 0.022, and native context recovery of 0.918. These results are limited to one direction, one model family, 4K teacher-forced continuations, and a geometry-matched pair; free-generation equivalence and a general state interface remain unproven.

State precision is another systems dimension. “Low-Bit Recurrent States in Hybrid LLMs” derives quantization weights from future observability and state ranges (Chen et al., 25 Sep 2026). Its distortion coefficient is

$x=(x_1,\ldots,x_S)$7

where $x=(x_1,\ldots,x_S)$8 is a normalized state range and $x=(x_1,\ldots,x_S)$9 estimates persistence and erasure. Mixed-precision allocation gives more bits to large-range, slowly decaying, highly observable channels. At per-token write-back, the method achieved excess NLL of +0.040 on Kimi, +0.015 on Qwen3.8, and +0.086 on Qwen3.6 at approximately four-bit mean payloads, outperforming seven baselines in the evaluated windows. At six bits, excess NLL was at most approximately 0.0034 nats in the reported comparisons. At deferred write-back $h_i$0, the method was best in four of 15 model–budget pairs, tied in seven, and worse in four, demonstrating that persistence-aware allocation is not universally optimal.

The empirical and architectural evidence establishes several broad limitations:

• Compression versus exact binding: recurrent state can preserve semantic fields, response modes, and aggregated information, but may lose exact token identity or feature binding.
• Topology dependence: sequential and parallel hybrids respond differently to recurrent adaptation, fusion, and layer placement.
• Training imbalance: unrestricted SFT tends to exploit attention because it is easier to optimize than recurrent storage and retrieval.
• Inference dependence: chunking, replay, speculative verification, and state publication can determine practical long-context behavior as strongly as the underlying architecture.
• Approximate state operations: quantization, prefix replay, cross-model transfer, and tree verification require model-specific numerical and dynamical validation.
• Limited experimental coverage: many results concern sub-billion to single-digit-billion models, controlled synthetic tasks, teacher-forced continuations, or preliminary deployments.
• No universal replacement principle: recurrent state is not a general substitute for explicit attention, and attention is not a complete substitute for recurrent memory.

The accumulated findings support a division of labor rather than a single canonical hybrid architecture. Attention is particularly effective for exact, position-sensitive lookup and local disambiguation. Recurrence can provide compression, temporal continuity, long-range aggregation, key transport, response-mode control, and—in some architectures—content-addressed associative retrieval. Successful systems therefore combine architectural heterogeneity with mechanisms that explicitly coordinate, preserve, quantize, cache, transfer, and verify the two memory pathways.