MRFFN: Feed-Forward Memory Retrieval

• MRFFN is a family of feed-forward neural architectures that use explicit memory retrieval mechanisms instead of recurrent state propagation.
• Its design includes variants like FSMN for temporal retrieval and transformer-based key-value modules that leverage learned coefficients and fixed memory slots.
• Empirical results show MRFFN achieves competitive performance in language modeling and speech recognition with faster training and reduced computational latency.

Searching arXiv for the specified MRFFN-related papers and adjacent work to ground the article in the cited literature. Memory Retrieval Feed-Forward Network (MRFFN) is a feed-forward neural architecture in which memory is retrieved without recurrent feedback, but the term is used in more than one technically distinct sense across the literature. In the sequential modeling line, MRFFN is instantiated directly as a Feedforward Sequential Memory Network (FSMN), where hidden layers are augmented with tapped-delay memory blocks that aggregate neighboring hidden activations through learned coefficients and thereby model long-term dependencies without backpropagation through time (Zhang et al., 2015). In transformer analysis and design, MRFFN denotes a feed-forward module interpreted or trained as a key-value memory: either as a persistent pattern-sensitive memory within standard transformer FFNs (Geva et al., 2020), or, in MemoryLLM, as a context-free token-wise neural retrieval memory queried only by static token embeddings (Jaiswal et al., 30 Jan 2026). Related formulations further broaden the concept toward feed-forward attention-based retrieval (Raffel et al., 2015), associative retrieval derived from Hopfield-Fenchel-Young energies (Santos et al., 2024), kernelized minimum-norm memory systems (Iatropoulos et al., 2022), and feed-forward pathways coupled to associative attractor mechanisms (Ravichandran et al., 2022). Taken together, the literature presents MRFFN not as a single canonical architecture, but as a family of feed-forward memory mechanisms defined by explicit retrieval operators, bounded receptive fields or persistent memory slots, and the elimination or reduction of recurrent state dependence.

1. Historical emergence and scope of the term

The earliest direct precursor of MRFFN in the supplied literature is FSMN, introduced as “a new structure to learn long-term dependency” by equipping a standard fully connected feed-forward network with learnable memory blocks in hidden layers (Zhang et al., 2015). In that formulation, the core memory mechanism is positional and temporal: a tapped-delay line aggregates a finite neighborhood of hidden activations into a fixed-size contextual summary, and the resulting system remains strictly feed-forward because gradients flow only across layers, not through time (Zhang et al., 2015).

A closely related earlier formulation focused specifically on language modeling and characterized FSMN as a standard feedforward neural network augmented with learnable sequential memory blocks placed in hidden layers (Zhang et al., 2015). There, the memory block was explicitly cast as a high-order finite impulse response (FIR) filter over past hidden activations, contrasting with recurrent architectures viewed as first-order infinite impulse response (IIR) filters. This distinction was central: FIR filters were presented as always stable, while recurrent models required backpropagation through time and were associated with vanishing or exploding gradients (Zhang et al., 2015).

The term MRFFN later broadened beyond sequential FIR-style memory. One line of work argued that transformer feed-forward layers themselves operate as persistent key-value memories, where each slot consists of a key detecting a textual pattern and a value inducing a distribution over likely next tokens (Geva et al., 2020). Another line, MemoryLLM, made the memory interpretation explicit by decoupling FFNs from self-attention and training them as context-free token-wise neural retrieval memory, with the vocabulary itself forming the fixed, human-interpretable query space (Jaiswal et al., 30 Jan 2026).

This semantic expansion creates a potential misconception. MRFFN does not denote one universally standardized module. In the cited works it can refer to at least three non-identical mechanisms: FIR-like temporal retrieval over hidden-state neighborhoods (Zhang et al., 2015, Zhang et al., 2015), key-value memory internal to transformer FFNs (Geva et al., 2020), and context-free token-indexed FFN memory decoupled from self-attention (Jaiswal et al., 30 Jan 2026). A plausible implication is that MRFFN is best understood as an abstract design pattern—memory retrieval implemented inside a feed-forward computation graph—rather than as a single architecture.

2. Sequential MRFFN as feed-forward temporal retrieval

In the FSMN instantiation, MRFFN is a standard fully connected feed-forward network whose hidden layers are augmented with memory blocks. Each memory block retrieves contextual information from a window of past, and optionally future, hidden activations and encodes it into a fixed-size vector that is fed forward (Zhang et al., 2015). The retrieval operator is a tapped-delay line: at time $t$ and layer $l$, the memory block aggregates $\mathbf{h}_t^{(l)}$ together with its neighbors using learned coefficients, but no recurrent feedback is created and there is no back-in-time dependence through hidden states (Zhang et al., 2015).

The architectural embedding is explicit. If the $l$-th hidden layer produces $\mathbf{h}_t^{(l)} \in \mathbb{R}^{D_l}$, the memory block outputs a context vector $\tilde{\mathbf{h}}_t^{(l)} \in \mathbb{R}^{D_l}$ with the same dimensionality. The next layer consumes both the current activation and the retrieved memory via distinct weight matrices (Zhang et al., 2015). Two variants were defined. In scalar FSMN (sFSMN), all hidden dimensions share the same scalar coefficient per tap. In vectorized FSMN (vFSMN), each hidden dimension has its own coefficient vector per tap, enabling element-wise weighting (Zhang et al., 2015).

The causal scalar formulation is

$$\tilde{\mathbf{h}}_{t}^{(l)} = \sum_{i=0}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$

A residualized memory-retrieval form is

$$\mathbf{m}_{t}^{(l)} = \mathbf{h}_{t}^{(l)} + \sum_{i=1}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$

For bidirectional or non-causal models, past and future activations are both included: $$\tilde{\mathbf{h}}_{t}^{(l)} = \sum_{i=0}^{N_{1}} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)} + \sum_{j=1}^{N_{2}} c_{j}^{(l)} \, \mathbf{h}_{t+j}^{(l)}.$$ The vectorized variants replace scalar taps by element-wise coefficient vectors $\mathbf{A}_{i}^{(l)}$ and $l$0 combined through the Hadamard product (Zhang et al., 2015).

Forward propagation to the next layer is

$l$1

This construction is significant because the memory summary is fixed-size even when the contextual window is large, and because the retrieval path is purely feed-forward (Zhang et al., 2015).

The earlier language-modeling FSMN formulation used forward-only causal memory: $l$2 with scalar taps shared across units in the layer (Zhang et al., 2015). That work did not present backward or bidirectional memory formulas in its experiments, though it noted that such extensions could be defined analogously (Zhang et al., 2015).

3. Mathematical interpretation: FIR retrieval, stability, and optimization

A defining mathematical interpretation of sequential MRFFN is the FIR analogy. The FSMN memory block is described as a tapped-delay line with learned coefficients, computing a finite weighted sum over a bounded temporal window (Zhang et al., 2015). Recurrent layers in RNNs are contrasted with IIR filters because feedback causes $l$3 to depend on $l$4; FSMNs instead approximate such IIR behavior using sufficiently high-order FIR memory blocks while preserving feed-forward stability and efficient training (Zhang et al., 2015).

The earlier FSMN paper sharpened this comparison by writing the recurrent alternative as

$l$5

and emphasizing that FIR filters are always stable whereas training FSMNs requires only standard backpropagation in feedforward nets (Zhang et al., 2015). This is one of the central conceptual justifications for calling the mechanism a retrieval operator rather than a recurrent state update.

Optimization followed standard feed-forward training procedures. For speech recognition, the objective was frame-level cross-entropy against tied HMM state labels in a hybrid DNN-HMM acoustic model. For language modeling, the objective was to minimize perplexity, equivalently maximize next-word conditional likelihood (Zhang et al., 2015). Mini-batch SGD on GPUs, ReLU hidden units, and random initialization were used; for PTB and wiki9, batch sizes were 200 and 500 respectively, with initial learning rate 0.4 and halving after a validation plateau, while PTB used momentum 0.9 and weight decay $l$6 (Zhang et al., 2015). In the earlier FSMN language-modeling study, SGD with mini-batches, Glorot initialization, ReLU, a smaller learning rate for filter coefficients than for standard weights, and momentum plus weight decay on PTB were reported (Zhang et al., 2015).

The scalar memory operation admits an efficient matrix form. For a sequence, the memory computation is

$l$7

where $l$8 is a banded matrix containing the tap coefficients (Zhang et al., 2015). In the earlier formulation, the same idea was expressed for a sentence of length $l$9 using a structured banded matrix $\mathbf{h}_t^{(l)}$0, and extended to mini-batches through a block-diagonal matrix $\mathbf{h}_t^{(l)}$1 (Zhang et al., 2015). This matrix view is not merely an implementation detail; it explains why FSMN training remains GPU-friendly and avoids backpropagation through time.

For scalar FSMN, backpropagation requires only standard BP: $\mathbf{h}_t^{(l)}$2 For vFSMN, the element-wise updates are

$\mathbf{h}_t^{(l)}$3

Bidirectional models add symmetric terms with $\mathbf{h}_t^{(l)}$4 (Zhang et al., 2015). The practical consequence reported in the paper is that training is typically several times faster than BLSTM on the same acoustic task and is more stable because there are no exploding or vanishing gradients through time (Zhang et al., 2015).

4. Transformer FFNs as persistent and context-free retrieval memory

In transformer literature, MRFFN shifts from temporal retrieval to parametric key-value retrieval. The paper “Transformer Feed-Forward Layers Are Key-Value Memories” decomposed the standard two-layer FFN

$\mathbf{h}_t^{(l)}$5

into a sum of memory slots

$\mathbf{h}_t^{(l)}$6

where the $\mathbf{h}_t^{(l)}$7-th row of $\mathbf{h}_t^{(l)}$8 is a key $\mathbf{h}_t^{(l)}$9 and the $l$0-th column of $l$1 is a value $l$2 (Geva et al., 2020). Retrieval occurs because $l$3 measures how strongly the current hidden state matches the textual pattern encoded by the key, while the nonlinearity gates the strength of the corresponding write-back through $l$4 (Geva et al., 2020).

This framework interprets FFNs as persistent memory slots rather than dynamic sequence processors. The matrix form

$l$5

was presented as equivalent to a key-value memory, with the important caveat that FFNs use unnormalized activations such as ReLU or GELU rather than a softmax over keys (Geva et al., 2020). In pre-LN transformers, the FFN operates on a normalized residual and writes back through the residual path: $l$6 This supports a layered composition process in which many memories are active in parallel and their effects are progressively refined across layers (Geva et al., 2020).

The same paper connected value vectors to vocabulary logits through the unembedding matrix $l$7: $l$8 Thus each value induces an output distribution over vocabulary items, and upper layers were reported to align more strongly with plausible next-token continuations (Geva et al., 2020). Concrete empirical support included the finding that human annotators identified at least one pattern for every sampled key, with an average of 3.6 patterns per key, and that 65–80% of top-25 trigger prefixes per key were covered by identified patterns (Geva et al., 2020). Lower layers predominantly captured shallow patterns such as n-grams, last-word cues, and punctuation, while upper layers captured more semantic patterns such as part-of relations, time ranges, and TV episode contexts (Geva et al., 2020).

MemoryLLM radicalized this memory interpretation by decoupling FFNs from self-attention entirely. In that architecture, the FFN is trained and used in isolation from self-attention such that its input is the static embedding of a vocabulary token; the query space is therefore the finite vocabulary itself (Jaiswal et al., 30 Jan 2026). The residual update is

$l$9

where $\mathbf{h}_t^{(l)} \in \mathbb{R}^{D_l}$0 are context-free token embeddings from the embedding layer (Jaiswal et al., 30 Jan 2026). Since $\mathbf{h}_t^{(l)} \in \mathbb{R}^{D_l}$1 depends only on token IDs, the MRFFN query space is fixed, human-interpretable, and independent of context or layer depth (Jaiswal et al., 30 Jan 2026).

MemoryLLM used SwiGLU parameterization: $\mathbf{h}_t^{(l)} \in \mathbb{R}^{D_l}$2 with per-layer LayerNorm on $\mathbf{h}_t^{(l)} \in \mathbb{R}^{D_l}$3 (Jaiswal et al., 30 Jan 2026). The FFN was interpreted as a token-key-value memory with keys given by columns of $\mathbf{h}_t^{(l)} \in \mathbb{R}^{D_l}$4, values by rows of $\mathbf{h}_t^{(l)} \in \mathbb{R}^{D_l}$5, and gates by columns of $\mathbf{h}_t^{(l)} \in \mathbb{R}^{D_l}$6 (Jaiswal et al., 30 Jan 2026). For a query token embedding $\mathbf{h}_t^{(l)} \in \mathbb{R}^{D_l}$7, the gated coefficients are

$\mathbf{h}_t^{(l)} \in \mathbb{R}^{D_l}$8

and retrieval becomes

$\mathbf{h}_t^{(l)} \in \mathbb{R}^{D_l}$9

This implements soft addressing over $\tilde{\mathbf{h}}_t^{(l)} \in \mathbb{R}^{D_l}$0 memory cells (Jaiswal et al., 30 Jan 2026).

A major systems consequence is token-wise lookups (ToLs). For each token and each layer, the MRFFN output is precomputed: $\tilde{\mathbf{h}}_t^{(l)} \in \mathbb{R}^{D_l}$1 and concatenated across layers into

$\tilde{\mathbf{h}}_t^{(l)} \in \mathbb{R}^{D_l}$2

Inference then becomes an index-and-add operation: $\tilde{\mathbf{h}}_t^{(l)} \in \mathbb{R}^{D_l}$3 For MemoryLLM-1B with $\tilde{\mathbf{h}}_t^{(l)} \in \mathbb{R}^{D_l}$4, $\tilde{\mathbf{h}}_t^{(l)} \in \mathbb{R}^{D_l}$5, and $\tilde{\mathbf{h}}_t^{(l)} \in \mathbb{R}^{D_l}$6, FP16 ToL storage was reported as approximately 12.6 GB, INT8 as approximately 6.3 GB, and INT4 as approximately 3.15 GB, with negligible degradation across several tasks (Jaiswal et al., 30 Jan 2026). On a $\tilde{\mathbf{h}}_t^{(l)} \in \mathbb{R}^{D_l}$7 with BF16 ToLs, Base was reported at 9.541 GB and 21.50 ms/token, while MemoryLLM used 6.041 GB and 14.42 ms/token (Jaiswal et al., 30 Jan 2026).

A possible misconception is that such MRFFNs replace attention universally. The data do not support that claim. MemoryLLM retained a standard self-attention sublayer trained conventionally, and its Flex-MemoryLLM variant explicitly reintroduced a context-aware compute FFN on the residual stream: $\tilde{\mathbf{h}}_t^{(l)} \in \mathbb{R}^{D_l}$8 This suggests that context-free FFN memory captures token-level parametric knowledge efficiently, but context-aware computation remains important for closing the performance gap to dense transformers (Jaiswal et al., 30 Jan 2026).

5. Empirical performance across speech recognition, language modeling, and transformer efficiency

The empirical record for sequential MRFFN is strongest in speech recognition and language modeling. On Switchboard plus CallHome, approximately 329 hours, evaluated on the NIST Hub5 2000 SWB portion, baselines were reported as 14.6% WER for a ReLU DNN, 14.2% for a unidirectional LSTM, and 13.5% for a BLSTM with training time per epoch of approximately 22.6 hours (Zhang et al., 2015). On the same task, sFSMN achieved 14.2% WER at approximately 6.7 hours per epoch, while vFSMN achieved 13.2% WER at approximately 7.1 hours per epoch, outperforming BLSTM with a training speedup greater than $\tilde{\mathbf{h}}_t^{(l)} \in \mathbb{R}^{D_l}$9 (Zhang et al., 2015). With lookback and lookahead orders $$\tilde{\mathbf{h}}_{t}^{(l)} = \sum_{i=0}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$0, vFSMN obtained 13.2% WER; reducing lookahead to $$\tilde{\mathbf{h}}_{t}^{(l)} = \sum_{i=0}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$1 yielded 13.7% WER, illustrating the latency–accuracy trade-off (Zhang et al., 2015).

In language modeling, the PTB results reported test perplexities of 105 for a two-layer LSTM-LM, 102 for sFSMN-LM, and 101 for vFSMN-LM (Zhang et al., 2015). On Wiki9, the reported values were 132 for Kneser-Ney 5-gram, 112 for RNN-LM, 104 for FOFE-LM, 92 for sFSMN-LM with two memory blocks, and 90 for vFSMN-LM with two memory blocks, described as state-of-the-art in that setup (Zhang et al., 2015). The paper also reported that FSMN-LMs converged in approximately 5 epochs versus more than 15 epochs for RNN-LM (Zhang et al., 2015).

The earlier FSMN language-modeling work reported closely aligned results. On PTB, FSMN-LM reached 102 perplexity, compared with 123 for RNNLM, 117 for LSTM, 111 for MemN2N, 113 for 6-gram FNNLM, and 108 for FOFE-FNNLM (Zhang et al., 2015). On LTCB (enwik9), memory placement mattered: memory in the first hidden layer gave 95 perplexity, in the second hidden layer 96, and in both first and second hidden layers 92, the best result (Zhang et al., 2015). The article’s own design guidance followed from these results: placing memory in multiple layers improves performance, and larger FIR order coincided with strong performance, though a direct ablation over order $$\tilde{\mathbf{h}}_{t}^{(l)} = \sum_{i=0}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$2 was not provided (Zhang et al., 2015).

Feed-forward attention supplied a different empirical argument for MRFFN: order-agnostic content retrieval over very long and variable-length sequences. On synthetic addition and multiplication tasks with sequence lengths up to 10,000, a model using simplified feed-forward attention achieved 99.9% accuracy on addition and 99.4% on multiplication after 100 epochs in the variable-length regime $$\tilde{\mathbf{h}}_{t}^{(l)} = \sum_{i=0}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$3 (Raffel et al., 2015). On an NVIDIA GTX 980 Ti, one epoch over 100,000 sequences with $$\tilde{\mathbf{h}}_{t}^{(l)} = \sum_{i=0}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$4 took 254 seconds for the feed-forward attention model versus 917 seconds for a comparable single-layer vanilla RNN with hidden size 100 (Raffel et al., 2015). Because the architecture has no positional encodings and aggregates through a softmax-weighted average, it is explicitly order-agnostic and only suited to tasks where this invariance is acceptable (Raffel et al., 2015).

The transformer-oriented MRFFN results are of a different type. MemoryLLM reported that, at 50B tokens, Base-750M active achieved C4 perplexity 19.730 and Wikitext-2 perplexity 25.491, while MemoryLLM with total 1208M and active 402M achieved 20.933 and 27.258, and MemoryLLM with total 737M and active 245M achieved 22.079 and 29.976 (Jaiswal et al., 30 Jan 2026). Relative to active parameters, MemoryLLM outperformed dense baselines; relative to total parameters, dense baselines performed better (Jaiswal et al., 30 Jan 2026). Flex-MemoryLLM-$$\tilde{\mathbf{h}}_{t}^{(l)} = \sum_{i=0}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$5 at 1B total parameters was reported to closely match Base-1B perplexity and could outperform Base-737M (Jaiswal et al., 30 Jan 2026). On efficiency, Base required 9.541 GB and 21.50 ms/token, whereas MemoryLLM required 6.041 GB and 14.42 ms/token; Flex-MemoryLLM $$\tilde{\mathbf{h}}_{t}^{(l)} = \sum_{i=0}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$6 required 7.025/7.409/7.825 GB and 18.75/20.28/21.47 ms/token (Jaiswal et al., 30 Jan 2026).

6. Design variants, extensions, and neighboring formulations

The design space of MRFFN is broader than the core FSMN and transformer instantiations. Within FSMN itself, several choices were identified as practically important. Typical ASR systems used six hidden layers with memory blocks inserted in selected layers such as the first, third, and fifth, while LM systems used two or three hidden layers (Zhang et al., 2015). Larger lookback and lookahead orders increase receptive field; on Switchboard, $$\tilde{\mathbf{h}}_{t}^{(l)} = \sum_{i=0}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$7 gave the best WER, but smaller $$\tilde{\mathbf{h}}_{t}^{(l)} = \sum_{i=0}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$8 could be chosen for latency-sensitive settings (Zhang et al., 2015). vFSMN increased capacity through per-dimension coefficients and was notably better for ASR than sFSMN, whereas for LM the two variants were similar, likely because learned per-dimension filters were highly similar (Zhang et al., 2015).

The data also describe possible extensions beyond the reported FSMN results: attention-based FSMN with context-dependent coefficients, dilated taps or multi-kernel memory blocks, matrix or subspace taps, combinations with CNNs or Transformers, and use in sequence-to-sequence or semi-supervised settings (Zhang et al., 2015). These are presented as extensions rather than experimentally established conclusions in that paper.

A separate feed-forward retrieval formulation was proposed through simplified attention. There, each time step is encoded independently,

$$\tilde{\mathbf{h}}_{t}^{(l)} = \sum_{i=0}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$9

scored by a content-only function

$$\mathbf{m}_{t}^{(l)} = \mathbf{h}_{t}^{(l)} + \sum_{i=1}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$0

and aggregated through

$$\mathbf{m}_{t}^{(l)} = \mathbf{h}_{t}^{(l)} + \sum_{i=1}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$1

A decoder then maps the context vector to the target (Raffel et al., 2015). The significance of this line is that it operationalizes content-based retrieval in a purely feed-forward and $$\mathbf{m}_{t}^{(l)} = \mathbf{h}_{t}^{(l)} + \sum_{i=1}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$2 manner, but without preserving temporal order (Raffel et al., 2015). This suggests an MRFFN subtype specialized to commutative retrieval or aggregation tasks.

Associative-memory theory provides yet another interpretation. Hopfield-Fenchel-Young networks define an energy

$$\mathbf{m}_{t}^{(l)} = \mathbf{h}_{t}^{(l)} + \sum_{i=1}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$3

and derive the update

$$\mathbf{m}_{t}^{(l)} = \mathbf{h}_{t}^{(l)} + \sum_{i=1}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$4

which can be read as a feed-forward sequence of similarity computation, separation, projection, and optional normalization (Santos et al., 2024). By choosing Tsallis or norm entropies, the framework yields sparse transformations with margins and exact retrieval guarantees in one step under separation conditions (Santos et al., 2024). A plausible implication is that this work supplies a principled energy-based foundation for sparse feed-forward memory retrieval modules closely related to attention and modern Hopfield updates.

Kernel memory networks give a different unifying account. In the feed-forward hetero-associative case, stored keys and values are retrieved through minimum-norm kernel interpolation: $$\mathbf{m}_{t}^{(l)} = \mathbf{h}_{t}^{(l)} + \sum_{i=1}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$5 Auto-associative one-step recall is

$$\mathbf{m}_{t}^{(l)} = \mathbf{h}_{t}^{(l)} + \sum_{i=1}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$6

The paper emphasizes that local translation-invariant kernels can provide finite basins of attraction and that kernels with exponentially large effective feature spaces can yield exponential capacity scaling (Iatropoulos et al., 2022). Because this formulation is feed-forward in the hetero-associative case, it broadens MRFFN toward optimal kernelized memory retrieval.

A biologically grounded but hybrid form couples a feedforward representation learner to a recurrent associative memory. In the cited work, the pipeline was input $$\mathbf{m}_{t}^{(l)} = \mathbf{h}_{t}^{(l)} + \sum_{i=1}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$7 feedforward representation $$\mathbf{m}_{t}^{(l)} = \mathbf{h}_{t}^{(l)} + \sum_{i=1}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$8 recurrent attractor settling to $$\mathbf{m}_{t}^{(l)} = \mathbf{h}_{t}^{(l)} + \sum_{i=1}^{N} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)}.$$9 readout, with the feedforward pathway producing sparse distributed codes using BCPNN learning and the recurrent component performing prototype extraction and denoising (Ravichandran et al., 2022). This is not a purely feed-forward MRFFN, but it clarifies a recurring theme in the literature: feed-forward stages can transform correlated inputs into representations more suitable for memory retrieval.

7. Limitations, misconceptions, and open directions

Across the cited literature, the limitations of MRFFN depend on which instantiation is considered. In FSMN, the receptive field is fixed: dependencies beyond $$\tilde{\mathbf{h}}_{t}^{(l)} = \sum_{i=0}^{N_{1}} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)} + \sum_{j=1}^{N_{2}} c_{j}^{(l)} \, \mathbf{h}_{t+j}^{(l)}.$$0 time steps are not directly captured, though stacking more memory blocks enlarges the effective field while keeping it bounded (Zhang et al., 2015). vFSMN increases parameters through per-dimension coefficients and thereby increases potential overfitting risk (Zhang et al., 2015). Non-causal taps introduce lookahead latency, even if that latency is tunable (Zhang et al., 2015). The taps in sFSMN and vFSMN are content-independent; they do not gate or adapt per time step unless attention-based FSMN is used (Zhang et al., 2015). The earlier FSMN formulation made the same point in different terms: the memory is position-based, not content-based, because scalar taps weight positions by lag rather than by content (Zhang et al., 2015).

For transformer-based MRFFN, a common misconception would be to treat FFN memory slots as isolated, modular, and singly decisive. The evidence argues otherwise. In the key-value-memory analysis of transformer FFNs, hundreds of memories were typically active per layer, and in at least approximately 68% of examples the layer’s top vocabulary prediction differed from every single memory’s top prediction, indicating true intra-layer composition rather than single-slot dominance (Geva et al., 2020). Moreover, strong co-adaptation with attention and residual paths means that the memories are not fully modular (Geva et al., 2020).

In MemoryLLM, the main limitation is the context-free assumption. The MRFFN cannot condition its retrieval on context within the block, and pure MemoryLLM trails dense Base models at equal total parameters, although Flex-MemoryLLM reduces this gap (Jaiswal et al., 30 Jan 2026). ToL storage is large but compressible; the paper identifies non-uniform learned compression and adaptive caching as promising directions (Jaiswal et al., 30 Jan 2026). It also points to knowledge editing, MoE-like routers that remain interpretable, and improved analysis of over-parameterization in FFNs as open problems (Jaiswal et al., 30 Jan 2026).

The simplified feed-forward attention model has a different failure mode: loss of temporal order. Because the model aggregates by a weighted average and uses no positional encodings, it fails on tasks where order matters, such as distinguishing “X,Y” from “Y,X” (Raffel et al., 2015). It therefore cannot be treated as a general substitute for recurrent or pairwise self-attention mechanisms.

More abstract retrieval theories identify additional open directions. Hopfield-Fenchel-Young networks indicate that sparse transformations such as entmax, normmax, and SparseMAP provide margins, exact retrieval guarantees, and structured retrieval capabilities including $$\tilde{\mathbf{h}}_{t}^{(l)} = \sum_{i=0}^{N_{1}} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)} + \sum_{j=1}^{N_{2}} c_{j}^{(l)} \, \mathbf{h}_{t+j}^{(l)}.$$1-subsets and sequential $$\tilde{\mathbf{h}}_{t}^{(l)} = \sum_{i=0}^{N_{1}} a_{i}^{(l)} \, \mathbf{h}_{t-i}^{(l)} + \sum_{j=1}^{N_{2}} c_{j}^{(l)} \, \mathbf{h}_{t+j}^{(l)}.$$2-subsets (Santos et al., 2024). Kernel memory networks point to tunable trade-offs among capacity, robustness, and basin geometry through the kernel choice and regularization (Iatropoulos et al., 2022). These lines suggest that future MRFFN research may increasingly be framed not only architecturally, but also in terms of generalized entropy, convex duality, structured sparsity, and capacity–robustness trade-offs.

In aggregate, the literature supports a precise but plural understanding of MRFFN. In its sequential form, it is a feed-forward temporal retrieval operator implemented by FIR-like memory blocks (Zhang et al., 2015, Zhang et al., 2015). In transformers, it is a persistent or explicitly decoupled parametric memory that maps token or residual representations to retrieved values through feed-forward key-value interactions (Geva et al., 2020, Jaiswal et al., 30 Jan 2026). In broader associative-memory theory, it denotes feed-forward retrieval modules derived from attention, convex energy minimization, or kernel interpolation (Raffel et al., 2015, Santos et al., 2024, Iatropoulos et al., 2022). This suggests that the unifying characteristic of MRFFN is not a particular network diagram, but the replacement of recurrent state propagation by explicit feed-forward retrieval mechanisms that expose memory structure, stability, efficiency, or interpretability in different ways.