---
title: Limited Memory Language Models Overview
url: https://www.emergentmind.com/topics/limited-memory-language-models-lmlms
type: topic
---

# Limited Memory Language Models Overview

Searching arXiv for recent and foundational papers on Limited Memory Language Models and closely related memory-constrained LM architectures.
Limited Memory Language Models (LMLMs) are language models in which memory access, memory capacity, or memory-resident knowledge is explicitly bounded rather than treated as an unstructured byproduct of large parameter counts or unrestricted context accumulation. In the recent literature, the term spans several related formulations: bounded-context language models that are equivalent to finite automata unless equipped with external read-write memory [2301.04589], recurrent or transformer architectures with constrained retrieval or decaying retention [2310.16142], [2508.05803], fixed-size latent memory pools updated online without expanding model size [2402.04624], external-memory or associative-memory systems that preserve a small on-device working set while storing larger context elsewhere [2306.07174], [2407.01437], [2402.13449], and models that externalize factual knowledge to editable databases during pretraining and inference [2505.15962], [2607.07707]. The common theme is not a single architecture but an explicit memory bottleneck that can be analyzed, controlled, or exploited.

## 1. Conceptual foundations

The strongest formal boundary for LMLMs is the bounded-context result: any deterministic language model whose next-token behavior depends only on strings of bounded length is computationally equivalent to a finite automaton, and therefore restricted to regular-language power [2301.04589]. In that formulation, the effective state space is the set of suffixes of length at most \(L\), and the context update is a sliding-window transition. This gives a precise sense in which a strictly bounded-memory language model is limited.

The same work also shows that the limitation is architectural rather than inevitable. When a frozen transformer-based language model is paired with an associative read-write memory and a finite-state pre/post-processing loop, the resulting system can simulate the universal Turing machine \(U_{15,2}\) exactly, without modifying model weights [2301.04589]. This sharp contrast has shaped later LMLM research: the central question is not whether memory matters, but how memory is bounded, where it resides, and which operations are permitted over it.

A recurring misconception is that “limited memory” simply means “shorter context.” The literature is broader. In some papers, limitation means a single retrieval operation per word [2310.16142]. In others, it means a fixed-size latent memory pool [2402.04624], a bounded natural-language hidden state under a token budget [2601.13352], a compressed set of learned memory embeddings substituting for long token sequences [2602.13466], or an external factual store that replaces parametric fact memorization [2505.15962]. This suggests that LMLMs are best understood as a design space organized around explicit memory constraints rather than a single model family.

## 2. Architectural formulations

One major strand implements limited memory as constrained internal state. The Long Term Memory network (LTM) introduces a recurrent cell with input state, cell state, and output state, but without an explicit forget mechanism; instead, the cell state is additively updated and then bounded by a sigmoid transform [2305.11462]. Its defining updates are
\[
L_{t1} = \sigma(W_1(h_{t-1} + x_t)), \quad
L_{t2} = \sigma(W_2(h_{t-1} + x_t)), \quad
L'_t = L_{t1} \cdot L_{t2},
\]
\[
C'_t = L'_t + C_{t-1}, \quad
C_t = \sigma(W_4 \cdot C'_t),
\]
\[
L_{t3} = (W'_3(h_{t-1} + input)), \quad
h_t = C_t \cdot L_{t3}.
\]
Within this formulation, “limited memory” is not erasure by truncation but bounded accumulation that prioritizes current inputs while retaining long-term content [2305.11462].

A second strand treats memory as an external bank coupled to a frozen or lightly adapted decoder. LongMem freezes the backbone language model as a memory encoder and adds a residual SideNet that retrieves and reads cached key-value pairs from long past context, using token-to-chunk retrieval and gated fusion [2306.07174]. Larimar pushes this further with an external associative memory that can be written and read on CPU while only a compact latent readout returns to GPU, allowing long-context recall at 128K to 1M tokens without increasing the decoder’s working sequence length [2407.01437]. CAMELoT also couples a frozen attention-based LLM to a per-layer associative memory, but replaces raw token caching with a consolidated non-parametric distribution model in which incoming keys and values are merged into slot-level modes under novelty and recency criteria [2402.13449].

A third formulation internalizes memory in a fixed latent pool. MEMORYLLM augments a transformer with a fixed-size memory pool \(\theta = \{\theta^l\}\), where each \(\theta^l \in \mathbb{R}^{N \times d}\) is a matrix of memory tokens for layer \(l\) [2402.04624]. During generation, hidden states attend over all memory tokens in \(\theta^l\); during self-update, only the last \(K\) memory tokens are concatenated with new text representations, transformed, and written back after random eviction of \(K\) existing slots. The update is forward-only after deployment, while the backbone parameters \(\phi\) remain frozen [2402.04624].

A fourth formulation replaces raw token histories with a small number of learned memory embeddings. “Language Model Memory and Memory Models for Language” formalizes a memory embedding \(E(X,\theta)\in \mathbb{R}^d\) as the model’s memory of an input sequence \(X\), and shows that standard next-token prediction produces information-poor embeddings, whereas autoencoder-style objectives produce information-rich ones [2602.13466]. The proposed encoder–decoder memory model splits the input into \(s\) chunks, encodes each chunk into one embedding, and lets the decoder consume these embeddings plus visible tokens. Its complexity argument sets encoder cost \(A=n^2/s\) and decoder cost \(B=s^2\), with the balancing condition
\[
\frac{n^2}{s} = s^2 \Rightarrow s^3 = n^2 \Rightarrow s = n^{2/3},
\]
yielding total cost on the order of \(n^{4/3}\) rather than \(n^2\) [2602.13466].

## 3. Cognitive and psycholinguistic LMLMs

The most explicitly cognitive LMLM in the current literature is the Cue-Based Retrieval/Recurrent Neural Network (CBR-RNN), a recurrent language model with a single self-attention head designed to parallel cue-based retrieval theories of human sentence processing [2310.16142]. The model enforces exactly one retrieval operation per timestep, uses a simple RNN rather than an LSTM, and only allows attention over prior timesteps, not the current one. Its retrieval mechanism is
\[
q_i = FF([w_i; h_{i-1}]),
\]
\[
\alpha_i = \mathrm{softmax}(q_i K_{1\ldots i-1}^T),
\]
\[
a_i = \alpha_i V_{1\ldots i-1} = \sum_{j=1}^{i-1} \alpha_{ij} v_j,
\]
\[
[h_i; k_i; v_i] = FF_2([a_i; w_i; q_i; h_{i-1}]).
\]
The model links expectation-based and retrieval-based difficulty through surprisal,
\[
S_t = -\log p(w_t \mid w_{<t}),
\]
and a retrieval proxy,
\[
\mathrm{RelAttn}(v,s) = \frac{\mathrm{Attn}(v,s)}{\mathrm{Attn}(v,s)+\mathrm{Attn}(v,n)}.
\]
Empirically, it captures both agreement and semantic attraction, with additive effects in double violations, while avoiding the cognitive implausibility of many parallel retrievals in multi-head transformers [2310.16142].

A related but distinct cognitive formulation imposes fleeting memory on transformers. In “Human-like fleeting memory improves language learning but impairs reading time prediction in transformer language models,” memory is limited by a fixed, parameter-free recency bias applied after softmax in attention [2508.05803]. With an echoic buffer of \(E\) tokens and power-law decay thereafter, attention becomes
\[
\mathrm{Attn}_{\mathrm{FM}}(Q,K,V) = (W \odot B)V,
\]
where \(W=\mathrm{softmax}(QK^\top/\sqrt{d_k})\) and
\[
B(d)=
\begin{cases}
1, & d<E,\\
1-\left(\frac{d-E+1}{n-E}\right)^{\frac{1}{e\alpha}}, & E \le d < n.
\end{cases}
\]
On BabyLM data, echoic-buffer versions improved validation loss and BLiMP syntactic accuracy, but worsened surprisal-based reading-time prediction on Natural Stories and Dundee [2508.05803]. A plausible implication is that memory limitations can aid acquisition and generalization without necessarily improving psychometric alignment.

A different line of work constrains memory to simulate human participants rather than human parsing. “Simulating Human Memory with Language Models” introduces a compactor agent with a \(K=4\) slot key–value working memory, motivated by Cowan’s four-chunk theory [2605.25680]. The system first compresses the input into four chunks and then answers only from those chunks. Across ten memory tasks, this moves language-model score distributions closer to human baselines, especially on working-memory tasks, and improves downstream user simulation in an education setting [2605.25680]. Here “limited memory” is not a property of network internals but an inference-time architectural bottleneck.

## 4. External knowledge, retrieval, and unlearning

A central contemporary meaning of LMLM is a language model that externalizes factual knowledge to a knowledge base during pretraining rather than memorizing those facts in its parameters [2505.15962]. In the original relational LMLM formulation, the training corpus is annotated with inline lookup segments of the form
`<|db_start|> entity <|sep|> relation <|db_retrieve|> value <|db_end|>`,
and the retrieved value tokens are masked from the loss. The core objective is
\[
\mathcal{L}(\theta) = - \sum_{t=1}^{T} m_t \log p_\theta(x_t \mid x_{<t}),
\]
where \(m_t=0\) for retrieved value tokens and `<|db_end|>` and \(m_t=1\) otherwise [2505.15962]. The effect is to teach the model to emit queries rather than memorize answers. In the reported setup, the external KB contains 54.6M triplets across 9.5M unique entities, 8.5M relations, and 16.2M unique values, and smaller LMLMs achieve competitive factual precision against much larger standard LLMs while preserving NLU performance [2505.15962].

Co-LMLM generalizes this idea from relational queries to continuous queries and free-form factual spans [2607.07707]. When the model emits `<FACT>`, the last-layer hidden state at that position becomes a dense retrieval query, and top-1 retrieval inserts a human-readable text span between `<FACT>` and `</FACT>`. Training combines masked next-token prediction with a bidirectional contrastive loss:
\[
\mathcal{L}_{\mathrm{NTP}}(\theta) = -\sum_{t\notin M}\log p(x_t\mid x_{<t};\theta),
\]
\[
\mathcal{L}_{\mathrm{CL}} = -\frac{1}{2B} \sum_{i=1}^{B} \big( \ell_{f \to q}^{(i)} + \ell_{q \to f}^{(i)} \big),
\]
\[
\mathcal{L} = \mathcal{L}_{\mathrm{NTP}} + \lambda \mathcal{L}_{\mathrm{CL}},
\]
with \(\lambda=0.25\) by default [2607.07707]. This removes the Wikipedia-specific restriction of subject–relation–object annotation, reduces query overhead to a single autoregressive step, and expands the KB to about 240M items for Wikipedia alone and 2.2B entries for a combined Wikipedia+FineWeb-Edu index [2607.07707].

Because knowledge resides in an explicit store, deletion-based unlearning becomes a database operation rather than a weight-editing procedure. “Auditing Forgetting in Limited Memory Language Models” formalizes three interventions—FULL, DEL-ON, and DEL-OFF—to separate normal retrieval, post-deletion retrieval, and pure parametric recall [2607.00605]. For a fact \(f=(s,r,o)\), the audit defines
\[
L(f) = \mathbb{I}[Y(f,\mathrm{DEL\text{-}OFF}) = o],
\]
\[
R(f) = \mathbb{I}[Y(f,\mathrm{DEL\text{-}ON}) = o \wedge Y(f,\mathrm{DEL\text{-}OFF}) \neq o].
\]
Across 12,228 alias-closure deletions and thirteen databases, the leakage rate is \(L=0.11\%\), while residual post-deletion correctness is explained almost entirely by retrieval-mediated reconstruction from near neighbors rather than parametric memory [2607.00605]. This makes the retrieval graph and database design, rather than the model weights, the primary unlearning boundary.

The broader retrieval-based memory literature intersects with LMLMs but does not always share the same training objective. LongMem and CAMELoT attach external memory to frozen LLMs for long-range conditioning [2306.07174], [2402.13449]; SeMem turns a semiparametric LM into a scalable continual learner by storing only difficult tokens in a non-parametric memory [2303.01421]; LLM-as-RNN constrains a frozen LLM to a bounded, mutable natural-language hidden state updated online under a token budget \(B\) [2601.13352]. These approaches share bounded or externalized memory, but the strongest version of the LMLM idea is the deliberate separation of linguistic competence and factual storage during pretraining [2505.15962], [2607.07707].

## 5. Limited memory in training and deployment

Not all LMLM work is about inference-time cognition or knowledge control. A substantial systems literature studies language modeling and adaptation under strict memory budgets. ZO2 is a training framework for fine-tuning extremely large LLMs with limited GPU memory by combining zeroth-order optimization with dynamic CPU–GPU offloading [2503.12668]. Its core estimator is the two-point central difference
\[
g = \frac{L(\theta+\epsilon z)-L(\theta-\epsilon z)}{2\epsilon},
\qquad
\hat{\nabla}L(\theta)=gz,
\qquad
\theta \leftarrow \theta - \eta gz.
\]
Because \(g\) is scalar and no backward-pass activations are stored, ZO2 can fine-tune OPT-175B on a single GPU with about 18GB memory in FP16 mode, with reported throughput of 37 tokens/sec and “absolutely no accuracy loss compared to standard zeroth-order methods” [2503.12668].

BlockLLM addresses memory from the optimizer side by updating only a small subset of parameters at each iteration, inspired by block coordinate descent [2406.17296]. If \(s\) is sparsity and \(n_s=(1-s)n\) is the trainable budget, layers are ranked by
\[
\mathrm{score}(l) = \frac{\|\tilde{G}_t^l\|}{f_l},
\]
and only the active block receives Adam updates. The optimizer-state memory therefore drops from \(\mathcal{O}(4N)\) to \(\mathcal{O}(4(1-s)N)\) [2406.17296]. In RoBERTa fine-tuning on GLUE with \(s=0.95\), BlockLLM updates less than 5% of parameters per iteration and reports an average memory reduction of about 13.5% versus baselines [2406.17296].

Inference under limited memory has generated a separate line of work. “LLM in a flash” stores model parameters on flash and loads them on demand, reducing transfer volume by “windowing” neuron reuse and “row-column bundling” of FFN weights [2312.11514]. The system runs models up to twice the size of available DRAM and reports 4–5x speedups on CPU and 20–25x on GPU over naive loading [2312.11514]. “LLM Vocabulary Compression for Low-Compute Environments” instead compresses the final vocabulary projection by grouping tokens based on BPE merges, avoiding materialization of the full logits tensor and reducing memory usage by up to 3.4x while improving throughput by up to 3x on TinyStories [2411.06371].

These systems papers use “limited memory” in a hardware sense rather than a cognitive or knowledge-structural sense. Even so, they illuminate a common engineering principle: effective language modeling under memory budgets often requires replacing unconstrained dense access with structured access, selective loading, or bounded active state.

## 6. Evaluation, trade-offs, and open problems

Evaluation protocols for LMLMs are heterogeneous because the field itself is heterogeneous. Cognitive models measure surprisal, relative attention, reading-time fit, or human-likeness under controlled tasks [2310.16142], [2508.05803], [2605.25680]. External-knowledge LMLMs use perplexity, factual precision, exact match, FactScore, SimpleQA-style verified evaluation, and deletion-based auditing [2505.15962], [2607.07707], [2607.00605]. Memory-augmented long-context models emphasize perplexity on long documents, long-context recall, or many-shot in-context learning [2306.07174], [2407.01437], [2402.13449]. Systems LMLMs prioritize memory footprint, throughput, and feasibility under constrained hardware [2503.12668], [2406.17296], [2312.11514], [2411.06371].

Several trade-offs recur across these evaluations. First, accuracy and interpretability are often in tension. The CBR-RNN’s single attention head yields an interpretable retrieval probability but can understate human implausibility judgments in no-attractor conditions, partly because attention remains broad over many prior representations [2310.16142]. Second, memory limitation can improve one objective while harming another: fleeting memory improves validation loss and BLiMP performance yet worsens reading-time prediction [2508.05803]. Third, externalization improves factual control but shifts responsibility to the database and retriever. The unlearning audit shows that when deletion fails, it is predominantly because the retrieval graph still reconstitutes the answer from near-neighbor entries, not because the model retained the fact parametrically [2607.00605].

A further common issue is underuse or misuse of memory. In encoder–decoder memory models, causal training alone yields information-poor embeddings, and decoders tend to ignore memories unless trained with blank-copy or copy-style retention objectives [2602.13466]. In Co-LMLM, enforced lookup improves factual performance, indicating that retrieval timing remains an unresolved problem [2607.07707]. In human-memory simulation, a four-slot compactor improves aggregate human-likeness but still produces non-human error microstructure, such as truncation patterns unlike those of human digit-span recall [2605.25680].

Open problems therefore cluster around four axes. One is representation: whether memory should consist of timestep vectors, explicit constituents, compressed chunk embeddings, latent slots, or human-readable spans [2310.16142], [2602.13466], [2402.04624], [2607.07707]. A second is dynamics: whether memory should decay, consolidate, rewrite, or remain fixed-size under random eviction [2508.05803], [2402.04624], [2601.13352]. A third is retrieval control: how to decide when to retrieve, how many items to retrieve, and how to avoid fragmentation or near-neighbor artifacts [2306.07174], [2402.13449], [2607.00605]. A fourth is evaluation: whether the target is perplexity, factual control, continual learning, psycholinguistic plausibility, or hardware efficiency, since different LMLM designs optimize different aspects of “memory.”

Taken together, the literature suggests that LMLMs are not merely smaller-context language models. They are a family of architectures that make memory explicit, bounded, and operationally meaningful. Depending on where the bottleneck is placed—retrieval count, latent state size, external knowledge store, optimizer state, or on-device working set—LMLMs can serve as cognitive models, editable factual systems, long-context architectures, or memory-efficient training and inference frameworks [2310.16142], [2505.15962], [2607.07707], [2503.12668].

Source: https://www.emergentmind.com/topics/limited-memory-language-models-lmlms