Limited Memory Language Models Overview

• Limited Memory Language Models (LMLMs) are models that use explicit, bounded memory constraints rather than unlimited context, defining a clear computational limit.
• They encompass diverse formulations—such as bounded-context, external memory systems, and fixed latent pools—that allow controlled retrieval and efficient inference.
• LMLMs enable precise analysis of how memory bottlenecks affect performance, offering benefits in factual control, efficient training, and cognitive modeling.

Searching arXiv for recent and foundational papers on Limited Memory LLMs and closely related memory-constrained LM architectures. Limited Memory LLMs (LMLMs) are LLMs in which memory access, memory capacity, or memory-resident knowledge is explicitly bounded rather than treated as an unstructured byproduct of large parameter counts or unrestricted context accumulation. In the recent literature, the term spans several related formulations: bounded-context LLMs that are equivalent to finite automata unless equipped with external read-write memory (Schuurmans, 2023), recurrent or transformer architectures with constrained retrieval or decaying retention (Timkey et al., 2023, Thamma et al., 7 Aug 2025), fixed-size latent memory pools updated online without expanding model size (Wang et al., 2024), external-memory or associative-memory systems that preserve a small on-device working set while storing larger context elsewhere (Wang et al., 2023, Nelson et al., 2024, He et al., 2024), and models that externalize factual knowledge to editable databases during pretraining and inference (Zhao et al., 21 May 2025, Feldman et al., 8 Jul 2026). The common theme is not a single architecture but an explicit memory bottleneck that can be analyzed, controlled, or exploited.

1. Conceptual foundations

The strongest formal boundary for LMLMs is the bounded-context result: any deterministic LLM whose next-token behavior depends only on strings of bounded length is computationally equivalent to a finite automaton, and therefore restricted to regular-language power (Schuurmans, 2023). In that formulation, the effective state space is the set of suffixes of length at most $L$, and the context update is a sliding-window transition. This gives a precise sense in which a strictly bounded-memory LLM is limited.

The same work also shows that the limitation is architectural rather than inevitable. When a frozen transformer-based LLM is paired with an associative read-write memory and a finite-state pre/post-processing loop, the resulting system can simulate the universal Turing machine $U_{15,2}$ exactly, without modifying model weights (Schuurmans, 2023). This sharp contrast has shaped later LMLM research: the central question is not whether memory matters, but how memory is bounded, where it resides, and which operations are permitted over it.

A recurring misconception is that “limited memory” simply means “shorter context.” The literature is broader. In some papers, limitation means a single retrieval operation per word (Timkey et al., 2023). In others, it means a fixed-size latent memory pool (Wang et al., 2024), a bounded natural-language hidden state under a token budget (Lu et al., 19 Jan 2026), a compressed set of learned memory embeddings substituting for long token sequences (Badger, 13 Feb 2026), or an external factual store that replaces parametric fact memorization (Zhao et al., 21 May 2025). This suggests that LMLMs are best understood as a design space organized around explicit memory constraints rather than a single model family.

2. Architectural formulations

One major strand implements limited memory as constrained internal state. The Long Term Memory network (LTM) introduces a recurrent cell with input state, cell state, and output state, but without an explicit forget mechanism; instead, the cell state is additively updated and then bounded by a sigmoid transform (Nugaliyadde, 2023). Its defining updates are

$$L_{t1} = \sigma(W_1(h_{t-1} + x_t)), \quad L_{t2} = \sigma(W_2(h_{t-1} + x_t)), \quad L'_t = L_{t1} \cdot L_{t2},$$

$$C'_t = L'_t + C_{t-1}, \quad C_t = \sigma(W_4 \cdot C'_t),$$

$$L_{t3} = (W'_3(h_{t-1} + input)), \quad h_t = C_t \cdot L_{t3}.$$

Within this formulation, “limited memory” is not erasure by truncation but bounded accumulation that prioritizes current inputs while retaining long-term content (Nugaliyadde, 2023).

A second strand treats memory as an external bank coupled to a frozen or lightly adapted decoder. LongMem freezes the backbone LLM as a memory encoder and adds a residual SideNet that retrieves and reads cached key-value pairs from long past context, using token-to-chunk retrieval and gated fusion (Wang et al., 2023). Larimar pushes this further with an external associative memory that can be written and read on CPU while only a compact latent readout returns to GPU, allowing long-context recall at 128K to 1M tokens without increasing the decoder’s working sequence length (Nelson et al., 2024). CAMELoT also couples a frozen attention-based LLM to a per-layer associative memory, but replaces raw token caching with a consolidated non-parametric distribution model in which incoming keys and values are merged into slot-level modes under novelty and recency criteria (He et al., 2024).

A third formulation internalizes memory in a fixed latent pool. MEMORYLLM augments a transformer with a fixed-size memory pool $\theta = \{\theta^l\}$, where each $\theta^l \in \mathbb{R}^{N \times d}$ is a matrix of memory tokens for layer $l$ (Wang et al., 2024). During generation, hidden states attend over all memory tokens in $\theta^l$; during self-update, only the last $K$ memory tokens are concatenated with new text representations, transformed, and written back after random eviction of $U_{15,2}$0 existing slots. The update is forward-only after deployment, while the backbone parameters $U_{15,2}$1 remain frozen (Wang et al., 2024).

A fourth formulation replaces raw token histories with a small number of learned memory embeddings. “LLM Memory and Memory Models for Language” formalizes a memory embedding $U_{15,2}$2 as the model’s memory of an input sequence $U_{15,2}$3, and shows that standard next-token prediction produces information-poor embeddings, whereas autoencoder-style objectives produce information-rich ones (Badger, 13 Feb 2026). The proposed encoder–decoder memory model splits the input into $U_{15,2}$4 chunks, encodes each chunk into one embedding, and lets the decoder consume these embeddings plus visible tokens. Its complexity argument sets encoder cost $U_{15,2}$5 and decoder cost $U_{15,2}$6, with the balancing condition

$U_{15,2}$7

yielding total cost on the order of $U_{15,2}$8 rather than $U_{15,2}$9 (Badger, 13 Feb 2026).

3. Cognitive and psycholinguistic LMLMs

The most explicitly cognitive LMLM in the current literature is the Cue-Based Retrieval/Recurrent Neural Network (CBR-RNN), a recurrent LLM with a single self-attention head designed to parallel cue-based retrieval theories of human sentence processing (Timkey et al., 2023). The model enforces exactly one retrieval operation per timestep, uses a simple RNN rather than an LSTM, and only allows attention over prior timesteps, not the current one. Its retrieval mechanism is

$$L_{t1} = \sigma(W_1(h_{t-1} + x_t)), \quad L_{t2} = \sigma(W_2(h_{t-1} + x_t)), \quad L'_t = L_{t1} \cdot L_{t2},$$0

$$L_{t1} = \sigma(W_1(h_{t-1} + x_t)), \quad L_{t2} = \sigma(W_2(h_{t-1} + x_t)), \quad L'_t = L_{t1} \cdot L_{t2},$$1

$$L_{t1} = \sigma(W_1(h_{t-1} + x_t)), \quad L_{t2} = \sigma(W_2(h_{t-1} + x_t)), \quad L'_t = L_{t1} \cdot L_{t2},$$2

$$L_{t1} = \sigma(W_1(h_{t-1} + x_t)), \quad L_{t2} = \sigma(W_2(h_{t-1} + x_t)), \quad L'_t = L_{t1} \cdot L_{t2},$$3

The model links expectation-based and retrieval-based difficulty through surprisal,

$$L_{t1} = \sigma(W_1(h_{t-1} + x_t)), \quad L_{t2} = \sigma(W_2(h_{t-1} + x_t)), \quad L'_t = L_{t1} \cdot L_{t2},$$4

and a retrieval proxy,

$$L_{t1} = \sigma(W_1(h_{t-1} + x_t)), \quad L_{t2} = \sigma(W_2(h_{t-1} + x_t)), \quad L'_t = L_{t1} \cdot L_{t2},$$5

Empirically, it captures both agreement and semantic attraction, with additive effects in double violations, while avoiding the cognitive implausibility of many parallel retrievals in multi-head transformers (Timkey et al., 2023).

A related but distinct cognitive formulation imposes fleeting memory on transformers. In “Human-like fleeting memory improves language learning but impairs reading time prediction in transformer LLMs,” memory is limited by a fixed, parameter-free recency bias applied after softmax in attention (Thamma et al., 7 Aug 2025). With an echoic buffer of $$L_{t1} = \sigma(W_1(h_{t-1} + x_t)), \quad L_{t2} = \sigma(W_2(h_{t-1} + x_t)), \quad L'_t = L_{t1} \cdot L_{t2},$$6 tokens and power-law decay thereafter, attention becomes

$$L_{t1} = \sigma(W_1(h_{t-1} + x_t)), \quad L_{t2} = \sigma(W_2(h_{t-1} + x_t)), \quad L'_t = L_{t1} \cdot L_{t2},$$7

where $$L_{t1} = \sigma(W_1(h_{t-1} + x_t)), \quad L_{t2} = \sigma(W_2(h_{t-1} + x_t)), \quad L'_t = L_{t1} \cdot L_{t2},$$8 and

$$L_{t1} = \sigma(W_1(h_{t-1} + x_t)), \quad L_{t2} = \sigma(W_2(h_{t-1} + x_t)), \quad L'_t = L_{t1} \cdot L_{t2},$$9

On BabyLM data, echoic-buffer versions improved validation loss and BLiMP syntactic accuracy, but worsened surprisal-based reading-time prediction on Natural Stories and Dundee (Thamma et al., 7 Aug 2025). A plausible implication is that memory limitations can aid acquisition and generalization without necessarily improving psychometric alignment.

A different line of work constrains memory to simulate human participants rather than human parsing. “Simulating Human Memory with LLMs” introduces a compactor agent with a $$C'_t = L'_t + C_{t-1}, \quad C_t = \sigma(W_4 \cdot C'_t),$$0 slot key–value working memory, motivated by Cowan’s four-chunk theory (Wang et al., 25 May 2026). The system first compresses the input into four chunks and then answers only from those chunks. Across ten memory tasks, this moves language-model score distributions closer to human baselines, especially on working-memory tasks, and improves downstream user simulation in an education setting (Wang et al., 25 May 2026). Here “limited memory” is not a property of network internals but an inference-time architectural bottleneck.

4. External knowledge, retrieval, and unlearning

A central contemporary meaning of LMLM is a LLM that externalizes factual knowledge to a knowledge base during pretraining rather than memorizing those facts in its parameters (Zhao et al., 21 May 2025). In the original relational LMLM formulation, the training corpus is annotated with inline lookup segments of the form <|db_start|> entity <|sep|> relation <|db_retrieve|> value <|db_end|>, and the retrieved value tokens are masked from the loss. The core objective is

$$C'_t = L'_t + C_{t-1}, \quad C_t = \sigma(W_4 \cdot C'_t),$$1

where $$C'_t = L'_t + C_{t-1}, \quad C_t = \sigma(W_4 \cdot C'_t),$$2 for retrieved value tokens and <|db_end|> and $$C'_t = L'_t + C_{t-1}, \quad C_t = \sigma(W_4 \cdot C'_t),$$3 otherwise (Zhao et al., 21 May 2025). The effect is to teach the model to emit queries rather than memorize answers. In the reported setup, the external KB contains 54.6M triplets across 9.5M unique entities, 8.5M relations, and 16.2M unique values, and smaller LMLMs achieve competitive factual precision against much larger standard LLMs while preserving NLU performance (Zhao et al., 21 May 2025).

Co-LMLM generalizes this idea from relational queries to continuous queries and free-form factual spans (Feldman et al., 8 Jul 2026). When the model emits <[FACT](https://www.emergentmind.com/topics/flow-matching-action-tokenizer-fact-89184bc9-9b7a-4183-88e6-ea72c5b47029)>, the last-layer hidden state at that position becomes a dense retrieval query, and top-1 retrieval inserts a human-readable text span between <FACT> and </FACT>. Training combines masked next-token prediction with a bidirectional contrastive loss: $$C'_t = L'_t + C_{t-1}, \quad C_t = \sigma(W_4 \cdot C'_t),$$4

$$C'_t = L'_t + C_{t-1}, \quad C_t = \sigma(W_4 \cdot C'_t),$$5

$$C'_t = L'_t + C_{t-1}, \quad C_t = \sigma(W_4 \cdot C'_t),$$6

with $$C'_t = L'_t + C_{t-1}, \quad C_t = \sigma(W_4 \cdot C'_t),$$7 by default (Feldman et al., 8 Jul 2026). This removes the Wikipedia-specific restriction of subject–relation–object annotation, reduces query overhead to a single autoregressive step, and expands the KB to about 240M items for Wikipedia alone and 2.2B entries for a combined Wikipedia+FineWeb-Edu index (Feldman et al., 8 Jul 2026).

Because knowledge resides in an explicit store, deletion-based unlearning becomes a database operation rather than a weight-editing procedure. “Auditing Forgetting in Limited Memory LLMs” formalizes three interventions—FULL, DEL-ON, and DEL-OFF—to separate normal retrieval, post-deletion retrieval, and pure parametric recall (Raeesi et al., 1 Jul 2026). For a fact $$C'_t = L'_t + C_{t-1}, \quad C_t = \sigma(W_4 \cdot C'_t),$$8, the audit defines

$$C'_t = L'_t + C_{t-1}, \quad C_t = \sigma(W_4 \cdot C'_t),$$9

$$L_{t3} = (W'_3(h_{t-1} + input)), \quad h_t = C_t \cdot L_{t3}.$$0

Across 12,228 alias-closure deletions and thirteen databases, the leakage rate is $$L_{t3} = (W'_3(h_{t-1} + input)), \quad h_t = C_t \cdot L_{t3}.$$1, while residual post-deletion correctness is explained almost entirely by retrieval-mediated reconstruction from near neighbors rather than parametric memory (Raeesi et al., 1 Jul 2026). This makes the retrieval graph and database design, rather than the model weights, the primary unlearning boundary.

The broader retrieval-based memory literature intersects with LMLMs but does not always share the same training objective. LongMem and CAMELoT attach external memory to frozen LLMs for long-range conditioning (Wang et al., 2023, He et al., 2024); SeMem turns a semiparametric LM into a scalable continual learner by storing only difficult tokens in a non-parametric memory (Peng et al., 2023); LLM-as-RNN constrains a frozen LLM to a bounded, mutable natural-language hidden state updated online under a token budget $$L_{t3} = (W'_3(h_{t-1} + input)), \quad h_t = C_t \cdot L_{t3}.$$2 (Lu et al., 19 Jan 2026). These approaches share bounded or externalized memory, but the strongest version of the LMLM idea is the deliberate separation of linguistic competence and factual storage during pretraining (Zhao et al., 21 May 2025, Feldman et al., 8 Jul 2026).

5. Limited memory in training and deployment

Not all LMLM work is about inference-time cognition or knowledge control. A substantial systems literature studies language modeling and adaptation under strict memory budgets. ZO2 is a training framework for fine-tuning extremely large LLMs with limited GPU memory by combining zeroth-order optimization with dynamic CPU–GPU offloading (Wang et al., 16 Mar 2025). Its core estimator is the two-point central difference

$$L_{t3} = (W'_3(h_{t-1} + input)), \quad h_t = C_t \cdot L_{t3}.$$3

Because $$L_{t3} = (W'_3(h_{t-1} + input)), \quad h_t = C_t \cdot L_{t3}.$$4 is scalar and no backward-pass activations are stored, ZO2 can fine-tune OPT-175B on a single GPU with about 18GB memory in FP16 mode, with reported throughput of 37 tokens/sec and “absolutely no accuracy loss compared to standard zeroth-order methods” (Wang et al., 16 Mar 2025).

BlockLLM addresses memory from the optimizer side by updating only a small subset of parameters at each iteration, inspired by block coordinate descent (Ramesh et al., 2024). If $$L_{t3} = (W'_3(h_{t-1} + input)), \quad h_t = C_t \cdot L_{t3}.$$5 is sparsity and $$L_{t3} = (W'_3(h_{t-1} + input)), \quad h_t = C_t \cdot L_{t3}.$$6 is the trainable budget, layers are ranked by

$$L_{t3} = (W'_3(h_{t-1} + input)), \quad h_t = C_t \cdot L_{t3}.$$7

and only the active block receives Adam updates. The optimizer-state memory therefore drops from $$L_{t3} = (W'_3(h_{t-1} + input)), \quad h_t = C_t \cdot L_{t3}.$$8 to $$L_{t3} = (W'_3(h_{t-1} + input)), \quad h_t = C_t \cdot L_{t3}.$$9 (Ramesh et al., 2024). In RoBERTa fine-tuning on GLUE with $\theta = \{\theta^l\}$0, BlockLLM updates less than 5% of parameters per iteration and reports an average memory reduction of about 13.5% versus baselines (Ramesh et al., 2024).

Inference under limited memory has generated a separate line of work. “LLM in a flash” stores model parameters on flash and loads them on demand, reducing transfer volume by “windowing” neuron reuse and “row-column bundling” of FFN weights (Alizadeh et al., 2023). The system runs models up to twice the size of available DRAM and reports 4–5x speedups on CPU and 20–25x on GPU over naive loading (Alizadeh et al., 2023). “LLM Vocabulary Compression for Low-Compute Environments” instead compresses the final vocabulary projection by grouping tokens based on BPE merges, avoiding materialization of the full logits tensor and reducing memory usage by up to 3.4x while improving throughput by up to 3x on TinyStories (Vennam et al., 2024).

These systems papers use “limited memory” in a hardware sense rather than a cognitive or knowledge-structural sense. Even so, they illuminate a common engineering principle: effective language modeling under memory budgets often requires replacing unconstrained dense access with structured access, selective loading, or bounded active state.

6. Evaluation, trade-offs, and open problems

Evaluation protocols for LMLMs are heterogeneous because the field itself is heterogeneous. Cognitive models measure surprisal, relative attention, reading-time fit, or human-likeness under controlled tasks (Timkey et al., 2023, Thamma et al., 7 Aug 2025, Wang et al., 25 May 2026). External-knowledge LMLMs use perplexity, factual precision, exact match, FactScore, SimpleQA-style verified evaluation, and deletion-based auditing (Zhao et al., 21 May 2025, Feldman et al., 8 Jul 2026, Raeesi et al., 1 Jul 2026). Memory-augmented long-context models emphasize perplexity on long documents, long-context recall, or many-shot in-context learning (Wang et al., 2023, Nelson et al., 2024, He et al., 2024). Systems LMLMs prioritize memory footprint, throughput, and feasibility under constrained hardware (Wang et al., 16 Mar 2025, Ramesh et al., 2024, Alizadeh et al., 2023, Vennam et al., 2024).

Several trade-offs recur across these evaluations. First, accuracy and interpretability are often in tension. The CBR-RNN’s single attention head yields an interpretable retrieval probability but can understate human implausibility judgments in no-attractor conditions, partly because attention remains broad over many prior representations (Timkey et al., 2023). Second, memory limitation can improve one objective while harming another: fleeting memory improves validation loss and BLiMP performance yet worsens reading-time prediction (Thamma et al., 7 Aug 2025). Third, externalization improves factual control but shifts responsibility to the database and retriever. The unlearning audit shows that when deletion fails, it is predominantly because the retrieval graph still reconstitutes the answer from near-neighbor entries, not because the model retained the fact parametrically (Raeesi et al., 1 Jul 2026).

A further common issue is underuse or misuse of memory. In encoder–decoder memory models, causal training alone yields information-poor embeddings, and decoders tend to ignore memories unless trained with blank-copy or copy-style retention objectives (Badger, 13 Feb 2026). In Co-LMLM, enforced lookup improves factual performance, indicating that retrieval timing remains an unresolved problem (Feldman et al., 8 Jul 2026). In human-memory simulation, a four-slot compactor improves aggregate human-likeness but still produces non-human error microstructure, such as truncation patterns unlike those of human digit-span recall (Wang et al., 25 May 2026).

Open problems therefore cluster around four axes. One is representation: whether memory should consist of timestep vectors, explicit constituents, compressed chunk embeddings, latent slots, or human-readable spans (Timkey et al., 2023, Badger, 13 Feb 2026, Wang et al., 2024, Feldman et al., 8 Jul 2026). A second is dynamics: whether memory should decay, consolidate, rewrite, or remain fixed-size under random eviction (Thamma et al., 7 Aug 2025, Wang et al., 2024, Lu et al., 19 Jan 2026). A third is retrieval control: how to decide when to retrieve, how many items to retrieve, and how to avoid fragmentation or near-neighbor artifacts (Wang et al., 2023, He et al., 2024, Raeesi et al., 1 Jul 2026). A fourth is evaluation: whether the target is perplexity, factual control, continual learning, psycholinguistic plausibility, or hardware efficiency, since different LMLM designs optimize different aspects of “memory.”

Taken together, the literature suggests that LMLMs are not merely smaller-context LLMs. They are a family of architectures that make memory explicit, bounded, and operationally meaningful. Depending on where the bottleneck is placed—retrieval count, latent state size, external knowledge store, optimizer state, or on-device working set—LMLMs can serve as cognitive models, editable factual systems, long-context architectures, or memory-efficient training and inference frameworks (Timkey et al., 2023, Zhao et al., 21 May 2025, Feldman et al., 8 Jul 2026, Wang et al., 16 Mar 2025).