Lngram: Latent-Space Memory Module

• Lngram is a latent-space conditional memory module that leverages learned discrete symbols from hidden states to decouple compositional reasoning from local static knowledge retrieval.
• It employs exact latent N-gram lookup over learned discrete symbols, avoiding token-based limitations and enabling its extension to non-text modalities.
• Empirical results demonstrate that integrating Lngram reduces perplexity in long-context language modeling and effectively injects domain knowledge.

Searching arXiv for the primary Lngram paper and closely related work on Engram/Tensorized Engram for accurate citation support. Searching arXiv for the primary Lngram paper and closely related work on Engram/Tensorized Engram for accurate citation support. Lngram is a latent-space conditional memory module for sequence models that replaces token-ID-based $N$-gram lookup with lookup over learned discrete symbols derived from hidden states. It is designed to separate “compositional reasoning” from “local static knowledge retrieval”: the former remains in the Transformer backbone, while the latter is handled by a learned memory over latent $N$-grams. In the evaluated settings, Lngram “outperforms Transformer and Engram baselines, consistently reduces perplexity in long-context language modeling, and effectively injects domain knowledge when added post hoc to pretrained models” (Zheng et al., 24 May 2026). Its construction is explicitly positioned against Engram-style token-based memories and against standard dense Transformer computation, while also relating to newer tensorized $N$-gram memories such as Tensorized Engram (Zhou et al., 6 Jun 2026).

1. Conceptual definition and motivation

Lngram is introduced from a functional decomposition of sequence modeling into “compositional reasoning” and “local static knowledge retrieval” (Zheng et al., 24 May 2026). The first involves context-dependent, multi-step computation and remains the province of attention and MLP layers. The second concerns “pattern-matching of short, frequently recurring patterns: multi-token entities, idioms, local collocations, short local reasoning templates,” which the paper characterizes as table-lookup-like rather than inherently deep computation (Zheng et al., 24 May 2026).

This framing leads to the claim that standard Transformers use dense computation for both roles, causing layers to spend capacity on “re-implementing LOOKUP-like operations” (Zheng et al., 24 May 2026). A plausible implication is that a model with an explicit retrieval pathway can reserve more effective depth for nonlocal or compositional processing.

The immediate precursor is Engram, which “partially decouples retrieval from the backbone,” but does so by building $N$-gram memories over tokenizer IDs and compressing the key space with deterministic hashing (Zheng et al., 24 May 2026). Lngram departs from this by learning discrete symbols directly from hidden states and performing exact latent $N$-gram lookup over those symbols (Zheng et al., 24 May 2026). This design is presented as removing “the dependence on tokenizer IDs” and as making the mechanism naturally extensible beyond text to “vision-language and vision-language-action tasks” (Zheng et al., 24 May 2026).

The paper identifies three limitations of Engram-style token-bound memories: token-based keys, hash compression and collisions, and limited adaptability because the hash functions are non-learnable (Zheng et al., 24 May 2026). A related later development, “Tensorized Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs” (Zhou et al., 6 Jun 2026), addresses Engram’s “separate hash tables per n-gram order” and “hash collisions” through CP-factorized shared latent structure. This suggests that the emerging research area is less about classical count-based $N$-grams than about explicit $N$-gram memory pathways inside neural sequence models.

2. Latent discretization and memory addressing

The defining operation in Lngram is the conversion of hidden states into discrete latent symbols (Zheng et al., 24 May 2026). For a layer input $$H^{(\ell)} = [h_1,\dots,h_T]^\top \in \mathbb{R}^{T\times d}$$, the module computes

$U = \mathrm{RMSNorm}(H), \quad Z = U W_q,$

followed by bitwise binarization

$b_{t,c} = \mathbb{I}[z_{t,c}>0].$

The $N$0 binary dimensions are grouped into routes of $N$1 bits, producing $N$2 routes, and each route is packed into an integer symbol

$N$3

These $N$4 define $N$5 parallel streams of learned discrete symbols (Zheng et al., 24 May 2026).

The paper’s central claim is that these symbols function as latent analogues of token IDs, but are learned from the model’s internal state rather than inherited from a tokenizer (Zheng et al., 24 May 2026). Because addressing happens in this latent discrete space, the module is not restricted to subword text segmentation. This is the basis for the paper’s statement that Lngram “removes the dependence on tokenizer IDs and naturally extends to non-text modalities” (Zheng et al., 24 May 2026).

For each $N$6-gram order $N$7, Lngram maintains learned memory tables indexed by route-specific latent $N$8-grams (Zheng et al., 24 May 2026). In the single-table version, the address for position $N$9, route $N$0, order $N$1 is

$N$2

where $N$3. The retrieved vector is

$N$4

and route-wise retrievals are concatenated into

$N$5

This is exact addressing in a learned code space, in contrast to Engram’s deterministic multi-head hashing over token IDs (Zheng et al., 24 May 2026).

The paper also defines a multi-table variant with $N$6 independent subtables, separate projections $N$7, route symbols $N$8, and per-subtable lookup tables $N$9 (Zheng et al., 24 May 2026). This increases memory capacity without changing the route code space. The mechanism is parameteric and learned end-to-end rather than built from offline count tables.

3. Readout, fusion, and placement within the Transformer

Lngram is inserted as an extra residual branch in selected Transformer layers (Zheng et al., 24 May 2026). The sequence of operations in a layer with Lngram is: compute the Lngram output $N$0, add it residually to the layer input, and then pass the updated representation through the usual self-attention and MLP stack (Zheng et al., 24 May 2026). The paper places Lngram at layers 2 and 12 in its main language-model experiments.

Because retrieved latent $N$1-gram vectors are static with respect to broader context, Lngram uses a context-aware readout to determine relevance (Zheng et al., 24 May 2026). In the single-table setting, the retrieved vector $N$2 is projected to key and value vectors,

$N$3

The current hidden state $N$4 acts as a dynamic query through the scalar gate

$N$5

and the final readout is

$N$6

In the multi-table case, branch scores are normalized with a softmax over $N$7 branches using temperature $N$8 (Zheng et al., 24 May 2026).

After gated aggregation, the sequence $N$9 is passed through a short causal depthwise-separable convolution,

$N$0

with kernel size $N$1 and dilation $N$2 (Zheng et al., 24 May 2026). The resulting $N$3 is added to the residual stream before attention (Zheng et al., 24 May 2026).

This design keeps Lngram explicitly subordinate to the backbone rather than replacing it. The retrieved branch provides local, prediction-relevant signals, while attention and MLP layers remain responsible for broader integration. The paper’s later analyses with LogitLens and CKA argue that this causes “prediction-relevant information to emerge earlier,” effectively increasing depth without large inference or memory cost (Zheng et al., 24 May 2026).

4. Training and surrogate gradients for discrete codes

Lngram is trained jointly with the backbone under the standard autoregressive language-model objective

$N$4

in text settings (Zheng et al., 24 May 2026). The discrete routing mechanism creates a non-differentiability problem, since the hard thresholds

$N$5

make the path from $N$6 to memory addresses piecewise constant (Zheng et al., 24 May 2026).

The paper states that “a naive straight-through estimator (STE) applied only to the threshold is insufficient” because the dependency between bit flips and retrieved table entries is structured by the address computation (Zheng et al., 24 May 2026). It therefore introduces a “counterfactual surrogate gradient” (Zheng et al., 24 May 2026).

In the exact local surrogate, for one route and one position, each possible code $N$7 has probability

$N$8

where $N$9, and the expected retrieval is

$N$0

With upstream gradient $N$1, the local surrogate objective is

$N$2

The resulting analytic gradient is

$N$3

Because $N$4 implies $N$5, exact enumeration is feasible, but the main experiments use a cheaper one-bit approximate surrogate (Zheng et al., 24 May 2026).

In the one-bit approximation, only counterfactual symbols produced by forcing bit $N$6 to 0 or 1 are considered, yielding

$N$7

The paper uses this approximate surrogate in its main experiments (Zheng et al., 24 May 2026). Memory tables, readout projections, and convolution layers still receive exact gradients through ordinary backpropagation (Zheng et al., 24 May 2026).

The training regimes include from-scratch language modeling, post hoc domain adaptation by training only Lngram on top of a frozen pretrained model, and multimodal training in vision-language and vision-language-action pipelines (Zheng et al., 24 May 2026). A plausible implication is that Lngram is intended not only as an architectural replacement during pretraining but also as a modular add-on for specialization.

5. Empirical results in language modeling and long-context settings

The paper evaluates three main 2B-scale models trained from scratch on 35B tokens from FineWeb-Edu: MOE, MOE+Engram, and MOE+Lngram (Zheng et al., 24 May 2026). The reported average benchmark scores are 0.5146 for MOE, 0.5225 for MOE+Engram, and 0.5288 for MOE+Lngram across HellaSwag, MMLU, SciQ, WinoGrande, and PIQA (Zheng et al., 24 May 2026). Thus, Lngram is reported as outperforming both the baseline MoE and Engram-enhanced variants (Zheng et al., 24 May 2026).

The detailed scores are as follows.

Model	HellaSwag	MMLU	SciQ	WinoGrande	PIQA	AVG
MOE	0.4394	0.2435	0.6830	0.5272	0.6801	0.5146
MOE+Engram	0.4418	0.2502	0.6980	0.5383	0.6844	0.5225
MOE+Lngram	0.4481	0.2619	0.7100	0.5312	0.6926	0.5288

The paper also reports paired-bootstrap significance tests for MOE+Lngram versus MOE, with statistically significant gains on HellaSwag, MMLU, SciQ, and PIQA, but not on WinoGrande (Zheng et al., 24 May 2026). A smaller 23-layer MOE+Lngram-23L model, with fewer total and active parameters than the baseline MOE, still achieves an average score of 0.5207, which exceeds the MOE baseline’s 0.5146 (Zheng et al., 24 May 2026). This is used by the authors as evidence that Lngram can partly substitute for dense backbone depth.

In scaling experiments, Lngram remains beneficial both with more training data and with larger sparse capacity (Zheng et al., 24 May 2026). For 2B models trained on 140B tokens, the average score rises from 0.4718 for MoE to 0.4756 for MoE+Lngram. For 8B total / 1.6B activated parameter models trained on 35B tokens, the average rises from 0.4847 to 0.4950 (Zheng et al., 24 May 2026).

For long-context language modeling, the paper applies a YaRN-based extension and evaluates prefix perplexity up to 64k tokens on PG-19 (Zheng et al., 24 May 2026). It reports that MoE+Lngram has consistently lower perplexity than MoE across all prefix lengths (Zheng et al., 24 May 2026). The description emphasizes that Lngram handles local static patterns and thereby frees the backbone to focus on long-range dependencies, rather than directly replacing long-context attention (Zheng et al., 24 May 2026).

This empirical emphasis distinguishes Lngram from count-based long-$N$8-gram systems such as Infini-gram, which scales explicit exact-match $N$9-gram probabilities to trillions of tokens via suffix arrays and defines an $N$0-gram LLM (Liu et al., 2024). Infini-gram modernizes count-based $N$1-grams by increasing both corpus scale and effective order (Liu et al., 2024), whereas Lngram modernizes $N$2-gram memory by moving lookup into learned latent space (Zheng et al., 24 May 2026). The two approaches address different levels of the modeling stack.

6. Domain adaptation, multimodal transfer, and interpretive analyses

A prominent result concerns domain knowledge injection into pretrained LLMs (Zheng et al., 24 May 2026). Starting from Qwen3-1.7B-Base, the paper compares four settings on driving-domain datasets BDD and CNK: the base model, base plus Lngram with only Lngram tuned, full model tuning without Lngram, and full tuning with Lngram (Zheng et al., 24 May 2026). The accuracies are 50.59/78.01 for the base model, 55.73/79.39 for Lngram-only tuning, 56.91/79.39 for full tuning, and 62.45/81.02 for full tuning plus Lngram (Zheng et al., 24 May 2026). The paper therefore claims that Lngram-only tuning can approach full fine-tuning and that joint tuning with Lngram surpasses it (Zheng et al., 24 May 2026).

The same experiments also show degradation on general benchmarks after domain adaptation for all adapted variants (Zheng et al., 24 May 2026). This is treated as a limitation rather than hidden. A plausible implication is that Lngram is well suited for specialized domain models, but not a mechanism that preserves general capability automatically.

Lngram is also applied to multimodal settings. In vision-language-action experiments on LIBERO, MindPI-MoE achieves an average success rate of 0.9825, while MindPI-MoE+Lngram achieves 0.9850 (Zheng et al., 24 May 2026). In vision-language experiments on SEED-Bench, the average score rises from 60.5 to 61.2, with gains on Visual Reasoning, Instance Location, and Instance Attributes, but a drop on Instance Interaction (Zheng et al., 24 May 2026). These results are used to support the claim that Lngram is modality-agnostic because its keys are derived from hidden states rather than tokenizer IDs (Zheng et al., 24 May 2026).

The interpretive analyses are central to the paper’s argument. Using LogitLens, the authors compare intermediate-layer output distributions to the final distribution through

$N$3

They report that after the first layer, Lngram yields lower KL than the baseline across most of the network, which they interpret as meaning that representations become “prediction-like earlier” (Zheng et al., 24 May 2026). Using CKA, they show that Lngram layer representations align most closely with deeper layers of the baseline network, producing an “effective depth gain” of about +2 to +3 layers in the middle of the model (Zheng et al., 24 May 2026).

Gate visualizations provide a more local explanation: in examples such as “Alexander the Great” and “The Princess of Wales,” the gate for the 3-gram branch peaks when the phrase is completed (Zheng et al., 24 May 2026). The paper interprets this as evidence that Lngram acts as a latent phrase memory that activates selectively on completed multi-token patterns (Zheng et al., 24 May 2026).

Ablations indicate that $N$4-gram orders work best, outperforming $N$5, $N$6, or $N$7 (Zheng et al., 24 May 2026). The paper also finds that allocating 25% of sparse capacity to Lngram and 75% to MoE-MLP is preferable to more extreme ratios (Zheng et al., 24 May 2026). This suggests that the module is most useful as a complement to, rather than replacement for, dense dynamic computation.

7. Relation to adjacent $N$8-gram traditions and limitations

Lngram belongs to a broader revival of $N$9-gram structures in modern neural systems, but it is distinct from several neighboring lines of work. Classical $$H^{(\ell)} = [h_1,\dots,h_T]^\top \in \mathbb{R}^{T\times d}$$0-gram language modeling estimates conditional probabilities from counts and smoothing, as in modified Kneser–Ney or power low rank ensembles (Parikh et al., 2013). Infini-gram modernizes this count-based tradition by defining an unbounded $$H^{(\ell)} = [h_1,\dots,h_T]^\top \in \mathbb{R}^{T\times d}$$1-gram model over massive corpora (Liu et al., 2024), and Infini-gram mini extends exact $$H^{(\ell)} = [h_1,\dots,h_T]^\top \in \mathbb{R}^{T\times d}$$2-gram search to Internet scale with FM-index compression (Xu et al., 13 Jun 2025). Those works retain explicit corpus-based statistics. Lngram, by contrast, stores a learned parametric memory and uses latent discrete codes rather than corpus counts (Zheng et al., 24 May 2026).

Engram is the closest architectural predecessor. It builds suffix $$H^{(\ell)} = [h_1,\dots,h_T]^\top \in \mathbb{R}^{T\times d}$$3-gram memories over token IDs and retrieves vectors through hashed tables (Zheng et al., 24 May 2026). Tensorized Engram later replaces those independent per-order hash tables with a CP-factorized shared latent representation to eliminate collisions and share structure across orders (Zhou et al., 6 Jun 2026). Lngram solves a different problem: it removes tokenizer dependence by learning the symbols themselves from hidden states (Zheng et al., 24 May 2026). This suggests two distinct research directions: improving token-based $$H^{(\ell)} = [h_1,\dots,h_T]^\top \in \mathbb{R}^{T\times d}$$4-gram memory representations, as in TN-gram (Zhou et al., 6 Jun 2026), and replacing token-space addressing with latent-space addressing, as in Lngram (Zheng et al., 24 May 2026).

The paper’s limitations are explicit. Memory tables scale with $$H^{(\ell)} = [h_1,\dots,h_T]^\top \in \mathbb{R}^{T\times d}$$5, so capacity can become large (Zheng et al., 24 May 2026). Training requires specialized surrogate gradients, since naive STE is inadequate (Zheng et al., 24 May 2026). The module captures local context rather than arbitrarily long-range structure (Zheng et al., 24 May 2026), and performance depends on hyperparameters such as $$H^{(\ell)} = [h_1,\dots,h_T]^\top \in \mathbb{R}^{T\times d}$$6-gram orders, bits per route, and capacity allocation (Zheng et al., 24 May 2026). In multimodal evaluation, some relational tasks do not improve, and domain adaptation can reduce general benchmark performance (Zheng et al., 24 May 2026).

These constraints counter a possible misconception that Lngram is a general replacement for attention or for dense neural reasoning. The paper instead presents it as a mechanism for “rewriting local static matching as latent lookup” (Zheng et al., 24 May 2026). A plausible implication is that its most natural role is architectural specialization: offloading local pattern retrieval so that the backbone can use its depth on problems for which dense computation is genuinely necessary.