--- title: 'Lngram: Latent-Space Memory Module' url: https://www.emergentmind.com/topics/lngram type: topic --- # Lngram: Latent-Space Memory Module Searching arXiv for the primary Lngram paper and closely related work on Engram/Tensorized Engram for accurate citation support. Searching arXiv for the primary Lngram paper and closely related work on Engram/Tensorized Engram for accurate citation support. Lngram is a latent-space conditional memory module for sequence models that replaces token-ID-based \(N\)-gram lookup with lookup over learned discrete symbols derived from hidden states. It is designed to separate “compositional reasoning” from “local static knowledge retrieval”: the former remains in the Transformer backbone, while the latter is handled by a learned memory over latent \(N\)-grams. In the evaluated settings, Lngram “outperforms Transformer and Engram baselines, consistently reduces perplexity in long-context language modeling, and effectively injects domain knowledge when added post hoc to pretrained models” [2605.24869]. Its construction is explicitly positioned against Engram-style token-based memories and against standard dense Transformer computation, while also relating to newer tensorized \(N\)-gram memories such as Tensorized Engram [2606.08347]. ## 1. Conceptual definition and motivation Lngram is introduced from a functional decomposition of sequence modeling into “compositional reasoning” and “local static knowledge retrieval” [2605.24869]. The first involves context-dependent, multi-step computation and remains the province of attention and MLP layers. The second concerns “pattern-matching of short, frequently recurring patterns: multi-token entities, idioms, local collocations, short local reasoning templates,” which the paper characterizes as table-lookup-like rather than inherently deep computation [2605.24869]. This framing leads to the claim that standard Transformers use dense computation for both roles, causing layers to spend capacity on “re-implementing LOOKUP-like operations” [2605.24869]. A plausible implication is that a model with an explicit retrieval pathway can reserve more effective depth for nonlocal or compositional processing. The immediate precursor is Engram, which “partially decouples retrieval from the backbone,” but does so by building \(N\)-gram memories over tokenizer IDs and compressing the key space with deterministic hashing [2605.24869]. Lngram departs from this by learning discrete symbols directly from hidden states and performing exact latent \(N\)-gram lookup over those symbols [2605.24869]. This design is presented as removing “the dependence on tokenizer IDs” and as making the mechanism naturally extensible beyond text to “vision-language and vision-language-action tasks” [2605.24869]. The paper identifies three limitations of Engram-style token-bound memories: token-based keys, hash compression and collisions, and limited adaptability because the hash functions are non-learnable [2605.24869]. A related later development, “Tensorized Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs” [2606.08347], addresses Engram’s “separate hash tables per n-gram order” and “hash collisions” through CP-factorized shared latent structure. This suggests that the emerging research area is less about classical count-based \(N\)-grams than about explicit \(N\)-gram memory pathways inside neural sequence models. ## 2. Latent discretization and memory addressing The defining operation in Lngram is the conversion of hidden states into discrete latent symbols [2605.24869]. For a layer input \(H^{(\ell)} = [h_1,\dots,h_T]^\top \in \mathbb{R}^{T\times d}\), the module computes \[ U = \mathrm{RMSNorm}(H), \quad Z = U W_q, \] followed by bitwise binarization \[ b_{t,c} = \mathbb{I}[z_{t,c}>0]. \] The \(d\) binary dimensions are grouped into routes of \(M\) bits, producing \(R=d/M\) routes, and each route is packed into an integer symbol \[ a_{t,r} = \sum_{j=0}^{M-1} b_{t,(r,j)} 2^j \in \{0,\dots, 2^M-1\}. \] These \(a_{t,r}\) define \(R\) parallel streams of learned discrete symbols [2605.24869]. The paper’s central claim is that these symbols function as latent analogues of token IDs, but are learned from the model’s internal state rather than inherited from a tokenizer [2605.24869]. Because addressing happens in this latent discrete space, the module is not restricted to subword text segmentation. This is the basis for the paper’s statement that Lngram “removes the dependence on tokenizer IDs and naturally extends to non-text modalities” [2605.24869]. For each \(N\)-gram order \(n\), Lngram maintains learned memory tables indexed by route-specific latent \(N\)-grams [2605.24869]. In the single-table version, the address for position \(t\), route \(r\), order \(n\) is \[ g_{t,r}^{(n)} = rK^n + \sum_{i=0}^{n-1} a_{t-n+1+i, r} K^i, \] where \(K=2^M\). The retrieved vector is \[ m_{t,r}^{(n)} = E^{(n)}[g_{t,r}^{(n)}] \in \mathbb{R}^{d_m}, \] and route-wise retrievals are concatenated into \[ e_t^{(n)} = \mathrm{Concat}_{r=0}^{R-1} m_{t,r}^{(n)}. \] This is exact addressing in a learned code space, in contrast to Engram’s deterministic multi-head hashing over token IDs [2605.24869]. The paper also defines a multi-table variant with \(S\) independent subtables, separate projections \(W_q^{(s)}\), route symbols \(a_{t,r}^{(s)}\), and per-subtable lookup tables \(E_s^{(n)}\) [2605.24869]. This increases memory capacity without changing the route code space. The mechanism is parameteric and learned end-to-end rather than built from offline count tables. ## 3. Readout, fusion, and placement within the Transformer Lngram is inserted as an extra residual branch in selected Transformer layers [2605.24869]. The sequence of operations in a layer with Lngram is: compute the Lngram output \(Y\), add it residually to the layer input, and then pass the updated representation through the usual self-attention and MLP stack [2605.24869]. The paper places Lngram at layers 2 and 12 in its main language-model experiments. Because retrieved latent \(N\)-gram vectors are static with respect to broader context, Lngram uses a context-aware readout to determine relevance [2605.24869]. In the single-table setting, the retrieved vector \(e_t^{(n)}\) is projected to key and value vectors, \[ k_t^{(n)} = W_K e_t^{(n)} + b_K, \qquad v_t^{(n)} = W_V e_t^{(n)} + b_V. \] The current hidden state \(h_t\) acts as a dynamic query through the scalar gate \[ \alpha_t^{(n)} = \sigma\!\left( \frac{ \mathrm{RMSNorm}(h_t)^\top \mathrm{RMSNorm}(k_t^{(n)}) }{\sqrt{d}} \right), \] and the final readout is \[ v_t = \sum_{n\in\mathcal{N}} \alpha_t^{(n)} v_t^{(n)}. \] In the multi-table case, branch scores are normalized with a softmax over \((s,n)\) branches using temperature \(\tau_f\) [2605.24869]. After gated aggregation, the sequence \(V=[v_1,\dots,v_T]^\top\) is passed through a short causal depthwise-separable convolution, \[ Y = V + \mathrm{SiLU}\!\bigl(\mathrm{DWConv1D}(\mathrm{RMSNorm}(V))\bigr), \] with kernel size \(w=4\) and dilation \(\delta = \max \mathcal{N}\) [2605.24869]. The resulting \(Y\) is added to the residual stream before attention [2605.24869]. This design keeps Lngram explicitly subordinate to the backbone rather than replacing it. The retrieved branch provides local, prediction-relevant signals, while attention and MLP layers remain responsible for broader integration. The paper’s later analyses with LogitLens and CKA argue that this causes “prediction-relevant information to emerge earlier,” effectively increasing depth without large inference or memory cost [2605.24869]. ## 4. Training and surrogate gradients for discrete codes Lngram is trained jointly with the backbone under the standard autoregressive language-model objective \[ \mathcal{L}_{\text{LM}} = - \sum_t \log p(x_t \mid x_{ 0] \] make the path from \(Z\) to memory addresses piecewise constant [2605.24869]. The paper states that “a naive straight-through estimator (STE) applied only to the threshold is insufficient” because the dependency between bit flips and retrieved table entries is structured by the address computation [2605.24869]. It therefore introduces a “counterfactual surrogate gradient” [2605.24869]. In the exact local surrogate, for one route and one position, each possible code \(c \in \{0,\dots,K-1\}\) has probability \[ P(c \mid z) = \prod_{j=0}^{M-1} p_j^{\beta_j(c)} (1-p_j)^{1-\beta_j(c)}, \] where \(p_j = \sigma(\tau z_j)\), and the expected retrieval is \[ \mu(z) = \sum_{c=0}^{K-1} P(c\mid z)\, E_c. \] With upstream gradient \(g = \partial \mathcal{L}/\partial E\), the local surrogate objective is \[ \mathcal{L}_{\text{local}}(z) = \langle g, \mu(z) \rangle. \] The resulting analytic gradient is \[ \frac{\partial \mathcal{L}_{\text{local}}}{\partial z_j} = \tau \sum_{c=0}^{K-1} P(c\mid z)\bigl(\beta_j(c)-p_j\bigr)\langle g, E_c\rangle. \] Because \(M=4\) implies \(K=16\), exact enumeration is feasible, but the main experiments use a cheaper one-bit approximate surrogate [2605.24869]. In the one-bit approximation, only counterfactual symbols produced by forcing bit \(j\) to 0 or 1 are considered, yielding \[ \frac{\partial \mathcal{L}_{\text{local}}}{\partial z_j} \approx \lambda \tau p_j(1-p_j) \langle g, E_j^{(1)} - E_j^{(0)} \rangle. \] The paper uses this approximate surrogate in its main experiments [2605.24869]. Memory tables, readout projections, and convolution layers still receive exact gradients through ordinary backpropagation [2605.24869]. The training regimes include from-scratch language modeling, post hoc domain adaptation by training only Lngram on top of a frozen pretrained model, and multimodal training in vision-language and vision-language-action pipelines [2605.24869]. A plausible implication is that Lngram is intended not only as an architectural replacement during pretraining but also as a modular add-on for specialization. ## 5. Empirical results in language modeling and long-context settings The paper evaluates three main 2B-scale models trained from scratch on 35B tokens from FineWeb-Edu: MOE, MOE+Engram, and MOE+Lngram [2605.24869]. The reported average benchmark scores are 0.5146 for MOE, 0.5225 for MOE+Engram, and 0.5288 for MOE+Lngram across HellaSwag, MMLU, SciQ, WinoGrande, and PIQA [2605.24869]. Thus, Lngram is reported as outperforming both the baseline MoE and Engram-enhanced variants [2605.24869]. The detailed scores are as follows. | Model | HellaSwag | MMLU | SciQ | WinoGrande | PIQA | AVG | |---|---:|---:|---:|---:|---:|---:| | MOE | 0.4394 | 0.2435 | 0.6830 | 0.5272 | 0.6801 | 0.5146 | | MOE+Engram | 0.4418 | 0.2502 | 0.6980 | 0.5383 | 0.6844 | 0.5225 | | MOE+Lngram | 0.4481 | 0.2619 | 0.7100 | 0.5312 | 0.6926 | 0.5288 | The paper also reports paired-bootstrap significance tests for MOE+Lngram versus MOE, with statistically significant gains on HellaSwag, MMLU, SciQ, and PIQA, but not on WinoGrande [2605.24869]. A smaller 23-layer MOE+Lngram-23L model, with fewer total and active parameters than the baseline MOE, still achieves an average score of 0.5207, which exceeds the MOE baseline’s 0.5146 [2605.24869]. This is used by the authors as evidence that Lngram can partly substitute for dense backbone depth. In scaling experiments, Lngram remains beneficial both with more training data and with larger sparse capacity [2605.24869]. For 2B models trained on 140B tokens, the average score rises from 0.4718 for MoE to 0.4756 for MoE+Lngram. For 8B total / 1.6B activated parameter models trained on 35B tokens, the average rises from 0.4847 to 0.4950 [2605.24869]. For long-context language modeling, the paper applies a YaRN-based extension and evaluates prefix perplexity up to 64k tokens on PG-19 [2605.24869]. It reports that MoE+Lngram has consistently lower perplexity than MoE across all prefix lengths [2605.24869]. The description emphasizes that Lngram handles local static patterns and thereby frees the backbone to focus on long-range dependencies, rather than directly replacing long-context attention [2605.24869]. This empirical emphasis distinguishes Lngram from count-based long-\(N\)-gram systems such as Infini-gram, which scales explicit exact-match \(n\)-gram probabilities to trillions of tokens via suffix arrays and defines an \(\infty\)-gram language model [2401.17377]. Infini-gram modernizes count-based \(n\)-grams by increasing both corpus scale and effective order [2401.17377], whereas Lngram modernizes \(N\)-gram memory by moving lookup into learned latent space [2605.24869]. The two approaches address different levels of the modeling stack. ## 6. Domain adaptation, multimodal transfer, and interpretive analyses A prominent result concerns domain knowledge injection into pretrained language models [2605.24869]. Starting from Qwen3-1.7B-Base, the paper compares four settings on driving-domain datasets BDD and CNK: the base model, base plus Lngram with only Lngram tuned, full model tuning without Lngram, and full tuning with Lngram [2605.24869]. The accuracies are 50.59/78.01 for the base model, 55.73/79.39 for Lngram-only tuning, 56.91/79.39 for full tuning, and 62.45/81.02 for full tuning plus Lngram [2605.24869]. The paper therefore claims that Lngram-only tuning can approach full fine-tuning and that joint tuning with Lngram surpasses it [2605.24869]. The same experiments also show degradation on general benchmarks after domain adaptation for all adapted variants [2605.24869]. This is treated as a limitation rather than hidden. A plausible implication is that Lngram is well suited for specialized domain models, but not a mechanism that preserves general capability automatically. Lngram is also applied to multimodal settings. In vision-language-action experiments on LIBERO, MindPI-MoE achieves an average success rate of 0.9825, while MindPI-MoE+Lngram achieves 0.9850 [2605.24869]. In vision-language experiments on SEED-Bench, the average score rises from 60.5 to 61.2, with gains on Visual Reasoning, Instance Location, and Instance Attributes, but a drop on Instance Interaction [2605.24869]. These results are used to support the claim that Lngram is modality-agnostic because its keys are derived from hidden states rather than tokenizer IDs [2605.24869]. The interpretive analyses are central to the paper’s argument. Using LogitLens, the authors compare intermediate-layer output distributions to the final distribution through \[ D_{\mathrm{KL}}(p_{\text{final}} \Vert p_l). \] They report that after the first layer, Lngram yields lower KL than the baseline across most of the network, which they interpret as meaning that representations become “prediction-like earlier” [2605.24869]. Using CKA, they show that Lngram layer representations align most closely with deeper layers of the baseline network, producing an “effective depth gain” of about +2 to +3 layers in the middle of the model [2605.24869]. Gate visualizations provide a more local explanation: in examples such as “Alexander the Great” and “The Princess of Wales,” the gate for the 3-gram branch peaks when the phrase is completed [2605.24869]. The paper interprets this as evidence that Lngram acts as a latent phrase memory that activates selectively on completed multi-token patterns [2605.24869]. Ablations indicate that \(\{2,3\}\)-gram orders work best, outperforming \(\{2\}\), \(\{3\}\), or \(\{1,2,3\}\) [2605.24869]. The paper also finds that allocating 25% of sparse capacity to Lngram and 75% to MoE-MLP is preferable to more extreme ratios [2605.24869]. This suggests that the module is most useful as a complement to, rather than replacement for, dense dynamic computation. ## 7. Relation to adjacent \(N\)-gram traditions and limitations Lngram belongs to a broader revival of \(N\)-gram structures in modern neural systems, but it is distinct from several neighboring lines of work. Classical \(n\)-gram language modeling estimates conditional probabilities from counts and smoothing, as in modified Kneser–Ney or power low rank ensembles [1312.7077]. Infini-gram modernizes this count-based tradition by defining an unbounded \(\infty\)-gram model over massive corpora [2401.17377], and Infini-gram mini extends exact \(n\)-gram search to Internet scale with FM-index compression [2506.12229]. Those works retain explicit corpus-based statistics. Lngram, by contrast, stores a learned parametric memory and uses latent discrete codes rather than corpus counts [2605.24869]. Engram is the closest architectural predecessor. It builds suffix \(N\)-gram memories over token IDs and retrieves vectors through hashed tables [2605.24869]. Tensorized Engram later replaces those independent per-order hash tables with a CP-factorized shared latent representation to eliminate collisions and share structure across orders [2606.08347]. Lngram solves a different problem: it removes tokenizer dependence by learning the symbols themselves from hidden states [2605.24869]. This suggests two distinct research directions: improving token-based \(N\)-gram memory representations, as in TN-gram [2606.08347], and replacing token-space addressing with latent-space addressing, as in Lngram [2605.24869]. The paper’s limitations are explicit. Memory tables scale with \(R K^n d_m\), so capacity can become large [2605.24869]. Training requires specialized surrogate gradients, since naive STE is inadequate [2605.24869]. The module captures local context rather than arbitrarily long-range structure [2605.24869], and performance depends on hyperparameters such as \(N\)-gram orders, bits per route, and capacity allocation [2605.24869]. In multimodal evaluation, some relational tasks do not improve, and domain adaptation can reduce general benchmark performance [2605.24869]. These constraints counter a possible misconception that Lngram is a general replacement for attention or for dense neural reasoning. The paper instead presents it as a mechanism for “rewriting local static matching as latent lookup” [2605.24869]. A plausible implication is that its most natural role is architectural specialization: offloading local pattern retrieval so that the backbone can use its depth on problems for which dense computation is genuinely necessary. Source: https://www.emergentmind.com/topics/lngram