---
title: 'X-GRAM: Efficient Token-Injection Scaling'
url: https://www.emergentmind.com/topics/x-gram
type: topic
---

# X-GRAM: Efficient Token-Injection Scaling

X-GRAM is, in its explicit contemporary usage, a **frequency-aware dynamic token-injection framework** for efficient embedding parameter scaling in large language models, introduced to make token-indexed memory a practical **memory-centric scaling axis** that decouples model capacity from FLOPs [2604.21724]. The same string also has an informal but technically natural interpretation in parallel-beam X-ray CT, where a **Gram-operator–based** pixel-basis reconstruction framework can be read as “X-GRAM,” with **X** denoting X-ray CT and **GRAM** denoting the normal operator \(G=A^T A\) and its associated Gram filter [2005.13471]. By contrast, the recommender-system model **GRAM** and the convex-algebraic notion of **Gram spectrahedra** are distinct concepts that share the lexical stem “Gram” but not the underlying methodology [2506.01673; 1910.06728].

## 1. Scope and nomenclature

The label **X-GRAM** is not monolithic across the supplied literature. One paper uses it explicitly as the title of a language-model memory architecture, while another provides a CT reconstruction framework that the source text identifies as the kind of method that could sit behind an **informally called “X-GRAM”** system. Other “GRAM” usages are separate research objects rather than variants of the same framework.

| Usage | Domain | Defining idea |
|---|---|---|
| **X-GRAM** [2604.21724] | Language modeling | Frequency-aware dynamic token injection for token-indexed memory |
| **Informal “X-GRAM”** [2005.13471] | Parallel-beam X-ray CT | Gram-operator–based Gram filtering and sinogram interpolation |
| **GRAM** [2506.01673] | Generative recommendation | Semantic-aware multi-granular late fusion |
| **Gram spectrahedra** [1910.06728] | Convex algebraic geometry | Facial structure of symmetric and Hermitian Gram spectrahedra |

For most current technical purposes, **X-GRAM** denotes the language-model framework introduced in “Beyond N-gram: Data-Aware X-GRAM Extraction for Efficient Embedding Parameter Scaling” [2604.21724]. Its central claim is that lookup-based scaling is constrained less by raw parameter count than by the mismatch between **table capacity**, **update density**, and **representational diversity**. The framework therefore couples **frequency-aware routing**, **alias mixing**, **normalized SwiGLU ShortConv**, and **depth-aware gating** to increase the utility of token-indexed memory under fixed compute and activation budgets [2604.21724].

A common source of confusion is the assumption that “X-GRAM” must refer to a single standardized acronym across fields. The supplied literature supports a narrower statement: the name is explicit in large-scale language modeling, while in CT it is a plausible and source-endorsed informal descriptor for **Gram-based processing for X-ray CT** [2005.13471].

## 2. X-GRAM in efficient embedding parameter scaling

The explicit X-GRAM framework addresses the long-standing appeal and difficulty of **token-indexed memory**. Large lookup tables provide a **compute-decoupled scaling path**, because capacity can be increased without proportionally increasing Transformer depth or width. The paper argues, however, that naïvely enlarged tables are often limited by three pathologies: **Zipfian under-training of the long tail**, **heterogeneous demand across layers**, and **slot collapse** that yields redundant embeddings [2604.21724].

The parameter-efficiency problem is stated concretely. MoLE/SCONE-style designs may require reparameterized tables **2.4–7.4×** larger than the backbone, and explicit n-gram tables can reach **7.6 TB for \(10^9\) n-gram embeddings**. Since such tables cannot remain entirely in HBM, they are offloaded to host or SSD, which increases parameter traffic during inference [2604.21724]. The framework therefore rejects direct n-gram-table inflation and instead learns **x-gram** features from compressed 1-gram memory.

The statistical motivation is explicitly Zipfian. Token frequencies follow a head-heavy, tail-heavy distribution in which a small head dominates occurrences. The paper reports that **average activation magnitude per token** strongly correlates with frequency, and that as table size grows, the fraction of “sufficiently updated” rows drops sharply, so additional rows often remain close to initialization. A related systems-level observation is that different layers and pathways do not demand the same form of auxiliary memory: **value-stream** and **inter-layer residual** injection are substantially more effective than **Q/K injection**, and early layers benefit strongly, although distributing injection across layers is better than restricting it to a single layer [2604.21724].

The formal optimization view places X-GRAM within a constrained augmentation problem:
$$
\min_{\theta,\varphi}
\mathbb{E}_{(\mathbf{x},y)\sim\mathcal{D}}
\left[\ell\!\left(f_{\theta,\varphi}(\mathbf{x}),y\right)\right]
\quad \text{s.t.}\quad
\mathcal{C}(\varphi)\le \mathcal{C}_0,\ \mathcal{A}(\varphi)\le \mathcal{A}_0,\ 
\mathcal{F}(\varphi)\le \mathcal{F}_0.
$$
Here \(\mathcal{C}\) is memory, \(\mathcal{A}\) is activation footprint, and \(\mathcal{F}\) is extra FLOPs. This places token-indexed memory on a separate scaling axis: X-GRAM seeks larger effective capacity by increasing table structure and retrieval diversity while holding the compute pathway relatively light [2604.21724].

## 3. Core operators and model mechanics

X-GRAM decomposes lookup-based augmentation into three operators: **routing/retrieval** \(\mathcal{R}_\phi\), **extraction/refinement** \(\mathcal{T}_\psi\), and **injection** \(\mathcal{I}_s\). At layer \(\ell\) and view \(m\),
$$
\mathbf{E}_{\ell}^{(m)}=\mathcal{R}_\phi^{(m)}(\mathbf{x};\mathbf{B}_{\ell}^{(m)}), 
\qquad 
\tilde{\mathbf{E}}_{\ell}^{(m)}=\mathcal{T}_\psi^{(m)}(\mathbf{E}_{\ell}^{(m)}),
$$
and the fused memory signal is
$$
\Delta_{\ell}=\sum_{m=1}^{M_\ell} g_{\ell}^{(m)}\,\tilde{\mathbf{E}}_{\ell}^{(m)}.
$$
This tri-operator factorization is important because X-GRAM does not merely add retrieved embeddings; it learns a structured sequence-to-sequence refinement over retrieved memory before injection [2604.21724].

### Frequency-aware routing: hybrid hashing and alias mixing

The routing mechanism begins with empirical token frequency \(p(\omega)\) and a smoothed mass
$$
s(\omega)=p(\omega)^{\alpha}, \qquad 0<\alpha\le 1.
$$
A VIP head set \(\mathcal{V}_{\mathrm{vip}}\) receives dedicated rows, while the remaining vocabulary is partitioned into \(B\) logical buckets satisfying
$$
\sum_{\omega\in \mathcal{G}_b} p(\omega)^{\alpha} \approx \frac{1}{B}\sum_{\omega\in\mathcal{V}\setminus \mathcal{V}_{\mathrm{vip}}} p(\omega)^{\alpha}.
$$
The compression ratio is
$$
\rho = S/|V|,
$$
with \(\rho=0.5\) in the 50% setting [2604.21724].

Within each bucket, sparse regions can map tokens directly to distinct physical rows, whereas dense regions use multi-path local hashing and **alias mixing**. The token representation is
$$
\mathbf{e}_{\ell}^{(m)}(\omega) = \sum_{j\in \mathcal{A}(\omega)} c_{\omega,j}\, \sigma\!\bigl(a_{\ell,j}^{(m)}\bigr)\, \mathbf{B}_{\ell}^{(m)}[j].
$$
Here \(\mathcal{A}(\omega)\) is a small alias set, \(c_{\omega,j}\) are fixed aggregation coefficients, \(a_{\ell,j}^{(m)}\) are row-wise learnable gates, and \(\sigma\) is a Sigmoid. This construction serves two explicit purposes: attenuating noisy collisions in dense buckets and recycling unused rows in sparse buckets [2604.21724].

### X-gram extraction: normalized SwiGLU ShortConv

The framework’s defining move beyond explicit n-gram tables is to transform retrieved 1-gram sequences into learned local **x-gram** features. Retrieved sequences are first RMS-normalized:
$$
\bar{\mathbf{E}}_{\ell}^{(m)}=\mathrm{RMSNorm}(\mathbf{E}_{\ell}^{(m)}).
$$
Each view then applies a depthwise, SwiGLU-like ShortConv:
$$
\mathbf{C}_{\ell}^{(m)} =
\mathrm{DWConv1D}_{\psi_c^{(m)},\,k^{(m)}}\!\left(\bar{\mathbf{E}}_{\ell}^{(m)}\right)
\odot
\mathrm{SiLU}\!\left(
\mathrm{DWConv1D}_{\psi_g^{(m)},\,k^{(m)}}\!\left(\bar{\mathbf{E}}_{\ell}^{(m)}\right)+\mathbf{b}^{(m)}
\right),
$$
followed by residual refinement,
$$
\tilde{\mathbf{E}}_{\ell}^{(m)}=\mathbf{E}_{\ell}^{(m)}+\mathbf{C}_{\ell}^{(m)}.
$$
Different views use different kernel sizes \(k^{(m)}\), yielding multi-scale coverage such as \(k\in\{3,5,7,9\}\). This design is meant to break slot symmetry, introduce contextual selectivity, and generate variable-length n-gram-like signals without explicit n-gram tables [2604.21724].

### Injection and depth-aware gating

After refinement, views are fused as
$$
\Delta_{\ell}=\frac{1}{\sqrt{M_\ell}}\sum_{m=1}^{M_\ell} g_{\ell}^{(m)}(u)\,\tilde{\mathbf{E}}_{\ell}^{(m)}.
$$
The preferred injection site is the attention value stream:
$$
\mathbf{V}_\ell \leftarrow \mathbf{H}_{\ell-1}\mathbf{W}_\ell^{V} + \Delta_\ell,
$$
and, under heavier compression, the signal may also be injected into the inter-layer residual:
$$
\mathbf{H}_\ell \leftarrow \mathrm{Block}_\ell(\mathbf{H}_{\ell-1} + \Delta_\ell).
$$
The gate itself is
$$
g_{\ell}^{(m)}(u)=\lambda_{\ell}^{(m)}\cdot \sqrt{\ell+1}\cdot w_{\text{warmup}}(u),
$$
where \(\lambda_{\ell}^{(m)}\) is learnable, \(\sqrt{\ell+1}\) provides heuristic depth scaling, and \(w_{\text{warmup}}(u)\) ramps from 0 to 1 early in training. The source analysis reports that **V** and **inter-layer residual** injection dominate **Q/K** injection in the accuracy/perplexity trade-off [2604.21724].

## 4. Optimization, budgets, and empirical behavior

X-GRAM is trained jointly with the backbone under the standard causal language-model objective. The routing maps are static during training: token frequency estimation, VIP selection, bucket boundaries, and hash functions are fixed offline, whereas lookup tables \(\mathbf{B}_{\ell}^{(m)}\), alias gates \(a_{\ell,j}^{(m)}\), ShortConv parameters \(\psi_c^{(m)},\psi_g^{(m)},\mathbf{b}^{(m)}\), gating scalars \(\lambda_{\ell}^{(m)}\), and backbone parameters \(\theta\) are learned [2604.21724].

Several stabilization devices are part of the framework rather than incidental implementation details. Retrieved sequences are RMS-normalized before ShortConv; multi-view fusion uses the \(1/\sqrt{M_\ell}\) factor; warmup gating reduces loss spikes; and lookup-table learning rates are scaled approximately as
$$
\eta_{\text{lookup}} \propto \sqrt{S},
$$
to compensate for reduced per-row update frequency as table size grows [2604.21724].

The budget accounting is also explicit. For each injection site and view, the parameter overhead is \(\Delta\text{Params}=S d_s\) or \(\rho S d_s\) when compressed, the activation overhead is \(\Delta\text{Dims}=d_s\), and the extra FLOPs are approximately
$$
\Delta\text{FLOPs} \approx (K+2k+1)d_s.
$$
For value-path injection, the usual choice is \(d_s=d_{\text{kv}}\), which is smaller than the full hidden width under grouped-query attention. This is one reason the paper presents value-stream injection as particularly efficient [2604.21724].

The utilization analysis quantifies the effect of hybrid hashing. Under frequency-agnostic routing (“No Hash”), the top **0.1%** rows absorb **57.74%** of hits and the \(p75/p25\) movement gap is **\(3.47\times\)**. Under frequency-aware hashing, the top **0.1%** rows absorb **43.11%** of hits, the movement gap drops to **\(1.14\times\)**, and the coefficient of variation drops from **0.587** to **0.159**. These diagnostics support the claim that X-GRAM redistributes updates more evenly under a fixed memory budget [2604.21724].

The ablation results assign substantial importance to the extraction and optimization mechanisms. Removing **X-gram Extraction** degrades perplexity from **17.702** to **18.420** and average accuracy from **49.68** to **47.80**. Removing retrieved-sequence normalization yields **18.282** perplexity and **47.38** average accuracy, while removing lookup learning-rate scaling yields **18.271** perplexity and **47.62** average accuracy. A multi-scale ShortConv with \(k\in\{3,5,7,9\}\) achieves the best reported **17.924** perplexity and **49.27** accuracy, with only about **2%** extra FLOPs over a baseline 3× ShortConv path [2604.21724].

## 5. Performance and scaling characteristics

The main experiments use decoder-only Transformer backbones at **0.73B** and **1.15B** scale, trained on **OLMo-mix-1124** and evaluated on the **lm-evaluation-harness** suite including SciQ, PIQA, WinoGrande, ARC-Easy, ARC-Challenge, HellaSwag, SocialIQa, BoolQ, OpenBookQA, and MMLU [2604.21724]. The baselines are the vanilla backbone and strong lookup/memory baselines including **Retoken**, **MoRT**, and **Engram**.

| Scale | Baseline avg. | Best X-GRAM avg. |
|---|---:|---:|
| 0.73B | 44.7 | 49.7 |
| 1.15B | 47.4 | 50.8 |

At **0.73B**, the baseline average is **44.7**. Under the 1× budget, **X-gram-50%** reaches **48.5** using a **1.17B** table, whereas **X-gram-100%** reaches **47.5** with a **2.33B** table. Under the 2× budget, **X-gram-100%** reaches **49.7**, and under the 4× budget **X-gram-50%** reaches **49.1** while **X-gram-100%** reaches **49.5**. The paper summarizes this as up to **4.4 points over the vanilla backbone** and **3.2 points over strong retrieval baselines**, with substantially smaller tables in the **50%** configuration [2604.21724].

At **1.15B**, the baseline average is **47.4**. The reported X-GRAM variants achieve **49.5** at 1×, **50.5** at 2× for **X-gram-50%**, and **50.8** at 4× for both **X-gram-50%** and **X-gram-100%**. The pattern emphasized in the paper is **scaling monotonicity**: X-GRAM continues to improve with greater capacity, whereas several baselines plateau or regress at higher table sizes [2604.21724].

The training-dynamics analysis complements the endpoint metrics. Under 4× capacity, X-GRAM maintains about **0.04 lower loss** than Retoken and MoRT throughout training, and it matches baseline performance using only about **57%** of pretraining tokens. This suggests higher data efficiency rather than merely higher asymptotic capacity [2604.21724].

The site-specific analyses reinforce the architectural claims. Injecting into **V** and/or **inter-layer residuals** yields the best perplexity trade-off; **Q/K injection** is consistently worse; and mixed **h+v** configurations dominate value-only or inter-layer-only variants in the accuracy/perplexity versus parameter-budget frontier. The depth analysis shows that injecting at any single layer improves over baseline, early layers are more sensitive, and full-layer injection is much stronger than any single-layer injection [2604.21724].

## 6. Informal X-GRAM in parallel-beam X-ray CT

In the CT literature supplied here, the phrase **X-GRAM** is not used as the paper’s formal title, but the source text states that the work is “exactly the type of work that could sit behind a method informally called **‘X-GRAM’**: Gram-based processing for X-ray CT” [2005.13471]. In this setting, the central object is the **Gram operator**
$$
G=A^T A=H^T H,
$$
for a parallel-beam X-ray CT system matrix defined on a **pixel basis** represented by a bivariate box spline. The normal equations for unregularized least squares are
$$
A^T A \mathbf{c}=A^T\mathbf{g},
$$
so accurate and efficient application of \(G\) is crucial for backprojection, iterative reconstruction, preconditioning, and conditioning analysis [2005.13471].

The paper’s main contribution is to derive the **Gram filter exactly for the pixel basis**, rather than under the bandlimited sinc assumptions of earlier Gram-based CT work. Because the Gram matrix is **Toeplitz-block-Toeplitz**, the large \(N^2\times N^2\) matrix can be represented by a \((2N-1)\times(2N-1)\) convolution kernel. The source text states: “we can compute \(\boldsymbol{Gx}\) exactly by applying a \((2N-1)\times(2N-1)\) filter on \(\boldsymbol{x}\).” This is the CT analogue of replacing repeated forward/backprojection with a precomputed convolutional normal operator [2005.13471].

The same framework incorporates **detector blur** by representing blur as a univariate box spline. In the unblurred model, the projected pixel basis produces a **two-direction univariate box spline**; with blur, the forward kernel becomes a **three-direction univariate box spline** and the Gram kernel becomes a **six-direction univariate box spline**. The source emphasizes that this preserves the same structural efficiency: detector blur “just increases the number of box spline directions by one and has no substantial effect on speed” [2005.13471].

A second pillar is **optimal sinogram interpolation**. Instead of sinc-based interpolation under a bandlimited assumption, the method uses Strang–Fix theory and the exact continuity of the sinogram signal to choose B-spline interpolation kernels: **degree 0** for axis-aligned projections, **degree 1** for generic projections without blur, and **degree 1 or 2** when blur increases smoothness. For degree 0 and 1, the digital correction filter is simply \(q[m]=\delta[m]\); for degree 2,
$$
q[m]=(2\sqrt{2}-3)^{|m|},
$$
with \(q[40]\approx 3.38\times 10^{-31}\), permitting negligible-error truncation [2005.13471].

The empirical results reported in the source are specific. For image sizes from \(64^2\) to \(1024^2\), the box-spline backprojection method is **at least 30% faster** than oblique interpolation, while the orthogonal method is too slow to be practical. For \(64\times 64\) reconstruction with downsampling rates between 0.5 and 2, the box-spline backprojection achieves about **5 dB higher SNR** than methods under bandlimited assumptions. In real CT examples, box-spline reconstructions show **SSIM 0.84 vs. 0.75** compared with oblique methods. Including detector blur adds only about **20% computational overhead** [2005.13471]. A plausible implication is that, within CT, an “X-GRAM” label naturally denotes a **Gram-operator–centric, filter-based reconstruction pipeline** with exact pixel-basis modeling.

## 7. Related but distinct GRAM and Gram constructs

The recommender-system model **GRAM** is not an X-GRAM variant in the language-model-memory sense, although the two names can be confused. GRAM stands for **Generative Recommendation via semantic-Aware Multi-granular late fusion** and addresses a different problem: casting recommendation as a **text-to-text generation** task over item textual IDs. Its two main innovations are **semantic-to-lexical translation**, which turns hierarchical and collaborative item relations into vocabulary-space signals, and **multi-granular late fusion**, which separately encodes coarse user prompts and fine-grained item prompts with a shared T5 encoder and fuses them at decoding time [2506.01673].

GRAM’s reported improvements are recommendation metrics rather than language-model benchmark averages. On four benchmark datasets, it outperforms eight state-of-the-art generative recommenders, with **11.5–16.0%** gains in **Recall@5** and **5.3–13.6%** gains in **NDCG@5**. The model uses **T5-small (60M parameters)** in the main experiments, and its late-fusion design reduces online encoder cost by encoding only the user prompt at inference while caching item-prompt encodings [2506.01673]. The shared lexical stem therefore reflects naming coincidence rather than methodological identity.

The convex-algebraic notion of **Gram spectrahedra** is more distant still. For a nonnegative binary form \(f\in\mathbb{R}[x,y]_{2d}\), the symmetric Gram spectrahedron \(\mathrm{Gram}(f)\) and the Hermitian Gram spectrahedron \(\mathcal{H}^+(f)\) are spectrahedra of positive semidefinite Gram tensors representing \(f\). The paper in this area studies the existence and structure of **polyhedral faces**, proving, for example, that if \(F\subseteq \mathcal{H}^+(f)\) is a polyhedral face of dimension \(k\), then
$$
\binom{k+1}{2}\le d,
$$
and establishing converse generic existence results for simplex faces in both Hermitian and symmetric settings [1910.06728]. Here “Gram” refers to SOS and PSD representation geometry, not to token-indexed memory or CT normal operators.

Taken together, these usages show that **X-GRAM** currently has one explicit meaning in large-scale language modeling and one source-endorsed informal reading in X-ray CT, while other nearby “GRAM” constructs belong to separate literatures. The technically substantive core of the explicit term is the language-model framework: **data-aware routing**, **dynamic x-gram extraction**, and **depth-aware injection** as a practical route to memory-augmented scaling under constrained FLOPs [2604.21724].

Source: https://www.emergentmind.com/topics/x-gram