X-GRAM: Efficient Token-Injection Scaling

• X-GRAM is a dynamic token-injection framework that efficiently scales language models by leveraging frequency-aware token-indexed memory.
• It employs methods like alias mixing, normalized SwiGLU ShortConv, and depth-aware gating to optimize embedding parameter allocation.
• Empirical results show improved perplexity and data efficiency compared to baselines, highlighting its practical impact on large-scale model training.

X-GRAM is, in its explicit contemporary usage, a frequency-aware dynamic token-injection framework for efficient embedding parameter scaling in LLMs, introduced to make token-indexed memory a practical memory-centric scaling axis that decouples model capacity from FLOPs (Chen et al., 23 Apr 2026). The same string also has an informal but technically natural interpretation in parallel-beam X-ray CT, where a Gram-operator–based pixel-basis reconstruction framework can be read as “X-GRAM,” with X denoting X-ray CT and GRAM denoting the normal operator $G=A^T A$ and its associated Gram filter (Shu et al., 2020). By contrast, the recommender-system model GRAM and the convex-algebraic notion of Gram spectrahedra are distinct concepts that share the lexical stem “Gram” but not the underlying methodology (Lee et al., 2 Jun 2025, Mayer, 2019).

1. Scope and nomenclature

The label X-GRAM is not monolithic across the supplied literature. One paper uses it explicitly as the title of a language-model memory architecture, while another provides a CT reconstruction framework that the source text identifies as the kind of method that could sit behind an informally called “X-GRAM” system. Other “GRAM” usages are separate research objects rather than variants of the same framework.

Usage	Domain	Defining idea
X-GRAM (Chen et al., 23 Apr 2026)	Language modeling	Frequency-aware dynamic token injection for token-indexed memory
Informal “X-GRAM” (Shu et al., 2020)	Parallel-beam X-ray CT	Gram-operator–based Gram filtering and sinogram interpolation
GRAM (Lee et al., 2 Jun 2025)	Generative recommendation	Semantic-aware multi-granular late fusion
Gram spectrahedra (Mayer, 2019)	Convex algebraic geometry	Facial structure of symmetric and Hermitian Gram spectrahedra

For most current technical purposes, X-GRAM denotes the language-model framework introduced in “Beyond N-gram: Data-Aware X-GRAM Extraction for Efficient Embedding Parameter Scaling” (Chen et al., 23 Apr 2026). Its central claim is that lookup-based scaling is constrained less by raw parameter count than by the mismatch between table capacity, update density, and representational diversity. The framework therefore couples frequency-aware routing, alias mixing, normalized SwiGLU ShortConv, and depth-aware gating to increase the utility of token-indexed memory under fixed compute and activation budgets (Chen et al., 23 Apr 2026).

A common source of confusion is the assumption that “X-GRAM” must refer to a single standardized acronym across fields. The supplied literature supports a narrower statement: the name is explicit in large-scale language modeling, while in CT it is a plausible and source-endorsed informal descriptor for Gram-based processing for X-ray CT (Shu et al., 2020).

2. X-GRAM in efficient embedding parameter scaling

The explicit X-GRAM framework addresses the long-standing appeal and difficulty of token-indexed memory. Large lookup tables provide a compute-decoupled scaling path, because capacity can be increased without proportionally increasing Transformer depth or width. The paper argues, however, that naïvely enlarged tables are often limited by three pathologies: Zipfian under-training of the long tail, heterogeneous demand across layers, and slot collapse that yields redundant embeddings (Chen et al., 23 Apr 2026).

The parameter-efficiency problem is stated concretely. MoLE/SCONE-style designs may require reparameterized tables 2.4–7.4× larger than the backbone, and explicit n-gram tables can reach 7.6 TB for $10^9$ n-gram embeddings. Since such tables cannot remain entirely in HBM, they are offloaded to host or SSD, which increases parameter traffic during inference (Chen et al., 23 Apr 2026). The framework therefore rejects direct n-gram-table inflation and instead learns x-gram features from compressed 1-gram memory.

The statistical motivation is explicitly Zipfian. Token frequencies follow a head-heavy, tail-heavy distribution in which a small head dominates occurrences. The paper reports that average activation magnitude per token strongly correlates with frequency, and that as table size grows, the fraction of “sufficiently updated” rows drops sharply, so additional rows often remain close to initialization. A related systems-level observation is that different layers and pathways do not demand the same form of auxiliary memory: value-stream and inter-layer residual injection are substantially more effective than Q/K injection, and early layers benefit strongly, although distributing injection across layers is better than restricting it to a single layer (Chen et al., 23 Apr 2026).

The formal optimization view places X-GRAM within a constrained augmentation problem:

$$\min_{\theta,\varphi} \mathbb{E}_{(\mathbf{x},y)\sim\mathcal{D}} \left[\ell\!\left(f_{\theta,\varphi}(\mathbf{x}),y\right)\right] \quad \text{s.t.}\quad \mathcal{C}(\varphi)\le \mathcal{C}_0,\ \mathcal{A}(\varphi)\le \mathcal{A}_0,\ \mathcal{F}(\varphi)\le \mathcal{F}_0.$$

Here $\mathcal{C}$ is memory, $\mathcal{A}$ is activation footprint, and $\mathcal{F}$ is extra FLOPs. This places token-indexed memory on a separate scaling axis: X-GRAM seeks larger effective capacity by increasing table structure and retrieval diversity while holding the compute pathway relatively light (Chen et al., 23 Apr 2026).

3. Core operators and model mechanics

X-GRAM decomposes lookup-based augmentation into three operators: routing/retrieval $\mathcal{R}_\phi$, extraction/refinement $\mathcal{T}_\psi$, and injection $\mathcal{I}_s$. At layer $\ell$ and view $10^9$0,

$10^9$1

and the fused memory signal is

$10^9$2

This tri-operator factorization is important because X-GRAM does not merely add retrieved embeddings; it learns a structured sequence-to-sequence refinement over retrieved memory before injection (Chen et al., 23 Apr 2026).

Frequency-aware routing: hybrid hashing and alias mixing

The routing mechanism begins with empirical token frequency $10^9$3 and a smoothed mass

$10^9$4

A VIP head set $10^9$5 receives dedicated rows, while the remaining vocabulary is partitioned into $10^9$6 logical buckets satisfying

$10^9$7

The compression ratio is

$10^9$8

with $10^9$9 in the 50% setting (Chen et al., 23 Apr 2026).

Within each bucket, sparse regions can map tokens directly to distinct physical rows, whereas dense regions use multi-path local hashing and alias mixing. The token representation is

$$\min_{\theta,\varphi} \mathbb{E}_{(\mathbf{x},y)\sim\mathcal{D}} \left[\ell\!\left(f_{\theta,\varphi}(\mathbf{x}),y\right)\right] \quad \text{s.t.}\quad \mathcal{C}(\varphi)\le \mathcal{C}_0,\ \mathcal{A}(\varphi)\le \mathcal{A}_0,\ \mathcal{F}(\varphi)\le \mathcal{F}_0.$$0

Here $$\min_{\theta,\varphi} \mathbb{E}_{(\mathbf{x},y)\sim\mathcal{D}} \left[\ell\!\left(f_{\theta,\varphi}(\mathbf{x}),y\right)\right] \quad \text{s.t.}\quad \mathcal{C}(\varphi)\le \mathcal{C}_0,\ \mathcal{A}(\varphi)\le \mathcal{A}_0,\ \mathcal{F}(\varphi)\le \mathcal{F}_0.$$1 is a small alias set, $$\min_{\theta,\varphi} \mathbb{E}_{(\mathbf{x},y)\sim\mathcal{D}} \left[\ell\!\left(f_{\theta,\varphi}(\mathbf{x}),y\right)\right] \quad \text{s.t.}\quad \mathcal{C}(\varphi)\le \mathcal{C}_0,\ \mathcal{A}(\varphi)\le \mathcal{A}_0,\ \mathcal{F}(\varphi)\le \mathcal{F}_0.$$2 are fixed aggregation coefficients, $$\min_{\theta,\varphi} \mathbb{E}_{(\mathbf{x},y)\sim\mathcal{D}} \left[\ell\!\left(f_{\theta,\varphi}(\mathbf{x}),y\right)\right] \quad \text{s.t.}\quad \mathcal{C}(\varphi)\le \mathcal{C}_0,\ \mathcal{A}(\varphi)\le \mathcal{A}_0,\ \mathcal{F}(\varphi)\le \mathcal{F}_0.$$3 are row-wise learnable gates, and $$\min_{\theta,\varphi} \mathbb{E}_{(\mathbf{x},y)\sim\mathcal{D}} \left[\ell\!\left(f_{\theta,\varphi}(\mathbf{x}),y\right)\right] \quad \text{s.t.}\quad \mathcal{C}(\varphi)\le \mathcal{C}_0,\ \mathcal{A}(\varphi)\le \mathcal{A}_0,\ \mathcal{F}(\varphi)\le \mathcal{F}_0.$$4 is a Sigmoid. This construction serves two explicit purposes: attenuating noisy collisions in dense buckets and recycling unused rows in sparse buckets (Chen et al., 23 Apr 2026).

X-gram extraction: normalized SwiGLU ShortConv

The framework’s defining move beyond explicit n-gram tables is to transform retrieved 1-gram sequences into learned local x-gram features. Retrieved sequences are first RMS-normalized:

$$\min_{\theta,\varphi} \mathbb{E}_{(\mathbf{x},y)\sim\mathcal{D}} \left[\ell\!\left(f_{\theta,\varphi}(\mathbf{x}),y\right)\right] \quad \text{s.t.}\quad \mathcal{C}(\varphi)\le \mathcal{C}_0,\ \mathcal{A}(\varphi)\le \mathcal{A}_0,\ \mathcal{F}(\varphi)\le \mathcal{F}_0.$$5

Each view then applies a depthwise, SwiGLU-like ShortConv:

$$\min_{\theta,\varphi} \mathbb{E}_{(\mathbf{x},y)\sim\mathcal{D}} \left[\ell\!\left(f_{\theta,\varphi}(\mathbf{x}),y\right)\right] \quad \text{s.t.}\quad \mathcal{C}(\varphi)\le \mathcal{C}_0,\ \mathcal{A}(\varphi)\le \mathcal{A}_0,\ \mathcal{F}(\varphi)\le \mathcal{F}_0.$$6

followed by residual refinement,

$$\min_{\theta,\varphi} \mathbb{E}_{(\mathbf{x},y)\sim\mathcal{D}} \left[\ell\!\left(f_{\theta,\varphi}(\mathbf{x}),y\right)\right] \quad \text{s.t.}\quad \mathcal{C}(\varphi)\le \mathcal{C}_0,\ \mathcal{A}(\varphi)\le \mathcal{A}_0,\ \mathcal{F}(\varphi)\le \mathcal{F}_0.$$7

Different views use different kernel sizes $$\min_{\theta,\varphi} \mathbb{E}_{(\mathbf{x},y)\sim\mathcal{D}} \left[\ell\!\left(f_{\theta,\varphi}(\mathbf{x}),y\right)\right] \quad \text{s.t.}\quad \mathcal{C}(\varphi)\le \mathcal{C}_0,\ \mathcal{A}(\varphi)\le \mathcal{A}_0,\ \mathcal{F}(\varphi)\le \mathcal{F}_0.$$8, yielding multi-scale coverage such as $$\min_{\theta,\varphi} \mathbb{E}_{(\mathbf{x},y)\sim\mathcal{D}} \left[\ell\!\left(f_{\theta,\varphi}(\mathbf{x}),y\right)\right] \quad \text{s.t.}\quad \mathcal{C}(\varphi)\le \mathcal{C}_0,\ \mathcal{A}(\varphi)\le \mathcal{A}_0,\ \mathcal{F}(\varphi)\le \mathcal{F}_0.$$9. This design is meant to break slot symmetry, introduce contextual selectivity, and generate variable-length n-gram-like signals without explicit n-gram tables (Chen et al., 23 Apr 2026).

Injection and depth-aware gating

After refinement, views are fused as

$\mathcal{C}$0

The preferred injection site is the attention value stream:

$\mathcal{C}$1

and, under heavier compression, the signal may also be injected into the inter-layer residual:

$\mathcal{C}$2

The gate itself is

$\mathcal{C}$3

where $\mathcal{C}$4 is learnable, $\mathcal{C}$5 provides heuristic depth scaling, and $\mathcal{C}$6 ramps from 0 to 1 early in training. The source analysis reports that V and inter-layer residual injection dominate Q/K injection in the accuracy/perplexity trade-off (Chen et al., 23 Apr 2026).

4. Optimization, budgets, and empirical behavior

X-GRAM is trained jointly with the backbone under the standard causal language-model objective. The routing maps are static during training: token frequency estimation, VIP selection, bucket boundaries, and hash functions are fixed offline, whereas lookup tables $\mathcal{C}$7, alias gates $\mathcal{C}$8, ShortConv parameters $\mathcal{C}$9, gating scalars $\mathcal{A}$0, and backbone parameters $\mathcal{A}$1 are learned (Chen et al., 23 Apr 2026).

Several stabilization devices are part of the framework rather than incidental implementation details. Retrieved sequences are RMS-normalized before ShortConv; multi-view fusion uses the $\mathcal{A}$2 factor; warmup gating reduces loss spikes; and lookup-table learning rates are scaled approximately as

$\mathcal{A}$3

to compensate for reduced per-row update frequency as table size grows (Chen et al., 23 Apr 2026).

The budget accounting is also explicit. For each injection site and view, the parameter overhead is $\mathcal{A}$4 or $\mathcal{A}$5 when compressed, the activation overhead is $\mathcal{A}$6, and the extra FLOPs are approximately

$\mathcal{A}$7

For value-path injection, the usual choice is $\mathcal{A}$8, which is smaller than the full hidden width under grouped-query attention. This is one reason the paper presents value-stream injection as particularly efficient (Chen et al., 23 Apr 2026).

The utilization analysis quantifies the effect of hybrid hashing. Under frequency-agnostic routing (“No Hash”), the top 0.1% rows absorb 57.74% of hits and the $\mathcal{A}$9 movement gap is $\mathcal{F}$0. Under frequency-aware hashing, the top 0.1% rows absorb 43.11% of hits, the movement gap drops to $\mathcal{F}$1, and the coefficient of variation drops from 0.587 to 0.159. These diagnostics support the claim that X-GRAM redistributes updates more evenly under a fixed memory budget (Chen et al., 23 Apr 2026).

The ablation results assign substantial importance to the extraction and optimization mechanisms. Removing X-gram Extraction degrades perplexity from 17.702 to 18.420 and average accuracy from 49.68 to 47.80. Removing retrieved-sequence normalization yields 18.282 perplexity and 47.38 average accuracy, while removing lookup learning-rate scaling yields 18.271 perplexity and 47.62 average accuracy. A multi-scale ShortConv with $\mathcal{F}$2 achieves the best reported 17.924 perplexity and 49.27 accuracy, with only about 2% extra FLOPs over a baseline 3× ShortConv path (Chen et al., 23 Apr 2026).

5. Performance and scaling characteristics

The main experiments use decoder-only Transformer backbones at 0.73B and 1.15B scale, trained on OLMo-mix-1124 and evaluated on the lm-evaluation-harness suite including SciQ, PIQA, WinoGrande, ARC-Easy, ARC-Challenge, HellaSwag, SocialIQa, BoolQ, OpenBookQA, and MMLU (Chen et al., 23 Apr 2026). The baselines are the vanilla backbone and strong lookup/memory baselines including Retoken, MoRT, and Engram.

Scale	Baseline avg.	Best X-GRAM avg.
0.73B	44.7	49.7
1.15B	47.4	50.8

At 0.73B, the baseline average is 44.7. Under the 1× budget, X-gram-50% reaches 48.5 using a 1.17B table, whereas X-gram-100% reaches 47.5 with a 2.33B table. Under the 2× budget, X-gram-100% reaches 49.7, and under the 4× budget X-gram-50% reaches 49.1 while X-gram-100% reaches 49.5. The paper summarizes this as up to 4.4 points over the vanilla backbone and 3.2 points over strong retrieval baselines, with substantially smaller tables in the 50% configuration (Chen et al., 23 Apr 2026).

At 1.15B, the baseline average is 47.4. The reported X-GRAM variants achieve 49.5 at 1×, 50.5 at 2× for X-gram-50%, and 50.8 at 4× for both X-gram-50% and X-gram-100%. The pattern emphasized in the paper is scaling monotonicity: X-GRAM continues to improve with greater capacity, whereas several baselines plateau or regress at higher table sizes (Chen et al., 23 Apr 2026).

The training-dynamics analysis complements the endpoint metrics. Under 4× capacity, X-GRAM maintains about 0.04 lower loss than Retoken and MoRT throughout training, and it matches baseline performance using only about 57% of pretraining tokens. This suggests higher data efficiency rather than merely higher asymptotic capacity (Chen et al., 23 Apr 2026).

The site-specific analyses reinforce the architectural claims. Injecting into V and/or inter-layer residuals yields the best perplexity trade-off; Q/K injection is consistently worse; and mixed h+v configurations dominate value-only or inter-layer-only variants in the accuracy/perplexity versus parameter-budget frontier. The depth analysis shows that injecting at any single layer improves over baseline, early layers are more sensitive, and full-layer injection is much stronger than any single-layer injection (Chen et al., 23 Apr 2026).

6. Informal X-GRAM in parallel-beam X-ray CT

In the CT literature supplied here, the phrase X-GRAM is not used as the paper’s formal title, but the source text states that the work is “exactly the type of work that could sit behind a method informally called ‘X-GRAM’: Gram-based processing for X-ray CT” (Shu et al., 2020). In this setting, the central object is the Gram operator

$\mathcal{F}$3

for a parallel-beam X-ray CT system matrix defined on a pixel basis represented by a bivariate box spline. The normal equations for unregularized least squares are

$\mathcal{F}$4

so accurate and efficient application of $\mathcal{F}$5 is crucial for backprojection, iterative reconstruction, preconditioning, and conditioning analysis (Shu et al., 2020).

The paper’s main contribution is to derive the Gram filter exactly for the pixel basis, rather than under the bandlimited sinc assumptions of earlier Gram-based CT work. Because the Gram matrix is Toeplitz-block-Toeplitz, the large $\mathcal{F}$6 matrix can be represented by a $\mathcal{F}$7 convolution kernel. The source text states: “we can compute $\mathcal{F}$8 exactly by applying a $\mathcal{F}$9 filter on $\mathcal{R}_\phi$0.” This is the CT analogue of replacing repeated forward/backprojection with a precomputed convolutional normal operator (Shu et al., 2020).

The same framework incorporates detector blur by representing blur as a univariate box spline. In the unblurred model, the projected pixel basis produces a two-direction univariate box spline; with blur, the forward kernel becomes a three-direction univariate box spline and the Gram kernel becomes a six-direction univariate box spline. The source emphasizes that this preserves the same structural efficiency: detector blur “just increases the number of box spline directions by one and has no substantial effect on speed” (Shu et al., 2020).

A second pillar is optimal sinogram interpolation. Instead of sinc-based interpolation under a bandlimited assumption, the method uses Strang–Fix theory and the exact continuity of the sinogram signal to choose B-spline interpolation kernels: degree 0 for axis-aligned projections, degree 1 for generic projections without blur, and degree 1 or 2 when blur increases smoothness. For degree 0 and 1, the digital correction filter is simply $\mathcal{R}_\phi$1; for degree 2,

$\mathcal{R}_\phi$2

with $\mathcal{R}_\phi$3, permitting negligible-error truncation (Shu et al., 2020).

The empirical results reported in the source are specific. For image sizes from $\mathcal{R}_\phi$4 to $\mathcal{R}_\phi$5, the box-spline backprojection method is at least 30% faster than oblique interpolation, while the orthogonal method is too slow to be practical. For $\mathcal{R}_\phi$6 reconstruction with downsampling rates between 0.5 and 2, the box-spline backprojection achieves about 5 dB higher SNR than methods under bandlimited assumptions. In real CT examples, box-spline reconstructions show SSIM 0.84 vs. 0.75 compared with oblique methods. Including detector blur adds only about 20% computational overhead (Shu et al., 2020). A plausible implication is that, within CT, an “X-GRAM” label naturally denotes a Gram-operator–centric, filter-based reconstruction pipeline with exact pixel-basis modeling.

7. Related but distinct GRAM and Gram constructs

The recommender-system model GRAM is not an X-GRAM variant in the language-model-memory sense, although the two names can be confused. GRAM stands for Generative Recommendation via semantic-Aware Multi-granular late fusion and addresses a different problem: casting recommendation as a text-to-text generation task over item textual IDs. Its two main innovations are semantic-to-lexical translation, which turns hierarchical and collaborative item relations into vocabulary-space signals, and multi-granular late fusion, which separately encodes coarse user prompts and fine-grained item prompts with a shared T5 encoder and fuses them at decoding time (Lee et al., 2 Jun 2025).

GRAM’s reported improvements are recommendation metrics rather than language-model benchmark averages. On four benchmark datasets, it outperforms eight state-of-the-art generative recommenders, with 11.5–16.0% gains in Recall@5 and 5.3–13.6% gains in NDCG@5. The model uses T5-small (60M parameters) in the main experiments, and its late-fusion design reduces online encoder cost by encoding only the user prompt at inference while caching item-prompt encodings (Lee et al., 2 Jun 2025). The shared lexical stem therefore reflects naming coincidence rather than methodological identity.

The convex-algebraic notion of Gram spectrahedra is more distant still. For a nonnegative binary form $\mathcal{R}_\phi$7, the symmetric Gram spectrahedron $\mathcal{R}_\phi$8 and the Hermitian Gram spectrahedron $\mathcal{R}_\phi$9 are spectrahedra of positive semidefinite Gram tensors representing $\mathcal{T}_\psi$0. The paper in this area studies the existence and structure of polyhedral faces, proving, for example, that if $\mathcal{T}_\psi$1 is a polyhedral face of dimension $\mathcal{T}_\psi$2, then

$\mathcal{T}_\psi$3

and establishing converse generic existence results for simplex faces in both Hermitian and symmetric settings (Mayer, 2019). Here “Gram” refers to SOS and PSD representation geometry, not to token-indexed memory or CT normal operators.

Taken together, these usages show that X-GRAM currently has one explicit meaning in large-scale language modeling and one source-endorsed informal reading in X-ray CT, while other nearby “GRAM” constructs belong to separate literatures. The technically substantive core of the explicit term is the language-model framework: data-aware routing, dynamic x-gram extraction, and depth-aware injection as a practical route to memory-augmented scaling under constrained FLOPs (Chen et al., 23 Apr 2026).