---
title: Push-Forward Language Modeling
url: https://www.emergentmind.com/topics/push-forward-language-modeling
type: topic
---

# Push-Forward Language Modeling

Push-forward language modeling denotes a family of constructions in which a language model’s predictive distribution is modified by an auxiliary transformation acting on context, time, noise, or output representation. In the literature considered here, the phrase covers at least four technically distinct formulations: selecting informative spans from “ancient history” outside a fixed context window and copying them into the active window [2104.08742]; conditioning token prediction on temporal document history in order to model future text [2404.10297]; learning a deterministic mapping from independent uniform noise variables to a joint sample of the next \(k\) tokens [2606.10820]; and inducing a new autoregressive model over transformed strings by pushing a base language model through a deterministic string-to-string map, typically represented by a finite-state transducer [2603.05193]. Taken together, these works suggest that push-forward language modeling is less a single architecture than a general strategy for altering how probability mass is conditioned, transported, or read out.

## 1. Formal scope and recurring mathematical structure

One explicit formalization appears in transducer-based work. Let \(X \sim p_X\) be a random source string over alphabet \(\Sigma\), and let \(f:\Sigma^* \to \Gamma^*\) be a deterministic string-to-string map. The induced target-side distribution is

$$
p_Y(y)=\Pr[f(X)=y]=\sum_{\{x\in\Sigma^*:f(x)=y\}} p_X(x).
$$

Here, the push-forward operation is literal: a distribution over source strings is transformed into a distribution over target strings without changing the underlying model parameters [2603.05193].

A second formalization appears in joint next-\(k\)-token decoding. There, a push-forward language model defines an implicit joint conditional through a deterministic map

$$
G_\theta:(x_{\le t}, z_1,\dots,z_k)\mapsto (\hat x_{t+1},\dots,\hat x_{t+k}),
$$

where \(z_1,\dots,z_k \sim \mathrm{Uniform}(0,1)\) independently. The map transforms simple noise into a joint sample from the target conditional \(p(x_{t+1:t+k}\mid x_{\le t})\) in a single forward pass [2606.10820].

A third pattern rewrites the conditioning context rather than the output distribution or the sampling path. In ancient-history modeling, the original window \(C_t=x_{t-M:t-1}\) is replaced by a modified window \(C'_t\) that contains retrieved spans from the older history \(H_t=x_{0:t-M-1}\), after which the frozen language model is queried in the usual way:

$$
P(x_t \mid C'_t) \equiv P_{LM}(x_t \mid x'_{t-M:t-1}).
$$

This formulation preserves the base model call while altering the effective prompt seen at inference time [2104.08742].

These formulations are mathematically different. The shared feature is the introduction of an auxiliary map between the base language model and the final predictive object. Depending on the construction, that object may be a new context window, a future-conditioned token distribution, a block sampler, or a transformed-output language model.

## 2. Ancient-history span selection in fixed-window language models

Koncel-Kedziorski and Smith formulate document-level language modeling with a fixed window of size \(M\), where a standard model defines \(P_{LM}(x_t \mid x_{t-M:t-1})\). They split the history into the current window \(C_t=x_{t-M:t-1}\) and the “ancient history” \(H_t=x_{0:t-M-1}\). The goal is to replace a small subspan of \(C_t\) with spans drawn from \(H_t\), yielding a modified context that is more predictive of future text [2104.08742].

The method breaks \(H_t\) into overlapping candidate spans of fixed length \(\ell\), for example \(\ell=64\). For each candidate \(x_{i:j}\subset H_t\), an auxiliary scoring function is defined as
\(f(x_{i:j};\theta)=\mathrm{score}_\theta(x_{i:j},x_{t-M:t-1})\).
Training uses an oracle gain for a small future segment \(x_{t:t+s}\):

$$
G_{i:j}=\ln P_{LM}(x_{t:t+s}\mid x_{i:j},x_{t-M:t-1})-\ln P_{LM}(x_{t:t+s}\mid x_{t-M:t-1}),
$$

and \(\theta\) is learned by minimizing \(E[(f(x_{i:j};\theta)-G_{i:j})^2]\). At inference time, all candidates are ranked by \(f\), and the top-\(K\) spans are selected:
\(S=\arg\mathrm{top}_K\{f(x_{i:j};\theta):x_{i:j}\subset H_t\}\) [2104.08742].

Replacement is performed by overwriting the leftmost \(J \le M\) tokens of the current window. If \(K\ell=J\), the new context is

$$
C'_t=[s_1 \,\|\, s_2 \,\|\, \cdots \,\|\, s_K \,\|\, x_{t-M+J:t-1}],
$$

where each \(s_k\) has length \(\ell\). The base LM is frozen throughout; only the selector is trained. In the reported implementation, the selector was trained on \(\sim 728\)K span/prefix pairs using GPT2-large to approximate \(G_{i:j}\), optimized with AdamW at \(1\mathrm{e}{-5}\) [2104.08742].

The empirical results are reported for GPT2-large with \(M=1024\), \(J=512\), \(\ell=64\), and \(K=8\). On WikiText-2 validation, baseline perplexity was \(16.56\), oracle-span selection reached \(16.43\), and the learned selector reached \(15.64\), corresponding to a \(7.2\%\) relative reduction. On S2ORC scientific papers, using the same GPT2-large and a selector trained on WikiText-2, the baseline was \(14.12\) and AHLM was \(12.62\), corresponding to a \(12.5\%\) relative reduction. Smaller models and finetuning regimes yielded similar gains [2104.08742].

The principal significance of this line is that it improves perplexity “with no updates to the LM’s own parameters.” The paper also states that the technique is compatible with closed-API LMs, because only the small selector is trained. Its limitations are equally explicit: fixed span length \(\ell\), fixed \(J\) per time step, and per-step re-selection. Future work might adaptively choose \(J\) or \(\ell\), or cache selector scores across strides [2104.08742].

## 3. Temporal document history and future language modeling

A distinct use of push-forward methodology appears in future language modeling from temporally indexed corpora. The corpus is organized into discrete time steps \(i=1,2,\dots,T\), each with a set of documents \(D_i\). A future language model at time \(i\) assigns probability

$$
P\bigl(x_{ijk}\mid x_{ij1},\dots,x_{ij(k-1)},D_1,\dots,D_{i-1}\bigr),
$$

so that held-out documents in year \(i\) are modeled using only the textual history \(D_1,\dots,D_{i-1}\). The maximum-likelihood training objective is \(L(\theta)=\sum_{i=2}^T L_i(\theta)\), with each \(L_i\) a cross-entropy over the documents in year \(i\) [2404.10297].

All three reported architectures start from a base autoregressive transformer language model, GPT-2, and add a temporal bias term \(B\) to the softmax. In the word-frequency model, each vocabulary word \(w\) has yearly raw frequency \(f_{iw}\). A recent-history window of size \(m\), for example \(m=3\), is fed as log-frequencies into a shared LSTM to predict a scalar bias \(B_{iw}\). In the temporal-contextual model, yearly contextual token embeddings are pooled into \(V_{jw}\), passed through an LSTM, and projected to \(B_{iw}\). In the doubly-contextualized model, the year-wise bias is further gated by the decoder state \(H_k\), yielding a token- and position-specific bias \(B_{ik,w}\). This lets the model choose when to use the historical bias and when to rely on the in-context LM [2404.10297].

Training proceeds year by year. A pretrained GPT-2 is loaded; LSTM-based bias predictors and any gating parameters are initialized; and for each year \(i\), the system builds the relevant histories from \(D_{i-m},\dots,D_{i-1}\) and fine-tunes the entire network on \(D_i\) using cross-entropy. All parameters are updated by Adam. In the reported experiments, the batch size was \(2\), the learning rate was tuned on dev perplexity, and training ran for up to \(10\) epochs per year. The concrete test bed was ACL Anthology abstracts filtered to English, with training on 2003–2019, dev on 2020, and test on 2021; GPT-2’s tokenizer supplied the vocabulary, and \(m\) was tuned between \(1\) and \(10\), with \(m=3\) working well [2404.10297].

On held-out year 2021, three automatic metrics were reported: perplexity (PPL), content-perplexity (CPL), and content-Meteor (CM). The test results were: Baseline-all \(22.76\) PPL, \(16.22\) CM, \(102.03\) CPL; Baseline-3 \(21.06\), \(19.01\), \(92.06\); Frequency \(20.20\), \(19.98\), \(87.68\); Context \(23.21\), \(18.20\), \(102.11\); and Context\(^2\) \(19.81\), \(20.12\), \(82.43\). All improvements over Baseline-all were significant \((p<.001)\). In human evaluation over \(100\) generated abstracts per system and six binary criteria, Baseline-all averaged \(46\%\) while Context\(^2\) averaged \(63\%\) [2404.10297].

The reported limitations are important. The model “does not produce truly novel content outside the support of history,” and factual accuracy is “not guaranteed.” The paper instead frames the contribution as a formalization of probabilistic future-text prediction from temporal document history, together with evidence that temporal biasing can improve both fluency and content relevance on a temporally split corpus [2404.10297].

## 4. Joint next-\(k\)-token decoding via push-forward mappings

K-Forcing defines push-forward language modeling as a response to the inefficiency of sequential autoregressive decoding. In a standard AR model,
\(p(x_1\dots x_T)=\prod_t p(x_t\mid x_{<t})\),
and inference produces exactly one token per forward evaluation. The paper argues that this regime becomes memory-bound on modern accelerators because each step has low arithmetic intensity, repeatedly loads weights, and grows the KV cache [2606.10820].

The push-forward alternative is to learn a conditional mapping from context and independent uniform noise to a block of \(k\) future tokens. The theoretical starting point is an inverse-CDF construction. Given an AR oracle \(q_{AR}(\cdot\mid x_{\le t})\), with cumulative distribution function \(F_{AR}\) and quantile function \(F_{AR}^{-1}\), one can draw \(z_1,\dots,z_k \sim \mathrm{Uniform}(0,1)\) and define
\(\hat x_{t+1}=F_{AR}^{-1}(z_1\mid x_{\le t})\),
\(\hat x_{t+2}=F_{AR}^{-1}(z_2\mid x_{\le t},\hat x_{t+1})\),
and so on up to \(\hat x_{t+k}\). This closed-form map \(G^\star\) exactly reproduces the AR joint \(p(x_{t+1:t+k}\mid x_{\le t})\). In practice, a student PFLM is distilled from a teacher by maximizing the likelihood of teacher-generated target blocks under sampled noise [2606.10820].

Training uses progressive self-forcing distillation. Stage 1 performs AR \(\rightarrow\) PFLM(\(k=1\)) forward distillation. Stage 2 recursively expands the window via PFLM(\(k\)) \(\rightarrow\) PFLM(\(2k\)) self-forcing. The reported curriculum is AR \(\rightarrow k=1 \rightarrow k=2 \rightarrow k=4\), each stage for \(500\)K steps with batch size \(512\). This is designed to avoid the train-inference mismatch associated with direct “noise inversion” of real tokens [2606.10820].

Both teacher and student use a standard \(12\)-layer Transformer decoder backbone of approximately \(100\)M parameters. Noise is injected by a small sinusoidal plus two-layer MLP noise encoder. Two conditioning architectures were studied: a standard MTP variant and a fully causal variant, with the fully causal architecture preferred. The KV-cache procedure moves in stride-\(k\) blocks: prefill on prompt tokens plus \(k\) fresh noise tokens, append only real-token KV entries, then iterate by feeding the last \(k\) real tokens and \(k\) new noise tokens [2606.10820].

Experiments were conducted on LM1B and OpenWebText with a \(12\)-layer, \(d=768\), \(12\)-head Transformer, context lengths \(128\) and \(1024\), and throughput measured on H100 with FlashAttention-v2. On LM1B at batch size \(16\), AR achieved \(2.03\) k tok/s and Gen-PPL \(104.8\). K-Forcing at \(k=2,3,4\) achieved \(3.48\), \(4.99\), and \(6.77\) k tok/s with Gen-PPL \(107.2\), \(117.1\), and \(127.6\), and win rates against AR of \(50.2\%\), \(45.1\%\), and \(42.9\%\). On OpenWebText at batch size \(16\), AR achieved \(1.99\) k tok/s and Gen-PPL \(42.64\); K-Forcing at \(k=2,3,4\) achieved \(3.36\), \(5.03\), and \(6.91\) k tok/s with Gen-PPL \(32.82\), \(29.67\), and \(24.97\), and win rates \(46.9\%\), \(42.8\%\), and \(39.4\%\). The abstract summarizes the aggressive \(k=4\) regime as delivering approximately \(2.4\)–\(3.5\times\) speedup across batch sizes with modest quality degradation relative to the AR teacher [2606.10820].

The production claim is central: because K-Forcing emits exactly \(k\) tokens per forward pass in lockstep, it remains compatible with standard AR KV-cache batching schedulers and avoids the “ragged tensor” issue of speculative decoding. The prediction window \(k\) is tunable at inference time for any \(k \le k_{train}\), so speed and quality can be traded without retraining [2606.10820].

## 5. Push-forward distributions induced by finite-state transduction

“Transducing Language Models” develops the most explicit probability-theoretic account of push-forward language modeling. The setting assumes a base LM \(p_X\) over source strings and a deterministic string-to-string map \(f\) represented by a finite-state transducer \(\mathbb{T}=(S,\Sigma,\Gamma,I,F,T)\). Because \(\mathbb{T}\) is deterministic, it realizes a partial function from source strings to target strings. The induced LM over target strings is obtained by marginalizing over all source strings that map to a given target string [2603.05193].

Exact inference requires summing \(p_X(x)\) over all \(x\) whose output under \(f\) has a specified target prefix \(y\). The method constructs a “pre-cover” acceptor by composing \(\mathbb{T}\) with the copy-transducer for prefix \(y\), then determinizing and trimming to obtain a DFA \(\mathbb{D}_y\). A decomposition procedure, \(\mathrm{decompose}(y)\), returns a finite prefix-free quotient set \(Q(y)\) and a finite remainder set \(R(y)\), such that the union of their cylinders covers the preimage of \(y\) [2603.05193].

The exact algorithm performs a BFS over source-string prefixes and classifies each prefix \(s\) into one of three cases. A “Cylinder” prefix is one for which every extension maps to a string with prefix \(y\), in which case \(s \in Q(y)\). A “Member” prefix maps to a \(y\)-prefix itself but not all of its extensions do, in which case \(s \in R(y)\). A “Live” prefix is one for which some extension may still map to a \(y\)-prefix, in which case the search continues by enqueuing \(s \cdot \sigma\) for each \(\sigma \in \Sigma\). Under strict-prefix-monotonicity, the reported time complexity per target prefix \(y\) is \(O(|Q_y|\cdot |\Sigma|\cdot C)\), where \(C\) is the cost of one base-LM call plus a small automaton BFS, and space is \(O(|Q_y|)\). Termination and correctness are guaranteed under the paper’s safety conditions and the absence of \(\epsilon\)-output cycles [2603.05193].

Because \(Q(y)\cup R(y)\) can be large, the paper introduces a pruning approximation: retain only the highest-mass prefixes whose cumulative \(p_X\) mass is at least \(1-\tau\), or up to a capacity \(\mathrm{max}\). The resulting prefix probability is a lower bound on the true \(p_Y(y)\), with missing mass bounded by \(\tau \cdot Z\), where \(Z\) is the total precover mass. A backtracking heuristic halves \(\tau\) when approximate decomposition fails to cover an observed symbol [2603.05193].

Conditioning on transformed outputs is then immediate. Once the exact preimage set \(\mathrm{Pre}(y)=\{x\mid f(x)=y\}\) has been enumerated, one computes

$$
p_X(x\mid f(x)=y)=\frac{p_X(x)}{p_Y(y)} \qquad \text{for } x\in \mathrm{Pre}(y).
$$

This produces fully functional autoregressive LMs over transformed units without retraining [2603.05193].

The experiments span three domains. For tokens \(\rightarrow\) bytes, the paper uses \(\mathbb{T}_{token2byte}\) and base LMs including GPT-2 Large, LLaMA-1B, LLaMA-8B, and Phi-4; \(\tau=1\mathrm{e}{-3}\) yields JSD approximately \(1\mathrm{e}{-3}\) at roughly \(10\)–\(70\) bytes/s, and \(\tau=1\mathrm{e}{-4}\) drives JSD below \(1\mathrm{e}{-4}\). For tokens \(\rightarrow\) words using PTB segmentation, an FST chain of approximately \(500\) states yields JSD approximately \(1\mathrm{e}{-5}\) at \(\tau=3\mathrm{e}{-4}\) and about \(1\)–\(5\) bytes/s, while \(\tau\approx 1\mathrm{e}{-3}\) gives JSD approximately \(1\mathrm{e}{-3}\) at about \(10\)–\(15\) bytes/s. For DNA \(\rightarrow\) amino acids, \(\mathbb{T}_{dna2aa}\) with a GPT-2 small base model reaches JSD approximately \(1\mathrm{e}{-5}\) at \(\tau \approx 3\mathrm{e}{-5}\) and capacity \(\mathrm{max}=5000\), at approximately \(1\) AA/s [2603.05193].

## 6. Comparative interpretation, limitations, and common misconceptions

A first misconception is that push-forward language modeling names a single algorithm. The surveyed literature does not support that reading. One line changes the effective context while leaving the base LM frozen [2104.08742]. Another adds temporal biases and fine-tunes the full network on future-year prediction [2404.10297]. A third distills a new student capable of blockwise generation from uniform noise [2606.10820]. A fourth keeps model parameters fixed and changes the output space by exact or approximate marginalization through an FST [2603.05193].

A second misconception is that push-forward methods are uniformly inference-only. That is true for ancient-history span replacement, where “no LM parameters are altered,” and for transducer-induced output adaptation, which is explicitly performed “without retraining” [2104.08742] [2603.05193]. It is not true for temporal future language modeling, which fine-tunes GPT-2 together with LSTM-based bias predictors, nor for K-Forcing, which requires multi-stage distillation of a student PFLM [2404.10297] [2606.10820].

A third misconception is that push-forward necessarily improves both quality and efficiency simultaneously. The evidence is mixed by design. Ancient-history modeling reports \(7.2\%\) and \(12.5\%\) relative perplexity reductions on WikiText-2 and S2ORC, respectively, but incurs non-trivial overhead because scoring all candidate spans dominates computation [2104.08742]. K-Forcing produces substantial throughput gains, especially at \(k=4\), but the paper characterizes the quality change as “modest degradation relative to its AR teacher” [2606.10820]. Future language modeling improves PPL, CPL, CM, and human judgments on the reported temporal split, yet explicitly does not guarantee factual accuracy or genuinely novel content beyond the support of history [2404.10297]. Transducer-based push-forward LMs offer exactness in principle, but practical inference may require pruning because the prefix sets can be combinatorially large [2603.05193].

The broader significance is methodological. These papers collectively show that useful LM behavior can be obtained by transforming context windows, injecting temporal statistics, amortizing multi-token sampling through a learned map, or marginalizing through structured symbolic transformations. This suggests a broader research program in which the language model itself need not be the only locus of modeling power: auxiliary push-forward mechanisms can alter what counts as context, what counts as an output unit, and how generation is staged, while preserving compatibility with existing pretrained models or with standard autoregressive serving infrastructure.

Source: https://www.emergentmind.com/topics/push-forward-language-modeling