Push-Forward Language Modeling

• Push-Forward Language Modeling is a strategy that transforms a model’s predictive distribution via auxiliary mappings on context or output to enhance generation.
• It encompasses methods like ancient-history span selection, temporal future conditioning, blockwise next-token decoding, and finite-state transduction to adjust probability mass.
• These approaches improve performance metrics such as perplexity and throughput while balancing efficiency and quality without altering the core model parameters.

Push-forward language modeling denotes a family of constructions in which a LLM’s predictive distribution is modified by an auxiliary transformation acting on context, time, noise, or output representation. In the literature considered here, the phrase covers at least four technically distinct formulations: selecting informative spans from “ancient history” outside a fixed context window and copying them into the active window (Koncel-Kedziorski et al., 2021); conditioning token prediction on temporal document history in order to model future text (Li et al., 2024); learning a deterministic mapping from independent uniform noise variables to a joint sample of the next $k$ tokens (Tang et al., 9 Jun 2026); and inducing a new autoregressive model over transformed strings by pushing a base LLM through a deterministic string-to-string map, typically represented by a finite-state transducer (Snæbjarnarson et al., 5 Mar 2026). Taken together, these works suggest that push-forward language modeling is less a single architecture than a general strategy for altering how probability mass is conditioned, transported, or read out.

1. Formal scope and recurring mathematical structure

One explicit formalization appears in transducer-based work. Let $X \sim p_X$ be a random source string over alphabet $\Sigma$, and let $f:\Sigma^* \to \Gamma^*$ be a deterministic string-to-string map. The induced target-side distribution is

$$p_Y(y)=\Pr[f(X)=y]=\sum_{\{x\in\Sigma^*:f(x)=y\}} p_X(x).$$

Here, the push-forward operation is literal: a distribution over source strings is transformed into a distribution over target strings without changing the underlying model parameters (Snæbjarnarson et al., 5 Mar 2026).

A second formalization appears in joint next-$k$-token decoding. There, a push-forward LLM defines an implicit joint conditional through a deterministic map

$$G_\theta:(x_{\le t}, z_1,\dots,z_k)\mapsto (\hat x_{t+1},\dots,\hat x_{t+k}),$$

where $z_1,\dots,z_k \sim \mathrm{Uniform}(0,1)$ independently. The map transforms simple noise into a joint sample from the target conditional $p(x_{t+1:t+k}\mid x_{\le t})$ in a single forward pass (Tang et al., 9 Jun 2026).

A third pattern rewrites the conditioning context rather than the output distribution or the sampling path. In ancient-history modeling, the original window $C_t=x_{t-M:t-1}$ is replaced by a modified window $X \sim p_X$0 that contains retrieved spans from the older history $X \sim p_X$1, after which the frozen LLM is queried in the usual way:

$X \sim p_X$2

This formulation preserves the base model call while altering the effective prompt seen at inference time (Koncel-Kedziorski et al., 2021).

These formulations are mathematically different. The shared feature is the introduction of an auxiliary map between the base LLM and the final predictive object. Depending on the construction, that object may be a new context window, a future-conditioned token distribution, a block sampler, or a transformed-output LLM.

2. Ancient-history span selection in fixed-window LLMs

Koncel-Kedziorski and Smith formulate document-level language modeling with a fixed window of size $X \sim p_X$3, where a standard model defines $X \sim p_X$4. They split the history into the current window $X \sim p_X$5 and the “ancient history” $X \sim p_X$6. The goal is to replace a small subspan of $X \sim p_X$7 with spans drawn from $X \sim p_X$8, yielding a modified context that is more predictive of future text (Koncel-Kedziorski et al., 2021).

The method breaks $X \sim p_X$9 into overlapping candidate spans of fixed length $\Sigma$0, for example $\Sigma$1. For each candidate $\Sigma$2, an auxiliary scoring function is defined as $\Sigma$3. Training uses an oracle gain for a small future segment $\Sigma$4:

$\Sigma$5

and $\Sigma$6 is learned by minimizing $\Sigma$7. At inference time, all candidates are ranked by $\Sigma$8, and the top-$\Sigma$9 spans are selected: $f:\Sigma^* \to \Gamma^*$0 (Koncel-Kedziorski et al., 2021).

Replacement is performed by overwriting the leftmost $f:\Sigma^* \to \Gamma^*$1 tokens of the current window. If $f:\Sigma^* \to \Gamma^*$2, the new context is

$f:\Sigma^* \to \Gamma^*$3

where each $f:\Sigma^* \to \Gamma^*$4 has length $f:\Sigma^* \to \Gamma^*$5. The base LM is frozen throughout; only the selector is trained. In the reported implementation, the selector was trained on $f:\Sigma^* \to \Gamma^*$6K span/prefix pairs using GPT2-large to approximate $f:\Sigma^* \to \Gamma^*$7, optimized with AdamW at $f:\Sigma^* \to \Gamma^*$8 (Koncel-Kedziorski et al., 2021).

The empirical results are reported for GPT2-large with $f:\Sigma^* \to \Gamma^*$9, $$p_Y(y)=\Pr[f(X)=y]=\sum_{\{x\in\Sigma^*:f(x)=y\}} p_X(x).$$0, $$p_Y(y)=\Pr[f(X)=y]=\sum_{\{x\in\Sigma^*:f(x)=y\}} p_X(x).$$1, and $$p_Y(y)=\Pr[f(X)=y]=\sum_{\{x\in\Sigma^*:f(x)=y\}} p_X(x).$$2. On WikiText-2 validation, baseline perplexity was $$p_Y(y)=\Pr[f(X)=y]=\sum_{\{x\in\Sigma^*:f(x)=y\}} p_X(x).$$3, oracle-span selection reached $$p_Y(y)=\Pr[f(X)=y]=\sum_{\{x\in\Sigma^*:f(x)=y\}} p_X(x).$$4, and the learned selector reached $$p_Y(y)=\Pr[f(X)=y]=\sum_{\{x\in\Sigma^*:f(x)=y\}} p_X(x).$$5, corresponding to a $$p_Y(y)=\Pr[f(X)=y]=\sum_{\{x\in\Sigma^*:f(x)=y\}} p_X(x).$$6 relative reduction. On S2ORC scientific papers, using the same GPT2-large and a selector trained on WikiText-2, the baseline was $$p_Y(y)=\Pr[f(X)=y]=\sum_{\{x\in\Sigma^*:f(x)=y\}} p_X(x).$$7 and AHLM was $$p_Y(y)=\Pr[f(X)=y]=\sum_{\{x\in\Sigma^*:f(x)=y\}} p_X(x).$$8, corresponding to a $$p_Y(y)=\Pr[f(X)=y]=\sum_{\{x\in\Sigma^*:f(x)=y\}} p_X(x).$$9 relative reduction. Smaller models and finetuning regimes yielded similar gains (Koncel-Kedziorski et al., 2021).

The principal significance of this line is that it improves perplexity “with no updates to the LM’s own parameters.” The paper also states that the technique is compatible with closed-API LMs, because only the small selector is trained. Its limitations are equally explicit: fixed span length $k$0, fixed $k$1 per time step, and per-step re-selection. Future work might adaptively choose $k$2 or $k$3, or cache selector scores across strides (Koncel-Kedziorski et al., 2021).

3. Temporal document history and future language modeling

A distinct use of push-forward methodology appears in future language modeling from temporally indexed corpora. The corpus is organized into discrete time steps $k$4, each with a set of documents $k$5. A future LLM at time $k$6 assigns probability

$k$7

so that held-out documents in year $k$8 are modeled using only the textual history $k$9. The maximum-likelihood training objective is $$G_\theta:(x_{\le t}, z_1,\dots,z_k)\mapsto (\hat x_{t+1},\dots,\hat x_{t+k}),$$0, with each $$G_\theta:(x_{\le t}, z_1,\dots,z_k)\mapsto (\hat x_{t+1},\dots,\hat x_{t+k}),$$1 a cross-entropy over the documents in year $$G_\theta:(x_{\le t}, z_1,\dots,z_k)\mapsto (\hat x_{t+1},\dots,\hat x_{t+k}),$$2 (Li et al., 2024).

All three reported architectures start from a base autoregressive transformer LLM, GPT-2, and add a temporal bias term $$G_\theta:(x_{\le t}, z_1,\dots,z_k)\mapsto (\hat x_{t+1},\dots,\hat x_{t+k}),$$3 to the softmax. In the word-frequency model, each vocabulary word $$G_\theta:(x_{\le t}, z_1,\dots,z_k)\mapsto (\hat x_{t+1},\dots,\hat x_{t+k}),$$4 has yearly raw frequency $$G_\theta:(x_{\le t}, z_1,\dots,z_k)\mapsto (\hat x_{t+1},\dots,\hat x_{t+k}),$$5. A recent-history window of size $$G_\theta:(x_{\le t}, z_1,\dots,z_k)\mapsto (\hat x_{t+1},\dots,\hat x_{t+k}),$$6, for example $$G_\theta:(x_{\le t}, z_1,\dots,z_k)\mapsto (\hat x_{t+1},\dots,\hat x_{t+k}),$$7, is fed as log-frequencies into a shared LSTM to predict a scalar bias $$G_\theta:(x_{\le t}, z_1,\dots,z_k)\mapsto (\hat x_{t+1},\dots,\hat x_{t+k}),$$8. In the temporal-contextual model, yearly contextual token embeddings are pooled into $$G_\theta:(x_{\le t}, z_1,\dots,z_k)\mapsto (\hat x_{t+1},\dots,\hat x_{t+k}),$$9, passed through an LSTM, and projected to $z_1,\dots,z_k \sim \mathrm{Uniform}(0,1)$0. In the doubly-contextualized model, the year-wise bias is further gated by the decoder state $z_1,\dots,z_k \sim \mathrm{Uniform}(0,1)$1, yielding a token- and position-specific bias $z_1,\dots,z_k \sim \mathrm{Uniform}(0,1)$2. This lets the model choose when to use the historical bias and when to rely on the in-context LM (Li et al., 2024).

Training proceeds year by year. A pretrained GPT-2 is loaded; LSTM-based bias predictors and any gating parameters are initialized; and for each year $z_1,\dots,z_k \sim \mathrm{Uniform}(0,1)$3, the system builds the relevant histories from $z_1,\dots,z_k \sim \mathrm{Uniform}(0,1)$4 and fine-tunes the entire network on $z_1,\dots,z_k \sim \mathrm{Uniform}(0,1)$5 using cross-entropy. All parameters are updated by Adam. In the reported experiments, the batch size was $z_1,\dots,z_k \sim \mathrm{Uniform}(0,1)$6, the learning rate was tuned on dev perplexity, and training ran for up to $z_1,\dots,z_k \sim \mathrm{Uniform}(0,1)$7 epochs per year. The concrete test bed was ACL Anthology abstracts filtered to English, with training on 2003–2019, dev on 2020, and test on 2021; GPT-2’s tokenizer supplied the vocabulary, and $z_1,\dots,z_k \sim \mathrm{Uniform}(0,1)$8 was tuned between $z_1,\dots,z_k \sim \mathrm{Uniform}(0,1)$9 and $p(x_{t+1:t+k}\mid x_{\le t})$0, with $p(x_{t+1:t+k}\mid x_{\le t})$1 working well (Li et al., 2024).

On held-out year 2021, three automatic metrics were reported: perplexity (PPL), content-perplexity (CPL), and content-Meteor (CM). The test results were: Baseline-all $p(x_{t+1:t+k}\mid x_{\le t})$2 PPL, $p(x_{t+1:t+k}\mid x_{\le t})$3 CM, $p(x_{t+1:t+k}\mid x_{\le t})$4 CPL; Baseline-3 $p(x_{t+1:t+k}\mid x_{\le t})$5, $p(x_{t+1:t+k}\mid x_{\le t})$6, $p(x_{t+1:t+k}\mid x_{\le t})$7; Frequency $p(x_{t+1:t+k}\mid x_{\le t})$8, $p(x_{t+1:t+k}\mid x_{\le t})$9, $C_t=x_{t-M:t-1}$0; Context $C_t=x_{t-M:t-1}$1, $C_t=x_{t-M:t-1}$2, $C_t=x_{t-M:t-1}$3; and Context$C_t=x_{t-M:t-1}$4 $C_t=x_{t-M:t-1}$5, $C_t=x_{t-M:t-1}$6, $C_t=x_{t-M:t-1}$7. All improvements over Baseline-all were significant $C_t=x_{t-M:t-1}$8. In human evaluation over $C_t=x_{t-M:t-1}$9 generated abstracts per system and six binary criteria, Baseline-all averaged $X \sim p_X$00 while Context$X \sim p_X$01 averaged $X \sim p_X$02 (Li et al., 2024).

The reported limitations are important. The model “does not produce truly novel content outside the support of history,” and factual accuracy is “not guaranteed.” The paper instead frames the contribution as a formalization of probabilistic future-text prediction from temporal document history, together with evidence that temporal biasing can improve both fluency and content relevance on a temporally split corpus (Li et al., 2024).

4. Joint next-$X \sim p_X$03-token decoding via push-forward mappings

K-Forcing defines push-forward language modeling as a response to the inefficiency of sequential autoregressive decoding. In a standard AR model, $X \sim p_X$04, and inference produces exactly one token per forward evaluation. The paper argues that this regime becomes memory-bound on modern accelerators because each step has low arithmetic intensity, repeatedly loads weights, and grows the KV cache (Tang et al., 9 Jun 2026).

The push-forward alternative is to learn a conditional mapping from context and independent uniform noise to a block of $X \sim p_X$05 future tokens. The theoretical starting point is an inverse-CDF construction. Given an AR oracle $X \sim p_X$06, with cumulative distribution function $X \sim p_X$07 and quantile function $X \sim p_X$08, one can draw $X \sim p_X$09 and define $X \sim p_X$10, $X \sim p_X$11, and so on up to $X \sim p_X$12. This closed-form map $X \sim p_X$13 exactly reproduces the AR joint $X \sim p_X$14. In practice, a student PFLM is distilled from a teacher by maximizing the likelihood of teacher-generated target blocks under sampled noise (Tang et al., 9 Jun 2026).

Training uses progressive self-forcing distillation. Stage 1 performs AR $X \sim p_X$15 PFLM($X \sim p_X$16) forward distillation. Stage 2 recursively expands the window via PFLM($X \sim p_X$17) $X \sim p_X$18 PFLM($X \sim p_X$19) self-forcing. The reported curriculum is AR $X \sim p_X$20, each stage for $X \sim p_X$21K steps with batch size $X \sim p_X$22. This is designed to avoid the train-inference mismatch associated with direct “noise inversion” of real tokens (Tang et al., 9 Jun 2026).

Both teacher and student use a standard $X \sim p_X$23-layer Transformer decoder backbone of approximately $X \sim p_X$24M parameters. Noise is injected by a small sinusoidal plus two-layer MLP noise encoder. Two conditioning architectures were studied: a standard MTP variant and a fully causal variant, with the fully causal architecture preferred. The KV-cache procedure moves in stride-$X \sim p_X$25 blocks: prefill on prompt tokens plus $X \sim p_X$26 fresh noise tokens, append only real-token KV entries, then iterate by feeding the last $X \sim p_X$27 real tokens and $X \sim p_X$28 new noise tokens (Tang et al., 9 Jun 2026).

Experiments were conducted on LM1B and OpenWebText with a $X \sim p_X$29-layer, $X \sim p_X$30, $X \sim p_X$31-head Transformer, context lengths $X \sim p_X$32 and $X \sim p_X$33, and throughput measured on H100 with FlashAttention-v2. On LM1B at batch size $X \sim p_X$34, AR achieved $X \sim p_X$35 k tok/s and Gen-PPL $X \sim p_X$36. K-Forcing at $X \sim p_X$37 achieved $X \sim p_X$38, $X \sim p_X$39, and $X \sim p_X$40 k tok/s with Gen-PPL $X \sim p_X$41, $X \sim p_X$42, and $X \sim p_X$43, and win rates against AR of $X \sim p_X$44, $X \sim p_X$45, and $X \sim p_X$46. On OpenWebText at batch size $X \sim p_X$47, AR achieved $X \sim p_X$48 k tok/s and Gen-PPL $X \sim p_X$49; K-Forcing at $X \sim p_X$50 achieved $X \sim p_X$51, $X \sim p_X$52, and $X \sim p_X$53 k tok/s with Gen-PPL $X \sim p_X$54, $X \sim p_X$55, and $X \sim p_X$56, and win rates $X \sim p_X$57, $X \sim p_X$58, and $X \sim p_X$59. The abstract summarizes the aggressive $X \sim p_X$60 regime as delivering approximately $X \sim p_X$61–$X \sim p_X$62 speedup across batch sizes with modest quality degradation relative to the AR teacher (Tang et al., 9 Jun 2026).

The production claim is central: because K-Forcing emits exactly $X \sim p_X$63 tokens per forward pass in lockstep, it remains compatible with standard AR KV-cache batching schedulers and avoids the “ragged tensor” issue of speculative decoding. The prediction window $X \sim p_X$64 is tunable at inference time for any $X \sim p_X$65, so speed and quality can be traded without retraining (Tang et al., 9 Jun 2026).

5. Push-forward distributions induced by finite-state transduction

“Transducing LLMs” develops the most explicit probability-theoretic account of push-forward language modeling. The setting assumes a base LM $X \sim p_X$66 over source strings and a deterministic string-to-string map $X \sim p_X$67 represented by a finite-state transducer $X \sim p_X$68. Because $X \sim p_X$69 is deterministic, it realizes a partial function from source strings to target strings. The induced LM over target strings is obtained by marginalizing over all source strings that map to a given target string (Snæbjarnarson et al., 5 Mar 2026).

Exact inference requires summing $X \sim p_X$70 over all $X \sim p_X$71 whose output under $X \sim p_X$72 has a specified target prefix $X \sim p_X$73. The method constructs a “pre-cover” acceptor by composing $X \sim p_X$74 with the copy-transducer for prefix $X \sim p_X$75, then determinizing and trimming to obtain a DFA $X \sim p_X$76. A decomposition procedure, $X \sim p_X$77, returns a finite prefix-free quotient set $X \sim p_X$78 and a finite remainder set $X \sim p_X$79, such that the union of their cylinders covers the preimage of $X \sim p_X$80 (Snæbjarnarson et al., 5 Mar 2026).

The exact algorithm performs a BFS over source-string prefixes and classifies each prefix $X \sim p_X$81 into one of three cases. A “Cylinder” prefix is one for which every extension maps to a string with prefix $X \sim p_X$82, in which case $X \sim p_X$83. A “Member” prefix maps to a $X \sim p_X$84-prefix itself but not all of its extensions do, in which case $X \sim p_X$85. A “Live” prefix is one for which some extension may still map to a $X \sim p_X$86-prefix, in which case the search continues by enqueuing $X \sim p_X$87 for each $X \sim p_X$88. Under strict-prefix-monotonicity, the reported time complexity per target prefix $X \sim p_X$89 is $X \sim p_X$90, where $X \sim p_X$91 is the cost of one base-LM call plus a small automaton BFS, and space is $X \sim p_X$92. Termination and correctness are guaranteed under the paper’s safety conditions and the absence of $X \sim p_X$93-output cycles (Snæbjarnarson et al., 5 Mar 2026).

Because $X \sim p_X$94 can be large, the paper introduces a pruning approximation: retain only the highest-mass prefixes whose cumulative $X \sim p_X$95 mass is at least $X \sim p_X$96, or up to a capacity $X \sim p_X$97. The resulting prefix probability is a lower bound on the true $X \sim p_X$98, with missing mass bounded by $X \sim p_X$99, where $\Sigma$00 is the total precover mass. A backtracking heuristic halves $\Sigma$01 when approximate decomposition fails to cover an observed symbol (Snæbjarnarson et al., 5 Mar 2026).

Conditioning on transformed outputs is then immediate. Once the exact preimage set $\Sigma$02 has been enumerated, one computes

$\Sigma$03

This produces fully functional autoregressive LMs over transformed units without retraining (Snæbjarnarson et al., 5 Mar 2026).

The experiments span three domains. For tokens $\Sigma$04 bytes, the paper uses $\Sigma$05 and base LMs including GPT-2 Large, LLaMA-1B, LLaMA-8B, and Phi-4; $\Sigma$06 yields JSD approximately $\Sigma$07 at roughly $\Sigma$08–$\Sigma$09 bytes/s, and $\Sigma$10 drives JSD below $\Sigma$11. For tokens $\Sigma$12 words using PTB segmentation, an FST chain of approximately $\Sigma$13 states yields JSD approximately $\Sigma$14 at $\Sigma$15 and about $\Sigma$16–$\Sigma$17 bytes/s, while $\Sigma$18 gives JSD approximately $\Sigma$19 at about $\Sigma$20–$\Sigma$21 bytes/s. For DNA $\Sigma$22 amino acids, $\Sigma$23 with a GPT-2 small base model reaches JSD approximately $\Sigma$24 at $\Sigma$25 and capacity $\Sigma$26, at approximately $\Sigma$27 AA/s (Snæbjarnarson et al., 5 Mar 2026).

6. Comparative interpretation, limitations, and common misconceptions

A first misconception is that push-forward language modeling names a single algorithm. The surveyed literature does not support that reading. One line changes the effective context while leaving the base LM frozen (Koncel-Kedziorski et al., 2021). Another adds temporal biases and fine-tunes the full network on future-year prediction (Li et al., 2024). A third distills a new student capable of blockwise generation from uniform noise (Tang et al., 9 Jun 2026). A fourth keeps model parameters fixed and changes the output space by exact or approximate marginalization through an FST (Snæbjarnarson et al., 5 Mar 2026).

A second misconception is that push-forward methods are uniformly inference-only. That is true for ancient-history span replacement, where “no LM parameters are altered,” and for transducer-induced output adaptation, which is explicitly performed “without retraining” (Koncel-Kedziorski et al., 2021, Snæbjarnarson et al., 5 Mar 2026). It is not true for temporal future language modeling, which fine-tunes GPT-2 together with LSTM-based bias predictors, nor for K-Forcing, which requires multi-stage distillation of a student PFLM (Li et al., 2024, Tang et al., 9 Jun 2026).

A third misconception is that push-forward necessarily improves both quality and efficiency simultaneously. The evidence is mixed by design. Ancient-history modeling reports $\Sigma$28 and $\Sigma$29 relative perplexity reductions on WikiText-2 and S2ORC, respectively, but incurs non-trivial overhead because scoring all candidate spans dominates computation (Koncel-Kedziorski et al., 2021). K-Forcing produces substantial throughput gains, especially at $\Sigma$30, but the paper characterizes the quality change as “modest degradation relative to its AR teacher” (Tang et al., 9 Jun 2026). Future language modeling improves PPL, CPL, CM, and human judgments on the reported temporal split, yet explicitly does not guarantee factual accuracy or genuinely novel content beyond the support of history (Li et al., 2024). Transducer-based push-forward LMs offer exactness in principle, but practical inference may require pruning because the prefix sets can be combinatorially large (Snæbjarnarson et al., 5 Mar 2026).

The broader significance is methodological. These papers collectively show that useful LM behavior can be obtained by transforming context windows, injecting temporal statistics, amortizing multi-token sampling through a learned map, or marginalizing through structured symbolic transformations. This suggests a broader research program in which the LLM itself need not be the only locus of modeling power: auxiliary push-forward mechanisms can alter what counts as context, what counts as an output unit, and how generation is staged, while preserving compatibility with existing pretrained models or with standard autoregressive serving infrastructure.