PonderLM-2: Latent Thought Pretraining

• PonderLM-2 is a pretraining methodology that interleaves latent hidden states—used as intermediate 'thoughts'—between token emissions to refine predictions.
• It repurposes the last hidden state as a latent thought, enabling horizontal scaling via additional compute steps and Jacobi iterations without explicit natural-language reasoning.
• Empirical results show that PonderLM-2 achieves competitive perplexities and downstream performance compared to larger models while balancing inference cost through a fixed 2× compute budget.

PonderLM-2 is a pretraining methodology for autoregressive LLMs that inserts latent “thinking” steps before the model emits the next token. Instead of mapping one forward computation directly to one next-token prediction, it first generates an intermediate latent thought—defined as the last hidden state of the current position—and then reuses that state as input for subsequent refinement before predicting the actual token. The method is trained with the standard next-token language-modeling objective on general pretraining corpora, and it is presented as a way to scale computation per token in continuous space rather than through explicit natural-language reasoning traces (Zeng et al., 27 Sep 2025). Earlier continuous-space pondering work introduced probability-weighted token-embedding feedback but did not mention “PonderLM-2” explicitly; that earlier method is therefore best read as a precursor rather than a direct description of the later system (Zeng et al., 27 May 2025).

1. Conceptual framing and motivation

PonderLM-2 is motivated by an analogy to Chain-of-Thought. The motivating question is whether the scaling of computational steps that helps at test time in CoT can be brought into pretraining itself so that each token prediction becomes stronger. In the paper’s framing, standard language modeling applies one forward computation to produce one next-token prediction, whereas PonderLM-2 extends the generation process per token by inserting one or more latent thoughts before the actual token is produced (Zeng et al., 27 Sep 2025).

A central distinction is representational. In CoT, intermediate reasoning is expressed as explicit text tokens, constrained by vocabulary and syntax. In PonderLM-2, the intermediate object is a hidden vector in continuous space. The latent thought is not a word, a rationale, or a separately parameterized module; it is the model’s own last hidden state at the current position. This makes the extra computation internal, dense, and not directly human-readable. The method is also trained on ordinary next-token pretraining data rather than curated reasoning traces, supervised chain-of-thought annotations, or reinforcement-learning pipelines.

The paper characterizes this as a “horizontal scaling” strategy. Rather than increasing effective depth by recurrently reusing layers on the same representation, it lengthens the generation process for each token. That contrast matters because the method is positioned not only against CoT-like prompting, but also against looped-transformer and pause-token style approaches.

2. Latent-thought mechanism

Let the input sequence be

$x = (x_1, x_2, \dots, x_T),$

with token embedding lookup $\mathbf{e}(\cdot)$. A standard Transformer LM computes

$\mathbf{H}_t = \mathrm{Transformer}(\mathbf{E}_t),$

where

$$\mathbf{E}_t = [\mathbf{e}(x_1), \mathbf{e}(x_2), \dots, \mathbf{e}(x_t)].$$

The output $\mathbf{H}_t \in \mathbb{R}^{t \times d}$ contains last-layer hidden states, and the hidden state at position $t$ is

$\mathbf{h}_t = \mathbf{H}_t[t,:].$

PonderLM-2 defines the intermediate latent thought as this last hidden state $\mathbf{h}_t$. After processing token $x_t$, the model obtains $\mathbf{h}_t$ and feeds it back in as the next input embedding instead of immediately predicting $\mathbf{e}(\cdot)$0. With one latent thought per token, the model processes the current context, produces a latent hidden state, treats that hidden state as an input embedding for a new step, runs the Transformer again, and then predicts the actual next token (Zeng et al., 27 Sep 2025).

A compact rendering of the mechanism given in the paper’s description is:

$\mathbf{e}(\cdot)$1

$\mathbf{e}(\cdot)$2

$\mathbf{e}(\cdot)$3

and after $\mathbf{e}(\cdot)$4 latent-thought steps,

$\mathbf{e}(\cdot)$5

The architectural change is intentionally minimal. The base model remains a standard Transformer LM. No extra decoder head or auxiliary reasoning module is introduced; the principal change is how the sequence is constructed and consumed. During training, the model operates over an interleaved sequence of original token embeddings and latent hidden states. When a hidden state is fed back as input, it uses the same positional encoding as its original token embedding. The paper also reports that using shared position ids for a token and its latent thoughts versus sequential thought positions makes negligible difference, and it chooses shared positions to avoid requiring long-context capability.

3. Training objective and Jacobi-aligned pretraining

PonderLM-2 is pretrained with only the standard next-token prediction objective. The paper does not report auxiliary losses for latent states, reconstruction, consistency, or distillation. The supervision is simply to predict $\mathbf{e}(\cdot)$6 from the latent thought associated with $\mathbf{e}(\cdot)$7 (Zeng et al., 27 Sep 2025).

Exact autoregressive training with latent thoughts would be highly sequential, so the paper uses Jacobi iteration to make training parallelizable. It begins with an initial hidden-state estimate from the original token embeddings:

$\mathbf{e}(\cdot)$8

At iteration $\mathbf{e}(\cdot)$9, it forms the interleaved sequence

$\mathbf{H}_t = \mathrm{Transformer}(\mathbf{E}_t),$0

All latent states are then updated in parallel through a shared Transformer:

$\mathbf{H}_t = \mathrm{Transformer}(\mathbf{E}_t),$1

After $\mathbf{H}_t = \mathrm{Transformer}(\mathbf{E}_t),$2 Jacobi iterations, the language-modeling loss is computed at the positions corresponding to the hidden-state inputs:

$\mathbf{H}_t = \mathrm{Transformer}(\mathbf{E}_t),$3

where

$\mathbf{H}_t = \mathrm{Transformer}(\mathbf{E}_t),$4

A central procedural detail is that the number of Jacobi iterations $\mathbf{H}_t = \mathrm{Transformer}(\mathbf{E}_t),$5 is randomly sampled from $\mathbf{H}_t = \mathrm{Transformer}(\mathbf{E}_t),$6 for each training instance. The ablation later shows that increasing Jacobi iterations lowers loss at first, but gains saturate after about 3 iterations. The paper also distinguishes between the main one-latent-thought-per-token model used in large-scale experiments and an empirical extension in which multiple latent thoughts are chained before each actual token. The paper states that increasing the number of latent thoughts consistently improves performance, although it does not give a fully explicit standalone formalism for arbitrary chain length beyond the main interleaved setup.

4. Inference behavior and compute tradeoffs

At inference, PonderLM-2 performs extra forward computation per emitted token. With one latent thought per token, a vanilla LM emits one token after one forward pass, whereas PonderLM-2 first computes a latent hidden state, feeds it back, and then emits the token. The paper therefore treats inference FLOPs as approximately $\mathbf{H}_t = \mathrm{Transformer}(\mathbf{E}_t),$7 a vanilla model of the same parameter count (Zeng et al., 27 Sep 2025).

This leads to the paper’s core comparison protocol: equalize inference cost rather than parameter count. Under that framing, a PonderLM-2 model with one latent thought per token is compared against a vanilla model with roughly twice as many parameters. The LLaMA throughput table makes the comparison explicit: Vanilla LLaMA-1.4B is reported at $\mathbf{H}_t = \mathrm{Transformer}(\mathbf{E}_t),$8 relative inference FLOPs and 221.19 tokens/s, PonderLM-2-LLaMA-1.4B at $\mathbf{H}_t = \mathrm{Transformer}(\mathbf{E}_t),$9 and 111.55 tokens/s, and LLaMA-2.8B at $$\mathbf{E}_t = [\mathbf{e}(x_1), \mathbf{e}(x_2), \dots, \mathbf{e}(x_t)].$$0 and 110.50 tokens/s. The resulting throughput parity supports the claim that PonderLM-2-1.4B and vanilla 2.8B operate at nearly identical inference cost.

The same section also places PonderLM-2 beside other extra-compute methods at the same $$\mathbf{E}_t = [\mathbf{e}(x_1), \mathbf{e}(x_2), \dots, \mathbf{e}(x_t)].$$1 budget. Reported throughput values are 112.33 tok/s for Looped LLaMA-1.4B with 2 loops, 112.57 tok/s for Pause LLaMA-1.4B with 1 pause, 110.16 tok/s for PonderLM-LLaMA-1.4B with 1 step, and 111.55 tok/s for PonderLM-2-LLaMA-1.4B. The runtime claim is therefore not that PonderLM-2 is cheaper than other $$\mathbf{E}_t = [\mathbf{e}(x_1), \mathbf{e}(x_2), \dots, \mathbf{e}(x_t)].$$2 methods, but that it is competitive in throughput while obtaining better reported quality.

Later work identifies a structural limitation of this fixed-step design. PonderLM-3 describes PonderLM-2 as a backbone in which every token receives the same maximum number of extra pondering steps, so inference cost grows linearly with the step budget and becomes a uniform “fixed tax.” That successor paper motivates token-wise adaptive pondering partly by arguing that fixed-step allocation can waste compute on easy tokens, can cause “overthinking,” and can still under-allocate compute to harder tokens (Li et al., 2 Mar 2026).

5. Empirical results

The main large-scale experiments train PonderLM-2 with the Pythia architecture on 300B tokens from the Pile. The headline result is that PonderLM-2-Pythia-1.4B significantly surpasses vanilla Pythia-2.8B trained on the same data on both language modeling and a range of general downstream tasks, while the comparison is framed as being at identical inference cost (Zeng et al., 27 Sep 2025).

Several reported results define the method’s empirical profile. From the Pythia scaling curves, PonderLM-2-Pythia-1.26B matches Pythia-2.8B loss with 55% fewer parameters, and PonderLM-2-Pythia-1.4B reaches the baseline final performance with 62% less training data. Appendix experiments on GPT-2 and LLaMA report that PonderLM-2-GPT-834M and PonderLM-2-LLaMA-834M achieve validation loss comparable to vanilla models trained with about $$\mathbf{E}_t = [\mathbf{e}(x_1), \mathbf{e}(x_2), \dots, \mathbf{e}(x_t)].$$3 and $$\mathbf{E}_t = [\mathbf{e}(x_1), \mathbf{e}(x_2), \dots, \mathbf{e}(x_t)].$$4 the parameter-token product, respectively.

On the LLaMA comparison trained from scratch on 26B tokens, the paper reports the following perplexities in the $$\mathbf{E}_t = [\mathbf{e}(x_1), \mathbf{e}(x_2), \dots, \mathbf{e}(x_t)].$$5 inference-FLOPs category.

Comparison	Reported result
LLaMA-2.8B	Pile 8.23, Lambada OpenAI 8.93, Wikitext 18.09, Lambada Std 17.08
PonderLM-2-LLaMA-1.4B	Pile 7.89, Lambada OpenAI 7.39, Wikitext 16.99, Lambada Std 12.20

The downstream task evaluation covers nine tasks: LAMBADA OpenAI, ARC-E, LAMBADA Standard, ARC-C, WinoGrande, PIQA, HellaSwag, SciQ, and RACE. For PonderLM-2-Pythia-1.4B versus Pythia-2.8B, the reported averages are 58.5 versus 57.3 in 0-shot and 59.5 versus 57.6 in 5-shot. Relative to same-size Pythia-1.4B, PonderLM-2-Pythia-1.4B gains +4.4 average points in 0-shot and +5.4 in 5-shot. The paper also states that the smaller PonderLM-2-Pythia-410M exceeds larger baselines such as Pythia-1B and Bloom-1.7B.

The pretraining advantage transfers to instruction tuning. After Alpaca fine-tuning, the reported average MT-Bench improvement over official Pythia counterparts is 0.63 for the 410M model and 0.77 for the 1.4B model. The paper further reports that on LLaMA-1.4B trained from scratch on 26B tokens, PonderLM-2 beats Looped Transformer, Pause Token, PonderLM, and vanilla LLaMA-2.8B at the same $$\mathbf{E}_t = [\mathbf{e}(x_1), \mathbf{e}(x_2), \dots, \mathbf{e}(x_t)].$$6 inference budget, and also beats stronger baseline configurations running at $$\mathbf{E}_t = [\mathbf{e}(x_1), \mathbf{e}(x_2), \dots, \mathbf{e}(x_t)].$$7 inference FLOPs.

An ablation over the number of latent thoughts per token reports a consistent decrease in loss as more latent thoughts are chained before each actual token. This is one of the key empirical supports for the paper’s CoT analogy: one latent thought helps, and more latent thoughts help more.

6. Relation to earlier PonderLM, successor systems, and adjacent adaptive-compute work

PonderLM-2 is closely related to, but distinct from, the earlier Pondering LLM. The precursor method in “Pretraining LLMs to Ponder in Continuous Space” computes a probability-weighted token embedding

$$\mathbf{E}_t = [\mathbf{e}(x_1), \mathbf{e}(x_2), \dots, \mathbf{e}(x_t)].$$8

or, for all positions,

$$\mathbf{E}_t = [\mathbf{e}(x_1), \mathbf{e}(x_2), \dots, \mathbf{e}(x_t)].$$9

and reinserts it through a residual connection

$\mathbf{H}_t \in \mathbb{R}^{t \times d}$0

That model refines token predictions by repeatedly feeding back a soft embedding expectation derived from the prediction distribution. PonderLM-2 instead uses the last hidden state itself as the latent thought and extends the autoregressive generation process through interleaved latent states. The authors present the latter as both conceptually different and empirically stronger (Zeng et al., 27 May 2025).

Relative to explicit chain-of-thought prompting, PonderLM-2 operates in hidden-state space rather than text space, uses ordinary pretraining rather than specialized reasoning traces, and inserts latent thoughts before each token rather than usually once per question or answer. Relative to recurrent-depth methods, the paper contrasts looped transformers and similar approaches as “vertical scaling,” whereas PonderLM-2 is framed as “horizontal scaling.” Relative to pause or planning-token methods, the claimed difference is that PonderLM-2 performs the extra computation in continuous hidden space rather than discrete vocabulary space.

PonderLM-3 is presented explicitly as being “built on top of the PonderLM-2 backbone.” In that successor paper, the inherited components are latent pondering states $\mathbf{H}_t \in \mathbb{R}^{t \times d}$1 per token, inference-time recurrent refinement by feeding back the current position’s last-layer hidden state as the input embedding for later pondering steps, and efficient parallel pretraining via a Jacobi-aligned approximation. What PonderLM-3 adds is token-wise adaptive computation: a router-predicted step distribution, a tail-CDF continuation mask, differentiable attention masking via $\mathbf{H}_t \in \mathbb{R}^{t \times d}$2, weighted hidden-state integration

$\mathbf{H}_t \in \mathbb{R}^{t \times d}$3

and hard stopping at inference using

$\mathbf{H}_t \in \mathbb{R}^{t \times d}$4

with $\mathbf{H}_t \in \mathbb{R}^{t \times d}$5 (Li et al., 2 Mar 2026). In a controlled 410M downstream comparison at maximum $\mathbf{H}_t \in \mathbb{R}^{t \times d}$6, PonderLM-3 uses 8.86 G/token versus 9.84 G/token for PonderLM-2; in 5-shot evaluation it reports 46.4 versus 46.0 average, and in 0-shot 45.0 versus 45.7. The successor’s claim is therefore not universal superiority on every benchmark, but comparable downstream quality with fewer realized inference FLOPs and a stronger perplexity-compute Pareto frontier.

Other adaptive-compute lines are adjacent rather than direct descendants. PonderTTT allocates compute by deciding whether to perform a test-time parameter update in a TTT layer using a training-free threshold over reconstruction loss, and the paper explicitly distinguishes this from PonderLM-style depth allocation (Sim, 31 Dec 2025). FR-Ponder is a frozen-backbone latent-space method in which a small GRPO-trained controller decides whether to halt or apply a steering-vector update

$\mathbf{H}_t \in \mathbb{R}^{t \times d}$7

again making it conceptually related but methodologically different from the pretrained latent-thought mechanism of PonderLM-2 (He et al., 29 Sep 2025).

The principal caveats of PonderLM-2 are explicit. Inference cost rises with each added latent thought. Exact autoregressive training is infeasible, so the method relies on Jacobi iteration as an approximation. The latent thoughts are not human-readable, and the paper does not deeply analyze what they represent. It does not characterize in detail how far multi-thought scaling can continue before saturation, and it does not discuss memory or cache tradeoffs in depth. Those limitations are part of why later work on PonderLM-3 shifts attention toward token-adaptive allocation rather than uniform fixed-step pondering.