DeLTa: Logit Trajectory Decoding

• DeLTa is a decoding strategy that treats the evolution of token logits as a trajectory, using linear regression to extrapolate improved next-token predictions.
• It operates at decoding time without modifying model architecture, harnessing intermediate Transformer layers to refine logit probabilities.
• Empirical evaluations on TruthfulQA, StrategyQA, and GSM8K show gains up to 8.1%, demonstrating enhanced factuality and multi-step reasoning.

DeLTa, short for Decoding by Logit Trajectory-based approach, is a decoding strategy for LLMs that aims to improve factual accuracy and inferential reasoning without requiring any modifications to the architecture or pre-trained parameters of the underlying model. Its central idea is to treat the evolution of token logits from lower to higher Transformer layers as a trajectory, fit that trajectory with linear regression, and use the extrapolated logit at a virtual higher layer for next-token selection. Reported evaluations on TruthfulQA, StrategyQA, and GSM8K show gains of up to 4.9%, 8.1%, and 7.3%, respectively, over the baseline (He et al., 4 Mar 2025).

1. Definition and objective

DeLTa was introduced to address a recurrent limitation of LLMs: generated content may be factually incorrect or may exhibit deficiencies in logical reasoning. In contrast to approaches that rely on retraining, architectural modification, or additional data, DeLTa operates entirely at decoding time. The method adjusts next-token probabilities by analyzing the layerwise trajectory of logits and extrapolating that trajectory with a linear model (He et al., 4 Mar 2025).

The motivating empirical premise is that the logits for correct tokens tend to increase through higher Transformer layers. DeLTa therefore treats the sequence of logits across selected upper layers as a “time series” and predicts a refined logit beyond the final layer. This design places DeLTa in the class of post hoc decoding interventions rather than parameter-updating or architecture-changing methods. A plausible implication is that DeLTa uses information already present in the forward pass but not fully exploited by standard final-layer decoding.

2. Layerwise probability representation

For a Transformer with $N$ layers and a decoding position $t$, DeLTa defines token probabilities at arbitrary intermediate layers through the logit lens formulation. If $x_t$ is the token at position $t$, $x_{<t}$ are the preceding tokens, and $\mathcal{X}$ is the vocabulary, then for layer $\ell$,

$$P_{\ell}(x_t \mid x_{<t}) = \mathrm{softmax}\left(\mathbf{logit}^{(\ell)}\right)_{x_t},\quad x_t \in \mathcal{X}.$$

To construct the regression problem, DeLTa selects a contiguous set of layers from an intermediate layer $N_{mid}$ to the final layer $N$:

$t$0

and collects the corresponding logits

$t$1

These quantities are defined per decoding step and per candidate token, so the regression operates directly on token-level logit evolution rather than on hidden states or attention scores (He et al., 4 Mar 2025).

This formulation makes DeLTa structurally simple: it reuses existing layer outputs and requires no auxiliary model. The method is therefore compatible with standard Transformer inference pipelines so long as intermediate-layer logits can be exposed.

3. Logit trajectory prediction

The core predictive step fits a linear regression for each candidate token across the selected layers and extrapolates to a virtual layer $t$2, which may be fractional and may exceed $t$3. The extrapolated logit is written as

$t$4

with regression coefficients

$t$5

$t$6

The final probability distribution is then obtained by applying softmax to the extrapolated logits, but only over a filtered candidate set $t$7:

$t$8

The candidate set is specified as

$t$9

with the summary reporting that typically, $x_t$0. Tokens outside this set are assigned zero probability (He et al., 4 Mar 2025).

In operational terms, DeLTa combines two mechanisms: extrapolation of upper-layer logit dynamics and restriction of the softmax support. The paper’s analysis distinguishes these two contributions rather than treating the filter as the sole source of improvement.

4. Decoding procedure and implementation profile

During generation, DeLTa repeats the same sequence at each decoding step: it extracts token-level logits across the selected layers, performs linear regression per token, extrapolates logits at $x_t$1, applies the candidate filter, computes the softmax on the filtered set, and then samples or chooses the next token (He et al., 4 Mar 2025).

A defining property of the method is that no forward passes through extra model weights or retraining is required. The intervention is therefore a decoding-time wrapper around an existing post-trained model. The reported hyperparameter setting for the regression range $x_t$2 is described as dataset- and model-agnostic, based on ablation and validation studies. The reported computational overhead is moderate and model-dependent; for Qwen2.5-7B, the latency increase is reported as 1.4x.

This implementation profile makes DeLTa a plug-in decoding method rather than a fine-tuning method. A plausible implication is that its deployment cost is governed primarily by access to intermediate logits and the per-step regression calculation, not by model re-optimization.

5. Empirical evaluation

DeLTa was evaluated on Qwen2.5-7B, Mistral-7B, and Llama-3.1-8B, with comparisons against standard decoding, a filtering-only baseline, and DoLa in early-layer and late-layer variants. The benchmark suite includes TruthfulQA for factuality, StrategyQA and GSM8K for reasoning, and TriviaQA and Natural Questions for closed-question answering (He et al., 4 Mar 2025).

The headline improvements reported in the paper are summarized below.

Benchmark	Reported maximum gain over baseline	Example
TruthfulQA	up to 4.9%	Llama-3.1-8B: 44.0 $x_t$3 48.9
StrategyQA	up to 8.1%	Llama-3.1-8B: 64.0 $x_t$4 72.1
GSM8K	up to 7.3%	Llama-3.1-8B: 42.8 $x_t$5 50.1

The Qwen2.5-7B results reported in the summary illustrate the same pattern. On TruthfulQA (\%True*Info), the baseline is 64.1, filtering-only gives 60.7, DoLa early-layer gives 62.9, DoLa late-layer gives 55.0, and DeLTa gives 65.4. On StrategyQA, the baseline is 76.9 and DeLTa reaches 81.2. On GSM8K, the baseline is 78.7 and DeLTa reaches 81.6. On TriviaQA, both filtering-only and DeLTa reach 44.1, compared with a 39.1 baseline; on Natural Questions, the baseline is 11.5 and DeLTa reaches 13.0 (He et al., 4 Mar 2025).

The paper further reports that across all tasks, DeLTa consistently outperforms the baseline and other decoding strategies, with especially prominent gains on reasoning-intensive tasks such as GSM8K and StrategyQA. A qualitative example on a GSM8K math word problem is described as showing that DeLTa corrected multi-step arithmetic errors present in the baseline.

6. Analysis, interpretation, and scope

A central ablation finding is that the logit trajectories across layers were empirically found to be highly linear near upper layers, with the coefficient of determination $x_t$6 reported as reaching $x_t$7 toward the final layers. This is presented as empirical support for the use of linear regression in upper-layer logit prediction (He et al., 4 Mar 2025).

The paper also isolates the effect of candidate filtering. The filtering-only baseline can reduce some errors, but it is explicitly reported that it is not responsible for the entire improvement, and that more than 50% of DeLTa’s gains stem from logit trajectory modeling itself. By the same token, DoLa can help in some cases but is reported as less robust, especially on reasoning and math tasks, where DeLTa’s multi-layer formulation yields more stable gains.

These analyses constrain two common misunderstandings. First, DeLTa is not merely a head-pruning or candidate-pruning heuristic; its central mechanism is regression over layerwise logit dynamics. Second, its reported gains are not limited to factuality benchmarks, since the same decoding rule improves performance on reasoning benchmarks that require chain-of-thought-like competence. This suggests that upper-layer logit trajectories encode information relevant both to factual retrieval and to inferential consistency, while remaining accessible through decoding-time computation alone.