---
title: 'DeLTa: Logit Trajectory Decoding'
url: https://www.emergentmind.com/topics/delta-b8f2a03e-0972-4329-9b0a-3494f268457d
type: topic
---

# DeLTa: Logit Trajectory Decoding

DeLTa, short for **Decoding by Logit Trajectory-based approach**, is a decoding strategy for large language models that aims to improve factual accuracy and inferential reasoning **without requiring any modifications to the architecture or pre-trained parameters** of the underlying model. Its central idea is to treat the evolution of token logits from lower to higher Transformer layers as a trajectory, fit that trajectory with linear regression, and use the extrapolated logit at a virtual higher layer for next-token selection. Reported evaluations on TruthfulQA, StrategyQA, and GSM8K show gains of up to **4.9%**, **8.1%**, and **7.3%**, respectively, over the baseline [2503.02343].

## 1. Definition and objective

DeLTa was introduced to address a recurrent limitation of large language models: generated content may be factually incorrect or may exhibit deficiencies in logical reasoning. In contrast to approaches that rely on retraining, architectural modification, or additional data, DeLTa operates entirely at decoding time. The method adjusts next-token probabilities by analyzing the layerwise trajectory of logits and extrapolating that trajectory with a linear model [2503.02343].

The motivating empirical premise is that **the logits for correct tokens tend to increase through higher Transformer layers**. DeLTa therefore treats the sequence of logits across selected upper layers as a “time series” and predicts a refined logit beyond the final layer. This design places DeLTa in the class of post hoc decoding interventions rather than parameter-updating or architecture-changing methods. A plausible implication is that DeLTa uses information already present in the forward pass but not fully exploited by standard final-layer decoding.

## 2. Layerwise probability representation

For a Transformer with \(N\) layers and a decoding position \(t\), DeLTa defines token probabilities at arbitrary intermediate layers through the logit lens formulation. If \(x_t\) is the token at position \(t\), \(x_{<t}\) are the preceding tokens, and \(\mathcal{X}\) is the vocabulary, then for layer \(\ell\),

$$
P_{\ell}(x_t \mid x_{<t}) = \mathrm{softmax}\left(\mathbf{logit}^{(\ell)}\right)_{x_t},\quad x_t \in \mathcal{X}.
$$

To construct the regression problem, DeLTa selects a contiguous set of layers from an intermediate layer \(N_{mid}\) to the final layer \(N\):

$$
\bm{X}_{\text{reg}} = [N_{mid}, N_{mid}+1, \ldots, N]
$$

and collects the corresponding logits

$$
\bm{Y}_{\text{reg}} = [\mathbf{logit}^{(N_{mid})}, \mathbf{logit}^{(N_{mid}+1)}, \ldots, \mathbf{logit}^{(N)}].
$$

These quantities are defined per decoding step and per candidate token, so the regression operates directly on token-level logit evolution rather than on hidden states or attention scores [2503.02343].

This formulation makes DeLTa structurally simple: it reuses existing layer outputs and requires no auxiliary model. The method is therefore compatible with standard Transformer inference pipelines so long as intermediate-layer logits can be exposed.

## 3. Logit trajectory prediction

The core predictive step fits a linear regression for each candidate token across the selected layers and extrapolates to a virtual layer \(L\), which may be fractional and may exceed \(N\). The extrapolated logit is written as

$$
\widehat{\mathbf{logit}^{(L)}} = \hat{\bm{\beta}}_0 + \hat{\bm{\beta}}_1 L,
$$

with regression coefficients

$$
\hat{\bm{\beta}}_1 = \frac{C(\bm{X}_{\text{reg}}, \bm{Y}_{\text{reg}})}{V(\bm{X}_{\text{reg}})},
$$

$$
\hat{\bm{\beta}}_0 = E(\bm{Y}_{\text{reg}}) - \hat{\bm{\beta}}_1 E(\bm{X}_{\text{reg}}).
$$

The final probability distribution is then obtained by applying softmax to the extrapolated logits, but only over a filtered candidate set \(\mathcal{V}_{\text{head}}\):

$$
\hat{P}_L(x_t \mid x_{<t}) = \mathrm{softmax}\left(\widehat{\mathbf{logit}^{(L)}}_{\mathcal{V}_{\text{head}}}\right)_{x_t}.
$$

The candidate set is specified as

$$
\mathcal{V}_{\text{head}} =
\left\{
x_t \in \mathcal{X} :
\hat{P}(x_t \mid x_{<t}) \ge \alpha \max_{w} P_N(w \mid x_{<t})
\right\},
$$

with the summary reporting that **typically, \(\alpha = 0.1\)**. Tokens outside this set are assigned zero probability [2503.02343].

In operational terms, DeLTa combines two mechanisms: extrapolation of upper-layer logit dynamics and restriction of the softmax support. The paper’s analysis distinguishes these two contributions rather than treating the filter as the sole source of improvement.

## 4. Decoding procedure and implementation profile

During generation, DeLTa repeats the same sequence at each decoding step: it extracts token-level logits across the selected layers, performs linear regression per token, extrapolates logits at \(L\), applies the candidate filter, computes the softmax on the filtered set, and then samples or chooses the next token [2503.02343].

A defining property of the method is that **no forward passes through extra model weights or retraining is required**. The intervention is therefore a decoding-time wrapper around an existing post-trained model. The reported hyperparameter setting for the regression range \((N_{mid}, L)\) is described as **dataset- and model-agnostic**, based on ablation and validation studies. The reported computational overhead is moderate and model-dependent; for **Qwen2.5-7B**, the latency increase is reported as **1.4x**.

This implementation profile makes DeLTa a plug-in decoding method rather than a fine-tuning method. A plausible implication is that its deployment cost is governed primarily by access to intermediate logits and the per-step regression calculation, not by model re-optimization.

## 5. Empirical evaluation

DeLTa was evaluated on **Qwen2.5-7B**, **Mistral-7B**, and **Llama-3.1-8B**, with comparisons against standard decoding, a filtering-only baseline, and **DoLa** in early-layer and late-layer variants. The benchmark suite includes **TruthfulQA** for factuality, **StrategyQA** and **GSM8K** for reasoning, and **TriviaQA** and **Natural Questions** for closed-question answering [2503.02343].

The headline improvements reported in the paper are summarized below.

| Benchmark | Reported maximum gain over baseline | Example |
|---|---:|---|
| TruthfulQA | up to 4.9% | Llama-3.1-8B: 44.0 \(\rightarrow\) 48.9 |
| StrategyQA | up to 8.1% | Llama-3.1-8B: 64.0 \(\rightarrow\) 72.1 |
| GSM8K | up to 7.3% | Llama-3.1-8B: 42.8 \(\rightarrow\) 50.1 |

The Qwen2.5-7B results reported in the summary illustrate the same pattern. On **TruthfulQA (\%True*Info)**, the baseline is **64.1**, filtering-only gives **60.7**, DoLa early-layer gives **62.9**, DoLa late-layer gives **55.0**, and DeLTa gives **65.4**. On **StrategyQA**, the baseline is **76.9** and DeLTa reaches **81.2**. On **GSM8K**, the baseline is **78.7** and DeLTa reaches **81.6**. On **TriviaQA**, both filtering-only and DeLTa reach **44.1**, compared with a **39.1** baseline; on **Natural Questions**, the baseline is **11.5** and DeLTa reaches **13.0** [2503.02343].

The paper further reports that **across all tasks, DeLTa consistently outperforms the baseline and other decoding strategies**, with especially prominent gains on reasoning-intensive tasks such as GSM8K and StrategyQA. A qualitative example on a GSM8K math word problem is described as showing that DeLTa corrected multi-step arithmetic errors present in the baseline.

## 6. Analysis, interpretation, and scope

A central ablation finding is that the **logit trajectories across layers were empirically found to be highly linear near upper layers**, with the coefficient of determination \(R^2\) reported as reaching **\(\sim 0.9\)** toward the final layers. This is presented as empirical support for the use of linear regression in upper-layer logit prediction [2503.02343].

The paper also isolates the effect of candidate filtering. The filtering-only baseline can reduce some errors, but it is explicitly reported that it is **not responsible for the entire improvement**, and that **more than 50% of DeLTa’s gains stem from logit trajectory modeling itself**. By the same token, **DoLa** can help in some cases but is reported as **less robust**, especially on reasoning and math tasks, where DeLTa’s multi-layer formulation yields more stable gains.

These analyses constrain two common misunderstandings. First, DeLTa is not merely a head-pruning or candidate-pruning heuristic; its central mechanism is regression over layerwise logit dynamics. Second, its reported gains are not limited to factuality benchmarks, since the same decoding rule improves performance on reasoning benchmarks that require chain-of-thought-like competence. This suggests that upper-layer logit trajectories encode information relevant both to factual retrieval and to inferential consistency, while remaining accessible through decoding-time computation alone.

Source: https://www.emergentmind.com/topics/delta-b8f2a03e-0972-4329-9b0a-3494f268457d