Logit-Attention Divergence

• Logit-Attention Divergence is a phenomenon where internal attention maps faithfully reflect visual evidence while output logits are skewed by position-dependent biases.
• The framework uses attention-guided calibration in multi-image retrieval to separate semantic signals from structural biases in final decision layers.
• Empirical evaluations show significant accuracy gains and diagnostic insights into hallucination detection, constrained decoding, and representational similarity.

Searching arXiv for papers on "Logit-Attention Divergence" and closely related terminology. Logit-Attention Divergence (LAD) denotes a mismatch between internal attention and final decision signals. In the explicit sense introduced for multimodal LLMs (MLLMs) in multi-image retrieval, LAD is the phenomenon in which internal attention maps remain aligned with the true visual evidence while final output logits are dominated by position-dependent structural priors, so the model may “look at” the correct answer but “say” the wrong one (Xian et al., 12 May 2026). Closely related literature uses neighboring but non-identical notions of divergence in attention space, logit space, and attention-logit dynamics, including KL-based hallucination detection from attention heads, KL-minimizing constrained decoding, symmetric logit-space distillation, representational logit distance, and step-to-step control of attention-logit change (Dijk, 6 May 2026, Lee, 23 Mar 2025, Dhakal et al., 14 Feb 2026, Nielsen et al., 17 Feb 2026, Anson et al., 26 Nov 2025).

1. Definition in multi-image retrieval

In the retrieval setting studied in "Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibration" (Xian et al., 12 May 2026), an instance is written as $x=(q,\mathcal{I})$, where $q$ is a text query and $\mathcal{I}=\{v_1,\dots,v_N\}$ is an ordered list of candidate images. The model predicts an index $y^*\in\{1,\dots,N\}$. LAD is identified by jointly inspecting two internal signals: the post-softmax attention weights from the last query token to the visual tokens associated with each image, and the final output logits over candidate indices. When these quantities are averaged over samples with the ground-truth image fixed at a given position, the attention distribution peaks at the true position with low variance, whereas the logits peak at a different, often preferred position (Xian et al., 12 May 2026).

The dataset-level signature is equally distinctive. Vanilla models produce confusion matrices with vertical stripes, indicating over-selection of certain positions regardless of content, whereas the proposed debiasing method restores a near-diagonal matrix. The paper interprets this as evidence that the dominant failure mode is not necessarily defective visual grounding inside the network, but distortion introduced at the final decision layer by positional structure (Xian et al., 12 May 2026).

This interpretation also frames LAD as a critique of purely logit-level calibration. The paper argues that methods such as PriDe assume that position bias is a static, content-independent offset represented by a single global prior vector $P_{\text{prior}}$. LAD instead implies that the bias is conditional on the ground-truth position and surrounding candidate configuration, so a single global prior cannot separate nearly identical conditional logit profiles (Xian et al., 12 May 2026).

2. Probabilistic decomposition and attention-guided calibration

The retrieval paper models the observed candidate probabilities as

$$P_{\text{obs}}(j \mid i) \propto P_{\text{bias}}(j \mid i)\cdot P_{\text{vis}}(j \mid i),$$

where $i$ is the true position and $j$ is the predicted position. The visual term is parameterized as

$$P_{\text{vis}}(j \mid i)=\gamma^{\mathbb{I}(j=i)}, \quad \gamma>1,$$

so that the correct position receives a multiplicative boost. In log form,

$$\log P_{\text{obs}}(j \mid i)=\log P_{\text{bias}}(j \mid i)+\log P_{\text{vis}}(j \mid i)+\text{const}.$$

Within this factorization, the logits are interpreted as the sum of a semantic signal and a conditional structural bias, while attention is treated as a more faithful proxy for the semantic part (Xian et al., 12 May 2026).

The proposed Attention-Guided Debiasing framework is training-free and has two stages: calibration and inference. In calibration, a small calibration set $q$0 of only 5 samples is symmetrized by cyclic permutations so that the correct answer appears equally often at every position. For the subset $q$1 with ground truth at position $q$2, the observed conditional distribution is estimated as

$q$3

where $q$4 is the tokenized index string for candidate $q$5. For multi-token candidate labels,

$q$6

A conservative visual gain is then estimated by

$q$7

and the conditional bias profile is recovered through

$q$8

The result is a conditional bias matrix rather than a single prior vector (Xian et al., 12 May 2026).

The same paper calibrates attention itself, because raw attention can still contain structural artifacts such as attention sinks and “lost in the middle” effects. For each layer $q$9 and head $\mathcal{I}=\{v_1,\dots,v_N\}$0, with post-softmax attention matrix $\mathcal{I}=\{v_1,\dots,v_N\}$1, attention from the query token $\mathcal{I}=\{v_1,\dots,v_N\}$2 to the token span $\mathcal{I}=\{v_1,\dots,v_N\}$3 of the $\mathcal{I}=\{v_1,\dots,v_N\}$4-th image is aggregated as

$\mathcal{I}=\{v_1,\dots,v_N\}$5

This yields

$\mathcal{I}=\{v_1,\dots,v_N\}$6

and the static attention prior is estimated by

$\mathcal{I}=\{v_1,\dots,v_N\}$7

At inference time, the evidence strength of layer $\mathcal{I}=\{v_1,\dots,v_N\}$8 is

$\mathcal{I}=\{v_1,\dots,v_N\}$9

and the top-$y^*\in\{1,\dots,N\}$0 layers $y^*\in\{1,\dots,N\}$1 with highest $y^*\in\{1,\dots,N\}$2 are selected; the experiments use $y^*\in\{1,\dots,N\}$3. Purified attention is computed by

$y^*\in\{1,\dots,N\}$4

followed by temperature sharpening,

$y^*\in\{1,\dots,N\}$5

with $y^*\in\{1,\dots,N\}$6. The dynamic bias prior is then

$y^*\in\{1,\dots,N\}$7

and the corrected scores are

$y^*\in\{1,\dots,N\}$8

The calibrated distribution $y^*\in\{1,\dots,N\}$9 is used for prediction (Xian et al., 12 May 2026).

3. Empirical profile and measured effects

The multi-image retrieval study evaluates three MLLM backbones—Qwen2.5-VL-3B, LLaVA-OneVision-8B, and InternVL3-8B—on MS-COCO-based benchmarks, with both a Random setting and an Adversarial setting in which hard negatives are mined using CLIP embeddings. The main paper uses candidate pool size $P_{\text{prior}}$0, and additional analysis covers $P_{\text{prior}}$1 and $P_{\text{prior}}$2. Each test instance is evaluated under $P_{\text{prior}}$3 random shuffles, and the reported metrics are Accuracy (Acc), Recall Standard Deviation (RStd), and Consistency (Cons.) (Xian et al., 12 May 2026).

For $P_{\text{prior}}$4, the reported numbers for LLaVA-OneVision-8B are 98.66% Acc, 0.88 RStd, 96.5 Cons. in the Random setting, and 71.06% Acc, 10.23 RStd, 51.3 Cons. in the Adversarial setting. The paper summarizes the gain as over 40% accuracy gain over baselines in challenging settings. For $P_{\text{prior}}$5, the same model reaches 94.92% on random and 55.34% on adversarial. Cross-domain experiments between MS-COCO and Flickr8k maintain high accuracy and consistency, which the paper interprets as evidence that the estimated structural bias is architecture-driven and transferable (Xian et al., 12 May 2026).

The ablations are designed to separate the contribution of attention readout, attention calibration, and conditional bias correction. In adversarial $P_{\text{prior}}$6, raw attention readout alone gives 42.36%, adding the static attention prior gives 64.38%, and the full method reaches 71.06%. Additional ablations show that performance saturates around 5 samples, that $P_{\text{prior}}$7 is optimal, and that selecting top-$P_{\text{prior}}$8 layers rather than averaging all layers is beneficial, with $P_{\text{prior}}$9 described as the sweet spot. The method also remains effective across candidate identifier formats such as “1”, “A”, “I”, and “first,” which the paper uses to argue that LAD is not an artifact of numeral tokenization (Xian et al., 12 May 2026).

A central qualitative result is that “Purified Attention” often outperforms vanilla logits, and that the gap widens as the candidate pool grows. This suggests that the model retains useful visual evidence internally even when the final logits are position-biased. A plausible implication is that LAD should be understood as a failure of calibration between internal evidence and output decision, rather than as a purely perceptual failure.

4. Attention-divergence as a hallucination signal

A distinct use of the term appears in "Detecting Hallucinations in LLMs via Internal Attention Divergence Signals" (Dijk, 6 May 2026). There, Logit-Attention Divergence refers to a white-box hallucination or uncertainty score derived from internal attention maps rather than from output probabilities alone. Each attention head at generation step $$P_{\text{obs}}(j \mid i) \propto P_{\text{bias}}(j \mid i)\cdot P_{\text{vis}}(j \mid i),$$0 produces a distribution

$$P_{\text{obs}}(j \mid i) \propto P_{\text{bias}}(j \mid i)\cdot P_{\text{vis}}(j \mid i),$$1

over previous tokens, and this distribution is compared to a uniform reference

$$P_{\text{obs}}(j \mid i) \propto P_{\text{bias}}(j \mid i)\cdot P_{\text{vis}}(j \mid i),$$2

The KL divergence feature is

$$P_{\text{obs}}(j \mid i) \propto P_{\text{bias}}(j \mid i)\cdot P_{\text{vis}}(j \mid i),$$3

Low KL divergence indicates near-uniform, diffuse attention; high KL divergence indicates concentrated attention (Dijk, 6 May 2026).

The example-level feature vector aggregates these per-head, per-step signals. For each generated answer, the procedure is: compute KL divergence for every head at every generation step; average those KL values across the generated answer tokens; and concatenate the per-head averages into

$$P_{\text{obs}}(j \mid i) \propto P_{\text{bias}}(j \mid i)\cdot P_{\text{vis}}(j \mid i),$$4

Correctness is then predicted by a lasso-regularized logistic regression probe,

$$P_{\text{obs}}(j \mid i) \propto P_{\text{bias}}(j \mid i)\cdot P_{\text{vis}}(j \mid i),$$5

optimized with cross-entropy plus an $$P_{\text{obs}}(j \mid i) \propto P_{\text{bias}}(j \mid i)\cdot P_{\text{vis}}(j \mid i),$$6 penalty. The paper emphasizes that the uncertainty signal comes from the attention divergence itself, while the probe serves as an aggregator and selector (Dijk, 6 May 2026).

This method is single-pass: no repeated sampling, no ensembles, and no external verifier model. The paper evaluates TruthfulQA, TriviaQA, HotpotQA, and GSM8K on Llama-3.2-3B-Instruct, Qwen3-4B-Instruct, and Mistral-7B-Instruct-v0.2. Reported AUROC ranges are around 0.89–0.91 for TruthfulQA, 0.83–0.85 for TriviaQA, 0.78–0.80 for HotpotQA, and, for GSM8K, Qwen3-4B reaches 0.945 AUROC. The strongest signal is concentrated in middle layers and on factual tokens such as named entities and numbers (Dijk, 6 May 2026).

Although this formulation differs from the retrieval-specific LAD of (Xian et al., 12 May 2026), both works treat internal attention as a white-box signal that can expose failures not visible from output probabilities alone. This suggests a broader research pattern in which attention-space diagnostics are used either for post-hoc calibration or for uncertainty estimation.

5. Broader logit-space divergence frameworks

Several nearby papers study divergence in logit space without defining LAD in the multi-image retrieval sense. They are relevant because they formalize how deviations in logits, projected token distributions, or attention scores can affect decoding, distillation, representational similarity, and training stability.

Source	Divergence object	Role
(Lee, 23 Mar 2025)	$$P_{\text{obs}}(j \mid i) \propto P_{\text{bias}}(j \mid i)\cdot P_{\text{vis}}(j \mid i),$$7 under token exclusion	constrained decoding
(Dhakal et al., 14 Feb 2026)	JSD / Jeffreys divergence in Logit Lens space	symmetric distillation
(Nielsen et al., 17 Feb 2026)	$$P_{\text{obs}}(j \mid i) \propto P_{\text{bias}}(j \mid i)\cdot P_{\text{vis}}(j \mid i),$$8	representational similarity bounds
(Anson et al., 26 Nov 2025)	step-to-step change in attention logits	training stability

In "(G)I-DLE: Generative Inference via Distribution-preserving Logit Exclusion with KL Divergence Minimization for Constrained Decoding" (Lee, 23 Mar 2025), constrained decoding is formulated as a KL minimization problem over distributions that assign zero mass to banned tokens: $$P_{\text{obs}}(j \mid i) \propto P_{\text{bias}}(j \mid i)\cdot P_{\text{vis}}(j \mid i),$$9 The optimizer is the conditional distribution

$i$0

implemented by adjusting logits so that relative probabilities among allowed tokens are preserved. The paper’s concise interpretation states that this mitigates logit-attention/probability divergence by ensuring that excluding tokens does not unnecessarily perturb the remaining distribution. On K$i$1-Eval with Qwen2.5 models from 1.5B to 14B, the method lowers variance and often improves mean score; for Qwen2.5-14B-Instruct, variance drops from 0.216729 to 0.055056, while mean score rises from 4.911111 to 4.966667 (Lee, 23 Mar 2025).

In "DistillLens: Symmetric Knowledge Distillation Through Logit Lens" (Dhakal et al., 14 Feb 2026), intermediate hidden states are projected into vocabulary space through the Logit Lens,

$i$2

and matched using a symmetric divergence objective. The intermediate loss averages Jensen-Shannon Divergence across mapped student-teacher layer pairs, and the paper also defines a Jeffreys divergence variant. The theoretical claim is that symmetry imposes a dual-sided penalty against both overconfidence and underconfidence while preserving high-entropy information conduits. DistillLens is not primarily about attention maps; it is a distributional alignment method in vocabulary space. Reported results include average Rouge-L 21.12 for GPT-2-120M, 23.72 for GPT-2-340M, and 25.48 for TinyLlama-1.1B, outperforming standard KD and feature-transfer baselines (Dhakal et al., 14 Feb 2026).

In "Logit Distance Bounds Representational Similarity" (Nielsen et al., 17 Feb 2026), the central object is the squared logit distance

$i$3

and the main theorem states that a representational dissimilarity based on the model’s identifiability class is bounded by logit distance. The paper proves that KL divergence universally lower-bounds logit distance and can upper-bound it only under probability lower bounds that are usually too weak in practice. Its empirical conclusion is that KL-based distillation can match a teacher’s predictions while failing to preserve linear representational properties and linearly recoverable concepts, whereas logit-distance distillation preserves them substantially better (Nielsen et al., 17 Feb 2026).

In "Controlling changes to attention logits" (Anson et al., 26 Nov 2025), the relevant divergence notion is dynamic rather than distributional. The attention score matrix

$i$4

is viewed as destabilizing when it moves too much from one optimization step to the next. The proposed method, QuacK, assigns parameter-dependent learning rates to query and key weights so that updates to $i$5 and $i$6 produce bounded changes in logits. The paper reports that QuacK is as stable as QK norm in practice, works in Multi-Latent Attention where QK norm is not compatible, and yields roughly a 10% training speedup in its setup (Anson et al., 26 Nov 2025).

6. Scope, limitations, and terminological variation

The retrieval formulation of LAD has clear operational limits. The method requires white-box access to attention weights, so it cannot be applied to closed-source APIs where intermediate activations are hidden. It is designed for discrete candidate retrieval and does not directly extend to open-ended generation. Its effectiveness is bounded by how semantically faithful the model’s attention is; on highly ambiguous tasks, the attention signal becomes less reliable (Xian et al., 12 May 2026).

The broader literature also shows that the phrase sits inside a larger family of divergence-based analyses rather than naming a single universally fixed construct. In the retrieval setting, LAD is the mismatch between semantically aligned attention and position-biased logits (Xian et al., 12 May 2026). In hallucination detection, the operative object is KL divergence between attention-head distributions and a uniform reference (Dijk, 6 May 2026). In constrained decoding, the relevant issue is distortion induced by banned-token exclusion and the KL-minimizing renormalization that preserves the conditional preference structure over allowed tokens (Lee, 23 Mar 2025). In distillation and representation theory, the focus shifts to symmetric divergence in vocabulary space and to logit distance as a proxy for representational similarity (Dhakal et al., 14 Feb 2026, Nielsen et al., 17 Feb 2026). In optimization, the analogous concern is the step-to-step divergence of attention logits during training (Anson et al., 26 Nov 2025).

A plausible implication is that LAD marks an emerging interface between interpretability, calibration, and stability. Across these formulations, attention and logits are treated not as interchangeable observables but as distinct layers of model behavior whose disagreement can be diagnostic: attention may remain semantically informative when logits are biased, logits may match while representations drift, and stable attention computation may depend more on controlling logit change than on controlling logit magnitude alone.