--- title: Logit-Attention Divergence url: https://www.emergentmind.com/topics/logit-attention-divergence type: topic --- # Logit-Attention Divergence Searching arXiv for recent papers on "Logit-Attention Divergence" and closely related terminology. Logit-Attention Divergence (LAD) denotes a mismatch between internal attention and final decision signals. In the explicit sense introduced for multimodal large language models (MLLMs) in multi-image retrieval, LAD is the phenomenon in which internal attention maps remain aligned with the true visual evidence while final output logits are dominated by position-dependent structural priors, so the model may “look at” the correct answer but “say” the wrong one [2605.11591]. Closely related literature uses neighboring but non-identical notions of divergence in attention space, logit space, and attention-logit dynamics, including KL-based hallucination detection from attention heads, KL-minimizing constrained decoding, symmetric logit-space distillation, representational logit distance, and step-to-step control of attention-logit change [2605.05025], [2503.18050], [2602.13567], [2602.15438], [2511.21377]. ## 1. Definition in multi-image retrieval In the retrieval setting studied in "Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibration" [2605.11591], an instance is written as \(x=(q,\mathcal{I})\), where \(q\) is a text query and \(\mathcal{I}=\{v_1,\dots,v_N\}\) is an ordered list of candidate images. The model predicts an index \(y^*\in\{1,\dots,N\}\). LAD is identified by jointly inspecting two internal signals: the post-softmax attention weights from the last query token to the visual tokens associated with each image, and the final output logits over candidate indices. When these quantities are averaged over samples with the ground-truth image fixed at a given position, the attention distribution peaks at the true position with low variance, whereas the logits peak at a different, often preferred position [2605.11591]. The dataset-level signature is equally distinctive. Vanilla models produce confusion matrices with vertical stripes, indicating over-selection of certain positions regardless of content, whereas the proposed debiasing method restores a near-diagonal matrix. The paper interprets this as evidence that the dominant failure mode is not necessarily defective visual grounding inside the network, but distortion introduced at the final decision layer by positional structure [2605.11591]. This interpretation also frames LAD as a critique of purely logit-level calibration. The paper argues that methods such as PriDe assume that position bias is a static, content-independent offset represented by a single global prior vector \(P_{\text{prior}}\). LAD instead implies that the bias is conditional on the ground-truth position and surrounding candidate configuration, so a single global prior cannot separate nearly identical conditional logit profiles [2605.11591]. ## 2. Probabilistic decomposition and attention-guided calibration The retrieval paper models the observed candidate probabilities as \[ P_{\text{obs}}(j \mid i) \propto P_{\text{bias}}(j \mid i)\cdot P_{\text{vis}}(j \mid i), \] where \(i\) is the true position and \(j\) is the predicted position. The visual term is parameterized as \[ P_{\text{vis}}(j \mid i)=\gamma^{\mathbb{I}(j=i)}, \quad \gamma>1, \] so that the correct position receives a multiplicative boost. In log form, \[ \log P_{\text{obs}}(j \mid i)=\log P_{\text{bias}}(j \mid i)+\log P_{\text{vis}}(j \mid i)+\text{const}. \] Within this factorization, the logits are interpreted as the sum of a semantic signal and a conditional structural bias, while attention is treated as a more faithful proxy for the semantic part [2605.11591]. The proposed Attention-Guided Debiasing framework is training-free and has two stages: calibration and inference. In calibration, a small calibration set \(\mathcal{D}_{\text{cal}}\) of only 5 samples is symmetrized by cyclic permutations so that the correct answer appears equally often at every position. For the subset \(\tilde{\mathcal{D}}_{\text{cal}}^{(i)}\) with ground truth at position \(i\), the observed conditional distribution is estimated as \[ \hat{P}_{\text{obs}}(j \mid i)=\frac{1}{|\tilde{\mathcal{D}}_{\text{cal}}^{(i)}|}\sum_{x\in\tilde{\mathcal{D}}_{\text{cal}}^{(i)}} P(c_j\mid x), \] where \(c_j\) is the tokenized index string for candidate \(j\). For multi-token candidate labels, \[ \log P(c_k \mid x)=\sum_{m=1}^{m_k}\log P\!\left(t_k^{(m)} \mid x, t_k^{(1)},\dots,t_k^{(m-1)}\right). \] A conservative visual gain is then estimated by \[ \hat{\gamma}=\max_{i,j\neq i}\frac{\hat{P}_{\text{obs}}(i\mid i)}{\hat{P}_{\text{obs}}(j\mid i)}, \] and the conditional bias profile is recovered through \[ \log \hat{P}_{\text{bias}}(j\mid i)=\log \hat{P}_{\text{obs}}(j\mid i)-\log P_{\text{vis}}(j\mid i). \] The result is a conditional bias matrix rather than a single prior vector [2605.11591]. The same paper calibrates attention itself, because raw attention can still contain structural artifacts such as attention sinks and “lost in the middle” effects. For each layer \(l\) and head \(h\), with post-softmax attention matrix \(\mathbf{A}^{(l,h)}(x)\in\mathbb{R}^{T\times T}\), attention from the query token \(t_q\) to the token span \(\mathcal{R}_k\) of the \(k\)-th image is aggregated as \[ a_k^{(l)}(x)=\frac{1}{H}\sum_{h=1}^{H}\sum_{t\in \mathcal{R}_k}\mathbf{A}_{t_q\to t}^{(l,h)}(x). \] This yields \[ \mathbf{a}^{(l)}(x)=\big[a_1^{(l)}(x),\dots,a_N^{(l)}(x)\big]^\top, \] and the static attention prior is estimated by \[ \hat{\mathbf{a}}_{\text{prior}}^{(l)}=\mathbb{E}_{x\sim \tilde{\mathcal{D}}_{\text{cal}}}\big[\mathbf{a}^{(l)}(x)\big]. \] At inference time, the evidence strength of layer \(l\) is \[ S^{(l)}(x)=\sum_{k=1}^{N} a_k^{(l)}(x), \] and the top-\(K\) layers \(\mathcal{L}^*\) with highest \(S^{(l)}(x)\) are selected; the experiments use \(K=2\). Purified attention is computed by \[ \log \tilde{\pi}_k(x)\propto \frac{1}{|\mathcal{L}^*|}\sum_{l\in\mathcal{L}^*}\Big(\log a_k^{(l)}(x)-\log \mathbf{a}^{(l)}_{\text{prior},k}\Big), \] followed by temperature sharpening, \[ \pi(x)=\mathrm{softmax}\big(\log \tilde{\pi}(x)\cdot \tau\big), \] with \(\tau=5\). The dynamic bias prior is then \[ P_{\text{prior}}(j\mid x)=\sum_{k=1}^{N}\pi_k(x)\hat{P}_{\text{bias}}(j\mid k), \] and the corrected scores are \[ \log \hat{P}_{\text{vis}}(c_j\mid x)=\log P_{\text{obs}}(c_j\mid x)-\log P_{\text{prior}}(j\mid x). \] The calibrated distribution \(\hat{P}_{\text{vis}}\) is used for prediction [2605.11591]. ## 3. Empirical profile and measured effects The multi-image retrieval study evaluates three MLLM backbones—Qwen2.5-VL-3B, LLaVA-OneVision-8B, and InternVL3-8B—on MS-COCO-based benchmarks, with both a Random setting and an Adversarial setting in which hard negatives are mined using CLIP embeddings. The main paper uses candidate pool size \(N=4\), and additional analysis covers \(N=8\) and \(N\in\{2,\dots,12\}\). Each test instance is evaluated under \(T=5\) random shuffles, and the reported metrics are Accuracy (Acc), Recall Standard Deviation (RStd), and Consistency (Cons.) [2605.11591]. For \(N=4\), the reported numbers for LLaVA-OneVision-8B are **98.66% Acc**, **0.88 RStd**, **96.5 Cons.** in the Random setting, and **71.06% Acc**, **10.23 RStd**, **51.3 Cons.** in the Adversarial setting. The paper summarizes the gain as **over 40% accuracy gain over baselines** in challenging settings. For \(N=8\), the same model reaches **94.92%** on random and **55.34%** on adversarial. Cross-domain experiments between MS-COCO and Flickr8k maintain high accuracy and consistency, which the paper interprets as evidence that the estimated structural bias is architecture-driven and transferable [2605.11591]. The ablations are designed to separate the contribution of attention readout, attention calibration, and conditional bias correction. In adversarial \(N=4\), raw attention readout alone gives **42.36%**, adding the static attention prior gives **64.38%**, and the full method reaches **71.06%**. Additional ablations show that performance saturates around **5 samples**, that \(\tau=5\) is optimal, and that selecting top-\(K\) layers rather than averaging all layers is beneficial, with \(K=2\) described as the sweet spot. The method also remains effective across candidate identifier formats such as “1”, “A”, “I”, and “first,” which the paper uses to argue that LAD is not an artifact of numeral tokenization [2605.11591]. A central qualitative result is that “Purified Attention” often outperforms vanilla logits, and that the gap widens as the candidate pool grows. This suggests that the model retains useful visual evidence internally even when the final logits are position-biased. A plausible implication is that LAD should be understood as a failure of calibration between internal evidence and output decision, rather than as a purely perceptual failure. ## 4. Attention-divergence as a hallucination signal A distinct use of the term appears in "Detecting Hallucinations in Large Language Models via Internal Attention Divergence Signals" [2605.05025]. There, Logit-Attention Divergence refers to a white-box hallucination or uncertainty score derived from internal attention maps rather than from output probabilities alone. Each attention head at generation step \(t\) produces a distribution \[ A_t^{(l,h)} \] over previous tokens, and this distribution is compared to a uniform reference \[ \mathcal{U}(x) = \frac{1}{T}, \quad x \in \{1,\dots,T\}. \] The KL divergence feature is \[ D_{\mathrm{KL}}(P \,\|\, \mathcal{U}) = \sum_x P(x)\log\frac{P(x)}{1/T} = \log T - H(P). \] Low KL divergence indicates near-uniform, diffuse attention; high KL divergence indicates concentrated attention [2605.05025]. The example-level feature vector aggregates these per-head, per-step signals. For each generated answer, the procedure is: compute KL divergence for every head at every generation step; average those KL values across the generated answer tokens; and concatenate the per-head averages into \[ x_i \in \mathbb{R}^{L \times H}. \] Correctness is then predicted by a lasso-regularized logistic regression probe, \[ P(y_i = 1 \mid x_i) = \sigma(w^\top x_i + b), \] optimized with cross-entropy plus an \(L_1\) penalty. The paper emphasizes that the uncertainty signal comes from the attention divergence itself, while the probe serves as an aggregator and selector [2605.05025]. This method is single-pass: no repeated sampling, no ensembles, and no external verifier model. The paper evaluates TruthfulQA, TriviaQA, HotpotQA, and GSM8K on Llama-3.2-3B-Instruct, Qwen3-4B-Instruct, and Mistral-7B-Instruct-v0.2. Reported AUROC ranges are around **0.89–0.91** for TruthfulQA, **0.83–0.85** for TriviaQA, **0.78–0.80** for HotpotQA, and, for GSM8K, Qwen3-4B reaches **0.945 AUROC**. The strongest signal is concentrated in middle layers and on factual tokens such as named entities and numbers [2605.05025]. Although this formulation differs from the retrieval-specific LAD of [2605.11591], both works treat internal attention as a white-box signal that can expose failures not visible from output probabilities alone. This suggests a broader research pattern in which attention-space diagnostics are used either for post-hoc calibration or for uncertainty estimation. ## 5. Broader logit-space divergence frameworks Several nearby papers study divergence in logit space without defining LAD in the multi-image retrieval sense. They are relevant because they formalize how deviations in logits, projected token distributions, or attention scores can affect decoding, distillation, representational similarity, and training stability. | Source | Divergence object | Role | |---|---|---| | [2503.18050] | \(D_{\mathrm{KL}}(Q\parallel P)\) under token exclusion | constrained decoding | | [2602.13567] | JSD / Jeffreys divergence in Logit Lens space | symmetric distillation | | [2602.15438] | \(d_{\mathrm{logit}}\) | representational similarity bounds | | [2511.21377] | step-to-step change in attention logits | training stability | In "(G)I-DLE: Generative Inference via Distribution-preserving Logit Exclusion with KL Divergence Minimization for Constrained Decoding" [2503.18050], constrained decoding is formulated as a KL minimization problem over distributions that assign zero mass to banned tokens: \[ \min_Q \; D_{\mathrm{KL}}(Q \parallel P) \quad \text{subject to } Q(x_t)=0 \ \forall x_t\in B,\quad \sum_{x_t\notin B} Q(x_t)=1. \] The optimizer is the conditional distribution \[ Q(x_t)=P(x_t\mid x_{