---
title: Attribution-Guided Decoding (AGD)
url: https://www.emergentmind.com/topics/attribution-guided-decoding-agd
type: topic
---

# Attribution-Guided Decoding (AGD)

Attribution-Guided Decoding (AGD) is an interpretability-based decoding strategy in which generation is steered at inference time by feature-attribution scores rather than by direct intervention on model activations. In its canonical formulation, AGD considers a set of high-probability output token candidates and selects the one that exhibits the highest attribution to a user-defined Region of Interest (ROI), where the ROI can be defined over input tokens, context documents, or internal model components such as attention heads [2509.26307]. Related work applies attribution-guided control in adjacent forms, including cross-attention-based semantic reranking for data-to-text generation and token-attribution-guided query rewriting for neural retrieval [2109.07043] [2602.11841].

## 1. Definition and conceptual scope

AGD was introduced to address a recurrent limitation of standard decoding methods: they often fail to robustly satisfy complex instructions and generate factually accurate text, while existing control techniques frequently degrade general output quality [2509.26307]. The central design choice is selection rather than intervention. AGD is a selectionist method operating at decoding time: it uses post-hoc interpretability methods inside the decoding loop to choose among plausible next tokens, and it does so without altering internal activations or requiring model retraining [2509.26307].

This framing places AGD within a broader family of attribution-guided inference procedures. In data-to-text NLG, SeA-GuiDe leverages information extracted from the cross-attention mechanism of transformer encoder-decoder models to infer which input attributes have been realized and then semantically reranks beam-search hypotheses; it requires no model modifications, no extra training data, no manual annotation, and no post-hoc rules [2109.07043]. In neural retrieval, attribution-guided query rewriting forms a closed loop in which token-level explanations from the retriever are used as soft guidance for LLM-based rewriting, with the rewritten query then re-issued to the same retriever [2602.11841]. This suggests that AGD is best understood not only as a single decoding algorithm, but also as a methodological pattern: infer what the model is relying on, then use that attribution signal to constrain inference-time choice.

## 2. Formal decoding mechanism

The canonical AGD formulation uses the standard autoregressive setup. Let \(x = (x_1, x_2, \dots, x_n)\) denote the prompt input tokens, \(y = (y_1, y_2, \dots)\) the generated output sequence, and \(\mathcal{V}\) the vocabulary. At generation step \(t\), the model produces \(p_\theta(y_t \mid x, y_{<t})\) [2509.26307].

AGD first constructs a candidate set \(\mathcal{C}_t\) as the set of up to \(k\) highest-probability tokens, further filtered so that \(p_\theta(c) \geq \pi_{\min}\). The purpose of this restriction is to preserve fluency and efficiency by ensuring that only plausible continuations are considered [2509.26307]. For each candidate \(c\), an attribution function produces relevance scores over attributable components \(\Omega\):
\[
\mathcal{A}_\theta(c \mid x, y_{<t}) \rightarrow \{ r_\omega \mid \omega \in \Omega \}.
\]
If \(R \subseteq \Omega\) is the user-defined ROI, the total attribution score is
\[
S(c, R) = \sum_{\omega \in R} r_\omega
= \sum_{\omega \in R} \mathcal{A}_{\theta}(c \mid x, y_{<t}; \omega).
\]
AGD then selects
\[
y_t = \arg\max_{c \in \mathcal{C}_t} S(c, R).
\]
In the reported implementation, Layerwise Relevance Propagation is the main attribution method used, adapted for transformers as AttnLRP; Input\(\times\)Gradient is also evaluated as an alternative [2509.26307].

A further variant is entropy-gated AGD. Because full AGD at every step is costly and can hurt text quality when the model is confident, AGD can be triggered only on high-entropy steps. If the entropy \(H_t\) of \(p_\theta(\cdot \mid x, y_{<t})\) exceeds threshold \(\tau\), AGD is applied; otherwise standard greedy decoding is used:
\[
y_t =
\begin{cases}
\arg\max_{c \in \mathcal{V}} p_\theta(c \mid x, y_{<t}), & H_t < \tau \\
\arg\max_{c \in \mathcal{C}_t} S(c, R), & H_t \geq \tau .
\end{cases}
\]
In the experiments, \(\tau\) is set to the 80th percentile of token-level entropy [2509.26307].

## 3. Regions of interest and attribution signals

The ROI determines what AGD is guided toward. For instruction following, the ROI is the input embeddings of instruction tokens,
\[
R_I = \{ e_i \mid x_i \in x_I \},
\]
so token selection is biased toward candidates whose logits are most attributable to the instruction portion of the prompt. For factuality, the ROI can be internal knowledge heads. For retrieval grounding, the ROI can be context document input embeddings or context-specific heads [2509.26307]. This modularity is a defining property of AGD: the same decoding rule can target prompt constraints, parametric knowledge, or external context simply by changing the attributable region.

In related attribution-guided inference, the attribution signal itself may differ. SeA-GuiDe uses cross-attention weights in encoder-decoder transformers to track slot realization during generation [2109.07043]. Empirical analysis of T5 and BART reveals three universal cross-attention patterns: verbatim slot mention in the lowest cross-attention layer, paraphrased slot mention in intermediate layers, and unrealized slot mention at the final decoding step when the EOS token is generated [2109.07043]. Let \(A_{\ell,h,t,i}\) denote cross-attention at layer \(\ell\), head \(h\), decoding step \(t\), and input token \(i\). The three mention-tracking components are:
\[
a_t^{(1)}(s) = \max_{h} \left( \max_{i \in I_s} \text{Atten}_{1,h,t,i} \right),
\]
\[
a_t^{(2)}(s) = \max_{h} \left( \frac{1}{L/2} \sum_{\ell=1}^{L/2} \left(\frac{1}{|I_s|}\sum_{i \in I_s} \text{Atten}_{\ell,h,t,i}\right) \right),
\]
\[
a_{EOS}^{(3)}(s) = \max_{h} \left( \frac{1}{L} \sum_{\ell=1}^L \left(\frac{1}{|I_s|}\sum_{i \in I_s} \text{Atten}_{\ell,h,EOS,i}\right) \right).
\]
If \(a_t^{(c)}(s) > \theta_c\), slot \(s\) is considered mentioned, or missed for the third component at EOS [2109.07043]. The thresholds reported are \(0.9\) for verbatim mention, \(0.3\)–\(0.4\) for paraphrased mention, and approximately \(0.1\) for unrealized mention [2109.07043].

In attribution-guided query rewriting, the attributable objects are query tokens rather than next-token candidates. Integrated Gradients is used to estimate the contribution of each query token to the retriever score. For each query, the top-\(k\) retrieved documents are collected with \(k=5\), and token attributions are averaged across those documents:
\[
\alpha_i = \frac{1}{k} \sum_{d \in D_q^{top}} \mathrm{IG}(t_i, s(q, d)).
\]
The attribution scores are normalized on a per-query basis; high positive \(\alpha_i\) indicates strong positive contribution, whereas low or negative \(\alpha_i\) indicates weak or misleading contribution [2602.11841]. The LLM receives the original query, the list of tokens with attribution scores, and explicit instructions to preserve original user intent, preserve or emphasize high-attribution tokens, clarify or disambiguate low-attribution tokens, and avoid adding new concepts not implied by the original query [2602.11841].

## 4. Empirical domains and reported performance

AGD has been evaluated on instruction following and factuality tasks in large language models. On instruction following, the paper reports that AGD significantly boosts adherence, improving the overall success rate on Llama 3.1 from \(66.0\%\) to \(79.1\%\) [2509.26307]. In the IHEval results summarized in the paper, Llama 3.1 with greedy decoding attains Prompt-Level Accuracy \(66.0\), while AGD-LRP attains \(79.1\); entropy-gated AGD-LRPe attains \(74.5\) Prompt-Level Accuracy with Quality Score \(76.4\) [2509.26307]. On SysBench, AGD-LRPe is reported at Constraint Satisfaction Rate \(74.3\), Instruction Satisfaction Rate \(58.2\), and Session Satisfaction Rate \(33.9\) [2509.26307].

For factuality, AGD is evaluated in both closed-book and open-book QA. In the closed-book setting, AGD-LRPh attains \(82.4\) on TQA and \(39.6\) on HPQA, compared with \(81.4\) and \(34.6\) under greedy decoding. In the open-book setting, AGD-LRPc attains \(91.4\) on TQA, \(87.9\) on NQ, and \(59.8\) on HPQA, outperforming the reported greedy and CAD baselines in that table [2509.26307]. The paper characterizes these gains as reductions in hallucinations and improvements in factual recall, with especially robust gains in open-book settings [2509.26307].

Related attribution-guided methods report improvements in other inference settings. In neural retrieval, attribution-guided LLM rewriting outperforms the original query, LLM-only rewriting, and hard attribution-based pruning across BEIR datasets and two retrievers [2602.11841]. For SPLADE, nDCG@10 gains are reported up to \(9\%\) over the original query and up to \(22\%\) over LLM-only rewriting; for TCT-ColBERT, the gains are reported up to \(10\%\) and \(16\%\), respectively [2602.11841]. The paper also states that LLM-only rewriting is inconsistent and sometimes worse than the original query, while hard pruning by retaining only tokens above the query-wise mean generally performs worst [2602.11841].

In data-to-text generation, SeA-GuiDe is evaluated with T5 and BART on ViGGO, E2E, and MultiWOZ 2.1 [2109.07043]. On ViGGO, SeA-GuiDe yields Semantic Error Rate \(0.49\%\) for T5-small and \(1.07\%\) for BART-base, described as approximately \(2\times\) reduction versus beam search and \(3\)–\(4\times\) reduction versus greedy search [2109.07043]. On E2E, SeA-GuiDe reduces SER to almost zero, with \(0.04\%\) reported on T5-small. On MultiWOZ, SER is reduced from approximately \(1.2\%\) under greedy decoding to as low as \(0.6\%\) [2109.07043]. The paper further reports that BLEU, METEOR, ROUGE-L, and CIDEr are maintained or marginally improved, and that runtime overhead is minor, at \(11\)–\(18\%\) on GPU compared to standard beam search with reranking [2109.07043].

## 5. Relation to other control methods and neighboring attribution interventions

AGD is explicitly contrasted with activation steering, logit steering, and contrastive decoding. The reported motivation is that direct manipulation of activations or logits can push models out of distribution, harming fluency or producing unnatural text, and may require probe training or additional hyperparameter tuning [2509.26307]. By contrast, AGD only selects among plausible tokens from the model’s own output distribution, and the paper argues that this preserves the natural distribution and output quality [2509.26307]. The same comparison is reflected in the factuality experiments against DoLA and in instruction-following experiments against CAD [2509.26307].

The relation between AGD and SeA-GuiDe is more specific. SeA-GuiDe is described as conceptually related to Attribution-Guided Decoding because it uses model attributions, specifically cross-attention distributions, to encourage faithful generation. However, SeA-GuiDe is described as more lightweight and plug-and-play: it extracts attribution information from unmodified models, infers semantic realization, and uses this externally for reranking, requiring no model retraining or architectural change [2109.07043]. A plausible implication is that attribution-guided methods span a spectrum from token-by-token next-token selection to sequence-level reranking.

A neighboring, though distinct, methodological concern is attributional validity. In non-invasive neural language decoding, the paper on source attribution argues that apparent gains can come from structural shortcuts, window-level stimulus-locked evidence, or cross-window contextual aggregation, and that reported decoding performance should therefore be source-attributed, not merely reported [2605.24524]. Its Group Context Bias (GCB) is an inference-time additive logit bias that pools sentence-consistent evidence across windows while leaving the base retrieval scores and candidate pool fixed; under fixed-window controls, R@1 shifts from \(44\%\) to \(52\%\) on Gwilliams and from \(22\%\) to \(29\%\) on MOUS [2605.24524]. The effect collapses under random-grouping perturbations and vanishes when local evidence is attenuated or near chance [2605.24524]. This neighboring work does not define AGD, but it sharpens a core methodological point shared with AGD research: inference-time control claims are more informative when the source of the gain is explicitly localized.

## 6. Limitations, failure modes, and terminological disambiguation

The main limitations reported for AGD are threefold. First, AGD has a selection limit: it cannot produce a token unless it is among the model’s top-\(k\) proposals. Second, it incurs computational cost because attribution requires a backward pass for each candidate at each intervention step. Third, ROI design is task-dependent and may be nontrivial [2509.26307]. The entropy-gated variant is proposed precisely to mitigate quality degradation and reduce computational overhead by applying guidance only when the model is uncertain [2509.26307].

Related attribution-guided methods exhibit their own failure modes. In attribution-guided query rewriting, hard pruning fails because contextual tokens, even if scoring low, are necessary for intent; the paper concludes that attribution as soft guidance is effective precisely because the LLM can clarify or specialize low-scoring terms rather than simply dropping them [2602.11841]. In SeA-GuiDe, the reported limitations are that it does not detect arbitrary hallucinations unattributed to any slot and might struggle with duplicate slot mentions or certain Boolean slot realizations [2109.07043]. More broadly, the source-attribution study in brain-to-language retrieval shows that variable-length decoding and stimulus leakage can inflate performance, with Gaussian noise reaching \(66.3\%\) Rank@1 under variable-length decoding but collapsing to near chance once fixed-duration windows and stimulus-identity splits are enforced [2605.24524]. This suggests that attribution-guided control and attribution-guided auditing are complementary: one steers inference, the other verifies what the gain should be attributed to.

The acronym “AGD” also has an unrelated meaning in mathematics. In the paper on “discretizations of AGD flows,” AGD refers to Adler–Gelfand–Dikii flows on scalar differential operators, with Hamiltonian functionals
\[
H(L) = \int_{S^1} \text{res}(L^{k/(m+1)}) dx,
\]
and conjectured discretizations in \(\mathbb{RP}^m\) by intersecting one \((k-1)\)-dimensional subspace and \(k-1\) \((m-1)\)-dimensional subspaces [1103.5047]. That usage is entirely separate from Attribution-Guided Decoding.

Source: https://www.emergentmind.com/topics/attribution-guided-decoding-agd