Attribution-Guided Decoding (AGD)

• Attribution-Guided Decoding is an interpretability-based method that selects high-probability tokens using feature-attribution scores rather than modifying model activations.
• It improves instruction following and factual recall by guiding token selection toward user-defined Regions of Interest without retraining or architectural changes.
• AGD employs techniques like Layerwise Relevance Propagation and entropy gating to balance fluency and efficiency, reducing hallucinations and ensuring output quality.

Attribution-Guided Decoding (AGD) is an interpretability-based decoding strategy in which generation is steered at inference time by feature-attribution scores rather than by direct intervention on model activations. In its canonical formulation, AGD considers a set of high-probability output token candidates and selects the one that exhibits the highest attribution to a user-defined Region of Interest (ROI), where the ROI can be defined over input tokens, context documents, or internal model components such as attention heads (Komorowski et al., 30 Sep 2025). Related work applies attribution-guided control in adjacent forms, including cross-attention-based semantic reranking for data-to-text generation and token-attribution-guided query rewriting for neural retrieval (Juraska et al., 2021, Garouani et al., 12 Feb 2026).

1. Definition and conceptual scope

AGD was introduced to address a recurrent limitation of standard decoding methods: they often fail to robustly satisfy complex instructions and generate factually accurate text, while existing control techniques frequently degrade general output quality (Komorowski et al., 30 Sep 2025). The central design choice is selection rather than intervention. AGD is a selectionist method operating at decoding time: it uses post-hoc interpretability methods inside the decoding loop to choose among plausible next tokens, and it does so without altering internal activations or requiring model retraining (Komorowski et al., 30 Sep 2025).

This framing places AGD within a broader family of attribution-guided inference procedures. In data-to-text NLG, SeA-GuiDe leverages information extracted from the cross-attention mechanism of transformer encoder-decoder models to infer which input attributes have been realized and then semantically reranks beam-search hypotheses; it requires no model modifications, no extra training data, no manual annotation, and no post-hoc rules (Juraska et al., 2021). In neural retrieval, attribution-guided query rewriting forms a closed loop in which token-level explanations from the retriever are used as soft guidance for LLM-based rewriting, with the rewritten query then re-issued to the same retriever (Garouani et al., 12 Feb 2026). This suggests that AGD is best understood not only as a single decoding algorithm, but also as a methodological pattern: infer what the model is relying on, then use that attribution signal to constrain inference-time choice.

2. Formal decoding mechanism

The canonical AGD formulation uses the standard autoregressive setup. Let $x = (x_1, x_2, \dots, x_n)$ denote the prompt input tokens, $y = (y_1, y_2, \dots)$ the generated output sequence, and $\mathcal{V}$ the vocabulary. At generation step $t$, the model produces $p_\theta(y_t \mid x, y_{<t})$ (Komorowski et al., 30 Sep 2025).

AGD first constructs a candidate set $\mathcal{C}_t$ as the set of up to $k$ highest-probability tokens, further filtered so that $p_\theta(c) \geq \pi_{\min}$. The purpose of this restriction is to preserve fluency and efficiency by ensuring that only plausible continuations are considered (Komorowski et al., 30 Sep 2025). For each candidate $c$, an attribution function produces relevance scores over attributable components $\Omega$: $y = (y_1, y_2, \dots)$0 If $y = (y_1, y_2, \dots)$1 is the user-defined ROI, the total attribution score is

$y = (y_1, y_2, \dots)$2

AGD then selects

$y = (y_1, y_2, \dots)$3

In the reported implementation, Layerwise Relevance Propagation is the main attribution method used, adapted for transformers as AttnLRP; Input$y = (y_1, y_2, \dots)$4Gradient is also evaluated as an alternative (Komorowski et al., 30 Sep 2025).

A further variant is entropy-gated AGD. Because full AGD at every step is costly and can hurt text quality when the model is confident, AGD can be triggered only on high-entropy steps. If the entropy $y = (y_1, y_2, \dots)$5 of $y = (y_1, y_2, \dots)$6 exceeds threshold $y = (y_1, y_2, \dots)$7, AGD is applied; otherwise standard greedy decoding is used: $y = (y_1, y_2, \dots)$8 In the experiments, $y = (y_1, y_2, \dots)$9 is set to the 80th percentile of token-level entropy (Komorowski et al., 30 Sep 2025).

3. Regions of interest and attribution signals

The ROI determines what AGD is guided toward. For instruction following, the ROI is the input embeddings of instruction tokens,

$\mathcal{V}$0

so token selection is biased toward candidates whose logits are most attributable to the instruction portion of the prompt. For factuality, the ROI can be internal knowledge heads. For retrieval grounding, the ROI can be context document input embeddings or context-specific heads (Komorowski et al., 30 Sep 2025). This modularity is a defining property of AGD: the same decoding rule can target prompt constraints, parametric knowledge, or external context simply by changing the attributable region.

In related attribution-guided inference, the attribution signal itself may differ. SeA-GuiDe uses cross-attention weights in encoder-decoder transformers to track slot realization during generation (Juraska et al., 2021). Empirical analysis of T5 and BART reveals three universal cross-attention patterns: verbatim slot mention in the lowest cross-attention layer, paraphrased slot mention in intermediate layers, and unrealized slot mention at the final decoding step when the EOS token is generated (Juraska et al., 2021). Let $\mathcal{V}$1 denote cross-attention at layer $\mathcal{V}$2, head $\mathcal{V}$3, decoding step $\mathcal{V}$4, and input token $\mathcal{V}$5. The three mention-tracking components are: $\mathcal{V}$6

$\mathcal{V}$7

$\mathcal{V}$8

If $\mathcal{V}$9, slot $t$0 is considered mentioned, or missed for the third component at EOS (Juraska et al., 2021). The thresholds reported are $t$1 for verbatim mention, $t$2–$t$3 for paraphrased mention, and approximately $t$4 for unrealized mention (Juraska et al., 2021).

In attribution-guided query rewriting, the attributable objects are query tokens rather than next-token candidates. Integrated Gradients is used to estimate the contribution of each query token to the retriever score. For each query, the top-$t$5 retrieved documents are collected with $t$6, and token attributions are averaged across those documents: $t$7 The attribution scores are normalized on a per-query basis; high positive $t$8 indicates strong positive contribution, whereas low or negative $t$9 indicates weak or misleading contribution (Garouani et al., 12 Feb 2026). The LLM receives the original query, the list of tokens with attribution scores, and explicit instructions to preserve original user intent, preserve or emphasize high-attribution tokens, clarify or disambiguate low-attribution tokens, and avoid adding new concepts not implied by the original query (Garouani et al., 12 Feb 2026).

4. Empirical domains and reported performance

AGD has been evaluated on instruction following and factuality tasks in LLMs. On instruction following, the paper reports that AGD significantly boosts adherence, improving the overall success rate on Llama 3.1 from $p_\theta(y_t \mid x, y_{<t})$0 to $p_\theta(y_t \mid x, y_{<t})$1 (Komorowski et al., 30 Sep 2025). In the IHEval results summarized in the paper, Llama 3.1 with greedy decoding attains Prompt-Level Accuracy $p_\theta(y_t \mid x, y_{<t})$2, while AGD-LRP attains $p_\theta(y_t \mid x, y_{<t})$3; entropy-gated AGD-LRPe attains $p_\theta(y_t \mid x, y_{<t})$4 Prompt-Level Accuracy with Quality Score $p_\theta(y_t \mid x, y_{<t})$5 (Komorowski et al., 30 Sep 2025). On SysBench, AGD-LRPe is reported at Constraint Satisfaction Rate $p_\theta(y_t \mid x, y_{<t})$6, Instruction Satisfaction Rate $p_\theta(y_t \mid x, y_{<t})$7, and Session Satisfaction Rate $p_\theta(y_t \mid x, y_{<t})$8 (Komorowski et al., 30 Sep 2025).

For factuality, AGD is evaluated in both closed-book and open-book QA. In the closed-book setting, AGD-LRPh attains $p_\theta(y_t \mid x, y_{<t})$9 on TQA and $\mathcal{C}_t$0 on HPQA, compared with $\mathcal{C}_t$1 and $\mathcal{C}_t$2 under greedy decoding. In the open-book setting, AGD-LRPc attains $\mathcal{C}_t$3 on TQA, $\mathcal{C}_t$4 on NQ, and $\mathcal{C}_t$5 on HPQA, outperforming the reported greedy and CAD baselines in that table (Komorowski et al., 30 Sep 2025). The paper characterizes these gains as reductions in hallucinations and improvements in factual recall, with especially robust gains in open-book settings (Komorowski et al., 30 Sep 2025).

Related attribution-guided methods report improvements in other inference settings. In neural retrieval, attribution-guided LLM rewriting outperforms the original query, LLM-only rewriting, and hard attribution-based pruning across BEIR datasets and two retrievers (Garouani et al., 12 Feb 2026). For SPLADE, nDCG@10 gains are reported up to $\mathcal{C}_t$6 over the original query and up to $\mathcal{C}_t$7 over LLM-only rewriting; for TCT-ColBERT, the gains are reported up to $\mathcal{C}_t$8 and $\mathcal{C}_t$9, respectively (Garouani et al., 12 Feb 2026). The paper also states that LLM-only rewriting is inconsistent and sometimes worse than the original query, while hard pruning by retaining only tokens above the query-wise mean generally performs worst (Garouani et al., 12 Feb 2026).

In data-to-text generation, SeA-GuiDe is evaluated with T5 and BART on ViGGO, E2E, and MultiWOZ 2.1 (Juraska et al., 2021). On ViGGO, SeA-GuiDe yields Semantic Error Rate $k$0 for T5-small and $k$1 for BART-base, described as approximately $k$2 reduction versus beam search and $k$3–$k$4 reduction versus greedy search (Juraska et al., 2021). On E2E, SeA-GuiDe reduces SER to almost zero, with $k$5 reported on T5-small. On MultiWOZ, SER is reduced from approximately $k$6 under greedy decoding to as low as $k$7 (Juraska et al., 2021). The paper further reports that BLEU, METEOR, ROUGE-L, and CIDEr are maintained or marginally improved, and that runtime overhead is minor, at $k$8–$k$9 on GPU compared to standard beam search with reranking (Juraska et al., 2021).

5. Relation to other control methods and neighboring attribution interventions

AGD is explicitly contrasted with activation steering, logit steering, and contrastive decoding. The reported motivation is that direct manipulation of activations or logits can push models out of distribution, harming fluency or producing unnatural text, and may require probe training or additional hyperparameter tuning (Komorowski et al., 30 Sep 2025). By contrast, AGD only selects among plausible tokens from the model’s own output distribution, and the paper argues that this preserves the natural distribution and output quality (Komorowski et al., 30 Sep 2025). The same comparison is reflected in the factuality experiments against DoLA and in instruction-following experiments against CAD (Komorowski et al., 30 Sep 2025).

The relation between AGD and SeA-GuiDe is more specific. SeA-GuiDe is described as conceptually related to Attribution-Guided Decoding because it uses model attributions, specifically cross-attention distributions, to encourage faithful generation. However, SeA-GuiDe is described as more lightweight and plug-and-play: it extracts attribution information from unmodified models, infers semantic realization, and uses this externally for reranking, requiring no model retraining or architectural change (Juraska et al., 2021). A plausible implication is that attribution-guided methods span a spectrum from token-by-token next-token selection to sequence-level reranking.

A neighboring, though distinct, methodological concern is attributional validity. In non-invasive neural language decoding, the paper on source attribution argues that apparent gains can come from structural shortcuts, window-level stimulus-locked evidence, or cross-window contextual aggregation, and that reported decoding performance should therefore be source-attributed, not merely reported (Zhang et al., 23 May 2026). Its Group Context Bias (GCB) is an inference-time additive logit bias that pools sentence-consistent evidence across windows while leaving the base retrieval scores and candidate pool fixed; under fixed-window controls, R@1 shifts from $p_\theta(c) \geq \pi_{\min}$0 to $p_\theta(c) \geq \pi_{\min}$1 on Gwilliams and from $p_\theta(c) \geq \pi_{\min}$2 to $p_\theta(c) \geq \pi_{\min}$3 on MOUS (Zhang et al., 23 May 2026). The effect collapses under random-grouping perturbations and vanishes when local evidence is attenuated or near chance (Zhang et al., 23 May 2026). This neighboring work does not define AGD, but it sharpens a core methodological point shared with AGD research: inference-time control claims are more informative when the source of the gain is explicitly localized.

6. Limitations, failure modes, and terminological disambiguation

The main limitations reported for AGD are threefold. First, AGD has a selection limit: it cannot produce a token unless it is among the model’s top-$p_\theta(c) \geq \pi_{\min}$4 proposals. Second, it incurs computational cost because attribution requires a backward pass for each candidate at each intervention step. Third, ROI design is task-dependent and may be nontrivial (Komorowski et al., 30 Sep 2025). The entropy-gated variant is proposed precisely to mitigate quality degradation and reduce computational overhead by applying guidance only when the model is uncertain (Komorowski et al., 30 Sep 2025).

Related attribution-guided methods exhibit their own failure modes. In attribution-guided query rewriting, hard pruning fails because contextual tokens, even if scoring low, are necessary for intent; the paper concludes that attribution as soft guidance is effective precisely because the LLM can clarify or specialize low-scoring terms rather than simply dropping them (Garouani et al., 12 Feb 2026). In SeA-GuiDe, the reported limitations are that it does not detect arbitrary hallucinations unattributed to any slot and might struggle with duplicate slot mentions or certain Boolean slot realizations (Juraska et al., 2021). More broadly, the source-attribution study in brain-to-language retrieval shows that variable-length decoding and stimulus leakage can inflate performance, with Gaussian noise reaching $p_\theta(c) \geq \pi_{\min}$5 Rank@1 under variable-length decoding but collapsing to near chance once fixed-duration windows and stimulus-identity splits are enforced (Zhang et al., 23 May 2026). This suggests that attribution-guided control and attribution-guided auditing are complementary: one steers inference, the other verifies what the gain should be attributed to.

The acronym “AGD” also has an unrelated meaning in mathematics. In the paper on “discretizations of AGD flows,” AGD refers to Adler–Gelfand–Dikii flows on scalar differential operators, with Hamiltonian functionals

$p_\theta(c) \geq \pi_{\min}$6

and conjectured discretizations in $p_\theta(c) \geq \pi_{\min}$7 by intersecting one $p_\theta(c) \geq \pi_{\min}$8-dimensional subspace and $p_\theta(c) \geq \pi_{\min}$9 $c$0-dimensional subspaces (Beffa, 2011). That usage is entirely separate from Attribution-Guided Decoding.