Alignment-Aware Decoding (AAD)

• Alignment-Aware Decoding (AAD) is an inference-time method that leverages DPO and SFT models to reweight token probabilities, ensuring outputs align with optimized human preferences.
• AAD restricts token selection to a plausibility set based on DPO probabilities, mitigating reference bias and preventing degenerate outputs during generation.
• Empirical studies demonstrate that AAD consistently outperforms standard decoding methods, achieving higher reward scores across diverse models and datasets.

Alignment-Aware Decoding (AAD) is an inference-time alignment method for LLMs that uses a DPO-aligned model together with its original SFT/reference model to steer generation toward more preferred outputs during decoding, without any additional alignment training. In the narrow and formal sense, the term refers to the method introduced in "Alignment-Aware Decoding" (Berdoz et al., 30 Sep 2025). More broadly, the phrase can also describe decoding rules that make token or sequence selection explicitly sensitive to an alignment signal, but the literature is terminologically overloaded: in one large audio-LLM paper, AAD explicitly stands for Audio-Aware Decoding, not Alignment-Aware Decoding (Hsu et al., 8 Jun 2025), and in EEG/speech work the same acronym commonly denotes auditory attention decoding (Geirnaert et al., 12 Mar 2025).

1. Definition and terminological scope

The defining claim of AAD in the language-model alignment literature is that ordinary decoding does not fully exploit the alignment information already encoded by preference optimization. The method therefore shifts the intervention point from training to generation: the aligned model is left unchanged, and only the token-selection rule is replaced (Berdoz et al., 30 Sep 2025).

This narrow definition matters because nearby papers use closely related but non-identical names. "Audio-Aware Decoding" (Hsu et al., 8 Jun 2025) also uses the acronym AAD, but its method contrasts logits with real audio versus blank audio in order to reduce object hallucination in large audio-LLMs. That paper explicitly states that it does not propose a method named Alignment-Aware Decoding. Likewise, the EEG/speech literature uses AAD to mean auditory attention decoding, a distinct research area concerned with identifying the attended speaker from neural recordings rather than aligning text generation to human preferences (Geirnaert et al., 12 Mar 2025).

A plausible implication is that the phrase “alignment-aware decoding” has become a family resemblance term rather than a uniquely stable label. In the formal usage of (Berdoz et al., 30 Sep 2025), however, AAD denotes a specific decoding rule for DPO-trained models and should not be conflated with cross-modal grounding methods, neural attention decoding, or generic safety-time reranking.

2. Theoretical basis in preference optimization

AAD is motivated by the standard KL-constrained reward-maximization view of alignment. Given prompt distribution $\rho$, SFT model $\pi_{\mathrm{sft}}$, and latent reward $r^*$, the target policy is

$$\pi^*=\arg\max_\pi \mathbb{E}_{x\sim \rho,\, y\sim \pi(\cdot | x)}[r^*(x,y)] - \beta \, \mathrm{KL}\bigl( \pi(\cdot \mid x) \Vert \pi_{\mathrm{sft}}(\cdot \mid x)\bigr),$$

with analytical solution

$$\pi^*(y\mid x) = \frac{1}{Z(x)}\pi_{\mathrm{sft}}(y\mid x)\exp\left(\frac{1}{\beta}r^*(x,y)\right).$$

This form implies that the optimal aligned policy is reward-reweighted by the SFT policy and can therefore retain reference-model biases (Berdoz et al., 30 Sep 2025).

The paper makes this explicit by comparing two completions $y_1,y_2$ with $r^*(x,y_1)\ge r^*(x,y_2)$. From the optimal-policy form,

$$\log\frac{\pi^*(y_1\mid x)}{\pi^*(y_2\mid x)} = \underbrace{\log \frac{\pi_{\mathrm{sft}}(y_1\mid x)}{\pi_{\mathrm{sft}}(y_2\mid x)}}_{:=\Delta_{\mathrm{sft}}} + \frac{1}{\beta} \underbrace{\bigl(r^*(x,y_1)-r^*(x,y_2)\bigr)}_{:=\Delta_r}.$$

If $\Delta_{\mathrm{sft}}<-\frac{1}{\beta}\Delta_r$, then $\pi^*(y_1\mid x)\le \pi^*(y_2\mid x)$ even though $\pi_{\mathrm{sft}}$0 has higher reward. AAD targets precisely this gap between learned reward structure and ordinary next-token decoding (Berdoz et al., 30 Sep 2025).

The method is built on the DPO interpretation of the aligned model as an implicit reward model. The paper states the intended reward parameterization in words and formula as

$\pi_{\mathrm{sft}}$1

Using autoregressive factorization, this induces the token-level score

$\pi_{\mathrm{sft}}$2

The factor $\pi_{\mathrm{sft}}$3 is omitted because it does not affect token ranking. The theoretical claim is therefore that AAD performs an approximate greedy optimization of the same DPO-derived reward signal that alignment training has already encoded (Berdoz et al., 30 Sep 2025).

3. Decoding rule and implementation

AAD runs both the DPO model and the original SFT/reference model at every decoding step on the same context $\pi_{\mathrm{sft}}$4. A naive rule would greedily maximize $\pi_{\mathrm{sft}}$5 over the full vocabulary, but the paper argues that this is unstable. Grammatically necessary tokens may have small ratios even when they are appropriate, while tokens given tiny probability by $\pi_{\mathrm{sft}}$6 can receive inflated ratios and cause degenerate outputs (Berdoz et al., 30 Sep 2025).

To prevent that over-optimization, AAD restricts token choice to a min-$\pi_{\mathrm{sft}}$7 plausibility set under $\pi_{\mathrm{sft}}$8:

$\pi_{\mathrm{sft}}$9

The next token is then selected by

$r^*$0

This is the defining AAD rule (Berdoz et al., 30 Sep 2025).

Operationally, the procedure is fixed. At step $r^*$1, both $r^*$2 and $r^*$3 are run on the current prefix. The plausible set $r^*$4 is built using only $r^*$5. For each candidate token in that set, the score

$r^*$6

is computed, and the token with largest $r^*$7 is emitted. The paper describes the method as deterministic greedy optimization of an adjusted score, not random sampling from a token distribution, and distinguishes it from ordinary greedy decoding, ordinary sampling, beam-search reranking over complete sequences, and explicit reward-guided decoding with a separate reward model (Berdoz et al., 30 Sep 2025).

Implementation requires two forward passes per token, one through the DPO model and one through the SFT/reference model. In the experiments, the paper sets $r^*$8, and the <user> token is treated as EOS for user-assistant data. The required ingredients are the frozen SFT/reference model, the DPO-aligned model, token-level log probabilities from both, the prompt and current prefix, and the filtering hyperparameter $r^*$9 (Berdoz et al., 30 Sep 2025).

4. Empirical behavior and synthetic-data use

The experimental study is deliberately broad. The paper uses six preference datasets: Ultrachat, Argilla, OpenRLHF Mixture, HHRLHF, Nectar, and Skywork. For the first four, both oracle reward models and weaker picker reward models are trained. For Nectar and Skywork, the evaluation uses external off-the-shelf oracle reward models: Starling for Nectar and the Skywork reward model for Skywork. Prompts are split 90/10 into train/eval; the oracle reward model is trained on the full training split, while only 10% of the training split is used to train the picker and the DPO-aligned model. The evaluated base models are Llama 3B, Llama 8B, Qwen 0.6B, and Qwen 4B, with appendix results also including fully fine-tuned Llama 1B and 3B models (Berdoz et al., 30 Sep 2025).

The main quantitative claim is that AAD consistently gives the highest oracle reward across all datasets and models in the main table. Representative results illustrate the scale. On Ultrachat, Llama 8B with AAD reaches $$\pi^*=\arg\max_\pi \mathbb{E}_{x\sim \rho,\, y\sim \pi(\cdot | x)}[r^*(x,y)] - \beta \, \mathrm{KL}\bigl( \pi(\cdot \mid x) \Vert \pi_{\mathrm{sft}}(\cdot \mid x)\bigr),$$0, versus Greedy DPO $$\pi^*=\arg\max_\pi \mathbb{E}_{x\sim \rho,\, y\sim \pi(\cdot | x)}[r^*(x,y)] - \beta \, \mathrm{KL}\bigl( \pi(\cdot \mid x) \Vert \pi_{\mathrm{sft}}(\cdot \mid x)\bigr),$$1, Bo2 $$\pi^*=\arg\max_\pi \mathbb{E}_{x\sim \rho,\, y\sim \pi(\cdot | x)}[r^*(x,y)] - \beta \, \mathrm{KL}\bigl( \pi(\cdot \mid x) \Vert \pi_{\mathrm{sft}}(\cdot \mid x)\bigr),$$2, and EFT $$\pi^*=\arg\max_\pi \mathbb{E}_{x\sim \rho,\, y\sim \pi(\cdot | x)}[r^*(x,y)] - \beta \, \mathrm{KL}\bigl( \pi(\cdot \mid x) \Vert \pi_{\mathrm{sft}}(\cdot \mid x)\bigr),$$3. On Argilla, Llama 8B with AAD reaches $$\pi^*=\arg\max_\pi \mathbb{E}_{x\sim \rho,\, y\sim \pi(\cdot | x)}[r^*(x,y)] - \beta \, \mathrm{KL}\bigl( \pi(\cdot \mid x) \Vert \pi_{\mathrm{sft}}(\cdot \mid x)\bigr),$$4, versus $$\pi^*=\arg\max_\pi \mathbb{E}_{x\sim \rho,\, y\sim \pi(\cdot | x)}[r^*(x,y)] - \beta \, \mathrm{KL}\bigl( \pi(\cdot \mid x) \Vert \pi_{\mathrm{sft}}(\cdot \mid x)\bigr),$$5, $$\pi^*=\arg\max_\pi \mathbb{E}_{x\sim \rho,\, y\sim \pi(\cdot | x)}[r^*(x,y)] - \beta \, \mathrm{KL}\bigl( \pi(\cdot \mid x) \Vert \pi_{\mathrm{sft}}(\cdot \mid x)\bigr),$$6, and $$\pi^*=\arg\max_\pi \mathbb{E}_{x\sim \rho,\, y\sim \pi(\cdot | x)}[r^*(x,y)] - \beta \, \mathrm{KL}\bigl( \pi(\cdot \mid x) \Vert \pi_{\mathrm{sft}}(\cdot \mid x)\bigr),$$7. On OpenRLHF Mixture, Llama 8B reaches $$\pi^*=\arg\max_\pi \mathbb{E}_{x\sim \rho,\, y\sim \pi(\cdot | x)}[r^*(x,y)] - \beta \, \mathrm{KL}\bigl( \pi(\cdot \mid x) \Vert \pi_{\mathrm{sft}}(\cdot \mid x)\bigr),$$8, versus $$\pi^*=\arg\max_\pi \mathbb{E}_{x\sim \rho,\, y\sim \pi(\cdot | x)}[r^*(x,y)] - \beta \, \mathrm{KL}\bigl( \pi(\cdot \mid x) \Vert \pi_{\mathrm{sft}}(\cdot \mid x)\bigr),$$9, $$\pi^*(y\mid x) = \frac{1}{Z(x)}\pi_{\mathrm{sft}}(y\mid x)\exp\left(\frac{1}{\beta}r^*(x,y)\right).$$0, and $$\pi^*(y\mid x) = \frac{1}{Z(x)}\pi_{\mathrm{sft}}(y\mid x)\exp\left(\frac{1}{\beta}r^*(x,y)\right).$$1. On Skywork, Llama 8B reaches $$\pi^*(y\mid x) = \frac{1}{Z(x)}\pi_{\mathrm{sft}}(y\mid x)\exp\left(\frac{1}{\beta}r^*(x,y)\right).$$2, versus $$\pi^*(y\mid x) = \frac{1}{Z(x)}\pi_{\mathrm{sft}}(y\mid x)\exp\left(\frac{1}{\beta}r^*(x,y)\right).$$3, $$\pi^*(y\mid x) = \frac{1}{Z(x)}\pi_{\mathrm{sft}}(y\mid x)\exp\left(\frac{1}{\beta}r^*(x,y)\right).$$4, and $$\pi^*(y\mid x) = \frac{1}{Z(x)}\pi_{\mathrm{sft}}(y\mid x)\exp\left(\frac{1}{\beta}r^*(x,y)\right).$$5. Win rates against baselines are often high; for example, on OpenRLHF Mixture with Llama 8B, AAD beats Greedy DPO 89% of the time and EFT 67% of the time (Berdoz et al., 30 Sep 2025).

The paper also reports that AAD remains competitive against best-of-$$\pi^*(y\mid x) = \frac{1}{Z(x)}\pi_{\mathrm{sft}}(y\mid x)\exp\left(\frac{1}{\beta}r^*(x,y)\right).$$6 selection with strong evaluators. On Skywork, Bo$$\pi^*(y\mid x) = \frac{1}{Z(x)}\pi_{\mathrm{sft}}(y\mid x)\exp\left(\frac{1}{\beta}r^*(x,y)\right).$$7 reaches AAD-level performance only around $$\pi^*(y\mid x) = \frac{1}{Z(x)}\pi_{\mathrm{sft}}(y\mid x)\exp\left(\frac{1}{\beta}r^*(x,y)\right).$$8, requiring roughly twice as much compute. On Argilla, even $$\pi^*(y\mid x) = \frac{1}{Z(x)}\pi_{\mathrm{sft}}(y\mid x)\exp\left(\frac{1}{\beta}r^*(x,y)\right).$$9 fails to match AAD. In a DPO-$y_1,y_2$0 sensitivity analysis on Argilla, larger $y_1,y_2$1 degrades alignment for all methods, but AAD shows the smallest relative loss. The paper interprets this as evidence that a decoding rule tied more directly to reward-like log-ratios is less brittle than one relying on raw policy likelihoods (Berdoz et al., 30 Sep 2025).

A major secondary contribution is synthetic data generation. Starting from a $y_1,y_2$2 trained on only 10% of the original preference dataset, the method generates a chosen response with AAD and a rejected response using nucleus sampling from $y_1,y_2$3 with $y_1,y_2$4. These pairs are used as a synthetic preference dataset, after which DPO is retrained either from the SFT model or from the previously aligned model. The paper reports that iterative DPO with AAD-generated data “substantially improves alignment,” nearly closing the gap to a model trained on the full dataset despite starting from only 10% of the preference labels. A histogram analysis in the appendix shows AAD strongly outperforming best-of-2 on the original model with a 72.8% win rate; over iterations the gap narrows because the retrained model itself becomes better, especially under standard decoding (Berdoz et al., 30 Sep 2025).

5. Relation to adjacent decoding-time methods

AAD belongs to a wider class of inference-time alignment interventions, but its signal source is specific: the token-level log-probability ratio between a DPO policy and its SFT reference. A closely related safety-oriented method is Alignment-Enhanced Decoding (AED), which treats jailbreaks as generation-time conflicts between helpfulness and harmlessness. AED computes a Competitive Index from the Top-$y_1,y_2$5 candidate set, obtains post-alignment logits from same-model self-evaluation, and interpolates those logits with the original logits. The method is inference-time only and adaptively refines only the first 30 tokens, but it is a distinct algorithm with different inputs and gating logic (Liu et al., 2024).

Another related line is sequence-level reranking rather than token-level decoding. DARC, or Disagreement-Aware Alignment via Risk-Constrained Decoding, is a retraining-free inference-time method that reranks a fixed candidate pool by maximizing a KL-robust entropic satisfaction objective under preference heterogeneity. It is therefore alignment-aware in a risk-sensitive, sequence-level sense rather than in the local token-ratio sense of AAD (Zou et al., 9 Mar 2026).

In multimodal and systems-oriented work, “alignment-aware” can refer to preserving agreement between compressed or pruned computations and a full target model. ParallelVLM, for example, introduces “Visual Alignment Aware Parallel Speculative Decoding,” where the alignment problem is draft-target agreement under long video contexts. Its alignment-aware component, UV-Prune, ranks visual tokens by cumulative layerwise increases in vision-text cosine similarity rather than by raw attention magnitude (Kong et al., 20 Mar 2026). This suggests a broader usage in which “alignment-aware” may refer to preference alignment, cross-modal grounding, or draft-target consistency, but the formal AAD method of (Berdoz et al., 30 Sep 2025) remains specifically tied to DPO/SFT reward ratios.

Finally, the acronym itself requires care. In "Reducing Object Hallucination in Large Audio-LLMs via Audio-Aware Decoding" (Hsu et al., 8 Jun 2025), AAD favors tokens whose probability increases when actual audio is present rather than blank audio. That method can be interpreted as encouraging cross-modal grounding, but the paper is explicit that its AAD stands for Audio-Aware Decoding, not Alignment-Aware Decoding (Hsu et al., 8 Jun 2025).

6. Limitations and open questions

AAD has two explicit operational limitations. First, it requires access to the original SFT model at inference; if only the final aligned model is available, AAD cannot be run as defined. Second, it needs two forward passes per token, increasing latency and memory cost. The paper notes that this compute is comparable to stronger inference-time baselines such as Bo2 and EFT, but it remains a real serving-time constraint (Berdoz et al., 30 Sep 2025).

The method is also narrow in its training dependency. It is designed around the standard DPO setup and reuses the DPO/SFT model pair as an implicit reward mechanism. This creates immediate practical simplicity—no separate reward model, no PPO stage, no extra training—but also leaves open how well the rule generalizes beyond DPO to ORPO-, KTO-, or IPO-like regimes, where reference dependence changes (Berdoz et al., 30 Sep 2025).

Several theoretical and empirical questions remain open. The paper does not present a theorem or formal sequence-level optimality guarantee for tokenwise ratio maximization. It raises the possibility of adaptive token filtering beyond fixed $y_1,y_2$6, asks whether entropy-aware gating should be integrated into the core greedy variant rather than treated as a beam-search stabilizer, and does not settle how AAD trades off alignment against diversity, creativity, and calibration in open-ended generation. A plausible implication is that AAD is best understood as a strong inference-time exploitation rule for already learned preference structure, rather than as a replacement for preference modeling itself (Berdoz et al., 30 Sep 2025).

Taken together, these features make Alignment-Aware Decoding a specific and technically well-defined response to a familiar alignment problem: even when preference optimization has succeeded at training time, raw next-token decoding may still select completions favored by the SFT prior rather than by the learned reward. AAD addresses that gap by decoding directly on the DPO/SFT log-ratio, restricted to tokens that remain plausible under the aligned policy.