Bid-Aware Decoding: Controlled Inference

• Bid-Aware Decoding is an inference-time paradigm that augments standard likelihood with auxiliary bids to optimize token selection.
• It is applied across language model alignment, robot action chunking, and generative recommendation to balance objectives like coherence and revenue.
• By reshaping the decoding process without retraining the model, it achieves improved performance metrics and targeted control over outputs.

Searching arXiv for the cited papers and related formulations to ground the article. Bid-Aware Decoding is an inference-time paradigm in which candidate tokens, action chunks, or item identifiers are selected using an auxiliary steering signal in addition to a model’s native likelihood. Recent work uses the term in three distinct but structurally related senses: as an implicit alignment reward in LLM decoding, as a bidirectional score over past and future plans in chunked robot policies, and as real-time monetary bid modulation in generative recommendation. The shared pattern is test-time intervention: the trained model is left unchanged, while decoding is reshaped to optimize alignment, temporal consistency, reactivity, or revenue under an explicit scoring rule (Berdoz et al., 30 Sep 2025, Liu et al., 2024, Jiang et al., 23 Mar 2026).

1. General formulation

Across these settings, bid-aware decoding replaces pure maximum-likelihood selection with a composite rule. One generic form is

$$\log p_{\text{new}}(y_t \mid y_{<t}, x) = \log p_{\text{base}}(y_t \mid y_{<t}, x) + \lambda \cdot b(y_{\le t}, x),$$

where $p_{\text{base}}$ is the base distribution, $b$ is an auxiliary bid or reward signal, and $\lambda$ controls steering strength. A second form is score-only decoding over a restricted candidate set, in which the auxiliary signal determines the argmax directly. A third form is sample-based search, where a finite set of candidates is generated first and then reranked by a composite objective (Berdoz et al., 30 Sep 2025, Liu et al., 2024, Jiang et al., 23 Mar 2026).

Setting	Base object	Bid-aware signal
LLM alignment	token probabilities from $\pi_{\mathrm{dpo}}$ and $\pi_{\mathrm{sft}}$	$$\nu = \log \frac{\pi_{\mathrm{dpo}}}{\pi_{\mathrm{sft}}}$$
Robot action chunking	sampled action chunks from a strong policy	$L_B + L_F$
Generative recommendation	control-token and semantic-ID logits	$\lambda \log(1+b_{\max})$, $\lambda \log(1+\mathcal{B}(c))$

The implementations differ in where the auxiliary signal enters. Alignment-Aware Decoding uses a filtered candidate set and then takes a hard argmax of the alignment log-ratio. Bidirectional Decoding samples from strong and weak policies and minimizes a sum of backward coherence and forward contrast. GEM-Rec performs logit shaping at the slot level and semantic-prefix level, then runs beam search. This suggests an umbrella usage in which a “bid” is any inference-time steering score that is external to the base decoding rule, whether it represents preference alignment, trajectory compatibility, or monetary value.

2. Alignment-aware decoding in LLMs

Alignment-Aware Decoding (AAD) assumes a supervised fine-tuned model $p_{\text{base}}$0 and a DPO-aligned model $p_{\text{base}}$1. DPO theory yields an implicit reward model through the log-likelihood ratio

$p_{\text{base}}$2

AAD makes this structure explicit at inference time by defining the token-level reward

$p_{\text{base}}$3

filtering to a plausible set

$p_{\text{base}}$4

and decoding with

$p_{\text{base}}$5

The search space is therefore determined by $p_{\text{base}}$6, while selection is determined by the alignment log-ratio. The paper default is $p_{\text{base}}$7, and the basic greedy implementation requires two forward passes per step, one for each model (Berdoz et al., 30 Sep 2025).

The theoretical motivation comes from KL-constrained preference optimization. Training-time preference optimization learns a policy that trades off reward and proximity to $p_{\text{base}}$8, whereas AAD uses $p_{\text{base}}$9 “exactly as intended during training, that is, in combination with $b$0 as an approximate token-level reward function.” The paper emphasizes that $b$1 inherits the bias of $b$2, so training-time PO is a compromise. AAD “undoes” some of that compromise at inference by discarding the direct $b$3 term in token selection and optimizing the reward-like ratio over high-probability tokens. In this sense, AAD is an inference-time analogue of implicit reward optimization.

Empirically, the method is evaluated on LLaMA-3.2 variants (1B, 3B, 8B) and Qwen (0.6B, 4B), with either LoRA or full finetuning, over Ultrachat, Argilla, OpenRLHF Mixture, HHRLHF, Nectar, and Skywork. The reported metrics are average oracle reward $b$4 and pairwise win-rate $b$5. Across all datasets and model sizes, AAD achieves the highest oracle reward. Representative examples include Skywork on Llama 8B LoRA, where mean reward is 13.64 for greedy DPO, 15.57 for EFT, and 19.27 for AAD; and Argilla on Llama 3B LoRA, where the corresponding values are 2.55, 3.16, 4.65, and 5.90. The paper also reports that AAD improves difficult HHRLHF settings, is relatively flat across data fractions in Skywork, degrades less when DPO $b$6 is suboptimal, and can generate high-quality synthetic preference pairs for iterative DPO when only 10% of real labels are available. In that synthetic-data pipeline, the chosen response is generated by AAD and the rejected response by nucleus sampling from $b$7 with top-$b$8.

3. Bidirectional decoding in robot learning

In robot behavioral cloning with action chunking, the policy predicts a sequence

$b$9

then executes only an action horizon $\lambda$0 before replanning. Longer horizons can better capture persistent latent variables such as handedness, “idle” pauses, and multi-step strategies, but they reduce reactivity to unexpected states. The paper formalizes this as a consistency–reactivity trade-off: larger $\lambda$1 preserves temporal coherence and latent strategy information, while smaller $\lambda$2 improves closed-loop adaptation. The authors show that deterministic environments favor longer horizons, highly stochastic environments favor shorter horizons, and intermediate regimes can be non-monotonic (Liu et al., 2024).

Bidirectional Decoding (BID) is a test-time inference algorithm that seeks to resolve this trade-off without retraining the policy. At each timestep it samples $\lambda$3 candidate chunks $\lambda$4 from a strong policy $\lambda$5 and $\lambda$6 candidates $\lambda$7 from a weak policy $\lambda$8. It then scores each candidate with two terms. The backward coherence term is

$\lambda$9

which penalizes deviation from the previously selected chunk $\pi_{\mathrm{dpo}}$0. The forward contrast term is

$\pi_{\mathrm{dpo}}$1

where $\pi_{\mathrm{dpo}}$2 and $\pi_{\mathrm{dpo}}$3 are the $\pi_{\mathrm{dpo}}$4 strong-policy and weak-policy samples with smallest backward loss. Selection is

$\pi_{\mathrm{dpo}}$5

Only the first action is executed, so BID operates with closed-loop $\pi_{\mathrm{dpo}}$6, while the chosen chunk is stored as decision memory for the next timestep.

The algorithm is architecture-agnostic and is demonstrated with Diffusion Policy and VQ-BeT. Typical experimental choices are prediction length $\pi_{\mathrm{dpo}}$7 for Diffusion Policy, batch size $\pi_{\mathrm{dpo}}$8, and mode size $\pi_{\mathrm{dpo}}$9. Loss computation is roughly $\pi_{\mathrm{sft}}$0, and the sampling and scoring are embarrassingly parallel across candidates on GPU. On Push-T with VQ-BeT, the reported inference time per step is 13.2 ms for vanilla $\pi_{\mathrm{sft}}$1, 25.2 ms for BID with $\pi_{\mathrm{sft}}$2, and 26.8 ms for BID with $\pi_{\mathrm{sft}}$3. On seven simulation benchmarks, BID yields an average relative improvement over vanilla of $\pi_{\mathrm{sft}}$4, and BID+EMA reaches $\pi_{\mathrm{sft}}$5 relative improvement. Under temporally correlated Gaussian action noise in Push-T, BID closed-loop attains 54.4, 45.3, and 31.7 success for noise levels 0.0, 1.0, and 1.5, compared with 48.9, 38.3, and 29.5 for vanilla closed-loop. In a dynamic cup-picking task, the paper reports that BID attains $\pi_{\mathrm{sft}}$6 the success rate of all other baselines in the dynamic case.

4. Bid-aware decoding in generative recommendation

In generative recommendation, Bid-Aware Decoding appears in GEM-Rec, a semantic-ID model that unifies organic and sponsored content in one autoregressive sequence. Each item $\pi_{\mathrm{sft}}$7 has a hierarchical semantic code $\pi_{\mathrm{sft}}$8, and GEM-Rec introduces control tokens

$\pi_{\mathrm{sft}}$9

to factorize each interaction segment as

$$\nu = \log \frac{\pi_{\mathrm{dpo}}}{\pi_{\mathrm{sft}}}$$0

This decouples “whether to show an ad” from “which item to show.” Historical logs contain successful interactions that already reflect platform filtering and user interaction, so the base model learns a feasibility policy rather than an explicit monetization objective (Jiang et al., 23 Mar 2026).

Bid-Aware Decoding modifies logits at two levels. At the slot level, if $$\nu = \log \frac{\pi_{\mathrm{dpo}}}{\pi_{\mathrm{sft}}}$$1 is the eligible sponsored set and

$$\nu = \log \frac{\pi_{\mathrm{dpo}}}{\pi_{\mathrm{sft}}}$$2

then the ad flag is boosted by

$$\nu = \log \frac{\pi_{\mathrm{dpo}}}{\pi_{\mathrm{sft}}}$$3

while $$\nu = \log \frac{\pi_{\mathrm{dpo}}}{\pi_{\mathrm{sft}}}$$4 is unchanged. At the item level, once $$\nu = \log \frac{\pi_{\mathrm{dpo}}}{\pi_{\mathrm{sft}}}$$5 has been chosen, each semantic token $$\nu = \log \frac{\pi_{\mathrm{dpo}}}{\pi_{\mathrm{sft}}}$$6 is modulated by a prefix-aware bid table

$$\nu = \log \frac{\pi_{\mathrm{dpo}}}{\pi_{\mathrm{sft}}}$$7

giving

$$\nu = \log \frac{\pi_{\mathrm{dpo}}}{\pi_{\mathrm{sft}}}$$8

Equivalently,

$$\nu = \log \frac{\pi_{\mathrm{dpo}}}{\pi_{\mathrm{sft}}}$$9

The decoding procedure is therefore two-stage per slot: first choose $L_B + L_F$0 or $L_B + L_F$1, then run standard beam search for organic content or bid-modulated beam search for ads. Organic semantic logits are never modulated once $L_B + L_F$2 is selected.

The central theoretical guarantee is allocation monotonicity. For the allocation rule

$L_B + L_F$3

Proposition 1 states that the exposure probability $L_B + L_F$4 is non-decreasing in item $L_B + L_F$5’s own bid. The proof relies on a fixed eligible set, a one-to-one mapping between items and semantic IDs, deterministic beam search with fixed tie-breaking, and prefix aggregation by subtree maximum. The model also has an Organic Integrity property: for any context $L_B + L_F$6, the relative ranking of two organic items is invariant to $L_B + L_F$7. Empirically, the method is evaluated on Steam, Amazon Beauty, Amazon Sports, and Amazon Toys. On Steam, GEM-Rec with $L_B + L_F$8 has Ad Rate 2.5% and Revenue $L_B + L_F$9, while $\lambda \log(1+b_{\max})$0 increases Ad Rate to 4.7% and Revenue to $\lambda \log(1+b_{\max})$1; Total NDCG decreases from 0.1411 to 0.1381, while O.NDCG@10 is 0.1468 at $\lambda \log(1+b_{\max})$2 and 0.1467 at $\lambda \log(1+b_{\max})$3. Under a “Bid Shock” in which 5% of inventory has bids multiplied by 10, $\lambda \log(1+b_{\max})$4 or higher raises the High-Value Share among ads to 80–99%, and on Steam the paper reports up to $\lambda \log(1+b_{\max})$5 revenue at $\lambda \log(1+b_{\max})$6.

5. Common patterns, misconceptions, and limits

A recurring misconception is that bid-aware decoding necessarily refers to monetary bidding. In current usage, that is only one case. In AAD, the steering quantity is the token-level log-ratio $\lambda \log(1+b_{\max})$7, derived from the implicit reward structure of DPO. In BID, the steering quantity is a composite score over backward coherence and forward contrast. In GEM-Rec, the steering quantity is an actual advertiser bid injected into logits. A plausible implication is that the term “bid” is best understood operationally: it denotes an auxiliary score that competes with or augments the base model score at inference time, not a fixed semantic category.

A second misconception is that these methods are simply alternative samplers over an unchanged single-model distribution. AAD requires access to both $\lambda \log(1+b_{\max})$8 and $\lambda \log(1+b_{\max})$9, with approximately double per-token runtime versus standard greedy decoding. BID requires multiple samples from a strong policy and a weak policy at every timestep, plus a previous-decision memory $\lambda \log(1+\mathcal{B}(c))$0. GEM-Rec requires an eligible candidate set, a prefix bid table, and a two-stage slot/content decoding procedure. The extra flexibility is therefore purchased with serving-time structure and compute.

The guarantees are also domain-specific rather than universal. AAD has no formal safety guarantee, remains bounded by the quality of the DPO reward signal, can over-steer if $\lambda \log(1+\mathcal{B}(c))$1 is too low, and is locally greedy. BID uses hand-designed metrics $\lambda \log(1+\mathcal{B}(c))$2 and $\lambda \log(1+\mathcal{B}(c))$3, depends on a weak policy checkpoint, and may be difficult to deploy on very low-cost or very high-frequency platforms. GEM-Rec proves monotonicity only under a fixed eligible set, deterministic beam search, and single-slot allocation, does not guarantee DSIC, and optimizes immediate revenue rather than long-term user satisfaction (Berdoz et al., 30 Sep 2025, Liu et al., 2024, Jiang et al., 23 Mar 2026).

6. Relation to adjacent methods and likely extensions

Each of the three strands is positioned against neighboring inference-time methods. AAD is compared with greedy decoding from $\lambda \log(1+\mathcal{B}(c))$4 or $\lambda \log(1+\mathcal{B}(c))$5, Best-of-$\lambda \log(1+\mathcal{B}(c))$6 sampling with a reward model, EFT-style decoding, and reward-guided beam search; it is presented as cheaper than large-$\lambda \log(1+\mathcal{B}(c))$7 Best-of-$\lambda \log(1+\mathcal{B}(c))$8 and simpler than explicit reward-model search. BID is compared with vanilla one-sample execution, receding horizon execution, warmstart, temporal ensembling, and MPC-like planning; its distinctive feature is that it couples successive chunks through $\lambda \log(1+\mathcal{B}(c))$9 while remaining closed-loop. GEM-Rec is contrasted with GSP-style auctions, learning-to-rank with bid features, post-hoc reranking, contextual bandits, and RL-based ad insertion; its distinctive claim is that bids enter the generative decoder itself, with a monotonicity proof at the allocation level.

The papers also point toward a broader design template. One may choose a candidate set from a base model, define an auxiliary bid function, and then either reweight probabilities or perform score-only search. AAD explicitly proposes replacing $p_{\text{base}}$00 with a learned reward model, a safety classifier score, a personalization score, or an economic bid from an external system. BID proposes better scoring functions, including value functions or external reward models, and integration with MPC or model-based look-ahead. GEM-Rec proposes DSIC mechanisms, multi-slot auctions, separate $p_{\text{base}}$01 and $p_{\text{base}}$02, and integration with production-scale retrieval. Taken together, these directions suggest that bid-aware decoding is emerging less as a single algorithm than as a reusable inference-time pattern for controlled generation under auxiliary objectives (Berdoz et al., 30 Sep 2025, Liu et al., 2024, Jiang et al., 23 Mar 2026).