BoNBoN: Distilling Best-of-N for LLM Alignment

Updated 5 July 2026

BoNBoN is a method that amortizes Best-of-N sampling by distilling optimal responses from multiple completions into a single-pass model.
It leverages a dual training approach, combining supervised fine-tuning on best-of-N winners with contrastive learning through Iterative Preference Optimization.
BoNBoN achieves near-optimal win-rates against the base model while significantly reducing the inference-time computational cost.

BoNBoN is a term used primarily in recent large-language-model alignment literature for a procedure that amortizes Best-of- $n$ (BoN) sampling into a single-pass policy, and more broadly for BoN-family methods that emulate or distill inference-time BoN behavior (Gui et al., 2024, Verdun et al., 6 May 2025). In its most specific sense, BoNBoN denotes the method introduced in "BoNBoN Alignment for LLMs and the Sweetness of Best-of-n Sampling," which trains a model to mimic the distribution induced by selecting the best completion among $n$ samples from a base policy (Gui et al., 2024). The term is not fully uniform across the literature: some papers use it to denote a broader family of BoN-inspired alignment methods, while one unrelated reinforcement-learning context interprets it as shorthand for bonus-based exploration rather than language-model alignment (Verdun et al., 6 May 2025, Taïga et al., 2021).

1. Terminological scope

In the LLM literature, BoNBoN is most precisely associated with a training method that replaces inference-time reranking by a learned single-sample policy. The underlying motivation is that BoN sampling is often effective for alignment but requires drawing $n$ samples, ranking them with a reward model or preference signal, and returning only the best one, which imposes a substantial inference-time cost (Gui et al., 2024).

A broader usage treats “BoNBoN” as shorthand for the Best-of- $N$ alignment family, especially methods that distill or emulate BoN-selected outputs into a standalone model. This broader sense appears in work on Soft Best-of- $n$ , which places BoNBoN alongside related approaches such as Bond, variational BoN, TreeBoN, and speculative BoN (Verdun et al., 6 May 2025). In that framing, the common thread is not a single objective but the attempt to recover BoN-like preference optimization without paying the full multi-sample decoding cost at deployment.

The term also has a separate context-specific meaning outside LLM alignment. In Atari exploration research, “BoNBoN” is used as a natural shorthand for bonus-based exploration methods such as pseudo-counts, Intrinsic Curiosity Module (ICM), and Random Network Distillation (RND). That usage refers to shaped rewards of the form

$r'_t = r_t + \beta\, b(s_t,a_t),$

and is terminologically distinct from BoNBoN alignment for LLMs (Taïga et al., 2021).

2. Best-of- $n$ foundations and the common tilting view

BoNBoN is built on the structure of Best-of- $n$ sampling. Given a prompt $x$ , a base or reference model $\pi_0(y\mid x)$ generates $n$ 0 i.i.d. candidates, a reward function $n$ 1 scores them, and the highest-scoring candidate is returned. Under the continuity assumptions used in the BoNBoN analysis, the induced density of the BoN policy is

$n$ 2

where $n$ 3 is the CDF of $n$ 4 under $n$ 5 (Gui et al., 2024).

The BoNBoN paper places BoN and standard alignment methods inside a common class of reward-weighted tiltings of the base model: $n$ 6 with $n$ 7 increasing and nonnegative. In the same framework, RLHF and DPO target the exponential tilt

$n$ 8

under a correctly specified Bradley–Terry model, whereas BoN corresponds to a power tilt in the reward CDF rather than a direct exponential tilt in the reward itself (Gui et al., 2024).

This common formulation is important because it makes BoNBoN more than a heuristic imitation scheme. The method exploits the analytic form of the BoN-induced distribution and uses that structure to derive a training objective tailored to BoN winners and losers, rather than optimizing a generic KL-regularized reward objective (Gui et al., 2024).

3. BoNBoN Alignment as a training procedure

The BoNBoN method trains a single-pass model $n$ 9 to mimic the BoN sampling distribution $n$ 0. The original procedure begins with a reference model $n$ 1 obtained by standard supervised fine-tuning on human-preferred data. For each prompt $n$ 2, it samples $n$ 3 completions from $n$ 4, scores them with a reward model, identifies the best completion $n$ 5 and the worst completion $n$ 6, and uses these order-statistic samples to construct the training signal (Gui et al., 2024).

BoNBoN combines two losses. The first is SFT-BoN, a supervised term on BoN winners: $n$ 7 The second is IPO-BoN, a contrastive term that matches the model’s log-likelihood ratio between best and worst samples to the BoN-implied ratio, with an analytically derived offset. The combined objective is

$n$ 8

The key calibration constant is

$n$ 9

This analytic $N$ 0 is one of the method’s defining features, because it removes the need to tune the IPO scale by sweep-based hyperparameter search (Gui et al., 2024).

The paper’s interpretation of the two terms is explicit. SFT-BoN alone has $N$ 1 as a minimizer in the infinite-data limit but is data-inefficient by itself. IPO-BoN uses loser information to improve data efficiency and enforce the correct winner-versus-loser likelihood ratio. The combination is intended to prevent “cheating” behavior that could arise from ratio matching alone and to keep both losses aligned to the same target distribution (Gui et al., 2024).

In the reward-free code-alignment study, BoNBoN is described in similar procedural terms but without reproducing the original loss in LaTeX. There it is characterized as a “dual training strategy: supervised fine-tuning (SFT) on the best responses from n-sample generations, combined with Iterative Preference Optimization (IPO) training that contrasts the best versus worst responses from the same sample sets,” and as a “data distribution-matching approach” that “trains models to directly mimic the best-of-n distribution” (Uddin et al., 27 Jun 2026).

4. Theoretical characterization

A central result of the BoNBoN paper is that BoN is essentially optimal in the trade-off between win-rate against the base model and KL divergence from the base model, within the class of reward-weighted tiltings described above. The formal optimization problem fixes a KL budget and maximizes win-rate against $N$ 2; the resulting optimum is an exponential tilt by the reward CDF: $N$ 3 Within this analysis, BoN lies very close to the optimal Pareto frontier. The paper states that the maximum win-rate gap, at $N$ 4, is less than one percentage point, and that the gap shrinks for larger $N$ 5 (Gui et al., 2024).

For BoN itself, the context-conditional win-rate and KL divergence admit closed forms: $N$ 6 These values are constants across prompts under the paper’s assumptions, and they also characterize the overall win-rate and KL on any prompt distribution $N$ 7 (Gui et al., 2024).

This theory motivates BoNBoN’s design. If BoN already sits essentially on the optimal win-rate-versus-KL frontier, then a model trained to mimic the BoN distribution inherits a target that is already near-optimal for that specific trade-off. The theoretical argument is therefore not merely that BoNBoN approximates a strong heuristic; it approximates a distribution that the paper argues is itself essentially optimal for win-rate at fixed KL (Gui et al., 2024).

The BoNBoN paper also gives an idealized benchmark for $N$ 8: $N$ 9 This serves as the reference frontier point for the paper’s experiments using $n$ 0 (Gui et al., 2024).

5. Empirical evidence across domains

The empirical literature treats BoNBoN both as a standalone alignment method and as a baseline for other BoN-inspired objectives. The major reported settings are summarized below.

Study	Setting	Reported BoNBoN pattern
(Gui et al., 2024)	Anthropic Helpful and Harmless; OpenAI TL;DR	Better win-rate vs off-target trade-offs than DPO/IPO; SFT-BoN and IPO-BoN alone underperform the combined objective
(Amini et al., 2024)	IMDB controlled generation baseline comparison	Used as a concurrent BoN-inspired baseline; vBoN exceeds it on reported Pareto-front frequency
(Uddin et al., 27 Jun 2026)	Code alignment across five open models, PTA and FTA	Larger relative gains from pretrained-to-aligned pathways; mixed or degrading results can occur from finetuned-to-aligned pathways

In the original BoNBoN paper, experiments use single-turn dialogue generation on Anthropic Helpful and Harmless and summarization on OpenAI TL;DR. The setup uses Pythia-2.8b fine-tuned with SFT on human-preferred completions as the base model, draws $n$ 1 completions per prompt from $n$ 2, uses the off-the-shelf preference scorer OpenAssistant/reward-model-deberta-v3-large-v2 to relabel pairs, and trains with RMSprop at learning rate $n$ 3 on 6 GPUs, evaluating the 20k-step checkpoint. The reported finding is that BoNBoN achieves much better win-rate versus off-target deviation trade-offs than DPO or IPO on both the original datasets and best/worst-of-8 datasets; baselines matched to BoNBoN win-rate produce large off-target drift, including cases where average response length nearly doubles, whereas BoNBoN minimally changes length and other off-target attributes (Gui et al., 2024).

The same paper reports that SFT-BoN and IPO-BoN alone underperform BoNBoN at all tested $n$ 4 values, which it interprets as evidence that both level-matching and ratio-matching are necessary. It also describes BoNBoN as nearly hyperparameter-free: $n$ 5 is fixed analytically, and $n$ 6 is chosen so that the two losses contribute roughly equally, with $n$ 7 reported in the main experiments (Gui et al., 2024).

In variational Best-of- $n$ 8 work, BoNBoN is included as a concurrent baseline rather than the primary method. On IMDB positive-sentiment control with GPT2-IMDB and a DistilBERT IMDB reward model, the paper reports Pareto-front shares of roughly 50% vBoN, 42% DPO, 4% BoNBoN, and 4% PPO for win-rate vs KL, and roughly 50% vBoN, 33% PPO, 10% DPO, and 7% BoNBoN for reward vs KL. In that setting, the reported conclusion is that vBoN comes closest to BoN’s performance while recovering the factor- $n$ 9 inference speedup (Amini et al., 2024).

The code-alignment study extends BoNBoN to a reward-free setting using SelfCodeAlign-derived preference pairs across Meta-Llama-3-8B, Qwen2.5-Coder-7B, CodeLlama-7B, deepseek-coder-1.3B, and deepseek-coder-7B, each evaluated in pretrained-to-aligned (PTA) and finetuned-to-aligned (FTA) pathways. The strongest reported BoNBoN gains occur in PTA settings. For Meta-Llama-3-8B PTA, HumanEval rises from 0.372 to 0.518, HumanEval+ from 0.323 to 0.476, and EvalPerf from 6.5% to 11.8%. For CodeLlama-7B PTA, CODAL average rises from 0.48 to 0.84, a reported +75% relative improvement. For Qwen2.5-Coder-7B-Instruct FTA, EvalPerf rises from 71.1% to 73.5%, which the paper identifies as the best across all models. The same study also reports degradations: Meta-Llama-3-8B-Instruct falls from 0.610 to 0.543 on HumanEval under BoNBoN, and CodeLlama-7B-Instruct CODAL average falls from 0.60 to 0.37 (Uddin et al., 27 Jun 2026).

Across those code experiments, the paper’s high-level conclusion is that pretrained-to-aligned pathways achieve larger improvements over pretrained baselines, while finetuned-to-aligned pathways offer smaller improvements or, in some cases, degradation. It further states that non-functional code quality improves more reliably than functional correctness under BoNBoN alignment, with readability and style often showing the largest relative gains (Uddin et al., 27 Jun 2026).

6. Relation to adjacent methods, misconceptions, and limitations

BoNBoN is not synonymous with all BoN-style decoding methods. A common source of confusion is to treat it as a generic label for any reward-guided reranking or best-of- $r'_t = r_t + \beta\, b(s_t,a_t),$ 0 regularization. The literature draws sharper distinctions.

First, BoNBoN is distinct from MBR-BoN, also called WD-RBoN, which is an inference-time decoding rule rather than a distillation objective. MBR-BoN selects

$r'_t = r_t + \beta\, b(s_t,a_t),$ 1

where the MBR term is the expected semantic distance from a candidate to samples from the reference policy. The corresponding paper explicitly states that it does not reference a method explicitly named “BoNBoN” (Jinnai et al., 2024).

Second, BoNBoN is distinct from Soft Best-of- $r'_t = r_t + \beta\, b(s_t,a_t),$ 2. Soft BoN replaces hard argmax selection with a softmax over sampled rewards controlled by a temperature $r'_t = r_t + \beta\, b(s_t,a_t),$ 3, interpolating between the base model and hard BoN. Its main contribution is a sharp $r'_t = r_t + \beta\, b(s_t,a_t),$ 4 convergence guarantee in KL and relative reward to the exponentially tilted target distribution, not a distillation objective of the BoNBoN type (Verdun et al., 6 May 2025).

Third, BoNBoN is distinct from variational BoN. vBoN derives the exact BoN-induced distribution $r'_t = r_t + \beta\, b(s_t,a_t),$ 5, then fine-tunes a model by minimizing the backward KL divergence

$r'_t = r_t + \beta\, b(s_t,a_t),$ 6

The vBoN paper explicitly notes that BoNBoN uses a pairwise IPO-like loss derived from Best-of- $r'_t = r_t + \beta\, b(s_t,a_t),$ 7 versus Worst-of- $r'_t = r_t + \beta\, b(s_t,a_t),$ 8 likelihood gaps, whereas vBoN directly targets the full BoN distribution via backward KL (Amini et al., 2024).

A further misconception is that the name implies “BoN applied twice” or a hierarchical best-of-best procedure. The variational BoN paper notes that such a reading might seem natural, but the actual BoNBoN method instead uses a pairwise objective plus NLL on BoN winners; it is not a second-stage BoN sampler (Amini et al., 2024).

The main limitations reported for BoNBoN are also consistent across studies. In the original alignment setting, the method still depends on the quality of the reward model used to identify BoN winners and losers, and regular KL estimates may fail to capture salient off-target changes such as response length, even though BoNBoN empirically minimizes such drift relative to baselines (Gui et al., 2024). In the code-alignment study, performance depends strongly on the starting point: pretrained models are more plastic and can improve more, while finetuned models may exhibit diminishing returns or degradation, with early SFT-stage degradation acting as a warning sign for final failure (Uddin et al., 27 Jun 2026).

Taken together, these results position BoNBoN as a specific and technically structured answer to a concrete problem in BoN alignment: how to preserve the empirical strength of best-of- $r'_t = r_t + \beta\, b(s_t,a_t),$ 9 selection while removing its deployment-time sampling cost. The term can refer more broadly to BoN-family distillation or emulation strategies, but its canonical meaning remains the SFT-plus-IPO procedure introduced for single-pass imitation of the BoN distribution (Gui et al., 2024, Verdun et al., 6 May 2025).