---
title: 'BoNBoN: Distilling Best-of-N for LLM Alignment'
url: https://www.emergentmind.com/topics/bonbon
type: topic
---

# BoNBoN: Distilling Best-of-N for LLM Alignment

BoNBoN is a term used primarily in recent large-language-model alignment literature for a procedure that amortizes Best-of-$n$ (BoN) sampling into a single-pass policy, and more broadly for BoN-family methods that emulate or distill inference-time BoN behavior [2406.00832, 2505.03156]. In its most specific sense, BoNBoN denotes the method introduced in "BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling," which trains a model to mimic the distribution induced by selecting the best completion among $n$ samples from a base policy [2406.00832]. The term is not fully uniform across the literature: some papers use it to denote a broader family of BoN-inspired alignment methods, while one unrelated reinforcement-learning context interprets it as shorthand for bonus-based exploration rather than language-model alignment [2505.03156, 2109.11052].

## 1. Terminological scope

In the LLM literature, BoNBoN is most precisely associated with a training method that replaces inference-time reranking by a learned single-sample policy. The underlying motivation is that BoN sampling is often effective for alignment but requires drawing $n$ samples, ranking them with a reward model or preference signal, and returning only the best one, which imposes a substantial inference-time cost [2406.00832].

A broader usage treats “BoNBoN” as shorthand for the Best-of-$N$ alignment family, especially methods that distill or emulate BoN-selected outputs into a standalone model. This broader sense appears in work on Soft Best-of-$n$, which places BoNBoN alongside related approaches such as Bond, variational BoN, TreeBoN, and speculative BoN [2505.03156]. In that framing, the common thread is not a single objective but the attempt to recover BoN-like preference optimization without paying the full multi-sample decoding cost at deployment.

The term also has a separate context-specific meaning outside LLM alignment. In Atari exploration research, “BoNBoN” is used as a natural shorthand for bonus-based exploration methods such as pseudo-counts, Intrinsic Curiosity Module (ICM), and Random Network Distillation (RND). That usage refers to shaped rewards of the form
\[
r'_t = r_t + \beta\, b(s_t,a_t),
\]
and is terminologically distinct from BoNBoN alignment for language models [2109.11052].

## 2. Best-of-$n$ foundations and the common tilting view

BoNBoN is built on the structure of Best-of-$n$ sampling. Given a prompt $x$, a base or reference model $\pi_0(y\mid x)$ generates $n$ i.i.d. candidates, a reward function $r(x,y)$ scores them, and the highest-scoring candidate is returned. Under the continuity assumptions used in the BoNBoN analysis, the induced density of the BoN policy is
\[
\pi^{(n)}_r(y\mid x) := n\, Q_x(r(x,y))^{n-1}\,\pi_0(y\mid x),
\]
where $Q_x$ is the CDF of $r(x,Y_0)$ under $Y_0\sim \pi_0(\cdot\mid x)$ [2406.00832].

The BoNBoN paper places BoN and standard alignment methods inside a common class of reward-weighted tiltings of the base model:
\[
\pi_r(y\mid x) = \frac{1}{Z_r}\,\pi_0(y\mid x)\, f(Q_x(r(x,y))),
\]
with $f$ increasing and nonnegative. In the same framework, RLHF and DPO target the exponential tilt
\[
\pi^{RLHF}_r(y\mid x) \propto \exp\{(1/\beta)\, r(x,y)\}\,\pi_0(y\mid x),
\]
under a correctly specified Bradley–Terry model, whereas BoN corresponds to a power tilt in the reward CDF rather than a direct exponential tilt in the reward itself [2406.00832].

This common formulation is important because it makes BoNBoN more than a heuristic imitation scheme. The method exploits the analytic form of the BoN-induced distribution and uses that structure to derive a training objective tailored to BoN winners and losers, rather than optimizing a generic KL-regularized reward objective [2406.00832].

## 3. BoNBoN Alignment as a training procedure

The BoNBoN method trains a single-pass model $\pi_\theta$ to mimic the BoN sampling distribution $\pi^{(n)}_r$. The original procedure begins with a reference model $\pi_0$ obtained by standard supervised fine-tuning on human-preferred data. For each prompt $x$, it samples $n$ completions from $\pi_0$, scores them with a reward model, identifies the best completion $y_{(n)}$ and the worst completion $y_{(1)}$, and uses these order-statistic samples to construct the training signal [2406.00832].

BoNBoN combines two losses. The first is SFT-BoN, a supervised term on BoN winners:
\[
\mathcal{L}_{SFT\text{-}BoN}(\pi_\theta) =
- \mathbb{E}_{x\sim D,\; y_{(n)}\sim \pi^{(n)}}[\log \pi_\theta(y_{(n)}\mid x)].
\]
The second is IPO-BoN, a contrastive term that matches the model’s log-likelihood ratio between best and worst samples to the BoN-implied ratio, with an analytically derived offset. The combined objective is
\[
\mathcal{L}_{BoNBoN}(\pi_\theta)
= \alpha\, \mathcal{L}_{SFT\text{-}BoN}(\pi_\theta)
+ (1-\alpha)\,\mathcal{L}_{IPO\text{-}BoN}(\pi_\theta).
\]
The key calibration constant is
\[
\beta_n^* = \frac{1}{2(n-1)\sum_{k=1}^{n-1} 1/k}.
\]
This analytic $\beta_n^*$ is one of the method’s defining features, because it removes the need to tune the IPO scale by sweep-based hyperparameter search [2406.00832].

The paper’s interpretation of the two terms is explicit. SFT-BoN alone has $\pi^{(n)}_r$ as a minimizer in the infinite-data limit but is data-inefficient by itself. IPO-BoN uses loser information to improve data efficiency and enforce the correct winner-versus-loser likelihood ratio. The combination is intended to prevent “cheating” behavior that could arise from ratio matching alone and to keep both losses aligned to the same target distribution [2406.00832].

In the reward-free code-alignment study, BoNBoN is described in similar procedural terms but without reproducing the original loss in LaTeX. There it is characterized as a “dual training strategy: supervised fine-tuning (SFT) on the best responses from n-sample generations, combined with Iterative Preference Optimization (IPO) training that contrasts the best versus worst responses from the same sample sets,” and as a “data distribution-matching approach” that “trains models to directly mimic the best-of-n distribution” [2606.28998].

## 4. Theoretical characterization

A central result of the BoNBoN paper is that BoN is essentially optimal in the trade-off between win-rate against the base model and KL divergence from the base model, within the class of reward-weighted tiltings described above. The formal optimization problem fixes a KL budget and maximizes win-rate against $\pi_0$; the resulting optimum is an exponential tilt by the reward CDF:
\[
\pi_{r,c}(y\mid x)=\frac{\pi_0(y\mid x)\exp\{c\,Q_x(r(x,y))\}}{Z_r^c}.
\]
Within this analysis, BoN lies very close to the optimal Pareto frontier. The paper states that the maximum win-rate gap, at $n=2$, is less than one percentage point, and that the gap shrinks for larger $n$ [2406.00832].

For BoN itself, the context-conditional win-rate and KL divergence admit closed forms:
\[
p_{(n)\succ x}=\frac{n}{n+1},\qquad
D\!\left(\pi^{(n)}\Vert \pi_0 \mid x\right)=\log(n)-\frac{n-1}{n}.
\]
These values are constants across prompts under the paper’s assumptions, and they also characterize the overall win-rate and KL on any prompt distribution $D$ [2406.00832].

This theory motivates BoNBoN’s design. If BoN already sits essentially on the optimal win-rate-versus-KL frontier, then a model trained to mimic the BoN distribution inherits a target that is already near-optimal for that specific trade-off. The theoretical argument is therefore not merely that BoNBoN approximates a strong heuristic; it approximates a distribution that the paper argues is itself essentially optimal for win-rate at fixed KL [2406.00832].

The BoNBoN paper also gives an idealized benchmark for $n=8$:
\[
p_{(8)\succ}=\frac{8}{9}\approx 0.888,\qquad
D\!\left(\pi^{(8)}\Vert \pi_0\right)=\log(8)-\frac{7}{8}.
\]
This serves as the reference frontier point for the paper’s experiments using $n=8$ [2406.00832].

## 5. Empirical evidence across domains

The empirical literature treats BoNBoN both as a standalone alignment method and as a baseline for other BoN-inspired objectives. The major reported settings are summarized below.

| Study | Setting | Reported BoNBoN pattern |
|---|---|---|
| [2406.00832] | Anthropic Helpful and Harmless; OpenAI TL;DR | Better win-rate vs off-target trade-offs than DPO/IPO; SFT-BoN and IPO-BoN alone underperform the combined objective |
| [2407.06057] | IMDB controlled generation baseline comparison | Used as a concurrent BoN-inspired baseline; vBoN exceeds it on reported Pareto-front frequency |
| [2606.28998] | Code alignment across five open models, PTA and FTA | Larger relative gains from pretrained-to-aligned pathways; mixed or degrading results can occur from finetuned-to-aligned pathways |

In the original BoNBoN paper, experiments use single-turn dialogue generation on Anthropic Helpful and Harmless and summarization on OpenAI TL;DR. The setup uses Pythia-2.8b fine-tuned with SFT on human-preferred completions as the base model, draws $n=8$ completions per prompt from $\pi_0$, uses the off-the-shelf preference scorer `OpenAssistant/reward-model-deberta-v3-large-v2` to relabel pairs, and trains with RMSprop at learning rate $5\times 10^{-7}$ on 6 GPUs, evaluating the 20k-step checkpoint. The reported finding is that BoNBoN achieves much better win-rate versus off-target deviation trade-offs than DPO or IPO on both the original datasets and best/worst-of-8 datasets; baselines matched to BoNBoN win-rate produce large off-target drift, including cases where average response length nearly doubles, whereas BoNBoN minimally changes length and other off-target attributes [2406.00832].

The same paper reports that SFT-BoN and IPO-BoN alone underperform BoNBoN at all tested $\beta$ values, which it interprets as evidence that both level-matching and ratio-matching are necessary. It also describes BoNBoN as nearly hyperparameter-free: $\beta_n^*$ is fixed analytically, and $\alpha$ is chosen so that the two losses contribute roughly equally, with $\alpha=0.005$ reported in the main experiments [2406.00832].

In variational Best-of-$N$ work, BoNBoN is included as a concurrent baseline rather than the primary method. On IMDB positive-sentiment control with GPT2-IMDB and a DistilBERT IMDB reward model, the paper reports Pareto-front shares of roughly 50% vBoN, 42% DPO, 4% BoNBoN, and 4% PPO for win-rate vs KL, and roughly 50% vBoN, 33% PPO, 10% DPO, and 7% BoNBoN for reward vs KL. In that setting, the reported conclusion is that vBoN comes closest to BoN’s performance while recovering the factor-$N$ inference speedup [2407.06057].

The code-alignment study extends BoNBoN to a reward-free setting using SelfCodeAlign-derived preference pairs across Meta-Llama-3-8B, Qwen2.5-Coder-7B, CodeLlama-7B, deepseek-coder-1.3B, and deepseek-coder-7B, each evaluated in pretrained-to-aligned (PTA) and finetuned-to-aligned (FTA) pathways. The strongest reported BoNBoN gains occur in PTA settings. For Meta-Llama-3-8B PTA, HumanEval rises from 0.372 to 0.518, HumanEval+ from 0.323 to 0.476, and EvalPerf from 6.5% to 11.8%. For CodeLlama-7B PTA, CODAL average rises from 0.48 to 0.84, a reported +75% relative improvement. For Qwen2.5-Coder-7B-Instruct FTA, EvalPerf rises from 71.1% to 73.5%, which the paper identifies as the best across all models. The same study also reports degradations: Meta-Llama-3-8B-Instruct falls from 0.610 to 0.543 on HumanEval under BoNBoN, and CodeLlama-7B-Instruct CODAL average falls from 0.60 to 0.37 [2606.28998].

Across those code experiments, the paper’s high-level conclusion is that pretrained-to-aligned pathways achieve larger improvements over pretrained baselines, while finetuned-to-aligned pathways offer smaller improvements or, in some cases, degradation. It further states that non-functional code quality improves more reliably than functional correctness under BoNBoN alignment, with readability and style often showing the largest relative gains [2606.28998].

## 6. Relation to adjacent methods, misconceptions, and limitations

BoNBoN is not synonymous with all BoN-style decoding methods. A common source of confusion is to treat it as a generic label for any reward-guided reranking or best-of-$n$ regularization. The literature draws sharper distinctions.

First, BoNBoN is distinct from MBR-BoN, also called WD-RBoN, which is an inference-time decoding rule rather than a distillation objective. MBR-BoN selects
\[
y^*(x)=\arg\max_{y\in ref}\bigl[R(x,y)-\lambda\cdot MBR(y;x)\bigr],
\]
where the MBR term is the expected semantic distance from a candidate to samples from the reference policy. The corresponding paper explicitly states that it does not reference a method explicitly named “BoNBoN” [2404.01054].

Second, BoNBoN is distinct from Soft Best-of-$n$. Soft BoN replaces hard argmax selection with a softmax over sampled rewards controlled by a temperature $\lambda$, interpolating between the base model and hard BoN. Its main contribution is a sharp $O(1/n)$ convergence guarantee in KL and relative reward to the exponentially tilted target distribution, not a distillation objective of the BoNBoN type [2505.03156].

Third, BoNBoN is distinct from variational BoN. vBoN derives the exact BoN-induced distribution $q_{BoN}(y\mid x)$, then fine-tunes a model by minimizing the backward KL divergence
\[
KL\bigl(p_\phi(\cdot\mid x)\,\|\, q_{BoN}(\cdot\mid x)\bigr).
\]
The vBoN paper explicitly notes that BoNBoN uses a pairwise IPO-like loss derived from Best-of-$N$ versus Worst-of-$N$ likelihood gaps, whereas vBoN directly targets the full BoN distribution via backward KL [2407.06057].

A further misconception is that the name implies “BoN applied twice” or a hierarchical best-of-best procedure. The variational BoN paper notes that such a reading might seem natural, but the actual BoNBoN method instead uses a pairwise objective plus NLL on BoN winners; it is not a second-stage BoN sampler [2407.06057].

The main limitations reported for BoNBoN are also consistent across studies. In the original alignment setting, the method still depends on the quality of the reward model used to identify BoN winners and losers, and regular KL estimates may fail to capture salient off-target changes such as response length, even though BoNBoN empirically minimizes such drift relative to baselines [2406.00832]. In the code-alignment study, performance depends strongly on the starting point: pretrained models are more plastic and can improve more, while finetuned models may exhibit diminishing returns or degradation, with early SFT-stage degradation acting as a warning sign for final failure [2606.28998].

Taken together, these results position BoNBoN as a specific and technically structured answer to a concrete problem in BoN alignment: how to preserve the empirical strength of best-of-$n$ selection while removing its deployment-time sampling cost. The term can refer more broadly to BoN-family distillation or emulation strategies, but its canonical meaning remains the SFT-plus-IPO procedure introduced for single-pass imitation of the BoN distribution [2406.00832, 2505.03156].

Source: https://www.emergentmind.com/topics/bonbon