---
title: Reward-Gated Rejection Sampling
url: https://www.emergentmind.com/topics/reward-gated-rejection-sampling
type: topic
---

# Reward-Gated Rejection Sampling

Searching arXiv for the cited papers to ground the article.
Reward-gated rejection sampling denotes a family of procedures in which samples from a proposal distribution, policy, or generator are not all treated equally: a reward-like signal, acceptability score, density ratio, or smooth acceptance function determines whether a sample is retained, rejected, continued, or allowed to influence optimization. Across recent work, the pattern appears in calibrated best-of-\(N\) inference, selective-thinking data filtering, early beam pruning, offline preference optimization, off-policy reinforcement learning, and variational posterior refinement. What unifies these otherwise heterogeneous methods is a two-stage structure—proposal followed by gating—even when the “gate” is implemented as a stochastic accept/reject rule, a rank-based survival criterion, or a differentiable weighting mechanism rather than literal hard rejection [2510.04087][2602.19519][2508.01969][2309.06657].

## 1. Core formulation and scope

A useful way to organize the literature is by the object being gated and by the semantics of the gating score. In some settings the gate is intended to represent contextual acceptability; in others it is an adaptive efficiency reward, a target-to-proposal ratio, or a smooth trustworthiness function of an importance ratio. This suggests that “reward-gated rejection sampling” is best treated as a structural family rather than a single algorithm.

| Variant | Gate | Retained object |
|---|---|---|
| Contextual acceptability gating | \(\hat R_\theta(x,y)>0\) or \(\hat R_{\max}>\tau_{N_l}\) | Inference candidate [2510.04087] |
| Adaptive stochastic filtering | \(p_{ij}\) from reward gaps or \(p_i\) from standardized rewards | Preference pairs or rollout members [2602.19519] |
| Early partial-reward rejection | Top \(N/M\) by partial reward after \(T\) tokens | Surviving beams [2508.01969] |
| Preference-policy rejection sampling | \(\exp((r_\psi-r_{\max})/\beta)\) | Responses from \(\pi_{r_\psi}\) [2309.06657] |
| Budgeted ratio gating | \(a(x)=\min\!\left(1,\frac{p_{\text{target}}(x)}{\lambda p_{\text{inf}}(x)}\right)\) | Accepted rollout tokens [2602.06107] |
| Differentiable rejection gate | \(\alpha_\theta(s,a)=g(r_\theta(s,a))\) | Gradient contribution [2604.14895] |
| Variational density-ratio gating | \(a(\mathbf z)=\frac{p_\theta(\mathbf{x}\mid \mathbf z)p(\mathbf z)}{p_\theta(\mathbf{x}\mid \mathbf z)p(\mathbf z)+M q_\phi(\mathbf z\mid \mathbf x)}\) | Accepted latent samples [2606.14235] |

Two distinctions recur throughout the literature. First, some methods gate **complete samples**, while others gate **partial trajectories** or even **individual tokens**. Second, some methods aim to preserve or approximate a target distribution, whereas others use gating instrumentally to improve reliability, reduce latency, or alter which examples contribute to learning. In that respect, the term covers both exact or approximate rejection sampling in the Monte Carlo sense and broader LLM-specific procedures that operationally behave as gated selection mechanisms.

## 2. Acceptability semantics and calibrated thresholds

A central problem for reward-gated inference is that pairwise preference models learn relative ordering but not whether any candidate is actually acceptable. "A Contextual Quality Reward Model for Reliable and Efficient Best-of-N Sampling" addresses this explicitly by introducing an outside option, \(y_0\), meaning “reject all responses,” and defining normalized reward relative to that option as
\[
R(x,y_i)\equiv \tilde R(x,y_i)-\tilde R(x,y_0).
\]
Under i.i.d. Gumbel noise, annotator choices over \(\mathcal Y=\{y_0,y_1,\dots,y_J\}\) follow a multinomial logit model,
\[
\Pr(d_i=1\mid x,\mathcal Y)=\frac{\exp(R(x,y_i))}{1+\sum_{j=1}^{J}\exp(R(x,y_j))},
\]
so that
\[
R(x,y_i)=\log(\Pr(d_i=1\mid x,\mathcal Y))-\log(\Pr(d_0=1\mid x,\mathcal Y)).
\]
This gives \(R(x,y)>0\) explicit acceptability semantics: the response is more likely to be chosen than the outside option. The same work defines hard prompts by
\[
p_g=\Pr_{y\sim \pi(\cdot\mid x)}(R(x,y)>0),
\]
and formalizes false acceptance for standard best-of-\(N\) reranking as
\[
P_{FA}(N)=\Pr(\hat R_\theta(x,y^*)>0\text{ and }R(x,y^*)<0\mid N).
\]
Its empirical result is that ordinary BoN can become less reliable as \(N\) increases: false positive count rises from \(104\) at \(N=1\) to \(210\) at \(N=32\) [2510.04087].

The same paper turns calibrated acceptability into a sequential gate through “best of mini-\(N\) in-loop.” In the accelerator setting, the gate is fixed at \(\tau=0\). In the more conservative guardrail setting, the gate is sample-count dependent:
\[
\hat R_{\max}>\tau_{N_l},
\]
with thresholds estimated from hard-prompt data via
\[
[\tilde F(\tau_N)]^N=\tilde F(0),\qquad
\tau_N=\max\left\{\tilde F^{-1}\left([\tilde F(0)]^{1/N}\right),0\right\}.
\]
This construction is notable because it corrects for maxima inflation as more samples are drawn. In the synthetic IMDB sentiment setup, the guardrail version at total budget \(N=32\), specifically “Best of mini-16 (2 loops),” reduces false positives from \(210\) to \(63\), a \(70\%\) reduction, while mean ground-truth reward declines only from \(0.486\) to \(0.466\). The accelerator configuration with \(\tau=0\) reduces average execution time from \(7.98\)s for standard BoN-32 to \(6.16\)s, over \(22\%\) faster, while preserving recall at \(95.4\%\) [2510.04087].

The broader significance is that rejection gating becomes reliable only when the reward scale is anchored. This paper’s outside-option normalization is an explicit attempt to make reward scores absolute or contextual rather than merely ordinal.

## 3. Inference-time rejection during generation

A second line of work places the gate inside generation rather than after full completion. "Accelerating LLM Reasoning via Early Rejection with Partial Reward Modeling" studies beam-search-style reasoning with Process Reward Models and hypothesizes that a PRM can also be used as a Partial Reward Model. For each step, candidate continuations are generated only up to a prefix length \(T\), partial rewards \(P_i\) are computed, the top \(N/M\) candidates are retained, and only survivors are completed. The survival rule is rank-based: if \(\tau\) is the \((1-1/M)\)-quantile of the partial reward distribution, beam \(i\) survives iff
\[
P_i\ge \tau.
\]
The paper models the relationship between partial and final reward as
\[
F_i=g(P_i)+n_i,
\]
with monotone increasing \(g\) and zero-mean noise, and proves
\[
\Pr(P_{i^*}<\tau)\le (N-1)\exp\!\left(-\frac{\Delta^2}{4\sigma^2}\right),
\]
where \(i^*=\arg\max_i F_i\). In the i.i.d. token-score toy model,
\[
\rho(P_i,F_i)=\sqrt{\frac{T}{L}},
\]
so correlation increases with decision-prefix length. Empirically, Pearson correlation already exceeds \(0.78\) at \(T=32\), both Pearson and Kendall’s Tau exceed \(0.9\) at \(T=64\), and the method yields \(1.4\times\)–\(9\times\) FLOPs reduction with MathShepherd-Mistral-7B and \(1.5\times\)–\(4\times\) with Skywork-PRM-1.5B, without degrading final performance [2508.01969].

A conceptually different but structurally related inference-time method is "Flipping Against All Odds: Reducing LLM Coin Flip Bias via Verbalized Rejection Sampling." Here the target is a Bernoulli distribution \(P\), the proposal is \(Q=\mathrm{Bern}(0.5)\), and the LLM itself verbalizes the accept/reject step. Classical acceptance probability is
\[
A(x)=\frac{P(x)}{M Q(x)},
\qquad
M=\max\left\{\frac{p}{q},\frac{1-p}{1-q}\right\},
\]
but in practice the model is prompted to reason and emit \(T/F\) rather than receiving an explicit uniform draw. The method is therefore approximate rather than exact, yet the paper proves bounded-error improvements under biased acceptance and reports substantial reductions in Bernoulli calibration error. For Llama-3.1-70B, mean STVD drops from \(15.73\) under direct sampling to \(6.20\) under VRS; for Qwen-2.5 72B, from \(20.27\) to \(5.47\). The method is best read as self-gated rejection sampling: the gate is not an external reward model but an LLM-generated binary accept/reject judgment intended to emulate \(A(x)\) [2506.09998].

Taken together, these works show two distinct inference-time uses of gating: **early compute allocation** based on partial reward, and **distribution correction** through repeated accept/reject decisions.

## 4. Training-time filtering and preference optimization

Another major interpretation of reward-gated rejection sampling is that the gate controls **which sampled trajectories enter training**, rather than which output is returned at inference. "Ada-RS: Adaptive Rejection Sampling for Selective Thinking" is explicit on this point. For each context \(x\), multiple completions \(y_i=(<think>~t_i~</think>,a_i)\) are sampled, and each receives the adaptive length-penalized reward
\[
r(y_i,x)=\mathbbm{1}(y_i,x)-\alpha\cdot s_K(x)\cdot |t_i|,
\qquad
s_K(x)=\frac{1}{K}\sum_{i=1}^K \mathbbm{1}(y_i,x).
\]
In the DPO setting, candidate pairs are accepted stochastically with probability
\[
p_{ij}=\exp\!\left(\frac{\Delta_{ij}-\Delta_{\max}}{\beta_{\mathrm{rs}}}\right),
\qquad
\Delta_{ij}=r_i-r_j,
\]
while in the DAPO setting each group member is retained with
\[
p_i=\min\left(\exp\!\left(\frac{(r_i-\mu)/\sigma}{\beta_{\mathrm{rs}}}\right),1\right).
\]
This is reward-gated rejection in a training-signal sense: higher-reward, shorter-on-easy-prompts trajectories are more likely to survive. On the synthetic e-commerce benchmark with Qwen3-8B and LoRA, Ada-RS reduces average output tokens by up to \(\sim 80\%\) and thinking rate by up to \(\sim 95\%\) while maintaining or improving tool-call accuracy [2602.19519].

In offline preference optimization, "Statistical Rejection Sampling Improves Preference Optimization" gives a more classical target-distribution construction. Starting from an SFT policy \(\pi_{\text{sft}}\) and a learned reward \(r_\psi\), it defines the KL-regularized optimal policy
\[
\pi_{r_\psi}(y\mid x)=\frac{1}{Z_\psi(x)}\pi_{\text{sft}}(y\mid x)\exp\!\left(\frac{1}{\beta}r_\psi(x,y)\right),
\]
and samples from it by rejection sampling over candidates drawn from \(\pi_{\text{sft}}\). Acceptance probability on a finite candidate pool is
\[
\exp\!\left(\frac{1}{\beta}(r_\psi(x,y)-r_{\max})\right).
\]
As \(\beta\to 0\), this becomes top-\(k\)/best-of-\(N\)-like selection; as \(\beta\to\infty\), acceptance approaches \(1\). On Reddit TL;DR and AnthropicHH, the RSO variants consistently outperform direct DPO-style training and SFT-sampled pair generation, supporting the claim that the sampling distribution of preference pairs matters at least as much as the form of the preference loss [2309.06657].

A more minimalist perspective appears in "A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce." RAFT is formalized as rejection-sampling fine-tuning: sample \(n\) responses per prompt, keep only those with maximal reward, and maximize
\[
\mathcal L^{\text{RAFT}}(\theta)=\sum_{(x,a)\in\mathcal D}\log \pi_\theta(a\mid x).
\]
RAFT++ keeps the same reward gate,
\[
\mathcal I\big(r(x,a)=\max_i r(x,a_i)\big),
\]
but adds importance sampling and PPO-style clipping. The paper’s central ablation is at prompt level: removing prompts whose sampled responses are all wrong explains much of GRPO’s advantage over vanilla REINFORCE, while mean-zero and standard-deviation normalization explain little. On Qwen2.5-Math-7B-base, average accuracy over Math500, Minerva Math, and Olympiad Bench is \(49.9\) for RAFT, \(52.5\) for RAFT++, \(52.9\) for Reinforce, and \(53.9\) for GRPO; on LLaMA-3.2-3B-instruct, the corresponding numbers are \(26.3\), \(27.5\), \(23.4\), and \(29.4\) [2504.11343].

These training-time methods differ in objective, but all implement the same operational principle: the training distribution is shaped by a reward-gated retention mechanism rather than by using all samples uniformly.

## 5. Ratio-gated generalizations in RL and variational inference

Several papers broaden the idea beyond explicit reward scores and treat the gate as a function of a target-to-proposal ratio. This is not reward gating in the narrow RLHF sense, but it is a mathematically close generalization.

"Refined \(\alpha\)-Divergence Variational Inference via Rejection Sampling" connects proposal learning and rejection efficiency through
\[
D_\infty(p\|q_\theta)=\log M(\theta),
\]
where \(M(\theta)\) is the optimal rejection constant for proposal \(q_\theta\). The method first learns \(q_\theta\) by minimizing a Monte Carlo Rényi-divergence objective and then refines it with acceptance probability
\[
a_{\hat\theta}(x\mid T)=\frac{1}{1+\left(\frac{q_{\hat\theta}(x)e^{-T}}{\tilde p(x)}\right)},
\]
a smoothed form of the exact clipped gate
\[
a_\theta(x\mid T)=\min\left[1,\frac{\tilde p(x)}{e^{-T}q_\theta(x)}\right].
\]
The refined distribution \(r(x)\propto q_\theta(x)a_\theta(x\mid T)\) satisfies
\[
D_\alpha(p\|r)\le D_\alpha(p\|q).
\]
The paper’s importance for reward-gated rejection sampling is conceptual: it shows that acceptance should often depend on score **relative to proposal probability**, not on score alone [1909.07627].

"Jackpot: Optimal Budgeted Rejection Sampling for Extreme Actor-Policy Mismatch Reinforcement Learning" makes that ratio interpretation explicit for off-policy RL in LLMs. For a rollout token \(x\sim p_{\text{inf}}\), Optimal Budget Rejection Sampling accepts with
\[
a(x)=\min\!\left(1,\frac{p_{\text{target}}(x)}{\lambda\,p_{\text{inf}}(x)}\right),
\]
producing
\[
P_{\text{OBRS}}(x)=\frac{p_{\text{inf}}(x)a(x)}{\sum_{x'}p_{\text{inf}}(x')a(x')}.
\]
The main theorem states that, under an average acceptance-rate budget, this is the unique rule minimizing forward KL to the target distribution. Rewards still drive the downstream PPO/GRPO updates, but the rejection gate itself is distribution-matching-gated rather than reward-gated. This is particularly relevant when actor-policy mismatch is severe [2602.06107].

"Beyond Importance Sampling: Rejection-Gated Policy Optimization" internalizes the gate directly into the objective. It replaces the importance ratio with a smooth acceptance gate
\[
\alpha_\theta(s,a)=g(r_\theta(s,a))\in[0,1],
\qquad
r_\theta(s,a)=\frac{\pi_\theta(a\mid s)}{\pi_{\text{old}}(a\mid s)},
\]
and yields effective gradient weight
\[
w(r)=g'(r)\,r.
\]
The RGPO objective is
\[
L_{\text{RGPO}}(\theta)=\mathbb E_{\pi_{\text{old}}}\big[g(r_\theta(s,a))A_{\text{old}}(s,a)\big].
\]
The paper proves finite bounded gradient variance even when standard IS variance diverges under heavy-tailed ratios, and gives an approximate monotonic policy-improvement guarantee analogous to TRPO. In online preference fine-tuning of Qwen2.5-1.5B-Instruct on Anthropic HH-RLHF, RGPO achieves the highest reward among online RL methods, \(+14.8\%\) vs. PPO-RLHF, and the lowest KL divergence to the reference model, \(-16.0\%\) vs. PPO-RLHF and \(-53.1\%\) vs. GRPO [2604.14895].

Finally, "Implicit Variational Rejection Sampling" applies a smooth acceptance gate to implicit variational inference:
\[
a(\mathbf z;\mathbf x,\theta,\phi)=
\frac{p_\theta(\mathbf x\mid \mathbf z)p(\mathbf z)}
{p_\theta(\mathbf x\mid \mathbf z)p(\mathbf z)+M q_\phi(\mathbf z\mid \mathbf x)}.
\]
With discriminator \(T^*(\mathbf x,\mathbf z)=\log q_\phi(\mathbf z\mid \mathbf x)-\log p(\mathbf z)\), the same gate can be written
\[
a(\mathbf z;\mathbf x,\theta,\phi)=
\frac{p_\theta(\mathbf x\mid \mathbf z)}
{p_\theta(\mathbf x\mid \mathbf z)+M\exp(T^*(\mathbf x,\mathbf z))}.
\]
The accepted distribution
\[
r_{\theta,\phi}(\mathbf z\mid \mathbf x)=
\frac{q_\phi(\mathbf z\mid \mathbf x)a(\mathbf z;\mathbf x,\theta,\phi)}
{Z_{\theta,\phi}(\mathbf x)}
\]
is shown, in the idealized setting, to move monotonically closer to the true posterior as \(M\) increases. This again realizes the same proposal-plus-gate architecture, but with log density ratio rather than externally defined reward as the scoring signal [2606.14235].

## 6. Conceptual boundaries, failure modes, and ongoing questions

The literature makes clear that “reward-gated rejection sampling” should not be conflated with a single canonical algorithm. Some methods are genuine rejection samplers over proposal distributions; others are gated rerankers, beam-pruning rules, or differentiable policy objectives. "A Contextual Quality Reward Model for Reliable and Efficient Best-of-N Sampling" is explicit that its method is not foregrounded as “rejection sampling,” yet operationally it is a sequential gated reranking policy with early stopping. "Accelerating LLM Reasoning via Early Rejection with Partial Reward Modeling" similarly occupies a middle ground between beam pruning and reward-gated rejection, because it rejects partial trajectories rather than completed samples. "Beyond Importance Sampling: Rejection-Gated Policy Optimization" pushes the idea further by making rejection differentiable and internal to gradient computation rather than a data preprocessing step [2510.04087][2508.01969][2604.14895].

Several failure modes recur. Gate calibration is decisive: outside-option labels must truly mean “none of these are acceptable in context,” or the absolute offset of \(R(x,y)\) is corrupted [2510.04087]. In Ada-RS, the adaptive solve-rate term and the selectivity temperature \(\beta_{\mathrm{rs}}\) are sensitive hyperparameters; naive rejection sampling without the auxiliary NLL loss produces poor accuracy and always-think behavior [2602.19519]. In VRS, prompt framing matters, exact symbolic computation of \(M\) does not explain success, and the procedure remains approximate because the acceptance coin is itself mediated by the LLM [2506.09998]. Early-rejection methods can fail when utility is delayed or non-monotonic, as in code synthesis with backtracking or tasks where early text is ambiguous but later recovery is possible [2508.01969]. RSO inherits the usual support limitation of rejection sampling: if the proposal policy rarely generates good responses, reweighting cannot recover them [2309.06657]. IVRS shows an analogous pathology on the inference side: overtraining the discriminator can drive acceptance rates toward collapse and worsen performance [2606.14235].

A broader interpretive question is whether the gate should depend on reward alone or on reward relative to proposal probability. The variational and budgeted-RL papers argue strongly for the latter. A plausible implication is that many heuristic reward-thresholding schemes are incomplete when the actual objective is to approximate a target distribution rather than merely keep “good-looking” samples. Conversely, the acceptability-model and selective-thinking papers show that in LLM applications, an anchored or contextualized scalar can be sufficient when the operational objective is reliability, efficiency, or selective training rather than exact distributional correction [1909.07627][2602.06107].

The common lesson is not that all useful rejection gates must be identical, but that sample selection becomes principled only when the meaning of the gating score is explicit. In one branch, that meaning is **acceptability relative to an outside option**. In another, it is **efficiency-aware reward**. In another, it is **target-to-proposal ratio** or **policy trustworthiness**. The modern literature therefore treats reward-gated rejection sampling less as a narrow Monte Carlo primitive and more as a general design pattern for allocating probability mass, computation, and gradient influence under uncertainty.

Source: https://www.emergentmind.com/topics/reward-gated-rejection-sampling