Reward-Gated Rejection Sampling

• Reward-gated rejection sampling is a structural family of methods that use reward signals to selectively accept or reject samples in a two-stage process.
• The approach integrates proposal generation with gating mechanisms—ranging from stochastic filtering to beam pruning—to improve efficiency and accuracy in both inference and training.
• Practical applications include calibrated best-of-N inference, offline preference optimization, and reinforcement learning adjustments, demonstrating significant compute savings and reduced error rates.

Searching arXiv for the cited papers to ground the article. Reward-gated rejection sampling denotes a family of procedures in which samples from a proposal distribution, policy, or generator are not all treated equally: a reward-like signal, acceptability score, density ratio, or smooth acceptance function determines whether a sample is retained, rejected, continued, or allowed to influence optimization. Across recent work, the pattern appears in calibrated best-of-$N$ inference, selective-thinking data filtering, early beam pruning, offline preference optimization, off-policy reinforcement learning, and variational posterior refinement. What unifies these otherwise heterogeneous methods is a two-stage structure—proposal followed by gating—even when the “gate” is implemented as a stochastic accept/reject rule, a rank-based survival criterion, or a differentiable weighting mechanism rather than literal hard rejection (Rho, 5 Oct 2025, Ge et al., 23 Feb 2026, Cheshmi et al., 4 Aug 2025, Liu et al., 2023).

1. Core formulation and scope

A useful way to organize the literature is by the object being gated and by the semantics of the gating score. In some settings the gate is intended to represent contextual acceptability; in others it is an adaptive efficiency reward, a target-to-proposal ratio, or a smooth trustworthiness function of an importance ratio. This suggests that “reward-gated rejection sampling” is best treated as a structural family rather than a single algorithm.

Variant	Gate	Retained object
Contextual acceptability gating	$\hat R_\theta(x,y)>0$ or $\hat R_{\max}>\tau_{N_l}$	Inference candidate (Rho, 5 Oct 2025)
Adaptive stochastic filtering	$p_{ij}$ from reward gaps or $p_i$ from standardized rewards	Preference pairs or rollout members (Ge et al., 23 Feb 2026)
Early partial-reward rejection	Top $N/M$ by partial reward after $T$ tokens	Surviving beams (Cheshmi et al., 4 Aug 2025)
Preference-policy rejection sampling	$\exp((r_\psi-r_{\max})/\beta)$	Responses from $\pi_{r_\psi}$ (Liu et al., 2023)
Budgeted ratio gating	$$a(x)=\min\!\left(1,\frac{p_{\text{target}}(x)}{\lambda p_{\text{inf}}(x)}\right)$$	Accepted rollout tokens (Chen et al., 5 Feb 2026)
Differentiable rejection gate	$\hat R_\theta(x,y)>0$0	Gradient contribution (Sun et al., 16 Apr 2026)
Variational density-ratio gating	$\hat R_\theta(x,y)>0$1	Accepted latent samples (Xu et al., 12 Jun 2026)

Two distinctions recur throughout the literature. First, some methods gate complete samples, while others gate partial trajectories or even individual tokens. Second, some methods aim to preserve or approximate a target distribution, whereas others use gating instrumentally to improve reliability, reduce latency, or alter which examples contribute to learning. In that respect, the term covers both exact or approximate rejection sampling in the Monte Carlo sense and broader LLM-specific procedures that operationally behave as gated selection mechanisms.

2. Acceptability semantics and calibrated thresholds

A central problem for reward-gated inference is that pairwise preference models learn relative ordering but not whether any candidate is actually acceptable. "A Contextual Quality Reward Model for Reliable and Efficient Best-of-N Sampling" addresses this explicitly by introducing an outside option, $\hat R_\theta(x,y)>0$2, meaning “reject all responses,” and defining normalized reward relative to that option as

$\hat R_\theta(x,y)>0$3

Under i.i.d. Gumbel noise, annotator choices over $\hat R_\theta(x,y)>0$4 follow a multinomial logit model,

$\hat R_\theta(x,y)>0$5

so that

$\hat R_\theta(x,y)>0$6

This gives $\hat R_\theta(x,y)>0$7 explicit acceptability semantics: the response is more likely to be chosen than the outside option. The same work defines hard prompts by

$\hat R_\theta(x,y)>0$8

and formalizes false acceptance for standard best-of-$\hat R_\theta(x,y)>0$9 reranking as

$\hat R_{\max}>\tau_{N_l}$0

Its empirical result is that ordinary BoN can become less reliable as $\hat R_{\max}>\tau_{N_l}$1 increases: false positive count rises from $\hat R_{\max}>\tau_{N_l}$2 at $\hat R_{\max}>\tau_{N_l}$3 to $\hat R_{\max}>\tau_{N_l}$4 at $\hat R_{\max}>\tau_{N_l}$5 (Rho, 5 Oct 2025).

The same paper turns calibrated acceptability into a sequential gate through “best of mini-$\hat R_{\max}>\tau_{N_l}$6 in-loop.” In the accelerator setting, the gate is fixed at $\hat R_{\max}>\tau_{N_l}$7. In the more conservative guardrail setting, the gate is sample-count dependent: $\hat R_{\max}>\tau_{N_l}$8 with thresholds estimated from hard-prompt data via

$\hat R_{\max}>\tau_{N_l}$9

This construction is notable because it corrects for maxima inflation as more samples are drawn. In the synthetic IMDB sentiment setup, the guardrail version at total budget $p_{ij}$0, specifically “Best of mini-16 (2 loops),” reduces false positives from $p_{ij}$1 to $p_{ij}$2, a $p_{ij}$3 reduction, while mean ground-truth reward declines only from $p_{ij}$4 to $p_{ij}$5. The accelerator configuration with $p_{ij}$6 reduces average execution time from $p_{ij}$7s for standard BoN-32 to $p_{ij}$8s, over $p_{ij}$9 faster, while preserving recall at $p_i$0 (Rho, 5 Oct 2025).

The broader significance is that rejection gating becomes reliable only when the reward scale is anchored. This paper’s outside-option normalization is an explicit attempt to make reward scores absolute or contextual rather than merely ordinal.

3. Inference-time rejection during generation

A second line of work places the gate inside generation rather than after full completion. "Accelerating LLM Reasoning via Early Rejection with Partial Reward Modeling" studies beam-search-style reasoning with Process Reward Models and hypothesizes that a PRM can also be used as a Partial Reward Model. For each step, candidate continuations are generated only up to a prefix length $p_i$1, partial rewards $p_i$2 are computed, the top $p_i$3 candidates are retained, and only survivors are completed. The survival rule is rank-based: if $p_i$4 is the $p_i$5-quantile of the partial reward distribution, beam $p_i$6 survives iff

$p_i$7

The paper models the relationship between partial and final reward as

$p_i$8

with monotone increasing $p_i$9 and zero-mean noise, and proves

$N/M$0

where $N/M$1. In the i.i.d. token-score toy model,

$N/M$2

so correlation increases with decision-prefix length. Empirically, Pearson correlation already exceeds $N/M$3 at $N/M$4, both Pearson and Kendall’s Tau exceed $N/M$5 at $N/M$6, and the method yields $N/M$7–$N/M$8 FLOPs reduction with MathShepherd-Mistral-7B and $N/M$9–$T$0 with Skywork-PRM-1.5B, without degrading final performance (Cheshmi et al., 4 Aug 2025).

A conceptually different but structurally related inference-time method is "Flipping Against All Odds: Reducing LLM Coin Flip Bias via Verbalized Rejection Sampling." Here the target is a Bernoulli distribution $T$1, the proposal is $T$2, and the LLM itself verbalizes the accept/reject step. Classical acceptance probability is

$T$3

but in practice the model is prompted to reason and emit $T$4 rather than receiving an explicit uniform draw. The method is therefore approximate rather than exact, yet the paper proves bounded-error improvements under biased acceptance and reports substantial reductions in Bernoulli calibration error. For Llama-3.1-70B, mean STVD drops from $T$5 under direct sampling to $T$6 under VRS; for Qwen-2.5 72B, from $T$7 to $T$8. The method is best read as self-gated rejection sampling: the gate is not an external reward model but an LLM-generated binary accept/reject judgment intended to emulate $T$9 (Xiao et al., 11 Jun 2025).

Taken together, these works show two distinct inference-time uses of gating: early compute allocation based on partial reward, and distribution correction through repeated accept/reject decisions.

4. Training-time filtering and preference optimization

Another major interpretation of reward-gated rejection sampling is that the gate controls which sampled trajectories enter training, rather than which output is returned at inference. "Ada-RS: Adaptive Rejection Sampling for Selective Thinking" is explicit on this point. For each context $\exp((r_\psi-r_{\max})/\beta)$0, multiple completions $\exp((r_\psi-r_{\max})/\beta)$1 are sampled, and each receives the adaptive length-penalized reward

$\exp((r_\psi-r_{\max})/\beta)$2

In the DPO setting, candidate pairs are accepted stochastically with probability

$\exp((r_\psi-r_{\max})/\beta)$3

while in the DAPO setting each group member is retained with

$\exp((r_\psi-r_{\max})/\beta)$4

This is reward-gated rejection in a training-signal sense: higher-reward, shorter-on-easy-prompts trajectories are more likely to survive. On the synthetic e-commerce benchmark with Qwen3-8B and LoRA, Ada-RS reduces average output tokens by up to $\exp((r_\psi-r_{\max})/\beta)$5 and thinking rate by up to $\exp((r_\psi-r_{\max})/\beta)$6 while maintaining or improving tool-call accuracy (Ge et al., 23 Feb 2026).

In offline preference optimization, "Statistical Rejection Sampling Improves Preference Optimization" gives a more classical target-distribution construction. Starting from an SFT policy $\exp((r_\psi-r_{\max})/\beta)$7 and a learned reward $\exp((r_\psi-r_{\max})/\beta)$8, it defines the KL-regularized optimal policy

$\exp((r_\psi-r_{\max})/\beta)$9

and samples from it by rejection sampling over candidates drawn from $\pi_{r_\psi}$0. Acceptance probability on a finite candidate pool is

$\pi_{r_\psi}$1

As $\pi_{r_\psi}$2, this becomes top-$\pi_{r_\psi}$3/best-of-$\pi_{r_\psi}$4-like selection; as $\pi_{r_\psi}$5, acceptance approaches $\pi_{r_\psi}$6. On Reddit TL;DR and AnthropicHH, the RSO variants consistently outperform direct DPO-style training and SFT-sampled pair generation, supporting the claim that the sampling distribution of preference pairs matters at least as much as the form of the preference loss (Liu et al., 2023).

A more minimalist perspective appears in "A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce." RAFT is formalized as rejection-sampling fine-tuning: sample $\pi_{r_\psi}$7 responses per prompt, keep only those with maximal reward, and maximize

$\pi_{r_\psi}$8

RAFT++ keeps the same reward gate,

$\pi_{r_\psi}$9

but adds importance sampling and PPO-style clipping. The paper’s central ablation is at prompt level: removing prompts whose sampled responses are all wrong explains much of GRPO’s advantage over vanilla REINFORCE, while mean-zero and standard-deviation normalization explain little. On Qwen2.5-Math-7B-base, average accuracy over Math500, Minerva Math, and Olympiad Bench is $$a(x)=\min\!\left(1,\frac{p_{\text{target}}(x)}{\lambda p_{\text{inf}}(x)}\right)$$0 for RAFT, $$a(x)=\min\!\left(1,\frac{p_{\text{target}}(x)}{\lambda p_{\text{inf}}(x)}\right)$$1 for RAFT++, $$a(x)=\min\!\left(1,\frac{p_{\text{target}}(x)}{\lambda p_{\text{inf}}(x)}\right)$$2 for Reinforce, and $$a(x)=\min\!\left(1,\frac{p_{\text{target}}(x)}{\lambda p_{\text{inf}}(x)}\right)$$3 for GRPO; on LLaMA-3.2-3B-instruct, the corresponding numbers are $$a(x)=\min\!\left(1,\frac{p_{\text{target}}(x)}{\lambda p_{\text{inf}}(x)}\right)$$4, $$a(x)=\min\!\left(1,\frac{p_{\text{target}}(x)}{\lambda p_{\text{inf}}(x)}\right)$$5, $$a(x)=\min\!\left(1,\frac{p_{\text{target}}(x)}{\lambda p_{\text{inf}}(x)}\right)$$6, and $$a(x)=\min\!\left(1,\frac{p_{\text{target}}(x)}{\lambda p_{\text{inf}}(x)}\right)$$7 (Xiong et al., 15 Apr 2025).

These training-time methods differ in objective, but all implement the same operational principle: the training distribution is shaped by a reward-gated retention mechanism rather than by using all samples uniformly.

5. Ratio-gated generalizations in RL and variational inference

Several papers broaden the idea beyond explicit reward scores and treat the gate as a function of a target-to-proposal ratio. This is not reward gating in the narrow RLHF sense, but it is a mathematically close generalization.

"Refined $$a(x)=\min\!\left(1,\frac{p_{\text{target}}(x)}{\lambda p_{\text{inf}}(x)}\right)$$8-Divergence Variational Inference via Rejection Sampling" connects proposal learning and rejection efficiency through

$$a(x)=\min\!\left(1,\frac{p_{\text{target}}(x)}{\lambda p_{\text{inf}}(x)}\right)$$9

where $\hat R_\theta(x,y)>0$00 is the optimal rejection constant for proposal $\hat R_\theta(x,y)>0$01. The method first learns $\hat R_\theta(x,y)>0$02 by minimizing a Monte Carlo Rényi-divergence objective and then refines it with acceptance probability

$\hat R_\theta(x,y)>0$03

a smoothed form of the exact clipped gate

$\hat R_\theta(x,y)>0$04

The refined distribution $\hat R_\theta(x,y)>0$05 satisfies

$\hat R_\theta(x,y)>0$06

The paper’s importance for reward-gated rejection sampling is conceptual: it shows that acceptance should often depend on score relative to proposal probability, not on score alone (Sharma et al., 2019).

"Jackpot: Optimal Budgeted Rejection Sampling for Extreme Actor-Policy Mismatch Reinforcement Learning" makes that ratio interpretation explicit for off-policy RL in LLMs. For a rollout token $\hat R_\theta(x,y)>0$07, Optimal Budget Rejection Sampling accepts with

$\hat R_\theta(x,y)>0$08

producing

$\hat R_\theta(x,y)>0$09

The main theorem states that, under an average acceptance-rate budget, this is the unique rule minimizing forward KL to the target distribution. Rewards still drive the downstream PPO/GRPO updates, but the rejection gate itself is distribution-matching-gated rather than reward-gated. This is particularly relevant when actor-policy mismatch is severe (Chen et al., 5 Feb 2026).

"Beyond Importance Sampling: Rejection-Gated Policy Optimization" internalizes the gate directly into the objective. It replaces the importance ratio with a smooth acceptance gate

$\hat R_\theta(x,y)>0$10

and yields effective gradient weight

$\hat R_\theta(x,y)>0$11

The RGPO objective is

$\hat R_\theta(x,y)>0$12

The paper proves finite bounded gradient variance even when standard IS variance diverges under heavy-tailed ratios, and gives an approximate monotonic policy-improvement guarantee analogous to TRPO. In online preference fine-tuning of Qwen2.5-1.5B-Instruct on Anthropic HH-RLHF, RGPO achieves the highest reward among online RL methods, $\hat R_\theta(x,y)>0$13 vs. PPO-RLHF, and the lowest KL divergence to the reference model, $\hat R_\theta(x,y)>0$14 vs. PPO-RLHF and $\hat R_\theta(x,y)>0$15 vs. GRPO (Sun et al., 16 Apr 2026).

Finally, "Implicit Variational Rejection Sampling" applies a smooth acceptance gate to implicit variational inference: $\hat R_\theta(x,y)>0$16 With discriminator $\hat R_\theta(x,y)>0$17, the same gate can be written

$\hat R_\theta(x,y)>0$18

The accepted distribution

$\hat R_\theta(x,y)>0$19

is shown, in the idealized setting, to move monotonically closer to the true posterior as $\hat R_\theta(x,y)>0$20 increases. This again realizes the same proposal-plus-gate architecture, but with log density ratio rather than externally defined reward as the scoring signal (Xu et al., 12 Jun 2026).

6. Conceptual boundaries, failure modes, and ongoing questions

The literature makes clear that “reward-gated rejection sampling” should not be conflated with a single canonical algorithm. Some methods are genuine rejection samplers over proposal distributions; others are gated rerankers, beam-pruning rules, or differentiable policy objectives. "A Contextual Quality Reward Model for Reliable and Efficient Best-of-N Sampling" is explicit that its method is not foregrounded as “rejection sampling,” yet operationally it is a sequential gated reranking policy with early stopping. "Accelerating LLM Reasoning via Early Rejection with Partial Reward Modeling" similarly occupies a middle ground between beam pruning and reward-gated rejection, because it rejects partial trajectories rather than completed samples. "Beyond Importance Sampling: Rejection-Gated Policy Optimization" pushes the idea further by making rejection differentiable and internal to gradient computation rather than a data preprocessing step (Rho, 5 Oct 2025, Cheshmi et al., 4 Aug 2025, Sun et al., 16 Apr 2026).

Several failure modes recur. Gate calibration is decisive: outside-option labels must truly mean “none of these are acceptable in context,” or the absolute offset of $\hat R_\theta(x,y)>0$21 is corrupted (Rho, 5 Oct 2025). In Ada-RS, the adaptive solve-rate term and the selectivity temperature $\hat R_\theta(x,y)>0$22 are sensitive hyperparameters; naive rejection sampling without the auxiliary NLL loss produces poor accuracy and always-think behavior (Ge et al., 23 Feb 2026). In VRS, prompt framing matters, exact symbolic computation of $\hat R_\theta(x,y)>0$23 does not explain success, and the procedure remains approximate because the acceptance coin is itself mediated by the LLM (Xiao et al., 11 Jun 2025). Early-rejection methods can fail when utility is delayed or non-monotonic, as in code synthesis with backtracking or tasks where early text is ambiguous but later recovery is possible (Cheshmi et al., 4 Aug 2025). RSO inherits the usual support limitation of rejection sampling: if the proposal policy rarely generates good responses, reweighting cannot recover them (Liu et al., 2023). IVRS shows an analogous pathology on the inference side: overtraining the discriminator can drive acceptance rates toward collapse and worsen performance (Xu et al., 12 Jun 2026).

A broader interpretive question is whether the gate should depend on reward alone or on reward relative to proposal probability. The variational and budgeted-RL papers argue strongly for the latter. A plausible implication is that many heuristic reward-thresholding schemes are incomplete when the actual objective is to approximate a target distribution rather than merely keep “good-looking” samples. Conversely, the acceptability-model and selective-thinking papers show that in LLM applications, an anchored or contextualized scalar can be sufficient when the operational objective is reliability, efficiency, or selective training rather than exact distributional correction (Sharma et al., 2019, Chen et al., 5 Feb 2026).

The common lesson is not that all useful rejection gates must be identical, but that sample selection becomes principled only when the meaning of the gating score is explicit. In one branch, that meaning is acceptability relative to an outside option. In another, it is efficiency-aware reward. In another, it is target-to-proposal ratio or policy trustworthiness. The modern literature therefore treats reward-gated rejection sampling less as a narrow Monte Carlo primitive and more as a general design pattern for allocating probability mass, computation, and gradient influence under uncertainty.