Self-Guided Rejection Sampling Finetuning

• Self-Guided Rejection Sampling Finetuning is a method that generates multiple candidate outputs, evaluates them with an automatic verifier, and fine-tunes on accepted positive samples.
• It is applied across domains such as mathematical reasoning and GUI-agent training, using clear acceptance criteria like final-answer correctness and action consistency.
• Variants like RS-DPO and RIFT enhance data efficiency by leveraging reward-informed objectives and pairwise comparisons to mitigate the loss incurred from discarding negatives.

Self-Guided Rejection Sampling Finetuning (RSF) is a family of self-improvement procedures in which a model generates multiple candidate outputs for a prompt, evaluates them with an automatically available signal, accepts only a subset of candidates, and then fine-tunes on the accepted outputs. In the recent literature, RSF is often treated as synonymous with Rejection Sampling Fine-Tuning (RFT), and, in the reasoning setting of AdaSTaR, with Self-Taught Reasoners (STaR). The core pattern is stable across domains: in mathematical reasoning it accepts correct self-generated solutions; in GUI-agent training it accepts self-generated guidelines that cause a frozen downstream agent to produce the correct action; and in hybrid alignment pipelines it serves as the data-construction stage preceding stronger objectives such as DPO. The central methodological question is how acceptance is defined, how much of the sampled pool is discarded, and whether negatives should remain unused or be repurposed by a generalized objective (Liu et al., 14 Jan 2026, Xu et al., 29 Sep 2025, Koh et al., 22 May 2025).

1. Terminology, scope, and canonical identity

RSF denotes a positives-only, self-guided finetuning regime. In the standard formulation described for LLM alignment, the current policy samples $K$ candidate responses per prompt, scores them with a reward or verifier, keeps only the top-scoring response or those above a threshold, and then performs SFT on the retained samples. RS-DPO characterizes this explicitly as “best-of-$n$ SFT”: generate from the current policy, score the pool, keep $y^\*(x)$, and maximize $\log \pi(y^\* \mid x)$ (Khaki et al., 2024).

In reasoning papers, the same template is presented under the names STaR or RFT. AdaSTaR states that Self-Guided Rejection Sampling Finetuning, STaR, and RFT refer to the same training paradigm: the model generates chains-of-thought and answers, accepts only generations whose final answer is correct, and fine-tunes on those accepted chains-of-thought (Koh et al., 22 May 2025). This naming convergence is not merely terminological. It indicates that RSF is best understood as a procedural pattern rather than a single loss function.

The same accept/reject logic appears outside text-only reasoning. In RAG-GUI, RSF is the second training stage for a guideline generator $f_\theta$. The model samples candidate guideline outputs from retrieved tutorials, injects each candidate into a frozen VLM agent $\pi$, and accepts a guideline only if the agent predicts the ground-truth next action. The accepted guidelines become new finetuning targets for $f_\theta$, again with only positives retained (Xu et al., 29 Sep 2025).

A common misconception is that RSF necessarily requires human preference labels or an external reward model. The cited systems show otherwise. Reported acceptance signals include final-answer exact match, execution correctness or unit tests, step-consistency checks when available, and exact equality between predicted and ground-truth environment actions. In RAG-GUI, no external reward model or pairwise preference labels are used; in math RFT/RSFT, a simple correctness signal suffices (Liu et al., 14 Jan 2026, Xu et al., 29 Sep 2025).

2. Formal setup and canonical procedure

A standard formalization begins with an autoregressive policy $p_\theta(y \mid x)$ over trajectories $y = (y_1,\dots,y_T)$, factorized as

$$p_\theta(y \mid x) = \prod_{t=1}^{T} p_\theta(y_t \mid x, y_{<t}).$$

In self-guided settings, one constructs an off-policy buffer

$n$0

where $n$1 candidates $n$2 are sampled for each prompt and scored by a scalar verifier-derived reward $n$3 (Liu et al., 14 Jan 2026).

The canonical RSF/RFT loop is:

1. sample $n$4 trajectories for each prompt;
2. evaluate a scalar reward or verifier outcome for each sample;
3. select the best sample or thresholded positives;
4. fine-tune with standard SFT loss on accepted positives only.

With top-1 selection, the RSF objective is

$n$5

or, tokenwise,

$n$6

This is the defining positive-only feature of RSF: rejected samples do not contribute to the update (Liu et al., 14 Jan 2026).

In reasoning-oriented STaR notation, each example is $n$7, where $n$8 is the query, $n$9 is the chain-of-thought, and $y^\*(x)$0 is the final answer. The model $y^\*(x)$1 generates $y^\*(x)$2 samples $y^\*(x)$3, and acceptance is outcome-only:

$y^\*(x)$4

Accepted samples form

$y^\*(x)$5

and the model minimizes an NLL loss on the accepted rationales and, optionally, answers (Koh et al., 22 May 2025).

Reported verifier choices are domain-specific. For mathematical reasoning, the cited experiments use final-answer correctness; in RIFT’s experiments on MATH and NuminaMath, the reward is $y^\*(x)$6 for correct and $y^\*(x)$7 for incorrect answers. In GUI-agent training, acceptance is the hard indicator $y^\*(x)$8, where $y^\*(x)$9 is the frozen agent’s action conditioned on the generated guideline (Liu et al., 14 Jan 2026, Xu et al., 29 Sep 2025).

3. Objective functions and theoretical interpretations

The baseline RSF objective is algorithmically simple but statistically lossy because it discards all negatives. RIFT isolates this limitation directly: when $\log \pi(y^\* \mid x)$0 is modest and base models are not extremely strong, a large fraction of generated data is thrown away, reducing gradient-signal diversity and removing information about failure modes (Liu et al., 14 Jan 2026).

One theoretical interpretation comes from RAG-GUI, which formulates RSF through an ELBO over the probability that the frozen agent selects the correct action given a retrieved tutorial:

$\log \pi(y^\* \mid x)$1

Under the acceptance assumption

$\log \pi(y^\* \mid x)$2

the posterior over sampled guidelines is proportional to the generator probability times the hard success indicator, and the finetuning loss reduces to NLL on the accepted set:

$\log \pi(y^\* \mid x)$3

This formalizes standard RSF as positive-only posterior concentration under a hard acceptance criterion (Xu et al., 29 Sep 2025).

A second interpretation is supplied by Statistical Rejection Sampling Optimization (RSO). RSO distinguishes deterministic RSF from probabilistic rejection sampling with an explicit target distribution $\log \pi(y^\* \mid x)$4. Its acceptance probability is

$\log \pi(y^\* \mid x)$5

Within this framework, deterministic top-1 RSF appears as the limit $\log \pi(y^\* \mid x)$6: only the argmax-reward sample is accepted. This casts standard RSF as a special case of a broader rejection-sampling family, but without the exact probabilistic acceptance semantics or associated consistency guarantees emphasized in RSO (Liu et al., 2023).

RIFT provides a third interpretation by generalizing RSF from positives-only filtering to a reward-informed objective that uses all sampled trajectories. Let $\log \pi(y^\* \mid x)$7 and $\log \pi(y^\* \mid x)$8 denote positive- and negative-reward samples. RIFT defines

$\log \pi(y^\* \mid x)$9

Its practical token-level version uses reward-weighted cross-entropy for positives and a reward-weighted linear-in-probability penalty for negatives. The paper’s motivation is that naive signed reward weighting with $f_\theta$0 collapses training because for $f_\theta$1 the objective is unbounded below as $f_\theta$2, whereas the negative RIFT term has bounded constant gradient with respect to $f_\theta$3 and therefore remains numerically stable (Liu et al., 14 Jan 2026).

4. Variants, generalizations, and hybridizations

The RSF design space has expanded along three main axes: pairwise preference learning, adaptive data sampling, and task-specific acceptance signals.

RS-DPO preserves RSF’s on-policy sample generation but replaces one-sided SFT on the best sample with DPO over contrastive pairs extracted from the same sampled pool. Starting from an SFT policy $f_\theta$4, it samples $f_\theta$5 responses per prompt, scores them with a scalar reward model, and accepts preference pairs when the temperature-scaled reward gap exceeds a margin threshold. The retained pairs are then optimized with the DPO loss using $f_\theta$6 as the reference model and $f_\theta$7. The paper emphasizes that this reuses substantially more of the sampled data than RSF and reduces sensitivity to absolute reward calibration because learning depends on pairwise gaps rather than top-1 selection alone (Khaki et al., 2024).

AdaSTaR modifies the sampling policy inside the STaR/RSF loop rather than the loss. It tracks, for each observation, the last-sampled iteration $f_\theta$8 and a win statistic $f_\theta$9, where

$\pi$0

Observations are prioritized by a hierarchical min-heap: first those not sampled recently, then those with lower win rates. A curriculum mechanism updates only the top $\pi$1 considered observations, where $\pi$2 is current training accuracy and $\pi$3 is the number of considered observations. This explicitly targets the observation-imbalance problem in vanilla RSF, where easy observations are repeatedly oversampled and hard ones remain undertrained (Koh et al., 22 May 2025).

RAG-GUI adapts RSF to a retrieval-augmented VLM system. Here the object being accepted is neither a final answer nor a preference winner but a generated guideline $\pi$4 derived from a retrieved tutorial. The frozen agent $\pi$5 consumes the candidate summary $\pi$6, and acceptance requires $\pi$7. Accepted candidates with conflicting relevance labels for the same tutorial are discarded to avoid ambiguity. This variant shows that RSF need not optimize the action policy directly; it can instead optimize an auxiliary component whose outputs causally affect a frozen downstream decision-maker (Xu et al., 29 Sep 2025).

RIFT is best viewed as a strict generalization of RSF rather than a replacement of its self-guided data-generation stage. The sampling procedure remains the same—sample from the current model, verify with automatic reward—but the finetuning stage changes from positive-only SFT to a stabilized mixed-sign objective that uses both correct and incorrect trajectories. A plausible implication is that “self-guided rejection sampling” increasingly refers to the data-construction stage, while the training objective applied to that data is now an active design dimension (Liu et al., 14 Jan 2026).

5. Empirical behavior across reasoning, alignment, and GUI tasks

The empirical record shows a consistent pattern: plain RSF is effective but often data-inefficient, and methods that preserve its self-guided sampling while extracting more signal from the sampled pool usually improve performance.

On mathematical reasoning benchmarks, RIFT consistently outperforms RFT. Using MATH training buffers and $\pi$8 samples per problem, Mean@8 improved from $\pi$9 to $f_\theta$0 for Qwen2.5-Math-1.5B, from $f_\theta$1 to $f_\theta$2 for Qwen2.5-Math-7B, and from $f_\theta$3 to $f_\theta$4 for Qwen3-1.7B. Pass@8 improved from $f_\theta$5 to $f_\theta$6 for Qwen2.5-Math-1.5B and from $f_\theta$7 to $f_\theta$8 for Qwen2.5-Math-7B. For DeepSeek-R1-Distill-Qwen-1.5B, RIFT reached Mean@8 $f_\theta$9 versus $p_\theta(y \mid x)$0 for RFT and Pass@8 $p_\theta(y \mid x)$1 versus $p_\theta(y \mid x)$2 for RFT. The same study reports that the positive fraction in generated buffers often ranged from roughly $p_\theta(y \mid x)$3 to $p_\theta(y \mid x)$4, underscoring how much trajectory mass plain RSF can discard (Liu et al., 14 Jan 2026).

The same paper also reports resource trade-offs. Peak memory for RIFT was about $p_\theta(y \mid x)$5–$p_\theta(y \mid x)$6 GB, compared with about $p_\theta(y \mid x)$7–$p_\theta(y \mid x)$8 GB for DPO and about $p_\theta(y \mid x)$9–$y = (y_1,\dots,y_T)$0 GB for RFT, while RIFT achieved higher Mean@8 than DPO and used only slightly more VRAM than SFT/RFT. Its gains depended on the fraction of mixed-reward problems: on MATH the mixed fraction decreased with stronger models from $y = (y_1,\dots,y_T)$1 to $y = (y_1,\dots,y_T)$2, and RIFT’s advantage over RFT shrank accordingly; on NuminaMath, where mixed fractions were higher, the gains increased (Liu et al., 14 Jan 2026).

In pairwise alignment, RS-DPO consistently exceeds RSF on both Anthropic HH helpfulness and WebGPT. On Anthropic HH with the pythia-6.9B-RM-OA reward model, RSF achieved MT-Bench $y = (y_1,\dots,y_T)$3 and AlpacaEval $y = (y_1,\dots,y_T)$4, whereas RS-DPO reached up to MT-Bench $y = (y_1,\dots,y_T)$5 and AlpacaEval $y = (y_1,\dots,y_T)$6 depending on the margin threshold $y = (y_1,\dots,y_T)$7. On WebGPT, RS-DPO remained stronger than RSF under both the weaker RM-WG judge and the stronger RM-OA judge. The reported explanation is that RS-DPO converts one sampling round into many high-margin comparisons, while RSF discards $y = (y_1,\dots,y_T)$8 samples per prompt (Khaki et al., 2024).

In GUI-agent training, the RSF stage in RAG-GUI yields consistent gains over an otherwise identical system without RSF. The ablation “w/o RSF” shows AndroidWorld success rate improving from $y = (y_1,\dots,y_T)$9 to $$p_\theta(y \mid x) = \prod_{t=1}^{T} p_\theta(y_t \mid x, y_{<t}).$$0 with the $$p_\theta(y \mid x) = \prod_{t=1}^{T} p_\theta(y_t \mid x, y_{<t}).$$1B backbone and from $$p_\theta(y \mid x) = \prod_{t=1}^{T} p_\theta(y_t \mid x, y_{<t}).$$2 to $$p_\theta(y \mid x) = \prod_{t=1}^{T} p_\theta(y_t \mid x, y_{<t}).$$3 with the $$p_\theta(y \mid x) = \prod_{t=1}^{T} p_\theta(y_t \mid x, y_{<t}).$$4B backbone; mm-Mind2Web cross-task Step SR improving from $$p_\theta(y \mid x) = \prod_{t=1}^{T} p_\theta(y_t \mid x, y_{<t}).$$5 to $$p_\theta(y \mid x) = \prod_{t=1}^{T} p_\theta(y_t \mid x, y_{<t}).$$6 for $$p_\theta(y \mid x) = \prod_{t=1}^{T} p_\theta(y_t \mid x, y_{<t}).$$7B and from $$p_\theta(y \mid x) = \prod_{t=1}^{T} p_\theta(y_t \mid x, y_{<t}).$$8 to $$p_\theta(y \mid x) = \prod_{t=1}^{T} p_\theta(y_t \mid x, y_{<t}).$$9 for $n$00B; and AndroidControl Step Acc improving from $n$01 to $n$02 for high-level $n$03B and from $n$04 to $n$05 for low-level $n$06B. Across three tasks and two model sizes, RAG-GUI improved over inference baselines by $n$07 to $n$08 absolute (Xu et al., 29 Sep 2025).

AdaSTaR shows that substantial RSF gains can come from changing the sampling schedule rather than the acceptance criterion. Across six reasoning benchmarks, it achieved the best test accuracy in all $n$09 cases while reducing training FLOPs by an average of $n$10 relative to the strongest accuracy baseline. Illustrative results include ARC-C at $n$11 accuracy with $n$12 PFLOPs versus STaR-Acc at $n$13 with $n$14 PFLOPs, ANLI-R1 at $n$15 with $n$16 PFLOPs versus STaR-Acc at $n$17 with $n$18 PFLOPs, and GSM8K with Qwen 2.5 3B at $n$19 with $n$20 PFLOPs versus $n$21 for STaR-Acc at $n$22 PFLOPs (Koh et al., 22 May 2025).

6. Limitations, misconceptions, and open directions

The most persistent limitation of standard RSF is that it uses only positives. This is computationally attractive and stable, but it throws away negative trajectories that may encode informative failure modes. RIFT’s entire contribution is motivated by this inefficiency, and its results suggest that standard RSF is not data-optimal when a verifier can provide mixed-sign scalar rewards (Liu et al., 14 Jan 2026).

A second limitation is verifier granularity. In AdaSTaR’s main experiments, acceptance is outcome-only: a sample is kept if the final answer is correct, regardless of whether the chain-of-thought is internally sound. The paper reports that diversity-only sampling increases false positives in accepted CoTs by about $n$23 on average across four datasets relative to STaR-Acc, showing that accepted trajectories need not be process-correct. RAG-GUI exhibits an analogous ambiguity problem: conflicting relevance labels among accepted samples for the same tutorial must be discarded, which reduces usable data (Koh et al., 22 May 2025, Xu et al., 29 Sep 2025).

A third misconception is that “rejection sampling” in RSF always refers to statistically exact rejection sampling. In much of the RSF/RFT literature, acceptance is deterministic top-1 selection or thresholding. RSO argues that this should be distinguished from probabilistic rejection sampling with an explicit acceptance probability and target distribution; within that framework, best-of-$n$24 RSF is the $n$25 limit, not the general case (Liu et al., 2023).

Reward and domain assumptions also constrain applicability. RIFT explicitly notes that its upper bound is limited by the discriminative power of the reward, that subjective tasks make reward design harder, and that long CoT trajectories may require step-wise rewards because a single local error can flip a sequence-level reward negative. RAG-GUI depends on retrieval quality and a frozen agent: irrelevant tutorials, domain shift, or sparse acceptance can weaken the RSF signal. RS-DPO remains dependent on reward-model quality and margin calibration, even though pairwise gaps mitigate absolute calibration errors (Liu et al., 14 Jan 2026, Xu et al., 29 Sep 2025, Khaki et al., 2024).

Current extensions point in several directions. The cited papers propose per-step or chain-of-thought rewards, dynamic sampling and curriculum strategies, adaptive $n$26 and temperature, richer verifier integration, optional KL constraints to manage drift, better handling of negatives and ambiguity, and joint adaptation of auxiliary modules and downstream agents rather than freezing one side of the system (Liu et al., 14 Jan 2026, Xu et al., 29 Sep 2025, Koh et al., 22 May 2025). This suggests that RSF is evolving from a simple best-of-$n$27 SFT heuristic into a broader design space centered on self-generated data, verifier-based acceptance, and increasingly explicit control over how rejected samples influence optimization.