---
title: Rejection Sampling Fine-Tuning (RFT)
url: https://www.emergentmind.com/topics/rejection-sampling-fine-tuning-rft
type: topic
---

# Rejection Sampling Fine-Tuning (RFT)

Rejection Sampling Fine-Tuning (RFT) refers to a set of methodologies that systematically enhance the efficacy, efficiency, and applicability of rejection sampling in statistical inference, generative modeling, and large-scale machine learning. While classical rejection sampling is simple but often inefficient—especially for complex, high-dimensional, or structure-constrained distributions—RFT approaches introduce data augmentations, adaptive mechanisms, statistical optimization, algorithmic improvements, or downstream synergy with learning procedures to realize more robust and resource-efficient sampling and learning.

## 1. Core Principles of Rejection Sampling and Its Fine-Tuning

Rejection sampling generates samples from a target density $p(x)$ (often up to normalization) by sampling from a proposal $q(x)$ and accepting with probability proportional to $p(x)/Mq(x)$ for a suitable envelope constant $M$. Rejection Sampling Fine-Tuning seeks to improve one or more of:

- **Sample efficiency:** reducing redundancies, improving acceptance probability, or leveraging bits/queries optimally [2509.24290][1810.09390][2310.00300].
- **Downstream utility:** augmenting data or learning proposals for inference or generative modeling (e.g., Bayesian inference for doubly intractable models [1406.6652], reinforcement learning of LLMs [2308.01825][2503.02875][2504.13145][2505.16322][2507.01679]).
- **Algorithmic generality and theoretical guarantees:** such as minimax-optimal rates [1810.09390], multi-level variance reduction [1702.03126], or optimal bit-revelation [2509.24290].
- **Statistical or preference optimization:** aligning sampling or fine-tuning with targets defined by preferences, divergences, or reward signals [1909.07627][2309.06657][2505.16761].

A unifying feature in RFT research is the explicit control and optimization of what gets retained, what gets discarded, and how the resulting information or data is used to further tune models or downstream algorithms.

## 2. Data Augmentation via Rejection Sampling: MCMC and Inference

In high-dimensional or doubly-intractable models, naively computing marginal likelihoods or normalization constants is often infeasible. Augmenting the observed data with the sequence of rejected proposals preceding every accepted sample, as in the method of [1406.6652], yields a tractable joint distribution:
$$
p(Y, x|\theta) = \frac{f(x, \theta)}{M}\prod_{i=1}^{|Y|} \left[q(y_i|\theta) - \frac{f(y_i,\theta)}{M}\right]
$$
where $Y = \{y_1, ..., y_r\}$ are rejected proposals, $x$ is the accepted sample, $q$ is the proposal, and $M$ is the envelope. This data augmentation:

- Eliminates the need for intractable normalization constants ($Z(\theta)$).
- Enables efficient MCMC and block-wise Gibbs updates, including for parameter regimes which are "doubly intractable".
- Empirically improves mixing rates and exploration efficiency in practical applications such as flow cytometry with truncation, Bayesian inference on matrix Langevin distributions, and nonparametric GP-modulated densities.
- Facilitates the use of gradient-based transition proposals, e.g., Hamiltonian Monte Carlo, by making the joint posterior fully differentiable.

This augmentation-based RFT can be seen as fundamentally altering the state-space—unconditionally transforming an otherwise intractable marginal inference problem into a conditionally tractable (augmented) posterior, enabling conventional sampling and learning techniques to be applied effectively.

## 3. Adaptive and Optimal Envelope Construction

Many RFT approaches focus directly on reducing rejection rates or optimizing sampling costs:

- **Adaptive Envelopes:** NNARS [1810.09390] constructs a nonparametric, nearest-neighbor-based upper-bound envelope that iteratively tightens using all previously evaluated (accepted/rejected) samples. By carefully bounding the deviation of the estimate from the true density (leveraging Hölder continuity), the algorithm achieves a near minimax-optimal expected loss in terms of rejected samples: $O(\log^2 n \cdot n^{1-s/d})$.
- **Gradient-Refined Proposals:** Instead of hand-designed proposals, gradient-based refinement of parameterized proposals (e.g., GMMs) directly minimizes $\max_x f(x)/g(x)$ using a softmax-based loss for tractable backpropagation [2310.00300]. This approach yields up to $7\times$ higher acceptance rates than state-of-the-art adaptive rejection sampling baselines in low dimensions, with only the requirement that $f$ is differentiable.
- **Bit-Efficient Sampling:** [2509.24290] analyzes bitwise rejection sampling for increasing densities, showing that an alternating "reveal and test" algorithm can (in the worst case) settle accept/reject using $O(n^2)$ bits for $n$-dimensional problems, establishing strong theoretical bounds on the minimal randomness consumption needed for correct decisions.

Adaptive envelope tuning and optimal bit revelation represent dual frontiers in RFT: one reduces functional evaluation/sample complexity, the other optimizes the information-theoretic cost of the sampling process itself.

## 4. RFT in Bayesian Inference, Generative Modeling, and Learning

RFT strategies now pervade statistical learning and generative modeling:

- **RDVI and RFT Hybridization:** By minimizing the Rényi $\alpha$-divergence ($D_\alpha(p \| q)$) between target $p$ and variational $q$—with the crucial observation that $\lim_{\alpha \to \infty} D_\alpha(p \| q) = \log M(\theta)$ (with $M(\theta)$ the optimal rejection constant)—optimal variational proposals are obtained [1909.07627]. A two-stage procedure first learns $q_\theta$ via RDVI, then samples using rejection sampling with tuned $M_\theta$, yielding posteriors that capture multi-modality and heavy tails more accurately than RDVI alone.
- **Preference and Reward Optimization:** In learning from preference data (human feedback, reward models), RFT frameworks such as Statistical Rejection Sampling Optimization (RSO) [2309.06657] generate preference pairs that more closely match the "optimal" policy by performing rejection sampling over candidate responses, thereby supporting more robust and scalable learning compared to Direct Preference Optimization or offline sequence likelihood approaches.
- **Rejection Filtering / Particle Methods:** Fusion of rejection sampling with particle filtering enables Bayesian inference in memory-constrained or online settings, with only summary statistics (mean, covariance) retained rather than all accepted samples [1511.06458]. This enables effective online tracking and classification (e.g., in embedded systems or active learning settings).
- **Generative Adversarial Models and Diffusion Models:** Discriminator Rejection Sampling [1810.06758] post-hoc filters GAN outputs using their critic's density ratio estimates, substantially enhancing fidelity and sample quality. Diffusion Rejection Sampling [2405.17880] aligns reverse process transitions with true kernels using learned discriminators and rejection tests per timestep, yielding state-of-the-art FID while being computationally competitive with (and synergistic to) fast ODE-based samplers.

## 5. RFT for Data Augmentation and Self-Improving Language Models

RFT underlies the "Self-Taught Reasoner" paradigm for Large Language Models (LLMs) and agentic pipelines [2308.01825][2503.02875][2504.13145][2505.16322][2507.01679]:

- **Self-Improving Reasoning:** RFT collects correct reasoning chains generated by a (supervised) model, filters them for correctness and diversity, and augments the training data for further fine-tuning. Diversity (distinct solution strategies, non-redundant equations) is explicitly promoted to enhance generalization and robustness.
- **Data Balancing and Adaptive Sampling:** AdaSTaR [2505.16322] addresses the inefficiency of classical RFT's random sampling (over-sampling "easy" solved examples and under-sampling hard ones) with adaptive heaps and curriculum mechanisms—leveraging performance statistics to prioritize examples, balance diversity, and align training difficulty with capability.
- **Exploring Failures and Negative Trajectories:** In LLM agent learning, RFT by default ignores failed expert trajectories. Complementary approaches mine such failures for recoverable states or beneficial actions (e.g., "Next" or "Back" navigation actions), adding carefully filtered segments to training and improving out-of-distribution/generalization performance [2504.13145].
- **Hybrid Imitation-Exploration:** Prefix-RFT [2507.01679] blends demonstration (SFT) prefixes with on-policy (RFT) completions in a unified fine-tuning framework, combining the stability of behavior cloning with the flexibility of reinforcement learning, and yielding new SOTA on mathematical reasoning.
- **Prefix Self-Consistency and Efficient Tuning:** Unsupervised Prefix Fine-Tuning (UPFT) [2503.02875] leverages the empirical observation that the initial reasoning steps are highly consistent across correct and incorrect solutions. Fine-tuning on only the prefix tokens reduces the need for full rejection sampling sweeps, dramatically lowering cost with negligible impact on accuracy.

## 6. RFT in Structured and Topologically-Constrained Generation

Recent advances in RFT extend to structured output spaces such as 3D mesh generation, where fine-grained reinforcement fine-tuning (e.g., Mesh-RFT [2505.16761]) leverages locally-aware preference optimization. Key features include:

- **Masked Direct Preference Optimization (M-DPO):** Refinement at the granularity of mesh faces, with local masking functions targeting only defective or low-quality regions and objective topology-aware metrics (e.g., Boundary Edge Ratio, Topology Score) guiding the update.
- **Objective Topology Metrics:** Fine-grained metrics (quad ratio, angle quality, aspect ratio, adjacent consistency) quantify and reward local topological and geometric integrity, enabling spatially precise improvements that are not readily achieved with global reinforcement signals.
- **Empirical Gains:** Masked RL yields substantial reductions in Hausdorff Distance and significant improvements in topological regularity, outperforming global RL and baseline DPO in both objective and subjective evaluation.

## 7. Theoretical and Practical Frontiers

RFT research connects theoretical optimality with practical algorithmics:

- **Variance Reduction and Multilevel Methods:** MLMC-ABC [1702.03126] exploits telescoping sum decompositions and level-wise couplings, accelerating Approximate Bayesian Computation while retaining i.i.d. sample properties and reducing simulation burden.
- **Kernel Design in MCMC:** Reducing rejection probability, via the design of explicit rejection control transition kernels, exponentially suppresses autocorrelation time in Markov chains, sometimes outperforming all traditional kernels when plotted against rejection rate [2208.03935].
- **Computational and Scaling Implications:** Empirical studies across reasoning LLMs [2308.01825][2505.16322], preference learning [2309.06657], and generative modeling [2405.17880][1810.06758] show RFT to be robust, often more data-parsimonious, and computationally tractable—especially when augmented with adaptive sampling, gradient refinement, and statistical calibration.

## 8. Limitations, Open Problems, and Future Directions

While RFT transforms the efficiency landscape for a broad class of problems, several frontiers remain:

- **High-dimensional challenges**: The "curse of dimensionality" persists for rejection-based methods; approaches like ERS [2310.00300], while effective up to $d\leq3$, show performance plateauing in higher dimensions. Extensions involving better initialization, hierarchical mixture models, or learned invertible proposals may offer gains.
- **Adaptive and unified loss frameworks**: Theoretical understanding of combined loss strategies (e.g., the unified DPO/SLiC view [2309.06657]) is maturing, but optimal balancing and robust scheduling of imitation/exploration or statistics-based hybrid losses remains an open research agenda.
- **Failure mining and negative supervision**: Methodologies for safely leveraging partial or negative trajectories (as in EEF [2504.13145]), without introducing bias or contamination, are critical for scalable agent and reasoning learning.
- **Integration with model distillation and hybrid generative frameworks**: The interface of RFT with diffusion model distillation, consistency models, and more generally, the distillation–augmentation pipeline, represents an emerging frontier with significant impact on high-quality, low-cost generation [2405.17880].
- **Optimal bitwise implementation and randomness reduction**: The fine-tuning of randomness consumption—beyond mere acceptance rates—has implications for hardware design, privacy-preserving computing, and quantum random number generation [2509.24290].

---

Rejection Sampling Fine-Tuning encompasses a diverse and rapidly evolving set of algorithmic strategies, statistical techniques, and practical implementations, all aimed at transforming rejection sampling from a generic but blunt tool into a sophisticated paradigm for efficient, expressive, and robust inference, generation, and learning in modern statistical and machine learning systems.

Source: https://www.emergentmind.com/topics/rejection-sampling-fine-tuning-rft