Cheap Adaptive Rejection Sampling (CARS)

Updated 19 August 2025

Cheap Adaptive Rejection Sampling (CARS) is a family of methods that efficiently generates independent samples from log-concave densities by fixing the number of support points to cap computational cost.
It adaptively refines the proposal distribution through node swapping and coarse partitioning to balance acceptance rates with bounded per-sample cost.
CARS is widely applied in large-scale Bayesian inference, particle filtering, and generative modeling to achieve significant runtime reductions over classic adaptive rejection sampling.

Cheap Adaptive Rejection Sampling (CARS) is a family of adaptive rejection sampling methodologies designed to efficiently generate independent samples from univariate, often log-concave, probability densities using a proposal distribution that adaptively tracks the shape of the target but with sharply controlled and bounded computational overhead. The distinguishing property of CARS, in contrast to classic adaptive rejection sampling (ARS), is its ability to cap complexity by limiting either the number, location, or adaptation of support points/nodes, or by leveraging computational "shortcuts" (such as partitioning, majorizing on coarse intervals, or proposal refinement) to keep per-sample or overall sampling costs constant, sublinear, or otherwise manageable. This design is motivated by the need for rapid, resource-efficient sampling in large-scale Bayesian inference, particle filtering, or other scenarios where classic ARS's adaptive complexity would be prohibitive.

1. Conceptual Foundations and Motivation

CARS arises from the recognition of a fundamental trade-off in ARS and its extensions: while progressively refining the proposal envelope by adding support points leads to higher acceptance rates and ever tighter proposals, it also results in escalating computational costs per sample due to increased envelope complexity. In many real-world applications—especially those requiring the generation of a large number of independent samples—the cost of proposal maintenance and sample extraction (including normalization and region selection in a piecewise-defined exponential envelope) can dominate overall runtime.

To address these limitations, CARS introduces a constraint: the structure of the proposal distribution (typically its number of support points or regions) is fixed a priori or adaptively bounded, and adaptation, if any, is performed via resource-efficient mechanisms such as node swapping, coarse partitioning, linear majorization, or batch proposal refinement. The objective is to maintain a balance where each sample can be generated in constant (or predictable, sublinear) time, even as acceptance rates remain high.

This foundational concept is instantiated in various forms—using piecewise exponential envelopes with bounded support points (Martino et al., 2015), mixture proposals over partitioned supports (Raim et al., 18 Jan 2024), greedy or hybrid-adaptive mechanisms (Flamich et al., 2023), and methods leveraging automatic parameterized proposals updated by targeted loss minimization (Raff et al., 2023).

2. Core Algorithmic Structures

The canonical CARS algorithm, as formulated for log-concave densities, operates as follows:

Initialization: Select a fixed number $M$ of support points $\mathcal{S}_0 = \{s_1,\ldots,s_M\}$ on the domain.
Proposal Construction: At iteration $t$ , build a piecewise exponential envelope $q_t(x)$ as the minimum over tangents to $\log \pi(x)$ at points in $\mathcal{S}_t$ , ensuring $q_t(x) \geq \pi(x)$ everywhere.
Sampling Step: Draw $x' \sim q_t(x)/c_t$ (with $c_t = \int q_t(x) dx$ ); accept $x'$ with probability $\pi(x')/q_t(x')$ .
Node Adaptation (on rejection): If $x'$ is rejected, identify the closest support point $s^*$ to $x'$ . Form a candidate node set $\mathcal{G} = \mathcal{S}_t \cup \{x'\} \setminus \{s^*\}$ . If the normalizing constant for the proposal based on $\mathcal{G}$ , denoted $c_g$ , is less than $c_t$ , accept the swap, else keep $\mathcal{S}_{t+1} = \mathcal{S}_t$ .
Iteration: Repeat until the required number of independent samples is generated.

Unlike classic ARS, CARS does not add new support points; instead, it adaptively relocates existing ones, striving to minimize the proposal's looseness (i.e., envelope normalizing constant) within a fixed budget of support points (Martino et al., 2015). As a result, the per-sample sampling and envelope-update costs are strictly bounded by $M$ , and the final average acceptance rate $\eta_\infty < 1$ is the best possible under this constraint.

Related CARS-type approaches adapt these principles for other target families: e.g., by partitioning the domain and majorizing the weighting function per region (vertical weighted strips, VWS) (Raim et al., 18 Jan 2024); by updating GMM proposals via gradient-based softmax loss targeting the acceptance threshold (ERS) (Raff et al., 2023); or by adaptive truncation/hybrid coding for channel simulation (Flamich et al., 2023).

3. Efficiency, Acceptance Rate, and Trade-offs

The crux of CARS's efficiency is that computational cost per sample is stabilized, yielding substantial speedups for large sample sizes compared to unconstrained ARS (Martino et al., 2015). Key trade-offs include:

Acceptance Rate Versus Proposal Complexity: With more support points (higher $M$ ), the proposal more closely approximates the target, yielding higher acceptance rates. However, sampling and normalization costs grow with $M$ in ARS but stay fixed in CARS.
Adaptation Criterion: Whereas ARS adds a new point after every rejection, CARS only relocates (swaps) points when this reduces the normalizing constant, preventing proposal overfitting and keeping region count constant.
Operating Regime: For applications demanding thousands or millions of samples, CARS's fixed cost per sample leads to dramatic runtime reductions (often 2–10x) even if acceptance rates remain slightly lower (typically 0.87–0.98 for modest $M$ ) (Martino et al., 2015).

Scheme	Complexity per Draw	Acceptance Rate	Notes
ARS	Grows with # nodes	→ 1 (as nodes ↑)	Increasingly costly, highest acceptance rate
CARS	Fixed (by design)	< 1 (bounded)	Acceptance rate adjusts given node budget

4. Generalizations and Recent Innovations

CARS has been generalized and extended in several notable directions:

Vertical Weighted Strips (VWS): To handle densities of the form $f(x) = w(x) g(x)$ , the VWS framework partitions the domain, majorizes $w(x)$ on each part, and forms a mixture proposal; efficiency guarantees and bounds for the rejection rate are computable a priori. The method yields controllable trade-offs between region granularity and sampler efficiency (Raim et al., 18 Jan 2024).
Parsimonious Adaptive Rejection Sampling (PARS): Rather than swapping nodes as in CARS, PARS only adds new support points if the local acceptance probability falls below a user-set threshold $\delta$ , balancing proposal complexity and runtime according to application requirements (Martino, 2017).
Gradient-Refined Proposals (ERS): Automatic proposal learning via gradient descent with softmax losses focused explicitly on minimizing the acceptance threshold, using GMMs as parameterized proposals; this leads to up to 7.3× higher acceptance rates on benchmark tasks than classic approaches (Raff et al., 2023).
Adaptive Channel Simulation (AGRS): In communication/channel simulation, adaptively truncated proposals (with carefully constructed bounds/partitions) substantially reduce the exponential expected runtime of classic GRS (governed by the Rényi $\infty$ -divergence) to linear or subexponential rates in the Kullback–Leibler divergence regime (Flamich et al., 2023).

A recurring theme is explicitly quantifying, bounding, and optimizing the rate-accuracy trade-off with minimal adaptation overhead.

5. Applications and Empirical Performance

CARS has found application in:

Large-scale Bayesian inference and simulation, where exact univariate conditional draws are required (e.g., Gibbs sampling, posterior inference of hierarchical or state-space models) (Martino et al., 2015).
Likelihood-free inference (ABC), where approximate posterior draws are needed efficiently; multilevel CARS-style estimators exploit telescopic sums and coupling for massive variance and runtime reduction (Warne et al., 2017).
Generative modeling (GAN/post-hoc sampling), where budgeted or acceptance-rate-constrained CARS algorithms bridge generator outputs to desired data distributions under arbitrary $f$ -divergence optimality (Verine et al., 2023).
LLM controlled generation, where adaptive weighted rejection sampling achieves constraint adherence with dramatically fewer constraint evaluations than full-vocabulary scans, enabled by dynamic per-sample computational focus (Lipkin et al., 7 Apr 2025).
Clustering/seeding: Approximate D²-sampling for accelerated k-means++ initialization, where CARS mitigates per-sample and per-iteration costs via efficient proposal and bounded iterations (Shah et al., 4 Feb 2025).

Empirically, across these domains, CARS yields near-identical statistical performance to ARS or other optimal sampling techniques, with order-of-magnitude improvements in runtime, especially as the number of required samples scales (Martino et al., 2015, Raim et al., 18 Jan 2024, Lipkin et al., 7 Apr 2025, Shah et al., 4 Feb 2025). Acceptance rates, although mathematically capped away from unity by proposal complexity constraints, approach 0.98 for moderate node/region counts.

6. Limitations and Open Directions

Despite these advantages, there are several considerations:

Curse of Dimensionality: Most CARS variants are best suited for univariate and low-dimensional targets; extension to high dimensions requires substantially different constructs, as the complexity of proposal maintenance scales exponentially unless structure (e.g., product form) is exploited (Raff et al., 2023).
Choice of Node Budget or Partition Granularity: The user must select the fixed node or region count, balancing runtime and accuracy. Automated tuning is an area of ongoing research.
Envelope Nonoptimality: The proposal can never exactly track the target, so acceptance is always strictly suboptimal to an unconstrained ARS with infinite adaptation.
Target Classes: CARS in its archetypal form is tailored to log-concave densities or densities amenable to envelope construction; non-convex, multimodal, or heavy-tailed targets may require adaptation (e.g., ratio-of-uniforms, majorization–minoration, VWS) (Martino et al., 2011, Raim et al., 18 Jan 2024).

Prospective improvements include automated gradient-based or information-theoretic adaptation of proposal supports, generalized partitioning schemes for multimodal or nonsmooth densities, and scalable, low-memory CARS variants for streaming or distributed environments (Raff et al., 2023, Flamich et al., 2023). There is also active interest in extending CARS concepts to privacy-aware or differentially private Monte Carlo schemes, where runtime leakage must be controlled (Awan et al., 2021).

7. Connection to the Broader Adaptive Rejection Sampling Literature

CARS is closely related to, but distinct from:

Adaptive Rejection Metropolis Sampling (ARMS): While ARMS adapts the proposal for non-log-concave densities within a Metropolis–Hastings framework, yielding correlated draws, CARS as formulated for log-concave targets retains the i.i.d. property, provided the envelope always bounds the target (Martino et al., 2012).
Probabilistic Divide-and-Conquer (PDC): PDC splits the simulation into stages to leverage conditioning and reduce rejection cost, achieving constant or sublinear per-sample cost. In the context of CARS, such divide-and-conquer strategies complement adaptation by minimizing the region of proposal refinement (Arratia et al., 2011).
Parsimonious ARS (PARS): PARS’s threshold-based support point addition can be viewed as a limit case of CARS when the threshold is set to enforce a stringent budget (Martino, 2017).
NNARS and Other Minimax Samplers: NNARS achieves minimax-optimal rejection rates via nearest-neighbor-based envelope estimation, but for general Hölder densities rather than log-concave ones; CARS, in its classic form, focuses on cost control rather than minimaxity (Achdou et al., 2018).

The overall landscape of adaptive rejection sampling techniques now includes a spectrum of compromises between acceptance rate, proposal complexity, adaptation overhead, and independence of samples, with CARS occupying a critical role where computational frugality and efficiency are paramount.

CARS represents an important methodological advance for practitioners seeking scalable, rapid, and statistically reliable exact sampling from challenging target distributions, especially when adaptation complexity and computational cost must be bounded in advance. Its continued evolution is being driven by both theoretical analysis—addressing optimality and adaptation efficiency—and a widening range of applications in statistical inference, signal processing, controlled generation, and optimization.