Create and explore imaginative 3D worlds with our new product Dreamfield (opens in a new tab)
Papers
Topics
Authors
Recent
Search
2000 character limit reached

Constrained Adaptive Rejection Sampling

Updated 4 July 2026
  • CARS is an adaptive rejection sampling method that efficiently preserves constrained target distributions by eliminating repeated evaluations of invalid proposals.
  • It operates at both the token and sequence levels, enhancing constrained generation in language models while maintaining global posterior accuracy and reducing computational overhead.
  • The approach bridges classical Monte Carlo techniques and modern constrained decoding, providing exact sampling and unbiased local normalizer estimates for improved performance.

Searching arXiv for the cited CARS papers and closely related context. arxiv_search(query="(Martino et al., 2015) OR (Lipkin et al., 7 Apr 2025) OR (Parys et al., 2 Oct 2025)", max_results=10) Constrained Adaptive Rejection Sampling (CARS) denotes adaptive rejection-based procedures that preserve a constrained target distribution while reducing repeated evaluation of invalid proposals. In the 2025 language-model literature, CARS refers to methods for exact or properly reweighted constrained generation under hard semantic or syntactic constraints, avoiding the full-vocabulary cost and posterior distortion associated with locally constrained decoding and token masking (Lipkin et al., 7 Apr 2025, Parys et al., 2 Oct 2025). The acronym also has an earlier and unrelated use in Monte Carlo sampling: “Cheap Adaptive Rejection Sampling,” a fixed-node variant of adaptive rejection sampling for univariate log-concave densities (Martino et al., 2015).

1. Constrained generation setting and target distributions

In constrained language-model generation, the basic objective is to sample complete strings s∈V+s \in \mathcal V^+ that satisfy a global constraint c:V∗→{0,1}c:\mathcal V^* \to \{0,1\}. Writing the unconstrained autoregressive probability as p(s)=∏tp(st∣s<t)p(s)=\prod_t p(s_t\mid s_{<t}), the constrained target is the posterior

P(s)=p(s) c(s)G,G=∑sp(s)c(s)>0.P(s)=\frac{p(s)\,c(s)}{G}, \qquad G=\sum_s p(s)c(s)>0.

Under prefix-based decoding, the local decision problem can be expressed through an oracle ℓτ(x)=c(τ⋅x ⋆)∈{0,1}\ell_\tau(x)=c(\tau\cdot x\,\star)\in\{0,1\}, indicating whether any valid completion begins with token xx after prefix τ\tau. The ideal next-token law is then

q(x)=p(x∣τ) ℓτ(x)Z,Z=∑x∈Vp(x∣τ)ℓτ(x).q(x)=\frac{p(x\mid \tau)\,\ell_\tau(x)}{Z}, \qquad Z=\sum_{x\in\mathcal V} p(x\mid \tau)\ell_\tau(x).

Naïve token masking evaluates ℓτ\ell_\tau on every x∈Vx\in\mathcal V, renormalizes, and samples. For large vocabularies this is expensive, and the 2025 formulation explicitly characterizes locally constrained decoding as myopic because it can distort the global posterior over strings (Lipkin et al., 7 Apr 2025).

A related but distinct formulation considers the constrained distribution over full sequences $x\in\Sigma^*$$,

pC(x)=P(x) I[x∈C]Z,Z=∑x′∈CP(x′).p_{\mathcal C}(x)=\frac{P(x)\,\mathbb I[x\in\mathcal C]}{Z}, \qquad Z=\sum_{x'\in\mathcal C}P(x').

This sequence-level perspective is central to the trie-based CARS construction, which operates by ruling out invalid prefixes while preserving the relative mass of all valid sequences (Parys et al., 2 Oct 2025).

A recurring misconception in this area is that enforcing validity during decoding is equivalent to sampling from the true conditional model. The cited works separate these notions sharply: masking-based methods enforce local admissibility, whereas CARS is designed either to sample exactly from the local constrained distribution qq and provide unbiased normalizer estimates for downstream correction, or to sample exactly from the full constrained distribution pCp_{\mathcal C} itself (Lipkin et al., 7 Apr 2025, Parys et al., 2 Oct 2025).

2. Token-level CARS and adaptive weighted rejection sampling

The 2025 adaptive weighted rejection-sampling formulation begins from simple rejection sampling at a single decoding step. One repeatedly draws x∼p(⋅∣τ)x\sim p(\cdot\mid\tau) until ℓτ(x)=1\ell_\tau(x)=1. The accepted token is exactly distributed as q(x)q(x), but the expected number of draws is $1/Z$, which becomes prohibitive when the valid mass ZZ is small (Lipkin et al., 7 Apr 2025).

Adaptive rejection sampling (ARS) improves this by maintaining a rejection set RR of already tested invalid tokens. At iteration ii, it samples from

qi(x)={p(x∣τ)1−∑r∈Rp(r∣τ),x∉R,0,otherwise,q_i(x)= \begin{cases} \dfrac{p(x\mid\tau)}{1-\sum_{r\in R}p(r\mid\tau)}, & x\notin R,\\[1ex] 0, & \text{otherwise}, \end{cases}

and adds any newly rejected token to RR. This guarantees that each non-conforming token is tested at most once, with expected cost

E[#evals]=∑x∉Cp(x∣τ)p(x∣τ)+Z.\mathbb E[\#\mathrm{evals}] = \sum_{x\notin \mathcal C}\frac{p(x\mid\tau)}{p(x\mid\tau)+Z}.

The same work then defines CARS by augmenting ARS with an additional loop so that the procedure outputs both an exact token sample xs∼q(⋅)x_s\sim q(\cdot) and an unbiased estimator Z^\hat Z of the local normalizer ZZ. In the single-loop M=1M=1 construction, the state variables are R←∅R\leftarrow\emptyset, ψ←0\psi\leftarrow 0, and n←0n\leftarrow 0; after rejection events, nn counts rejects and ψ\psi accumulates their prior mass. The returned estimator is

Z^=1−ψn+1.\hat Z=\frac{1-\psi}{n+1}.

The paper states that xs∼q(⋅)x_s\sim q(\cdot) exactly and E[Z^]=Z\mathbb E[\hat Z]=Z, via an auxiliary-variable / two-level RAVI analysis (Lipkin et al., 7 Apr 2025).

This unbiased local-normalizer estimate is the key addition that distinguishes token-level CARS from ordinary ARS in the sequential setting. In sequential Monte Carlo, the local importance-weight update is precisely ZZ. Replacing masking-based locally constrained decoding by CARS therefore allows the proposal to remain exact at the token level while using Z^\hat Z to preserve unbiasedness for the global posterior over strings (Lipkin et al., 7 Apr 2025).

3. Trie-based sequence-level CARS

A later 2025 formulation titled “Constrained Adaptive Rejection Sampling” defines CARS at the level of complete sequences rather than individual next-token draws. The method maintains a finite set W⊂Σ∗W\subset\Sigma^* of prefixes that are known invalid, meaning that they cannot lead to any x∈Cx\in\mathcal C. For each prefix uu, it stores a quantity

$p_u=\sum_{x\in\Sigma^*$\setminus\mathrm{ext}(W)} P(x\mid u),$

the total probability mass of completions of uu that avoid any extension of WW. These values are stored in a trie over prefix(W)\mathrm{prefix}(W), and when a new invalid prefix is inserted the update propagates upward so that the recursive identity

$p_u=\sum_{a\in\Sigma_$} P(a\mid u)\,p_{ua}$

remains valid (Parys et al., 2 Oct 2025).

Given the current trie, CARS samples from a reweighted distribution

RW(x)=P(x)pϵover x∉ext(W),pϵ=p∅.R^W(x)=\frac{P(x)}{p_\epsilon} \quad\text{over } x\notin \mathrm{ext}(W), \qquad p_\epsilon=p_{\emptyset}.

This can be implemented left-to-right through

RW(a∣u)=P(a∣u) puapu.R^W(a\mid u)=\frac{P(a\mid u)\,p_{ua}}{p_u}.

Any token with pua=0p_{ua}=0 is effectively masked out, but the masking is induced by accumulated structural knowledge about invalid prefixes rather than by exhaustively checking the full vocabulary at each step (Parys et al., 2 Oct 2025).

When a sampled sequence xx is invalid, the algorithm performs “maximal pruning.” It finds the shortest prefix uu of xx that cannot extend to any y∈Cy\in\mathcal C, adds uu to WW, and also adds each vava such that v⪯uv\preceq u and va∉prefix(C)va\notin \mathrm{prefix}(\mathcal C). The paper’s pseudocode specifies that each such insertion sets pva←0p_{va}\leftarrow 0 and updates the trie upward. According to the stated correctness argument, accepted samples remain exactly distributed as

pC(x)=P(x) I[x∈C]∑x′∈CP(x′),p_{\mathcal C}(x)=\frac{P(x)\,\mathbb I[x\in\mathcal C]}{\sum_{x'\in\mathcal C}P(x')},

because W∩C=∅W\cap\mathcal C=\emptyset and the relative weights of valid sequences continue to be proportional to P(x)P(x) (Parys et al., 2 Oct 2025).

This formulation is exact in a stronger sense than local constrained decoding: it does not merely sample locally admissible continuations, but returns full sequences from the conditional law pCp_{\mathcal C}. The paper presents this as a middle ground between greedy constrained decoding, which enforces validity but distorts the distribution, and vanilla rejection sampling, which preserves fidelity but may waste many complete draws (Parys et al., 2 Oct 2025).

4. Theoretical properties and runtime behavior

The token-level and sequence-level CARS formulations share a common design principle: adaptive elimination of invalid mass improves efficiency without changing the intended constrained law. In the token-level setting, the runtime analysis compares three schemes. Simple rejection sampling has

E[#evals]=1Z=exp⁡(DKL(q∥p)).\mathbb E[\#\mathrm{evals}]=\frac{1}{Z}=\exp(D_{KL}(q\Vert p)).

ARS reduces this to

E[#evals]=∑x∉condp(x)p(x)+Z≤∣V∖cond∣.\mathbb E[\#\mathrm{evals}] = \sum_{x\notin\mathrm{cond}}\frac{p(x)}{p(x)+Z} \le |\mathcal V\setminus \mathrm{cond}|.

For CARS with M=1M=1, the expected number of evaluations is

2+∑x∉cond[2⋅p(x)p(x)+Z−(p(x)p(x)+Z)2],2+\sum_{x\notin\mathrm{cond}} \left[ 2\cdot \frac{p(x)}{p(x)+Z} - \left(\frac{p(x)}{p(x)+Z}\right)^2 \right],

which is still O ⁣(∑x∉condp(x)/(p(x)+Z))O\!\left(\sum_{x\notin\mathrm{cond}} p(x)/(p(x)+Z)\right). The same analysis states that, in practice, sharply peaked “distractor” tokens often dominate the invalid mass, so ARS and CARS require only a handful of constraint checks per token, and that runtime improvements are greater for better models because larger ZZ corresponds to smaller divergence between the constrained and unconstrained distributions (Lipkin et al., 7 Apr 2025).

The trie-based formulation emphasizes exactness and monotonicity at the sequence level. Each time an invalid prefix is added, the allowed mass pϵp_\epsilon strictly decreases, so the acceptance rate improves monotonically. Sampling a sequence of length LL requires LL LM forward calls, one per next-token distribution. Trie updates after rejection may, in the worst case, add O(L⋅∣Σ∣)O(L\cdot |\Sigma|) nodes; updating each node’s pup_u and propagating upward costs O(L)O(L), yielding worst-case O(L2)O(L^2) work per rejection. The same source contrasts this with vanilla rejection sampling, which requires O(L/ρ)O(L/\rho) LM calls on average when ρ=∑x∈CP(x)\rho=\sum_{x\in\mathcal C} P(x) is the true acceptance probability under rejection sampling (Parys et al., 2 Oct 2025).

Both formulations also identify low-mass regimes as a limiting case. The token-level paper notes that when ZZ is extremely small, ARS and CARS may still need to exhaust nearly all non-conforming tokens, up to ∣V∖cond∣|\mathcal V\setminus \mathrm{cond}|. It further states that cutoff thresholds introduced to avoid tiny Z^\hat Z values make CARS biased, and it lists adaptive cutoffs, variance-controlled multi-loop schemes, batched and parallel CARS, and learned twist functions as open problems (Lipkin et al., 7 Apr 2025).

5. Empirical findings across domains

The empirical literature uses CARS in several distinct constrained-generation settings, but the reported comparisons are organized around a common baseline spectrum: unconstrained generation, locally constrained decoding with full-vocabulary masking, adaptive rejection-based constrained decoding, and more elaborate Monte Carlo corrections (Lipkin et al., 7 Apr 2025, Parys et al., 2 Oct 2025).

Paper Domains Reported emphasis
(Lipkin et al., 7 Apr 2025) Text-to-SQL, JSON, goal inference, molecular synthesis, pattern matching Faster constrained token sampling and unbiased local weighting for SMC
(Parys et al., 2 Oct 2025) Program fuzzing, molecular generation, PDDL planning Exact constrained sequence sampling with adaptive prefix pruning

In the adaptive weighted rejection-sampling study, pattern matching with ∣V∣≈100K|\mathcal V|\approx 100\text{K} is reported as follows: slow LCD achieves 0.978±0.010.978 \pm 0.01 accuracy at $6.9$ s/ex, fast LCD reaches 0.993±0.010.993 \pm 0.01 at $0.13$ s/ex, and SMC-CARS with $5$ particles gives 0.990±0.010.990 \pm 0.01 at $0.36$ s/ex. For Text-to-SQL, fast LCD runs in ∼1.07\sim 1.07 s at $0.569$ accuracy, versus $0.79$ s at $0.53$ for unconstrained generation; SMC-CARS improves this to $0.608$ accuracy with M=5M=5 in $5.3$ s. For JSON generation, SMC-CARS reaches $0.903$ accuracy, compared with $0.866$ for twist-only and $0.781$ for fast LCD, at runtime ∼10.5\sim 10.5 s/ex. The same paper states that goal inference and molecular design show >10×>10\times speedups over slow LCD together with accuracy gains over unconstrained and simple baselines (Lipkin et al., 7 Apr 2025).

The trie-based sequence-level study reports three structured-generation domains. In program fuzzing with XML and no grammar prompt, RS and ARS time out at >2000>2000 draws, RSFT requires $275$ draws for $100$ valid seeds, and CARS requires $215$ draws, described as 1.63×1.63\times fewer. In the same setting, branch coverage after one hour of fuzzing is reported as 9.9%9.9\% for CARS versus 7.2%7.2\% for GCD. In molecular generation with Llama-3.1-8B, the numbers for $100$ valid molecules are RS $793$, ARS $220$, CARS $183$, GCD $100$, and AWRS/MCMC $1000$; the paper states that CARS matches or exceeds ARS in quality metrics and uses 4.3×4.3\times fewer draws than RS and 1.3×1.3\times fewer than ARS. In PDDL planning with Qwen2.5-7B, RS/ARS achieve 38%38\%–54%54\% valid and time out on larger domains, RSFT gives 51%51\% valid, CARS gives 66%66\% valid with 6.3%6.3\% prefix validity, RSFT gives 6.4%6.4\%, ARS gives 4.3%4.3\%, and GCD gives 100%100\% valid but only 2.0%2.0\% prefix validity (Parys et al., 2 Oct 2025).

Taken together, these results support a consistent interpretation: exactness with respect to the constrained target need not imply the inefficiency of naïve rejection sampling, and validity alone need not imply preservation of the model’s constrained distribution. The empirical contrast with GCD, full-vocabulary masking, RS, and approximate MCMC-style methods is especially pronounced in large-vocabulary or low-acceptance regimes (Lipkin et al., 7 Apr 2025, Parys et al., 2 Oct 2025).

6. Relation to earlier “CARS” in classical Monte Carlo

Before the constrained-generation usage, the acronym CARS was introduced for “Cheap Adaptive Rejection Sampling,” a method for drawing from a univariate log-concave target density

f(x)=1Cπ π(x),π(x)=exp⁡(V(x)),V(x)=log⁡π(x),f(x)=\frac{1}{C_\pi}\,\pi(x), \qquad \pi(x)=\exp(V(x)), \qquad V(x)=\log \pi(x),

with V′′(x)≤0V''(x)\le 0. The method fixes in advance an integer M≥1M\ge 1 and a set of support points S={s1<⋯<sM}⊂XS=\{s_1<\dots<s_M\}\subset \mathcal X. At each sis_i, it constructs the tangent line

wi(x)=V(si)+V′(si)(x−si),w_i(x)=V(s_i)+V'(s_i)(x-s_i),

and defines the global upper hull

h(x)=min⁡1≤i≤Mwi(x).h(x)=\min_{1\le i\le M} w_i(x).

The resulting proposal q(x)=exp⁡(h(x))q(x)=\exp(h(x)) is piecewise exponential, and the acceptance probability at iteration tt is

αt=CπCq,0<αt≤1.\alpha_t=\frac{C_\pi}{C_q}, \qquad 0<\alpha_t\le 1.

Unlike classical ARS, which adds every rejected sample and therefore increases proposal complexity over time, Cheap Adaptive Rejection Sampling keeps ∣S∣=M|S|=M fixed. When a proposal X′X' is rejected, it identifies the closest node

s∗=arg⁡min⁡s∈S∣s−X′∣,s^*=\arg\min_{s\in S}|s-X'|,

forms

G=S∪{X′}∖{s∗},G=S\cup\{X'\}\setminus \{s^*\},

and performs the swap only if the new normalizer CgC_g is smaller than the current CqC_q. This “swap-if-it-helps” rule makes {Cq(t)}\{C_q^{(t)}\} non-increasing, so αt\alpha_t is non-decreasing and converges to a limiting α∞<1\alpha_\infty<1, the best achievable acceptance rate with MM tangent lines. The per-iteration cost remains O(M)O(M), uniformly in tt, in contrast to standard ARS where the number of nodes grows and the per-iteration cost scales like O(Mt)O(M_t) (Martino et al., 2015).

The reported experiments illustrate the intended trade-off. For a zero-mean Gaussian target with N=5×103N=5\times 10^3 and initial M=3M=3, ARS typically grows to E[Mt]≈32\mathbb E[M_t]\approx 32 nodes with E[α]≈0.994\mathbb E[\alpha]\approx 0.994 and normalized CPU time =1.00=1.00, whereas CARS with fixed M=3M=3 remains at α≈0.872\alpha\approx 0.872 and CPU time 0.96×0.96\times; with M=10M=10, CARS reaches α≈0.956\alpha\approx 0.956 and CPU time ≈0.97×\approx 0.97\times. For a Gamma(shape=2,scale=2)\mathrm{Gamma}(\text{shape}=2,\text{scale}=2) density on [0,∞)[0,\infty), ARS with N=105N=10^5 and initial M=10M=10 grows to roughly $70$ nodes with α≈0.999\alpha\approx 0.999, whereas CARS keeps M=10M=10 and α≈0.986\alpha\approx 0.986, with CPU time up to $20$–30%30\% lower for large NN (Martino et al., 2015).

This earlier usage is conceptually distinct from constrained language-model CARS. The shared acronym reflects a common adaptive-rejection theme, but the objects being adapted are different: tangent-line envelopes for continuous log-concave densities in the 2015 method, and invalid-token or invalid-prefix sets for constrained discrete generation in the 2025 methods. A plausible implication is that the later terminology emphasizes a methodological analogy rather than a direct algorithmic inheritance (Martino et al., 2015, Lipkin et al., 7 Apr 2025, Parys et al., 2 Oct 2025).

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to Constrained Adaptive Rejection Sampling (CARS).