Constrained Adaptive Rejection Sampling

• CARS is an adaptive rejection sampling method that efficiently preserves constrained target distributions by eliminating repeated evaluations of invalid proposals.
• It operates at both the token and sequence levels, enhancing constrained generation in language models while maintaining global posterior accuracy and reducing computational overhead.
• The approach bridges classical Monte Carlo techniques and modern constrained decoding, providing exact sampling and unbiased local normalizer estimates for improved performance.

Searching arXiv for the cited CARS papers and closely related context. arxiv_search(query="(Martino et al., 2015) OR (Lipkin et al., 7 Apr 2025) OR (Parys et al., 2 Oct 2025)", max_results=10) Constrained Adaptive Rejection Sampling (CARS) denotes adaptive rejection-based procedures that preserve a constrained target distribution while reducing repeated evaluation of invalid proposals. In the 2025 language-model literature, CARS refers to methods for exact or properly reweighted constrained generation under hard semantic or syntactic constraints, avoiding the full-vocabulary cost and posterior distortion associated with locally constrained decoding and token masking (Lipkin et al., 7 Apr 2025, Parys et al., 2 Oct 2025). The acronym also has an earlier and unrelated use in Monte Carlo sampling: “Cheap Adaptive Rejection Sampling,” a fixed-node variant of adaptive rejection sampling for univariate log-concave densities (Martino et al., 2015).

1. Constrained generation setting and target distributions

In constrained language-model generation, the basic objective is to sample complete strings $s \in \mathcal V^+$ that satisfy a global constraint $c:\mathcal V^* \to \{0,1\}$. Writing the unconstrained autoregressive probability as $p(s)=\prod_t p(s_t\mid s_{<t})$, the constrained target is the posterior

$$P(s)=\frac{p(s)\,c(s)}{G}, \qquad G=\sum_s p(s)c(s)>0.$$

Under prefix-based decoding, the local decision problem can be expressed through an oracle $\ell_\tau(x)=c(\tau\cdot x\,\star)\in\{0,1\}$, indicating whether any valid completion begins with token $x$ after prefix $\tau$. The ideal next-token law is then

$$q(x)=\frac{p(x\mid \tau)\,\ell_\tau(x)}{Z}, \qquad Z=\sum_{x\in\mathcal V} p(x\mid \tau)\ell_\tau(x).$$

Naïve token masking evaluates $\ell_\tau$ on every $x\in\mathcal V$, renormalizes, and samples. For large vocabularies this is expensive, and the 2025 formulation explicitly characterizes locally constrained decoding as myopic because it can distort the global posterior over strings (Lipkin et al., 7 Apr 2025).

A related but distinct formulation considers the constrained distribution over full sequences $c:\mathcal V^* \to \{0,1\}$0,

$c:\mathcal V^* \to \{0,1\}$1

This sequence-level perspective is central to the trie-based CARS construction, which operates by ruling out invalid prefixes while preserving the relative mass of all valid sequences (Parys et al., 2 Oct 2025).

A recurring misconception in this area is that enforcing validity during decoding is equivalent to sampling from the true conditional model. The cited works separate these notions sharply: masking-based methods enforce local admissibility, whereas CARS is designed either to sample exactly from the local constrained distribution $c:\mathcal V^* \to \{0,1\}$2 and provide unbiased normalizer estimates for downstream correction, or to sample exactly from the full constrained distribution $c:\mathcal V^* \to \{0,1\}$3 itself (Lipkin et al., 7 Apr 2025, Parys et al., 2 Oct 2025).

2. Token-level CARS and adaptive weighted rejection sampling

The 2025 adaptive weighted rejection-sampling formulation begins from simple rejection sampling at a single decoding step. One repeatedly draws $c:\mathcal V^* \to \{0,1\}$4 until $c:\mathcal V^* \to \{0,1\}$5. The accepted token is exactly distributed as $c:\mathcal V^* \to \{0,1\}$6, but the expected number of draws is $c:\mathcal V^* \to \{0,1\}$7, which becomes prohibitive when the valid mass $c:\mathcal V^* \to \{0,1\}$8 is small (Lipkin et al., 7 Apr 2025).

Adaptive rejection sampling (ARS) improves this by maintaining a rejection set $c:\mathcal V^* \to \{0,1\}$9 of already tested invalid tokens. At iteration $p(s)=\prod_t p(s_t\mid s_{<t})$0, it samples from

$p(s)=\prod_t p(s_t\mid s_{<t})$1

and adds any newly rejected token to $p(s)=\prod_t p(s_t\mid s_{<t})$2. This guarantees that each non-conforming token is tested at most once, with expected cost

$p(s)=\prod_t p(s_t\mid s_{<t})$3

The same work then defines CARS by augmenting ARS with an additional loop so that the procedure outputs both an exact token sample $p(s)=\prod_t p(s_t\mid s_{<t})$4 and an unbiased estimator $p(s)=\prod_t p(s_t\mid s_{<t})$5 of the local normalizer $p(s)=\prod_t p(s_t\mid s_{<t})$6. In the single-loop $p(s)=\prod_t p(s_t\mid s_{<t})$7 construction, the state variables are $p(s)=\prod_t p(s_t\mid s_{<t})$8, $p(s)=\prod_t p(s_t\mid s_{<t})$9, and $$P(s)=\frac{p(s)\,c(s)}{G}, \qquad G=\sum_s p(s)c(s)>0.$$0; after rejection events, $$P(s)=\frac{p(s)\,c(s)}{G}, \qquad G=\sum_s p(s)c(s)>0.$$1 counts rejects and $$P(s)=\frac{p(s)\,c(s)}{G}, \qquad G=\sum_s p(s)c(s)>0.$$2 accumulates their prior mass. The returned estimator is

$$P(s)=\frac{p(s)\,c(s)}{G}, \qquad G=\sum_s p(s)c(s)>0.$$3

The paper states that $$P(s)=\frac{p(s)\,c(s)}{G}, \qquad G=\sum_s p(s)c(s)>0.$$4 exactly and $$P(s)=\frac{p(s)\,c(s)}{G}, \qquad G=\sum_s p(s)c(s)>0.$$5, via an auxiliary-variable / two-level RAVI analysis (Lipkin et al., 7 Apr 2025).

This unbiased local-normalizer estimate is the key addition that distinguishes token-level CARS from ordinary ARS in the sequential setting. In sequential Monte Carlo, the local importance-weight update is precisely $$P(s)=\frac{p(s)\,c(s)}{G}, \qquad G=\sum_s p(s)c(s)>0.$$6. Replacing masking-based locally constrained decoding by CARS therefore allows the proposal to remain exact at the token level while using $$P(s)=\frac{p(s)\,c(s)}{G}, \qquad G=\sum_s p(s)c(s)>0.$$7 to preserve unbiasedness for the global posterior over strings (Lipkin et al., 7 Apr 2025).

3. Trie-based sequence-level CARS

A later 2025 formulation titled “Constrained Adaptive Rejection Sampling” defines CARS at the level of complete sequences rather than individual next-token draws. The method maintains a finite set $$P(s)=\frac{p(s)\,c(s)}{G}, \qquad G=\sum_s p(s)c(s)>0.$$8 of prefixes that are known invalid, meaning that they cannot lead to any $$P(s)=\frac{p(s)\,c(s)}{G}, \qquad G=\sum_s p(s)c(s)>0.$$9. For each prefix $\ell_\tau(x)=c(\tau\cdot x\,\star)\in\{0,1\}$0, it stores a quantity

$\ell_\tau(x)=c(\tau\cdot x\,\star)\in\{0,1\}$1

the total probability mass of completions of $\ell_\tau(x)=c(\tau\cdot x\,\star)\in\{0,1\}$2 that avoid any extension of $\ell_\tau(x)=c(\tau\cdot x\,\star)\in\{0,1\}$3. These values are stored in a trie over $\ell_\tau(x)=c(\tau\cdot x\,\star)\in\{0,1\}$4, and when a new invalid prefix is inserted the update propagates upward so that the recursive identity

$\ell_\tau(x)=c(\tau\cdot x\,\star)\in\{0,1\}$5

remains valid (Parys et al., 2 Oct 2025).

Given the current trie, CARS samples from a reweighted distribution

$\ell_\tau(x)=c(\tau\cdot x\,\star)\in\{0,1\}$6

This can be implemented left-to-right through

$\ell_\tau(x)=c(\tau\cdot x\,\star)\in\{0,1\}$7

Any token with $\ell_\tau(x)=c(\tau\cdot x\,\star)\in\{0,1\}$8 is effectively masked out, but the masking is induced by accumulated structural knowledge about invalid prefixes rather than by exhaustively checking the full vocabulary at each step (Parys et al., 2 Oct 2025).

When a sampled sequence $\ell_\tau(x)=c(\tau\cdot x\,\star)\in\{0,1\}$9 is invalid, the algorithm performs “maximal pruning.” It finds the shortest prefix $x$0 of $x$1 that cannot extend to any $x$2, adds $x$3 to $x$4, and also adds each $x$5 such that $x$6 and $x$7. The paper’s pseudocode specifies that each such insertion sets $x$8 and updates the trie upward. According to the stated correctness argument, accepted samples remain exactly distributed as

$x$9

because $\tau$0 and the relative weights of valid sequences continue to be proportional to $\tau$1 (Parys et al., 2 Oct 2025).

This formulation is exact in a stronger sense than local constrained decoding: it does not merely sample locally admissible continuations, but returns full sequences from the conditional law $\tau$2. The paper presents this as a middle ground between greedy constrained decoding, which enforces validity but distorts the distribution, and vanilla rejection sampling, which preserves fidelity but may waste many complete draws (Parys et al., 2 Oct 2025).

4. Theoretical properties and runtime behavior

The token-level and sequence-level CARS formulations share a common design principle: adaptive elimination of invalid mass improves efficiency without changing the intended constrained law. In the token-level setting, the runtime analysis compares three schemes. Simple rejection sampling has

$\tau$3

ARS reduces this to

$\tau$4

For CARS with $\tau$5, the expected number of evaluations is

$\tau$6

which is still $\tau$7. The same analysis states that, in practice, sharply peaked “distractor” tokens often dominate the invalid mass, so ARS and CARS require only a handful of constraint checks per token, and that runtime improvements are greater for better models because larger $\tau$8 corresponds to smaller divergence between the constrained and unconstrained distributions (Lipkin et al., 7 Apr 2025).

The trie-based formulation emphasizes exactness and monotonicity at the sequence level. Each time an invalid prefix is added, the allowed mass $\tau$9 strictly decreases, so the acceptance rate improves monotonically. Sampling a sequence of length $$q(x)=\frac{p(x\mid \tau)\,\ell_\tau(x)}{Z}, \qquad Z=\sum_{x\in\mathcal V} p(x\mid \tau)\ell_\tau(x).$$0 requires $$q(x)=\frac{p(x\mid \tau)\,\ell_\tau(x)}{Z}, \qquad Z=\sum_{x\in\mathcal V} p(x\mid \tau)\ell_\tau(x).$$1 LM forward calls, one per next-token distribution. Trie updates after rejection may, in the worst case, add $$q(x)=\frac{p(x\mid \tau)\,\ell_\tau(x)}{Z}, \qquad Z=\sum_{x\in\mathcal V} p(x\mid \tau)\ell_\tau(x).$$2 nodes; updating each node’s $$q(x)=\frac{p(x\mid \tau)\,\ell_\tau(x)}{Z}, \qquad Z=\sum_{x\in\mathcal V} p(x\mid \tau)\ell_\tau(x).$$3 and propagating upward costs $$q(x)=\frac{p(x\mid \tau)\,\ell_\tau(x)}{Z}, \qquad Z=\sum_{x\in\mathcal V} p(x\mid \tau)\ell_\tau(x).$$4, yielding worst-case $$q(x)=\frac{p(x\mid \tau)\,\ell_\tau(x)}{Z}, \qquad Z=\sum_{x\in\mathcal V} p(x\mid \tau)\ell_\tau(x).$$5 work per rejection. The same source contrasts this with vanilla rejection sampling, which requires $$q(x)=\frac{p(x\mid \tau)\,\ell_\tau(x)}{Z}, \qquad Z=\sum_{x\in\mathcal V} p(x\mid \tau)\ell_\tau(x).$$6 LM calls on average when $$q(x)=\frac{p(x\mid \tau)\,\ell_\tau(x)}{Z}, \qquad Z=\sum_{x\in\mathcal V} p(x\mid \tau)\ell_\tau(x).$$7 is the true acceptance probability under rejection sampling (Parys et al., 2 Oct 2025).

Both formulations also identify low-mass regimes as a limiting case. The token-level paper notes that when $$q(x)=\frac{p(x\mid \tau)\,\ell_\tau(x)}{Z}, \qquad Z=\sum_{x\in\mathcal V} p(x\mid \tau)\ell_\tau(x).$$8 is extremely small, ARS and CARS may still need to exhaust nearly all non-conforming tokens, up to $$q(x)=\frac{p(x\mid \tau)\,\ell_\tau(x)}{Z}, \qquad Z=\sum_{x\in\mathcal V} p(x\mid \tau)\ell_\tau(x).$$9. It further states that cutoff thresholds introduced to avoid tiny $\ell_\tau$0 values make CARS biased, and it lists adaptive cutoffs, variance-controlled multi-loop schemes, batched and parallel CARS, and learned twist functions as open problems (Lipkin et al., 7 Apr 2025).

5. Empirical findings across domains

The empirical literature uses CARS in several distinct constrained-generation settings, but the reported comparisons are organized around a common baseline spectrum: unconstrained generation, locally constrained decoding with full-vocabulary masking, adaptive rejection-based constrained decoding, and more elaborate Monte Carlo corrections (Lipkin et al., 7 Apr 2025, Parys et al., 2 Oct 2025).

Paper	Domains	Reported emphasis
(Lipkin et al., 7 Apr 2025)	Text-to-SQL, JSON, goal inference, molecular synthesis, pattern matching	Faster constrained token sampling and unbiased local weighting for SMC
(Parys et al., 2 Oct 2025)	Program fuzzing, molecular generation, PDDL planning	Exact constrained sequence sampling with adaptive prefix pruning

In the adaptive weighted rejection-sampling study, pattern matching with $\ell_\tau$1 is reported as follows: slow LCD achieves $\ell_\tau$2 accuracy at $\ell_\tau$3 s/ex, fast LCD reaches $\ell_\tau$4 at $\ell_\tau$5 s/ex, and SMC-CARS with $\ell_\tau$6 particles gives $\ell_\tau$7 at $\ell_\tau$8 s/ex. For Text-to-SQL, fast LCD runs in $\ell_\tau$9 s at $x\in\mathcal V$0 accuracy, versus $x\in\mathcal V$1 s at $x\in\mathcal V$2 for unconstrained generation; SMC-CARS improves this to $x\in\mathcal V$3 accuracy with $x\in\mathcal V$4 in $x\in\mathcal V$5 s. For JSON generation, SMC-CARS reaches $x\in\mathcal V$6 accuracy, compared with $x\in\mathcal V$7 for twist-only and $x\in\mathcal V$8 for fast LCD, at runtime $x\in\mathcal V$9 s/ex. The same paper states that goal inference and molecular design show $c:\mathcal V^* \to \{0,1\}$00 speedups over slow LCD together with accuracy gains over unconstrained and simple baselines (Lipkin et al., 7 Apr 2025).

The trie-based sequence-level study reports three structured-generation domains. In program fuzzing with XML and no grammar prompt, RS and ARS time out at $c:\mathcal V^* \to \{0,1\}$01 draws, RSFT requires $c:\mathcal V^* \to \{0,1\}$02 draws for $c:\mathcal V^* \to \{0,1\}$03 valid seeds, and CARS requires $c:\mathcal V^* \to \{0,1\}$04 draws, described as $c:\mathcal V^* \to \{0,1\}$05 fewer. In the same setting, branch coverage after one hour of fuzzing is reported as $c:\mathcal V^* \to \{0,1\}$06 for CARS versus $c:\mathcal V^* \to \{0,1\}$07 for GCD. In molecular generation with Llama-3.1-8B, the numbers for $c:\mathcal V^* \to \{0,1\}$08 valid molecules are RS $c:\mathcal V^* \to \{0,1\}$09, ARS $c:\mathcal V^* \to \{0,1\}$10, CARS $c:\mathcal V^* \to \{0,1\}$11, GCD $c:\mathcal V^* \to \{0,1\}$12, and AWRS/MCMC $c:\mathcal V^* \to \{0,1\}$13; the paper states that CARS matches or exceeds ARS in quality metrics and uses $c:\mathcal V^* \to \{0,1\}$14 fewer draws than RS and $c:\mathcal V^* \to \{0,1\}$15 fewer than ARS. In PDDL planning with Qwen2.5-7B, RS/ARS achieve $c:\mathcal V^* \to \{0,1\}$16–$c:\mathcal V^* \to \{0,1\}$17 valid and time out on larger domains, RSFT gives $c:\mathcal V^* \to \{0,1\}$18 valid, CARS gives $c:\mathcal V^* \to \{0,1\}$19 valid with $c:\mathcal V^* \to \{0,1\}$20 prefix validity, RSFT gives $c:\mathcal V^* \to \{0,1\}$21, ARS gives $c:\mathcal V^* \to \{0,1\}$22, and GCD gives $c:\mathcal V^* \to \{0,1\}$23 valid but only $c:\mathcal V^* \to \{0,1\}$24 prefix validity (Parys et al., 2 Oct 2025).

Taken together, these results support a consistent interpretation: exactness with respect to the constrained target need not imply the inefficiency of naïve rejection sampling, and validity alone need not imply preservation of the model’s constrained distribution. The empirical contrast with GCD, full-vocabulary masking, RS, and approximate MCMC-style methods is especially pronounced in large-vocabulary or low-acceptance regimes (Lipkin et al., 7 Apr 2025, Parys et al., 2 Oct 2025).

6. Relation to earlier “CARS” in classical Monte Carlo

Before the constrained-generation usage, the acronym CARS was introduced for “Cheap Adaptive Rejection Sampling,” a method for drawing from a univariate log-concave target density

$c:\mathcal V^* \to \{0,1\}$25

with $c:\mathcal V^* \to \{0,1\}$26. The method fixes in advance an integer $c:\mathcal V^* \to \{0,1\}$27 and a set of support points $c:\mathcal V^* \to \{0,1\}$28. At each $c:\mathcal V^* \to \{0,1\}$29, it constructs the tangent line

$c:\mathcal V^* \to \{0,1\}$30

and defines the global upper hull

$c:\mathcal V^* \to \{0,1\}$31

The resulting proposal $c:\mathcal V^* \to \{0,1\}$32 is piecewise exponential, and the acceptance probability at iteration $c:\mathcal V^* \to \{0,1\}$33 is

$c:\mathcal V^* \to \{0,1\}$34

Unlike classical ARS, which adds every rejected sample and therefore increases proposal complexity over time, Cheap Adaptive Rejection Sampling keeps $c:\mathcal V^* \to \{0,1\}$35 fixed. When a proposal $c:\mathcal V^* \to \{0,1\}$36 is rejected, it identifies the closest node

$c:\mathcal V^* \to \{0,1\}$37

forms

$c:\mathcal V^* \to \{0,1\}$38

and performs the swap only if the new normalizer $c:\mathcal V^* \to \{0,1\}$39 is smaller than the current $c:\mathcal V^* \to \{0,1\}$40. This “swap-if-it-helps” rule makes $c:\mathcal V^* \to \{0,1\}$41 non-increasing, so $c:\mathcal V^* \to \{0,1\}$42 is non-decreasing and converges to a limiting $c:\mathcal V^* \to \{0,1\}$43, the best achievable acceptance rate with $c:\mathcal V^* \to \{0,1\}$44 tangent lines. The per-iteration cost remains $c:\mathcal V^* \to \{0,1\}$45, uniformly in $c:\mathcal V^* \to \{0,1\}$46, in contrast to standard ARS where the number of nodes grows and the per-iteration cost scales like $c:\mathcal V^* \to \{0,1\}$47 (Martino et al., 2015).

The reported experiments illustrate the intended trade-off. For a zero-mean Gaussian target with $c:\mathcal V^* \to \{0,1\}$48 and initial $c:\mathcal V^* \to \{0,1\}$49, ARS typically grows to $c:\mathcal V^* \to \{0,1\}$50 nodes with $c:\mathcal V^* \to \{0,1\}$51 and normalized CPU time $c:\mathcal V^* \to \{0,1\}$52, whereas CARS with fixed $c:\mathcal V^* \to \{0,1\}$53 remains at $c:\mathcal V^* \to \{0,1\}$54 and CPU time $c:\mathcal V^* \to \{0,1\}$55; with $c:\mathcal V^* \to \{0,1\}$56, CARS reaches $c:\mathcal V^* \to \{0,1\}$57 and CPU time $c:\mathcal V^* \to \{0,1\}$58. For a $c:\mathcal V^* \to \{0,1\}$59 density on $c:\mathcal V^* \to \{0,1\}$60, ARS with $c:\mathcal V^* \to \{0,1\}$61 and initial $c:\mathcal V^* \to \{0,1\}$62 grows to roughly $c:\mathcal V^* \to \{0,1\}$63 nodes with $c:\mathcal V^* \to \{0,1\}$64, whereas CARS keeps $c:\mathcal V^* \to \{0,1\}$65 and $c:\mathcal V^* \to \{0,1\}$66, with CPU time up to $c:\mathcal V^* \to \{0,1\}$67–$c:\mathcal V^* \to \{0,1\}$68 lower for large $c:\mathcal V^* \to \{0,1\}$69 (Martino et al., 2015).

This earlier usage is conceptually distinct from constrained language-model CARS. The shared acronym reflects a common adaptive-rejection theme, but the objects being adapted are different: tangent-line envelopes for continuous log-concave densities in the 2015 method, and invalid-token or invalid-prefix sets for constrained discrete generation in the 2025 methods. A plausible implication is that the later terminology emphasizes a methodological analogy rather than a direct algorithmic inheritance (Martino et al., 2015, Lipkin et al., 7 Apr 2025, Parys et al., 2 Oct 2025).