--- title: Constrained Adaptive Rejection Sampling url: https://www.emergentmind.com/topics/constrained-adaptive-rejection-sampling-cars type: topic --- # Constrained Adaptive Rejection Sampling Searching arXiv for the cited CARS papers and closely related context. arxiv_search(query="1509.07985 OR 2504.05410 OR 2510.01902", max_results=10) Constrained Adaptive Rejection Sampling (CARS) denotes adaptive rejection-based procedures that preserve a constrained target distribution while reducing repeated evaluation of invalid proposals. In the 2025 language-model literature, CARS refers to methods for exact or properly reweighted constrained generation under hard semantic or syntactic constraints, avoiding the full-vocabulary cost and posterior distortion associated with locally constrained decoding and token masking [2504.05410][2510.01902]. The acronym also has an earlier and unrelated use in Monte Carlo sampling: “Cheap Adaptive Rejection Sampling,” a fixed-node variant of adaptive rejection sampling for univariate log-concave densities [1509.07985]. ## 1. Constrained generation setting and target distributions In constrained language-model generation, the basic objective is to sample complete strings \(s \in \mathcal V^+\) that satisfy a global constraint \(c:\mathcal V^* \to \{0,1\}\). Writing the unconstrained autoregressive probability as \(p(s)=\prod_t p(s_t\mid s_{0. \] Under prefix-based decoding, the local decision problem can be expressed through an oracle \(\ell_\tau(x)=c(\tau\cdot x\,\star)\in\{0,1\}\), indicating whether any valid completion begins with token \(x\) after prefix \(\tau\). The ideal next-token law is then \[ q(x)=\frac{p(x\mid \tau)\,\ell_\tau(x)}{Z}, \qquad Z=\sum_{x\in\mathcal V} p(x\mid \tau)\ell_\tau(x). \] Naïve token masking evaluates \(\ell_\tau\) on every \(x\in\mathcal V\), renormalizes, and samples. For large vocabularies this is expensive, and the 2025 formulation explicitly characterizes locally constrained decoding as myopic because it can distort the global posterior over strings [2504.05410]. A related but distinct formulation considers the constrained distribution over full sequences \(x\in\Sigma^*\$\), \[ p_{\mathcal C}(x)=\frac{P(x)\,\mathbb I[x\in\mathcal C]}{Z}, \qquad Z=\sum_{x'\in\mathcal C}P(x'). \] This sequence-level perspective is central to the trie-based CARS construction, which operates by ruling out invalid prefixes while preserving the relative mass of all valid sequences [2510.01902]. A recurring misconception in this area is that enforcing validity during decoding is equivalent to sampling from the true conditional model. The cited works separate these notions sharply: masking-based methods enforce local admissibility, whereas CARS is designed either to sample exactly from the local constrained distribution \(q\) and provide unbiased normalizer estimates for downstream correction, or to sample exactly from the full constrained distribution \(p_{\mathcal C}\) itself [2504.05410][2510.01902]. ## 2. Token-level CARS and adaptive weighted rejection sampling The 2025 adaptive weighted rejection-sampling formulation begins from simple rejection sampling at a single decoding step. One repeatedly draws \(x\sim p(\cdot\mid\tau)\) until \(\ell_\tau(x)=1\). The accepted token is exactly distributed as \(q(x)\), but the expected number of draws is \(1/Z\), which becomes prohibitive when the valid mass \(Z\) is small [2504.05410]. Adaptive rejection sampling (ARS) improves this by maintaining a rejection set \(R\) of already tested invalid tokens. At iteration \(i\), it samples from \[ q_i(x)= \begin{cases} \dfrac{p(x\mid\tau)}{1-\sum_{r\in R}p(r\mid\tau)}, & x\notin R,\\[1ex] 0, & \text{otherwise}, \end{cases} \] and adds any newly rejected token to \(R\). This guarantees that each non-conforming token is tested at most once, with expected cost \[ \mathbb E[\#\mathrm{evals}] = \sum_{x\notin \mathcal C}\frac{p(x\mid\tau)}{p(x\mid\tau)+Z}. \] The same work then defines CARS by augmenting ARS with an additional loop so that the procedure outputs both an exact token sample \(x_s\sim q(\cdot)\) and an unbiased estimator \(\hat Z\) of the local normalizer \(Z\). In the single-loop \(M=1\) construction, the state variables are \(R\leftarrow\emptyset\), \(\psi\leftarrow 0\), and \(n\leftarrow 0\); after rejection events, \(n\) counts rejects and \(\psi\) accumulates their prior mass. The returned estimator is \[ \hat Z=\frac{1-\psi}{n+1}. \] The paper states that \(x_s\sim q(\cdot)\) exactly and \(\mathbb E[\hat Z]=Z\), via an auxiliary-variable / two-level RAVI analysis [2504.05410]. This unbiased local-normalizer estimate is the key addition that distinguishes token-level CARS from ordinary ARS in the sequential setting. In sequential Monte Carlo, the local importance-weight update is precisely \(Z\). Replacing masking-based locally constrained decoding by CARS therefore allows the proposal to remain exact at the token level while using \(\hat Z\) to preserve unbiasedness for the global posterior over strings [2504.05410]. ## 3. Trie-based sequence-level CARS A later 2025 formulation titled “Constrained Adaptive Rejection Sampling” defines CARS at the level of complete sequences rather than individual next-token draws. The method maintains a finite set \(W\subset\Sigma^*\) of prefixes that are known invalid, meaning that they cannot lead to any \(x\in\mathcal C\). For each prefix \(u\), it stores a quantity \[ p_u=\sum_{x\in\Sigma^*\$\setminus\mathrm{ext}(W)} P(x\mid u), \] the total probability mass of completions of \(u\) that avoid any extension of \(W\). These values are stored in a trie over \(\mathrm{prefix}(W)\), and when a new invalid prefix is inserted the update propagates upward so that the recursive identity \[ p_u=\sum_{a\in\Sigma_\$} P(a\mid u)\,p_{ua} \] remains valid [2510.01902]. Given the current trie, CARS samples from a reweighted distribution \[ R^W(x)=\frac{P(x)}{p_\epsilon} \quad\text{over } x\notin \mathrm{ext}(W), \qquad p_\epsilon=p_{\emptyset}. \] This can be implemented left-to-right through \[ R^W(a\mid u)=\frac{P(a\mid u)\,p_{ua}}{p_u}. \] Any token with \(p_{ua}=0\) is effectively masked out, but the masking is induced by accumulated structural knowledge about invalid prefixes rather than by exhaustively checking the full vocabulary at each step [2510.01902]. When a sampled sequence \(x\) is invalid, the algorithm performs “maximal pruning.” It finds the shortest prefix \(u\) of \(x\) that cannot extend to any \(y\in\mathcal C\), adds \(u\) to \(W\), and also adds each \(va\) such that \(v\preceq u\) and \(va\notin \mathrm{prefix}(\mathcal C)\). The paper’s pseudocode specifies that each such insertion sets \(p_{va}\leftarrow 0\) and updates the trie upward. According to the stated correctness argument, accepted samples remain exactly distributed as \[ p_{\mathcal C}(x)=\frac{P(x)\,\mathbb I[x\in\mathcal C]}{\sum_{x'\in\mathcal C}P(x')}, \] because \(W\cap\mathcal C=\emptyset\) and the relative weights of valid sequences continue to be proportional to \(P(x)\) [2510.01902]. This formulation is exact in a stronger sense than local constrained decoding: it does not merely sample locally admissible continuations, but returns full sequences from the conditional law \(p_{\mathcal C}\). The paper presents this as a middle ground between greedy constrained decoding, which enforces validity but distorts the distribution, and vanilla rejection sampling, which preserves fidelity but may waste many complete draws [2510.01902]. ## 4. Theoretical properties and runtime behavior The token-level and sequence-level CARS formulations share a common design principle: adaptive elimination of invalid mass improves efficiency without changing the intended constrained law. In the token-level setting, the runtime analysis compares three schemes. Simple rejection sampling has \[ \mathbb E[\#\mathrm{evals}]=\frac{1}{Z}=\exp(D_{KL}(q\Vert p)). \] ARS reduces this to \[ \mathbb E[\#\mathrm{evals}] = \sum_{x\notin\mathrm{cond}}\frac{p(x)}{p(x)+Z} \le |\mathcal V\setminus \mathrm{cond}|. \] For CARS with \(M=1\), the expected number of evaluations is \[ 2+\sum_{x\notin\mathrm{cond}} \left[ 2\cdot \frac{p(x)}{p(x)+Z} - \left(\frac{p(x)}{p(x)+Z}\right)^2 \right], \] which is still \(O\!\left(\sum_{x\notin\mathrm{cond}} p(x)/(p(x)+Z)\right)\). The same analysis states that, in practice, sharply peaked “distractor” tokens often dominate the invalid mass, so ARS and CARS require only a handful of constraint checks per token, and that runtime improvements are greater for better models because larger \(Z\) corresponds to smaller divergence between the constrained and unconstrained distributions [2504.05410]. The trie-based formulation emphasizes exactness and monotonicity at the sequence level. Each time an invalid prefix is added, the allowed mass \(p_\epsilon\) strictly decreases, so the acceptance rate improves monotonically. Sampling a sequence of length \(L\) requires \(L\) LM forward calls, one per next-token distribution. Trie updates after rejection may, in the worst case, add \(O(L\cdot |\Sigma|)\) nodes; updating each node’s \(p_u\) and propagating upward costs \(O(L)\), yielding worst-case \(O(L^2)\) work per rejection. The same source contrasts this with vanilla rejection sampling, which requires \(O(L/\rho)\) LM calls on average when \(\rho=\sum_{x\in\mathcal C} P(x)\) is the true acceptance probability under rejection sampling [2510.01902]. Both formulations also identify low-mass regimes as a limiting case. The token-level paper notes that when \(Z\) is extremely small, ARS and CARS may still need to exhaust nearly all non-conforming tokens, up to \(|\mathcal V\setminus \mathrm{cond}|\). It further states that cutoff thresholds introduced to avoid tiny \(\hat Z\) values make CARS biased, and it lists adaptive cutoffs, variance-controlled multi-loop schemes, batched and parallel CARS, and learned twist functions as open problems [2504.05410]. ## 5. Empirical findings across domains The empirical literature uses CARS in several distinct constrained-generation settings, but the reported comparisons are organized around a common baseline spectrum: unconstrained generation, locally constrained decoding with full-vocabulary masking, adaptive rejection-based constrained decoding, and more elaborate Monte Carlo corrections [2504.05410][2510.01902]. | Paper | Domains | Reported emphasis | |---|---|---| | [2504.05410] | Text-to-SQL, JSON, goal inference, molecular synthesis, pattern matching | Faster constrained token sampling and unbiased local weighting for SMC | | [2510.01902] | Program fuzzing, molecular generation, PDDL planning | Exact constrained sequence sampling with adaptive prefix pruning | In the adaptive weighted rejection-sampling study, pattern matching with \(|\mathcal V|\approx 100\text{K}\) is reported as follows: slow LCD achieves \(0.978 \pm 0.01\) accuracy at \(6.9\) s/ex, fast LCD reaches \(0.993 \pm 0.01\) at \(0.13\) s/ex, and SMC-CARS with \(5\) particles gives \(0.990 \pm 0.01\) at \(0.36\) s/ex. For Text-to-SQL, fast LCD runs in \(\sim 1.07\) s at \(0.569\) accuracy, versus \(0.79\) s at \(0.53\) for unconstrained generation; SMC-CARS improves this to \(0.608\) accuracy with \(M=5\) in \(5.3\) s. For JSON generation, SMC-CARS reaches \(0.903\) accuracy, compared with \(0.866\) for twist-only and \(0.781\) for fast LCD, at runtime \(\sim 10.5\) s/ex. The same paper states that goal inference and molecular design show \(>10\times\) speedups over slow LCD together with accuracy gains over unconstrained and simple baselines [2504.05410]. The trie-based sequence-level study reports three structured-generation domains. In program fuzzing with XML and no grammar prompt, RS and ARS time out at \(>2000\) draws, RSFT requires \(275\) draws for \(100\) valid seeds, and CARS requires \(215\) draws, described as \(1.63\times\) fewer. In the same setting, branch coverage after one hour of fuzzing is reported as \(9.9\%\) for CARS versus \(7.2\%\) for GCD. In molecular generation with Llama-3.1-8B, the numbers for \(100\) valid molecules are RS \(793\), ARS \(220\), CARS \(183\), GCD \(100\), and AWRS/MCMC \(1000\); the paper states that CARS matches or exceeds ARS in quality metrics and uses \(4.3\times\) fewer draws than RS and \(1.3\times\) fewer than ARS. In PDDL planning with Qwen2.5-7B, RS/ARS achieve \(38\%\)–\(54\%\) valid and time out on larger domains, RSFT gives \(51\%\) valid, CARS gives \(66\%\) valid with \(6.3\%\) prefix validity, RSFT gives \(6.4\%\), ARS gives \(4.3\%\), and GCD gives \(100\%\) valid but only \(2.0\%\) prefix validity [2510.01902]. Taken together, these results support a consistent interpretation: exactness with respect to the constrained target need not imply the inefficiency of naïve rejection sampling, and validity alone need not imply preservation of the model’s constrained distribution. The empirical contrast with GCD, full-vocabulary masking, RS, and approximate MCMC-style methods is especially pronounced in large-vocabulary or low-acceptance regimes [2504.05410][2510.01902]. ## 6. Relation to earlier “CARS” in classical Monte Carlo Before the constrained-generation usage, the acronym CARS was introduced for “Cheap Adaptive Rejection Sampling,” a method for drawing from a univariate log-concave target density \[ f(x)=\frac{1}{C_\pi}\,\pi(x), \qquad \pi(x)=\exp(V(x)), \qquad V(x)=\log \pi(x), \] with \(V''(x)\le 0\). The method fixes in advance an integer \(M\ge 1\) and a set of support points \(S=\{s_1<\dots