--- title: Ensemble Rejection Sampling (ERS) url: https://www.emergentmind.com/topics/ensemble-rejection-sampling-ers type: topic --- # Ensemble Rejection Sampling (ERS) Searching arXiv for Ensemble Rejection Sampling and closely related papers. arxiv_search query: "Ensemble Rejection Sampling state-space models channel simulation distributed compression coupled rejection sampler" Ensemble Rejection Sampling (ERS) is a rejection-sampling framework in which a single proposal is replaced by an ensemble or batch, and exactness is recovered by performing accept/reject correction on an extended space. It was introduced for exact posterior simulation of latent state trajectories in nonlinear non-Gaussian state-space models, where ensembles of candidate states at each time induce \(N^T\) candidate paths and the expected cost, under regularity conditions, grows cubically with sequence length rather than exponentially as in standard rejection sampling [2001.09188]. Subsequent work recast the same ensemble principle for channel simulation and distributed compression, and extended it to coupled sampling schemes with asymptotically maximal coupling behavior [2510.05552; 2201.09585]. ## 1. Original formulation in state-space models In its original form, ERS targets the posterior distribution of a latent path \(X_{1:T}\) given observations \(Y_{1:T}=y_{1:T}\). The target density is \[ \pi(x_{1:T})=\frac{\gamma(x_{1:T})}{Z}, \qquad \gamma(x_{1:T})=\mu(x_1)g(y_1\mid x_1)\prod_{t=2}^T f(x_t\mid x_{t-1})g(y_t\mid x_t), \] with \(\mu\) the initial density, \(f\) the transition density, \(g\) the observation density, and \(Z=p(y_{1:T})\) the marginal likelihood. ERS introduces proposal densities \(q_t(x_t)\) and the associated incremental importance weights \[ w_1(x_1)=\frac{\mu(x_1)g(y_1\mid x_1)}{q_1(x_1)}, \qquad w_t(x_{t-1},x_t)=\frac{f(x_t\mid x_{t-1})g(y_t\mid x_t)}{q_t(x_t)},\quad t\ge 2. \] Rather than proposing one complete trajectory from \(\prod_{t=1}^T q_t(x_t)\), ERS draws \(N\) candidates at each time point, thereby defining a random grid of \(N^T\) possible trajectories, and then samples one path from a discrete approximation \(\hat\pi\) supported on that grid [2001.09188]. The key approximation is \[ \hat\pi(\cdot)=\sum_{i_{1:T}\in[N]^T}\frac{W_{i_{1:T}}}{N^T\hat Z}\, \delta_{X_{1:T}^{i_{1:T}}}(\cdot), \qquad \hat Z=\frac{1}{N^T}\sum_{i_{1:T}\in[N]^T}W_{i_{1:T}}, \] where \(W_{i_{1:T}}=w(X_{1:T}^{i_{1:T}})\) is the path weight of the trajectory indexed by \(i_{1:T}\). The induced proposal is therefore not a single factorized path proposal, but a proposal assembled from ensembles of state samples. This is the defining structural feature of ERS: the proposal distribution is built from exponentially many candidate paths while remaining algorithmically accessible through dynamic programming. This construction directly addresses the standard pathology of pathwise rejection sampling in hidden Markov and Feynman–Kac models. With a naively factorized proposal, the global rejection constant typically scales as a product of per-time-step bounds, and average acceptance deteriorates exponentially in \(T\). ERS replaces that single-path proposal with a random, ensemble-defined approximation that is much closer to the posterior path law. ## 2. Extended-space exactness and algorithmic mechanism ERS is exact because the rejection step is performed on an extended space that contains the ensemble variables, the selected index path, and the selected trajectory. The extended proposal is \[ \bar q(k_{1:T},\mathbf x_{1:T},x_{1:T}) = \left\{\prod_{t=1}^T\prod_{i=1}^N q_t(x_t^i)\right\} \frac{w(x_{1:T}^{k_{1:T}})} {\sum_{i_{1:T}\in[N]^T}w(x_{1:T}^{i_{1:T}})} \, \delta_{x_{1:T}^{k_{1:T}}}(x_{1:T}), \] and the extended target is \[ \bar\pi(k_{1:T},\mathbf x_{1:T},x_{1:T}) = \frac{\pi(x_{1:T})}{N^T} \, \delta_{x_{1:T}^{k_{1:T}}}(x_{1:T}) \prod_{t=1}^T\prod_{i\ne k_t}q_t(x_t^i). \] The central identity is \[ \bar q(k_{1:T},\mathbf x_{1:T},x_{1:T})\frac{\hat Z}{Z} = \bar\pi(k_{1:T},\mathbf x_{1:T},x_{1:T}), \] which implies that rejection sampling on the extended space yields an accepted marginal trajectory exactly distributed according to \(\pi(x_{1:T})\) [2001.09188]. Operationally, ERS proceeds in three stages. First, it generates the ensembles \(X_t^1,\dots,X_t^N\sim q_t\). Second, it samples an index path \(K_{1:T}\) from the discrete approximation on the random grid. Although the grid contains \(N^T\) paths, this index path can be drawn in \(O(N^2T)\) using forward-filtering backward-sampling on an embedded \(N\)-state hidden Markov model. Third, it computes a path-independent upper bound \(\bar Z\ge \hat Z\) and accepts the selected trajectory with probability \[ \frac{\hat Z}{\bar Z}. \] The exactness result is nonasymptotic: for any \(N\ge 1\), the algorithm returns an exact sample from the posterior. For fixed \(T\), the average acceptance probability \(P_{\mathrm{ERS}}\) satisfies \(P_{\mathrm{ERS}}\to 1\) as \(N\to\infty\) [2001.09188]. ERS is therefore not an approximate particle smoother, nor an MCMC kernel whose exact target is reached only in the limit; it is an exact sampler with an ensemble-defined proposal. ## 3. Acceptance behavior, complexity, and empirical regimes The main complexity result is derived under two-sided regularity bounds \(\underline w\le w_1,w_t\le \bar w\), with \[ \delta=\left(\frac{\bar w}{\underline w}\right)^2. \] Under these assumptions, ERS satisfies \[ P_{\mathrm{ERS}}\ge \left(1+\frac{\delta-1}{N}\right)^{-T}. \] If \(N=\lceil \beta T\rceil\) for some \(\beta>0\), then \[ \liminf_{T\to\infty}P_{\mathrm{ERS}} \ge \exp\{\beta^{-1}(1-\delta)\}. \] Because proposal generation and bound computation cost \(O(N^2T)\), choosing \(N=O(T)\) yields expected cost \(O(T^3)\) per exact posterior sample, rather than exponential cost in \(T\) for ordinary rejection sampling [2001.09188]. The original paper illustrates this scaling on several models. For a rare-event path-conditioning problem with \(S=[0,1]\), \(\lambda(x)=x\), and \(\sigma=0.2\), estimated average acceptance probabilities remained stable across \(T\) when \(N/T\) was fixed: for \(T=100,250,500\), the reported acceptances were approximately \(3.19\%,2.91\%,2.82\%\) for \(N=T\); \(17.29\%,16.92\%,16.64\%\) for \(N=2T\); and \(49.00\%,47.75\%,48.50\%\) for \(N=5T\). For a nonlinear autoregressive model with \(\phi=0.9\), \(\sigma_v=0.3\), and \(\sigma_w=0.1\), one realization with \(T=500\) produced estimated acceptances of about \(0.79\%\) for \(N=500\), \(7.34\%\) for \(N=1000\), and \(24.34\%\) for \(N=2000\). In a stochastic volatility application to S\&P 500 data with \(T=200\), \(\phi=0.95\), \(\beta=0.7\), and \(\sigma=0.3\), the reported acceptance at \(N=6000\) was approximately \(4.73\%\) [2001.09188]. These results delimit the practical profile of ERS. The method replaces exponential degradation by polynomial scaling, but it does not eliminate the need for good proposal design or sufficiently tight upper bounds. Its favorable complexity statement is contingent on bounded incremental weights and on ensemble sizes that typically grow with path length. ## 4. Finite-batch ERS for channel simulation and distributed compression A later formulation of ERS shifts the setting from latent-state smoothing to channel simulation and distributed matching. Here each rejection step is performed over a batch of \(N\) proposals. With proposal \(Q_Y\), target \(P_Y\), and weights \(\lambda_{ik}=P_Y(Y_{ik})/Q_Y(Y_{ik})\), the algorithm selects a within-batch candidate via a Gumbel-max or exponential-race rule, \[ K_i^{\mathrm{cand}}=\operatorname*{argmin}_{1\le k\le N}\frac{S_{ik}}{\lambda_{ik}}, \] computes \[ \hat Z(Y_{i,1:N})=\sum_{k=1}^N \lambda_{ik}, \qquad \bar Z(Y_{i,1:N},K_i^{\mathrm{cand}}) = \hat Z(Y_{i,1:N})+\omega-\lambda_{i,K_i^{\mathrm{cand}}}, \] and accepts the batch candidate if \[ U_i\le \frac{\hat Z}{\bar Z}\cdot \mathrm{scale}. \] For \(N=1\), this reduces exactly to standard rejection sampling; for large \(N\), the within-batch importance selection becomes increasingly dominant, and ERS approaches finite-batch importance-sampling or Gumbel-max behavior while preserving exactness [2510.05552]. This formulation makes precise the sense in which ERS interpolates between ordinary rejection sampling and finite-sample importance selection. In the channel-simulation setting it yields a coding rate \[ R \le I(X;Y) + 2\log(I(X;Y)+8) + 12, \] while the same paper also shows that standard rejection sampling itself can achieve \[ R \le I(X;Y)+\log(I(X;Y)+1)+9. \] The main theoretical advance, however, is a distributed matching lemma for ERS, positioned as the rejection-sampling counterpart to the Poisson Matching Lemma of Li and Anantharam (2021) and as a generalization of the importance matching lemma of Phan et al. (2024). The paper gives matching bounds of the form \[ \Pr (Y_A=Y_B\mid Y_A=y, X=x, Z=z) \ge \left( 1+\mu'_1(N)+\frac{P^A_Y(y)}{P^B_Y(y)}(1+\mu'_2(N)) \right)^{-1}, \] with correction terms \(\mu'_1(N),\mu'_2(N)\to 0\) as \(N\to\infty\), typically at rate \(N^{-1}\) under mild assumptions [2510.05552]. The same work applies ERS to distributed compression, including synthetic Gaussian sources and MNIST-based distributed image compression. Its practical claim is not merely that ERS is exact, but that it combines finite proposal sets, explicit termination, and matching guarantees close to Poisson-process methods, which is especially relevant when decoder-side likelihood ratios are learned rather than analytically available. ## 5. Coupled extensions and neighboring exact rejection frameworks ERS has also been extended to the coupling of two target distributions. The coupled rejection sampler of Jacob, Lindsten, and Schön is modified so that each marginal rejection step is replaced by an ensemble rejection sampler, while coupling is maintained across the two ensembles. The resulting ensemble coupled rejection-sampling method samples \(N\) coupled proposal pairs \((\hat X_i,\hat Y_i)\), computes marginal importance weights \(w_i^X=p(\hat X_i)/\hat p(\hat X_i)\) and \(w_i^Y=q(\hat Y_i)/\hat q(\hat Y_i)\), samples a pair of indices \((I,J)\) from a maximal coupling of the two categorical weight distributions, and then uses a shared uniform variable for the two marginal ERS acceptance tests. For any \(N>0\), the returned marginals are exact, \[ X\sim p,\qquad Y\sim q, \] and if \(\tau\) denotes the number of ensemble attempts, \[ \mathbb E[\tau] \le \frac{N+\min(M(p,\hat p),M(q,\hat q))-1}{N}. \] When \(p\) and \(q\) have no atoms, the diagonal mass satisfies \[ \mathbb P(X=Y)\underset{N\to\infty}{\longrightarrow} \mathbb P(\hat X=\hat Y)\int \min(p(x),q(x))\,dx, \] so a duplicated common proposal with \(\mathbb P(\hat X=\hat Y)=1\) asymptotically recovers the maximal coupling probability [2201.09585]. This coupled construction clarifies an important point about ERS. The ensemble mechanism is not only a device for improving marginal acceptance; it can also be used to improve synchronized selection and meeting probabilities in coupled samplers. In the coupled setting, the ensemble layer attacks both low acceptance and low coordination between marginals. A separate but adjacent line of work studies exact adaptive rejection without using ERS terminology. Constrained Adaptive Rejection Sampling (CARS) for autoregressive language models preserves the exact constrained distribution while adaptively ruling out invalid continuations by recording them in a trie and subtracting their probability mass from future draws. Its proposal \(R^W\) is a dynamically pruned version of the original language-model distribution, and acceptance rates improve monotonically as the cache of known-invalid prefixes grows [2510.01902]. This is not ERS in the original sense, but it illustrates a closely related design principle: rejected samples can be amortized structurally rather than discarded. ## 6. Theoretical limits, terminology, and open directions A recent theoretical analysis isolates a core ERS-like problem: how many random bits must be revealed, adaptively, to determine the accept/reject event in a rejection sampler when the threshold depends on many coupled variables. In the model considered, one samples \[ (X_1,\dots,X_n)\sim \mathrm{Unif}([0,1)^n), \qquad X_{n+1}\sim \mathrm{Unif}([0,1)), \] and accepts when \[ X_{n+1}