---
title: Nearest-Neighbor Adaptive Rejection Sampling
url: https://www.emergentmind.com/topics/nearest-neighbor-adaptive-rejection-sampling-nnars-aab1dba7-c927-4fa2-b2e2-dc0ac5aa8246
type: topic
---

# Nearest-Neighbor Adaptive Rejection Sampling

Nearest-Neighbor Adaptive Rejection Sampling (NNARS) is an adaptive Monte Carlo sampling algorithm designed for efficiently drawing independent samples from densities $f$ on $[0,1]^d$ that can be evaluated at any point but are otherwise unknown and potentially expensive to compute. NNARS achieves a minimax near-optimal rejection rate within logarithmic factors under Hölder smoothness assumptions on $f$. It advances the adaptive rejection sampling (ARS) literature by providing both tight theoretical guarantees and a practical, grid-based piecewise-constant proposal mechanism based on approximate nearest-neighbor estimation [1810.09390].

## 1. Problem Setting and Motivation

The task is to sample from an unknown density $f$ on $[0,1]^d$, assumed to satisfy $0 < c_f \leq f(x) \leq 1+H$ and a Hölder condition $|f(x)-f(y)| \leq H\|x-y\|_\infty^s$ for $0 < s \leq 1$ and constant $H$. Evaluating $f(x)$ is computationally costly. Standard rejection sampling draws from a proposal $g$ and accepts $X$ with probability $f(X)/(M g(X))$, requiring a tight envelope $M g(x) \geq f(x)$ everywhere to be efficient; otherwise, the acceptance rate can be prohibitively low. NNARS aims to automatically construct and refine such envelopes adaptively, based solely on observed $f(x)$ values, without strong parametric assumptions or requiring tractable decompositions.

## 2. Algorithmic Structure and Envelope Construction

NNARS proceeds in $K$ rounds. In each round $k$:

- A piecewise-constant proposal density $g_k$ over $[0,1]^d$ and associated rejection constant $M_k$ define the envelope $Q_k(x) = M_k g_k(x)$.
- $N_k$ candidate points are drawn using standard rejection sampling (RSS) with $(g_k, M_k)$. Accepted points are retained, and all proposals with their $f$-values are collected into the data set $\chi_k$.
- The algorithm constructs a new histogram estimator $\hat{f}(x)$ using an approximate nearest-neighbor rule: $[0,1]^d$ is partitioned by an $\ell_\infty$-grid with $|\chi_{1\cup\dots\cup k}|$-dependent resolution. For each $x$, its nearest grid cell center $C(x)$ is found, and the closest $f$-evaluated sample to that center sets $\hat{f}(x)$.
- A Hölder-based estimation error bound $\hat{r}$ is computed to ensure $| \hat{f}(x) - f(x) | \leq \hat{r}$. The new envelope is defined as $E_{k+1}(x) = \hat{f}(x) + \hat{r}$, so $f(x) \leq E_{k+1}(x)$ everywhere. The proposal is then $g_{k+1}(x) = E_{k+1}(x) / \int E_{k+1}$ and $M_{k+1} = \int E_{k+1}$.

| Step                | Key Operation                                   | Section Reference           |
|---------------------|-------------------------------------------------|-----------------------------|
| Envelope update     | Grid-based nearest-neighbor + error margin      | Envelope Construction       |
| Proposal sampling   | Piecewise-constant on adaptive grid             | Implementation Details      |
| Acceptance check    | $U \leq f(X) / Q_k(X)$ with $U \sim \mathrm{Unif}[0,1]$ | Algorithm Description       |

The procedure iteratively refines the envelope as more $f$-evaluations are acquired, shrinking the rejection constant and focusing sampling where $f$ is large.

## 3. Theoretical Guarantees and Minimax Optimality

The formal minimax risk is
$$
R_n^* = \inf_{A \in \mathrm{ARS}} \sup_{f \in F_0(s, H, c_f, d)} \mathbb{E}_f[L_n(A)],
$$
where $L_n(A, f)$ is the number of density evaluations used minus the number of accepted samples, given algorithm $A$.

- **Lower bound (Theorem 4.1):** For $n$ large enough,
$$
R_n^* \geq C_1(s, d) \cdot n^{1-s/d}
$$
for some constant $C_1 > 0$.
- **Upper bound for NNARS (Theorem 3.1):** For $n \geq O(\log^3 n)$,
$$
\mathbb{E}_f[L_n(\mathrm{NNARS})] \leq C_2 \log^2 n \cdot n^{1-s/d}
$$
with $C_2$ depending polynomially on $H, c_f^{-1}$, and $\sqrt{\log n}$.

Thus, NNARS is minimax-near-optimal for Hölder densities, matching the lower bound up to $\mathrm{polylog}(n)$ factors. This theoretical regime covers general multivariate densities without strong structural assumptions [1810.09390].

## 4. Implementation Details and Computational Complexity

The nearest-neighbor histogram is implemented via a cubic grid of side length $h_k = 1/(\lfloor |\mathcal{H}|^{1/d} \rfloor + 1)$. Each cell center in the grid stores the index of the closest sampled point. As new samples are added, updates affect only adjacent cells.

Sampling from $g_k$ consists of:

- Choosing a cell $u$ with probability proportional to $g_k(u) \cdot h_k^d$ (per-cell value times volume).
- Sampling a point uniformly within that cell.

- **Memory:** $O(n)$ for storing all sample points and their grid associations.
- **Envelope update per round:** $O(N_k \cdot |C|)$, reducible with spatial hashing; total $O(n \log n)$.
- **Sampling a proposal:** $O(\log|C|)$ for cell choice, $O(1)$ for within-cell draw.
- **Overall runtime:** $O(n \cdot \mathrm{polylog}(n))$ [1810.09390].

## 5. Empirical Comparisons and Sensitivity

Experiments benchmark NNARS against Pure Rejection Sampling (PRS), OS*/A* samplers, and Simple RS:

- For uni- and multi-modal densities with sharp peaks (e.g., $f(x) \propto e^{-x}/(1+x)^a$), NNARS achieves acceptance rates comparable to the best algorithms when no special structure is known; as $a$ increases, performance degrades in line with PRS.
- On high-dimensional product sine densities, NNARS outperforms all baselines for $d=1$ to $7$; OS*/A* samplers fail when decomposition structure is not available.
- For $f(x) \propto \exp(\sin x)$, acceptance rate approaches the asymptotic $O(1 - O(n^{-s}))$ regime after approximately $20$K density evaluations, with variance stabilizing at similar sample sizes.
- On two-dimensional real forest fire data, NNARS achieves acceptance rates near $45.7\%$, significantly above PRS ($\sim16\%$) and Simple RS ($\sim15.5\%$) [1810.09390].

These results demonstrate the robustness of NNARS, particularly in moderate to high dimensions under mild regularity.

## 6. Limitations, Parameter Choices, and Future Directions

NNARS requires specifying Hölder parameters $(s,H)$ and lower bound $c_f$ on $f$. Setting $s$ conservatively (below the true exponent) merely increases the envelope margin and worsens rates logarithmically. $c_f$ can be chosen as small as $1/\log\log n$ without affecting guarantees.

The grid-based piecewise-constant proposal incurs exponential complexity in $d$, so practical feasibility is limited to $d \lesssim 7$. Data structures such as kd-trees or spatial hashing can extend applicability to higher dimensions. *A plausible implication is that further algorithmic advances are necessary for scaling NNARS to very high-dimensional problems.*

Open research directions include:

- Extending the approach to densities that vanish (dropping the uniform lower bound $c_f>0$).
- Adapting to higher-order smoothness $(s>1)$ via polynomial envelopes instead of piecewise constant.
- Efficient nearest-neighbor updates in high dimension.
- Data-driven or adaptive estimation for $s$ and grid resolution, removing reliance on prior knowledge of $H$ [1810.09390].

Source: https://www.emergentmind.com/topics/nearest-neighbor-adaptive-rejection-sampling-nnars-aab1dba7-c927-4fa2-b2e2-dc0ac5aa8246