Adaptive Metropolis (AM)

• Adaptive Metropolis (AM) is a class of MCMC algorithms that iteratively adjusts the proposal distribution based on the empirical covariance of past samples.
• Notable variants like RAM, TSAM, DIAM, and cKAM address challenges in heavy-tailed, high-dimensional, and multimodal targets through specialized adaptation strategies.
• Theoretical guarantees such as diminishing adaptation and containment ensure ergodicity, while empirical studies demonstrate improved mixing and computational efficiency.

Adaptive Metropolis (AM) algorithms constitute a widely deployed class of Markov Chain Monte Carlo (MCMC) samplers that iteratively adapt the proposal distribution based on the evolving empirical covariance structure of the target distribution. The methodology is underpinned by rigorous theoretical analysis and continues to evolve, with extensions designed to address high-dimensionality, computational intractability, heavy tails, multimodality, and poor scaling in standard applications.

1. Core Principles of Adaptive Metropolis

Adaptive Metropolis algorithms generalize the Random Walk Metropolis-Hastings (RW-MH) approach by dynamically updating the proposal covariance using the path history of the chain. At iteration $n$, given the current state $x_n$, proposals are drawn as

$$q_n(x \mid x_n) = \mathcal{N}(x_n, s_d^2 \Sigma_n),$$

where $s_d^2$ is a scaling factor (typically $(2.38)^2 / d$ for dimension $d$, optimal for Gaussian targets) and $\Sigma_n$ is the empirical covariance of $\{x_0, ..., x_{n-1}\}$ plus a small regularization $\epsilon I$ for positive definiteness. Recursive updates for mean and covariance are employed for computational efficiency: $$\begin{aligned} \mu_{n+1} &= \mu_n + \frac{1}{n+1}(x_n - \mu_n), \ \Sigma_{n+1} &= \frac{n-1}{n}\Sigma_n + \frac{1}{n+1}(x_n - \mu_n)(x_n - \mu_n)^{T} + \epsilon I. \end{aligned}$$ Acceptance is governed by the Metropolis-Hastings ratio, which, for symmetric proposals, simplifies to $\alpha_n = \min\{1, \pi(x')/\pi(x_n)\}$.

For ergodicity, two conditions must hold: (1) diminishing adaptation ($\|\Sigma_{n+1}-\Sigma_n\| \to 0$), and (2) containment (boundedness of $\{\Sigma_n\}$). These guarantee convergence to the invariant target measure under standard regularity hypotheses (Li et al., 2022).

2. Notable Variants and Innovations

Substantial research has focused on improving classical AM to address its limitations regarding heavy-tailed distributions, dimension scaling, and mode exploration. Significant variants include:

Robust Adaptive Metropolis (RAM)

RAM replaces both sample-covariance adaptation and explicit scale adaptation with a rank-one update on a lower-triangular proposal shape matrix $S_n$, which is directly modified to simultaneously learn covariance structure and coerce the acceptance rate towards a user-specified $\alpha_*$.

$$S_n S_n^T = S_{n-1}\left[I + \eta_n (\alpha_n - \alpha_*) \frac{U_n U_n^T}{\|U_n\|^2}\right] S_{n-1}^T,$$

where $Y_n = X_{n-1} + S_{n-1} U_n$ with $U_n$ drawn from a reference (typically Gaussian or Student-$t$) distribution. RAM is specifically robust to heavy-tailed targets lacking finite second moments, where classical empirical covariance-based AM can become unstable (Vihola, 2010).

Two-Stage Adaptive Metropolis (TSAM)

TSAM targets scenarios with expensive posterior likelihoods by combining AM with a surrogate-based two-stage screening. The first stage uses a cheap $\tilde{\pi}$ to filter proposals, proceeding to the expensive evaluation $\pi(\cdot)$ only if stage 1 is accepted. Both stages involve independent accept-reject steps: $$\begin{aligned} \alpha_1(x_{t-1}, x^*) &= \min\left\{1, \frac{\tilde{\pi}(x^*)}{\tilde{\pi}(x_{t-1})}\right\}, \ \alpha_2(x_{t-1}, x') &= \min\left\{1, \frac{\pi(x') \tilde{\pi}(x_{t-1})}{\pi(x_{t-1}) \tilde{\pi}(x')}\right\}, \end{aligned}$$ with overall acceptance $\alpha = \alpha_1 \alpha_2$. TSAM realizes major computational gains in scenarios involving surrogates, subsampling, or coarser grid approximations (Mondal et al., 2021).

Dimension-Independent Adaptive Metropolis (DIAM)

DIAM adapts the proposal mechanism to achieve decorrelation times and acceptance rates independent of the dimension $d$ for Gaussian targets by employing a preconditioned Crank–Nicolson-type step: $$x_{n+1} = x_{\text{ref}} + \sqrt{1-\beta^2}(x_n - x_{\text{ref}}) + \beta A_n W_n, \quad W_n \sim \mathcal{N}(0, I), \quad A_n A_n^T = C_n,$$ where $C_n$ tracks the empirical covariance. Unlike classical AM, no factor of $1/\sqrt{d}$ is needed in scaling, and acceptance is asymptotically $1$ as $d \to \infty$ for the Gaussian case (Chen et al., 2015).

Cyclical Kernel Adaptive Metropolis (cKAM)

cKAM addresses the failure modes of adaptive methods on multimodal targets by periodically alternating between kernel-based exploration (with covariance estimated in reproducing kernel Hilbert space) and random-walk exploitation. A cyclical stepsize schedule with cosine annealing modulates exploration-exploitation balance: $$\nu_t = \frac{\nu^m_0}{2} \left[\cos(\pi r) + 1\right], \quad r = \frac{\mathrm{mod}(t-1, L)}{L},$$ where each cycle restarts the kernel adaptation on fresh samples, aiding mode switching (Li et al., 2022).

3. Theoretical Analysis and Ergodicity

Adaptive Metropolis algorithms are generally non-Markovian because the proposal depends on the entire chain history. However, under diminishing adaptation and containment conditions, ergodicity and a strong law of large numbers can be established. For TSAM, the main result states that, for bounded support and bounded target, the sample averages converge almost surely: $$\frac{1}{n+1} \sum_{t=0}^n f(X_t) \xrightarrow{\text{a.s.}} \int_D f(x) \pi(x)\, dx,$$ for every bounded measurable $f$ (Mondal et al., 2021). For RAM under appropriate tail assumptions on $\pi$ and step-size conditions, $S_n S_n^T$ converges almost surely to the target covariance shape (up to scale), and the chain is ergodic (Vihola, 2010). cKAM, as a composition of valid Metropolis-Hastings kernels, is ergodic under aperiodicity and irreducibility (Li et al., 2022).

4. Empirical Performance and Practical Implementation

Empirical performance comparisons are essential, particularly on high-dimensional, heavy-tailed, and multimodal targets:

• TSAM outperforms standard AM and Two-Stage MH on challenging simulated and real datasets by achieving matching convergence and autocorrelation, while reducing expensive evaluations via surrogate filtering. For Bayesian logistic regression with $N=41188$ observations, TSAM improved draws per minute by $\sim$50% over AM. In computational model calibration with expensive forward solves, TSAM attained a $\sim7\times$ speedup with indistinguishable posterior distributions (Mondal et al., 2021).
• RAM demonstrated superior stability over AM and ASWAM on heavy-tailed targets without finite second moments, where empirical covariance becomes ill-conditioned. It is also empirically competitive on Gaussian targets and poorly-scaled mixtures (Vihola, 2010).
• DIAM, especially with GPU acceleration and synchronized concurrent chains, enables practical sampling in hundreds to thousands of dimensions, outperforming AM, pCN, and standard RW in both mixing time and computational wall-clock time. In benchmarks at $d=1000$, DIAM achieved order-of-magnitude speedups in terms of integrated autocorrelation time and wall-clock convergence (Chen et al., 2015).
• cKAM escapes the mode-trapping issue of AM and kernel-based KAM, accurately discovering all regions of high density in multimodal distributions. On a 2D bimodal target, only cKAM was able to recover the correct posterior, with other adaptive methods adapting only to a single mode (Li et al., 2022).

Implementation considerations include $O(d^2)$ Cholesky updates for online covariance estimation, careful regularization to ensure positive definiteness, and tuning of adaptation schedules (e.g., $t_0$ for burn-in, small $\epsilon$, step-size annealing in RAM).

5. Limitations and Extensions

While AM and its family of variants provide significant improvements over static proposal MCMC in many settings, several limitations have been identified:

• For multimodal or locally-concentrated targets, global adaptation of proposal covariance may result in proposals that are too localized for effective exploration. AM, RAM, and ASWAM may require additional strategies (e.g., cyclical adaptation, local kernel covariance) to promote mode jumping (Li et al., 2022).
• The requirement of finite second moments in standard AM precludes stable adaptation in heavy-tailed target distributions, addressed by RAM.
• In very high dimensions, classical scaling of the proposal step-size ($\propto 1/\sqrt{d}$) can yield poor mixing; DIAM and pCN-inspired proposals achieve decorrelation times independent of dimension for Gaussian targets (Chen et al., 2015).
• The computational bottleneck associated with matrix operations at scale is mitigated using GPU-accelerated libraries and multi-chain parallelization, as implemented in DIAM (Chen et al., 2015).

A table of key AM variants and their main features:

Algorithm	Key Innovation	Notable Effect
AM	Online empirical covariance	Improved mixing in high dimension for sufficiently regular targets
RAM	Rank-one shape adaptation + acceptance coercion	Robustness to heavy tails, scale adaptation, affine invariance
TSAM	Surrogate-based two-stage acceptance	Reduces expensive likelihood evaluations in high-complexity models
DIAM	Dimension-independent scaling	Mixing and acceptance independent of $d$ for Gaussian targets
cKAM	Cyclical kernel-feature adaptation	Enhanced exploration in multimodal/posterior distributions

6. Application Domains and Impact

Adaptive Metropolis algorithms are central to Bayesian inference for hierarchical models, stochastic process calibration, and tall data regression where likelihoods are computationally demanding. Surrogates and subsampling in TSAM afford practical viability for simulation-based models. The RAM approach is routinely used in robust statistical inference with heavy-tailed data or in regions with poorly scaled covariance. DIAM and GPU-accelerated AM variants have enabled high-dimensional statistical computation for large-scale machine learning and computational physics (Chen et al., 2015). Mode-hopping mechanisms such as cKAM are increasingly crucial in probabilistic machine learning where targets are highly multimodal (Li et al., 2022).

7. Future Directions and Open Challenges

Ongoing work addresses theoretical and computational challenges in extending AM-style adaptation to infinite-dimensional spaces, sampling on manifolds, and non-Euclidean targets. The design of adaptive proposals for distributions with complex, non-elliptical, or spatially varying local structure remains a subject of research. Algorithmic innovations such as cyclical adaptation, kernel-based local proposals, and efficient parallelization represent promising strategies to overcome longstanding challenges of mode exploration, poor scaling, and computational cost.

Emerging directions also include rigorous diagnostics for adaptation quality and multimodality assessment, as well as automated selection of adaptation hyperparameters for robust performance across varying target structures.

---

References include (Mondal et al., 2021, Vihola, 2010, Chen et al., 2015, Li et al., 2022).