---
title: Affine-Invariant Ensemble MCMC
url: https://www.emergentmind.com/topics/affine-invariant-ensemble-mcmc-algorithm
type: topic
---

# Affine-Invariant Ensemble MCMC

An affine-invariant ensemble Markov chain Monte Carlo (MCMC) algorithm is a class of MCMC methods that maintains an ensemble of parallel “walkers”, with proposal moves and acceptance mechanisms designed so that the sampler’s behavior is unaffected by any invertible affine transformation of the target density. These methods have become canonical tools for efficiently sampling high-dimensional, strongly correlated, or highly anisotropic probability distributions, particularly when access to gradients or Hessians is unavailable. Their defining property is *affine invariance*: all proposal statistics and acceptance ratios transform properly under $x \mapsto Ax + b$ for invertible $A$ and arbitrary vector $b$.

## 1. Principles of Affine-Invariant Ensemble MCMC

The prototypical affine-invariant ensemble sampler is the Goodman–Weare (GW) “stretch move” [1104.2612, 2010.15181, 1202.3665, 1911.07688, 1509.02230]. Suppose the target is a density $\pi$ on $\mathbb{R}^M$. An ensemble of $L$ walkers $\vec{X} = (X_1, \dots, X_L) \in \mathbb{R}^{M \times L}$ is jointly updated according to a product law $\prod_{l=1}^L \pi(X_l)$. For each walker $X_i$:

- Pick another walker $X_j \ne X_i$ uniformly at random.
- Draw a stretch factor $Z \in [1/a, a]$ from $g(Z) \propto Z^{-1/2}$.
- Propose
  $$
  \tilde X_i = X_i + (1 - Z)(X_j - X_i) = X_j + Z^{-1}(X_i - X_j).
  $$
- Accept with probability
  $$
  \alpha = \min\left\{1, Z^{M-1} \frac{\pi(\tilde X_i)}{\pi(X_i)}\right\}.
  $$

Affine invariance follows because under any invertible $x \mapsto Ax + b$, the distribution and dynamics of proposals and acceptances are unchanged; the proposal and acceptance Jacobian terms exactly cancel [2010.15181, 1104.2612, 1202.3665, 1911.07688].

## 2. Algorithmic Variants and Extensions

Multiple affine-invariant ensemble algorithms have been proposed, often differing by choice of move type, adaptation scheme, or by targeting broader problem classes:

- **Stretch move (AIES):** The original GW algorithm as summarized above [2010.15181, 1104.2612, 1202.3665].
- **Ensemble Slice Sampler:** Proposes walker moves using adaptive, affine-invariant slice sampling directions determined by differences or covariance among walkers [2002.06212]. Parallel updates across walker subgroups and length-scale adaptation yield both affine invariance and robust mixing.
- **Penalised t-walk:** Extends the affine-invariant t-walk with specialized “penalty” moves to cross isolated modes, maintaining affine invariance under arbitrary full-rank affine maps [2012.02293].
- **Second-order and interacting Langevin ensemble dynamics:** Preconditioned ensemble-based Langevin samplers (e.g., EKHMC and ALDI) perform covariance-adapted diffusive sampling and are provably affine invariant, leveraging ensemble statistics for both drift and stochastic terms [2208.04506, 2506.20185].
- **Infinite-dimensional generalizations:** The functional ensemble sampler (FES) applies the AIES move on a fixed KL-truncated subspace and the pCN move on the infinite-dimensional orthogonal complement, achieving mesh-independent, gradient-free sampling in infinite-dimensional settings [2010.15181]. Other hybrid methods apply subspace-projected or covariance-inflated proposals to combine affine invariance and dimension-robustness [2202.11088].

## 3. Affine-Invariance: Theory and Proofs

For a proposal kernel $K(x, dx')$ and target density $\pi$, affine invariance requires
$$
K(Ax + b, d(Ax' + b)) = K(x, dx')
$$
for any invertible $A$, $b$, with the transformed target $\pi_T(Ax+b) = \pi(x) |\det A|^{-1}$. For the stretch move,
$$
\alpha = \min\left\{1, Z^{M-1} \frac{\pi(\tilde x)}{\pi(x)} \right\}
$$
transforms under $T(x)=Ax+b$ as
$$
\alpha_T = \min\left\{1, Z^{M-1} \frac{\pi_T(T\tilde x)}{\pi_T(Tx)} \right\} = \min\left\{1, Z^{M-1} \frac{\pi(\tilde x)}{\pi(x)} \right\}.
$$
Since the proposal and acceptance ratios are unchanged under the affine map, the process is invariant [2010.15181, 1911.07688, 1104.2612].

Generalizations (e.g., to covariance-preconditioned Langevin SDEs) use similar arguments: the drift and noise are adapted by the ensemble covariance, which transforms as $A \Sigma A^\top$ under $x \mapsto Ax + b$, ensuring affine invariance of the Fokker–Planck operator [2208.04506, 2506.20185].

## 4. Applications, Scalability, and Limitations

Affine-invariant ensemble samplers are widely used in astrophysics, Bayesian inverse problems, and high-dimensional data analysis due to their ability to handle strongly anisotropic targets without explicit covariance estimation [1104.2612, 1202.3665, 2010.15181]. 

In infinite-dimensional or function-space problems, the FES algorithm applies AIES to a fixed finite subspace (chosen via Karhunen–Loève expansion) and pCN or other schemes for the complement, ensuring mesh-independent mixing rates and robust performance as resolution increases [2010.15181]. Subspace-adjusted hybrid samplers extend these ideas to adaptively select the most informative modes for affine-invariant moves [2202.11088].

Empirically, for moderate dimension $M\lesssim 50$, ensemble stretch-move samplers offer order-of-magnitude efficiency gains (as measured by integrated autocorrelation time/IAT) over random-walk or non-adaptive slice samplers, and are robust to affine ill-conditioning [2002.06212, 1202.3665]. In very high dimensions, however, the AIES suffers from ensemble collapse—rapidly shrinking walker variance and inadequate exploration—leading to biased variance estimates and slow convergence, as rigorously characterized in [1509.02230]. Effective ensemble sizes shrink and long burn-in followed by slow “re-expansion” phases can degrade sampling unless supplemented by external regularization or more sophisticated dynamics.

## 5. Computational Structure and Implementation

A distinctive feature is ensemble parallelism: updates to walkers in one subset are independent given the positions of the complementary ensemble, enabling trivially parallel implementations. In practice, packages such as emcee [1202.3665, 1911.07688] realize these methods by splitting $L$ walkers into two groups, alternately updating each half using the current positions of the other, and exposing a modular interface for multiple affine-invariant and ensemble-adapted moves.

Tuning is minimal: the principal parameters are the stretch factor range parameter $a$ (usually $a=2$), the ensemble size $L$ (with $L > 2M$ recommended), and the dimension of the low-rank subspace in infinite-dimensional settings [2010.15181, 1202.3665]. Diagnostics rely on ensemble-wide summaries; acceptance rates between $0.2$–$0.5$ and comparison of empirical variance across independent runs are standard.

## 6. Benchmarks and Empirical Behavior

The performance of different affine-invariant ensemble algorithms is problem-dependent. Table 1 summarizes selected results from [2002.06212, 1509.02230, 2010.15181].

| Algorithm          | Setting (dim)           | IAT / Efficiency Gain      | Notes                               |
|--------------------|------------------------|---------------------------|--------------------------------------|
| AIES / GW stretch  | AR(1) Gaussian ($D=50$)| IAT ≈ 5×10⁴               | Collapse/re-expansion in $D\gg1$     |
| Ensemble Slice     | AR(1) Gaussian ($D=50$)| IAT ≈ 110 (×10-×20 gain)  | Robust, affine-invariant             |
| FES                | Advection (M=10 KL)    | IAT ≈ 1.5×10³ (×100 faster)| Infinite-dimensional, mesh-robust    |
| FES                | Langevin path rec.     | IAT(log α) ≈ 1.2×10⁴      | Beats hybrid/adaptive Gaussian RW    |

Empirical studies confirm that affine-invariant ensemble algorithms perform best for moderate-dimensional, highly anisotropic or correlated targets, and when gradient information is inaccessible or costly. In strongly multimodal/posteriors, extensions such as penalized moves or power/parallel tempering (penalized t-walk) enable improved global exploration [2012.02293]. In high dimension or infinite-dimensional inverse problems, subspace-based or hybrid ensemble-pCN methods and FES offer mixing times that do not grow with discretization size [2010.15181, 2202.11088].

## 7. Current Directions and Theoretical Developments

Active research continues on addressing high-dimensional pathologies, integrating ensemble affine-invariant dynamics with gradient-informed (Langevin or Hamiltonian) moves [2506.20185, 2208.04506], and extending affine-invariant adaptation to non-Gaussian priors, non-linear inverse problems, and rare-event or importance sampling regimes.

Theory now firmly establishes the dimension-robustness of affine-invariant moves on fixed-dimensional subspaces and in mean-field limits, but highlights the need for caution with naive application in $M\gg 50$ unless coupled with hybridization, annealing, or carefully designed subspace updates [1509.02230, 2010.15181, 2202.11088].

Affine-invariant ensemble MCMC remains indispensable for “black-box”, large-scale, and function-space Bayesian computations that require tuning-free, derivative-free, and computationally scalable sampling.

Source: https://www.emergentmind.com/topics/affine-invariant-ensemble-mcmc-algorithm