Probabilistic NES Algorithms

• Probabilistic NES is a class of black-box optimization methods that adapts a parameterized search distribution via natural gradient ascent guided by Fisher information.
• Variants like xNES, eNES, and SNES employ Gaussian and heavy-tailed distributions to achieve scalable updates while preserving invariance and reducing variance.
• Extensions incorporating Bayesian quadrature and discrete adaptations demonstrate ProbNES's flexibility in handling both continuous and combinatorial optimization challenges.

Searching arXiv for the core NES and ProbNES papers to ground the article in the current literature. Searching arXiv for "Natural Evolution Strategies" and related ProbNES work. arXiv search: Natural Evolution Strategies / ProbNES / related variants. Searching for core and papers on NES and ProbNES. Probabilistic Natural Evolutionary Strategy Algorithms (ProbNES) denote a probabilistic class of black-box optimization methods that maintain a parameterized search distribution over candidate solutions and adapt that distribution by ascending expected fitness in the geometry induced by the Fisher information. In the canonical Natural Evolution Strategies formulation, the objective is

$J(\theta)=\mathbb{E}_{x\sim p_\theta}[f(x)],$

and the update is driven by the natural gradient

$$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$

with $F(\theta)$ the Fisher information matrix of the search distribution (Wierstra et al., 2011). In this sense, NES is intrinsically probabilistic: optimization proceeds in distribution space rather than directly in solution space. The term “ProbNES” is therefore naturally applicable both to the original NES framework and, in later usage, to extensions that further enrich it with explicit probabilistic-numerical machinery such as Bayesian quadrature (Osselin et al., 9 Jul 2025).

1. Foundations and formal objective

ProbNES addresses black-box optimization problems in which a fitness function $f:\mathbb{R}^d\to\mathbb{R}$ can be evaluated at candidate points but is not analytically differentiated or otherwise modeled directly. The central design choice is to introduce a parametric search distribution $p_\theta(x)$ over candidate solutions and optimize the expected fitness under that distribution rather than the fitness at a single point (Wierstra et al., 2011).

The foundational objective is

$J(\theta)=\int f(x)\,p_\theta(x)\,dx,$

and its gradient follows from the log-likelihood trick: $$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[f(x)\,\nabla_\theta \log p_\theta(x)\right].$$ This is the score-function, or likelihood-ratio, estimator that underlies NES and many related stochastic-gradient procedures (Wierstra et al., 2011).

The defining step in NES is the replacement of the plain gradient by the natural gradient,

$$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$

where

$$F(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[ \nabla_\theta \log p_\theta(x)\,\nabla_\theta \log p_\theta(x)^\top \right].$$

This induces parameterization invariance: the update is normalized by the local information geometry of the distribution manifold rather than the arbitrary Euclidean coordinates of $\theta$ (Wierstra et al., 2011). Efficient Natural Evolution Strategies (eNES) formulates the same principle through the equivalent KL-constrained steepest-ascent interpretation, in which the natural gradient is the ascent direction under an infinitesimal KL trust region (Sun et al., 2012).

A Monte Carlo NES iteration samples a population $$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$0, evaluates fitness, assigns utilities $$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$1 by rank, and estimates the gradient by

$$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$2

The generic update is then

$$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$3

with learning rate $$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$4 (Wierstra et al., 2011). Rank-based utilities are typically shifted to zero mean, which acts as a baseline and reduces variance while making the algorithm invariant to monotone transformations of the fitness (Wierstra et al., 2011).

This probabilistic viewpoint links NES to broader information-geometric optimization. On Gaussian manifolds, continuous-trait replicator dynamics restricted to Gaussian distributions coincide with natural-gradient ascent of mean fitness under the Fisher–Rao metric, yielding the same parameter dynamics for mean and covariance (Jaćimović, 2022). This connection places ProbNES at the intersection of black-box optimization, information geometry, and evolutionary dynamics.

2. Core Gaussian formulations: xNES, eNES, and SNES

The best-known continuous ProbNES instantiations employ Gaussian search distributions. For a multivariate Gaussian $$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$5, the score functions are

$$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$6

$$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$7

(Wierstra et al., 2011).

The exponential Natural Evolution Strategy, or xNES, parameterizes $$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$8 as $$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$9 and performs updates in whitened coordinates $F(\theta)$0, where the Fisher matrix becomes identity for the local coordinates used by the algorithm. With utilities $F(\theta)$1, xNES forms

$F(\theta)$2

and updates

$F(\theta)$3

$F(\theta)$4

(Wierstra et al., 2011). The matrix exponential preserves symmetric positive definiteness and yields geodesic-style covariance updates. In the decomposed parameterization $F(\theta)$5 with $F(\theta)$6, separate learning rates may be used for mean, step size, and shape (Wierstra et al., 2011).

eNES takes a different route. It uses a full multivariate normal mutation distribution with covariance parameterized by an upper-triangular Cholesky factor $F(\theta)$7, $F(\theta)$8, and derives the exact Fisher information matrix for the Gaussian family in block-diagonal form (Sun et al., 2012). The key technical contribution is an $F(\theta)$9-time, $f:\mathbb{R}^d\to\mathbb{R}$0-space recursive algorithm for computing the exact inverse of the Fisher blocks, avoiding empirical Fisher estimation and its associated inversion pathologies (Sun et al., 2012). eNES also specializes the mean update to

$f:\mathbb{R}^d\to\mathbb{R}$1

while covariance-related blocks are updated analogously via exact Fisher-preconditioned score components (Sun et al., 2012).

For high-dimensional settings, separable NES (SNES) constrains the covariance to be diagonal,

$f:\mathbb{R}^d\to\mathbb{R}$2

so that sampling and updates scale as $f:\mathbb{R}^d\to\mathbb{R}$3 rather than $f:\mathbb{R}^d\to\mathbb{R}$4. With standard samples $f:\mathbb{R}^d\to\mathbb{R}$5 and $f:\mathbb{R}^d\to\mathbb{R}$6, the practical coordinate-wise updates are

$f:\mathbb{R}^d\to\mathbb{R}$7

$f:\mathbb{R}^d\to\mathbb{R}$8

(Wierstra et al., 2011). SNES sacrifices rotation invariance and cross-coordinate correlation modeling, but the linear-time structure enables scaling into thousands of dimensions (Wierstra et al., 2011).

A related low-rank formulation, Rank-One NES (R1-NES), uses

$f:\mathbb{R}^d\to\mathbb{R}$9

thereby retaining one predominant eigendirection while keeping per-sample computation linear in dimension. This allows efficient adaptation to high-dimensional non-separable problems and yielded, at publication time, the best reported 512-dimensional Rosenbrock result (Sun et al., 2011).

3. Variance reduction, invariance, and adaptive control

A defining feature of ProbNES is the systematic use of variance-reduction and robustness mechanisms around the score-function estimator. Rank-based utilities $p_\theta(x)$0 replace raw fitness values and are typically normalized to zero mean. A typical choice is the truncated logarithmic rule

$p_\theta(x)$1

(Wierstra et al., 2011). This confers invariance to monotone fitness transforms and reduces sensitivity to rescaling and shifting.

Baselines further reduce estimator variance. In the general NES formulation, zero-mean utilities already act as a baseline (Wierstra et al., 2011). eNES goes further by deriving optimal scalar and block-wise baselines that minimize the variance of the natural-gradient step without biasing it. Because the Gaussian Fisher matrix in eNES is block-diagonal, separate baselines $p_\theta(x)$2 can be assigned to parameter groups $p_\theta(x)$3, and the authors report that block baselines were robust, whereas uniform and per-parameter baselines sometimes led to premature convergence (Sun et al., 2012).

Importance mixing addresses sample reuse across consecutive search distributions. In NES, old samples from $p_\theta(x)$4 are accepted into the new batch with probability

$p_\theta(x)$5

while newly generated samples from $p_\theta(x)$6 are accepted with probability

$p_\theta(x)$7

(Wierstra et al., 2011). eNES proves that this two-stage procedure yields a batch distributed exactly according to the new search distribution and reports an empirical reduction in fitness evaluations by about a factor of $p_\theta(x)$8 (Sun et al., 2012).

Adaptation sampling provides online learning-rate control. NES tests whether a more aggressive hypothetical learning rate yields significantly better sample quality, using a weighted Mann–Whitney $p_\theta(x)$9-test, and increases or decreases the step accordingly (Wierstra et al., 2011). In the Gaussian replicator-dynamics interpretation, the natural gradient also appears as the KL-steepest-ascent direction, so KL-based step-size control is a natural complement to baseline-subtracted updates (Jaćimović, 2022).

These mechanisms clarify an important misconception: NES is not merely a Monte Carlo score estimator. Its practical behavior depends critically on natural-gradient preconditioning, fitness shaping, sample reuse, and adaptive step control. These components are part of what distinguishes ProbNES from naive derivative-free random search.

4. Heavy-tailed, low-rank, mirror, and trust-region variants

Beyond standard Gaussian families, ProbNES includes several extensions designed for specific geometric or algorithmic regimes. Heavy-tailed NES variants employ rotationally symmetric distributions such as the multivariate Cauchy. For the Cauchy case,

$J(\theta)=\int f(x)\,p_\theta(x)\,dx,$0

and the score functions in local coordinates become

$J(\theta)=\int f(x)\,p_\theta(x)\,dx,$1

(Wierstra et al., 2011). These heavy-tailed distributions increase the probability of long jumps and were reported to improve robustness on deceptive multimodal benchmarks such as double-funnel Rosenbrock and random-basin functions (Wierstra et al., 2011).

Mirror Natural Evolution Strategies (MiNES) modifies the optimization target itself. Rather than optimizing $J(\theta)=\int f(x)\,p_\theta(x)\,dx,$2 directly, MiNES introduces the regularized objective

$J(\theta)=\int f(x)\,p_\theta(x)\,dx,$3

with the search covariance written as $J(\theta)=\int f(x)\,p_\theta(x)\,dx,$4 (Ye et al., 2019). The covariance update is then performed by mirror descent in the precision domain,

$J(\theta)=\int f(x)\,p_\theta(x)\,dx,$5

followed by projection onto a spectral box (Ye et al., 2019). For quadratic objectives, MiNES proves that the learned covariance converges to the inverse Hessian with sublinear $J(\theta)=\int f(x)\,p_\theta(x)\,dx,$6 rate, giving a rigorous second-order interpretation that had previously been conjectural for related ES methods (Ye et al., 2019).

Convex Natural Evolutionary Strategies (CoNES) replaces explicit Fisher preconditioning by a finite-step KL trust-region subproblem. The update is defined as the solution to

$J(\theta)=\int f(x)\,p_\theta(x)\,dx,$7

(Veer et al., 2020). For Gaussian belief families, this becomes an SDP with exponential-cone constraints in the full-covariance case and an SOCP with exponential-cone constraints in the diagonal case (Veer et al., 2020). As $J(\theta)=\int f(x)\,p_\theta(x)\,dx,$8, the solution aligns with the natural gradient, but for finite $J(\theta)=\int f(x)\,p_\theta(x)\,dx,$9 the trust-region step is adapted to actual update length rather than only infinitesimal behavior (Veer et al., 2020).

Fast Moving NES (FM-NES) addresses ridge-structured and implicitly constrained black-box problems by augmenting DX-NES-IC with a CMA-ES-style rank-one shape update. The search distribution is parameterized as

$$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[f(x)\,\nabla_\theta \log p_\theta(x)\right].$$0

with mirrored sampling $$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[f(x)\,\nabla_\theta \log p_\theta(x)\right].$$1, $$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[f(x)\,\nabla_\theta \log p_\theta(x)\right].$$2 (Nomura et al., 2021). FM-NES performs a rank-one update only under a ridge-detection condition based on the eigenvalue ratio of $$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[f(x)\,\nabla_\theta \log p_\theta(x)\right].$$3, and resets the shape when an infeasible solution is sampled პირველად to protect performance on implicitly constrained problems (Nomura et al., 2021). This suggests a broader point: later NES variants increasingly incorporated structural priors about geometry, constraints, or curvature while preserving the underlying probabilistic natural-gradient logic.

5. Discrete and hybrid ProbNES

ProbNES is not limited to continuous parameter spaces. Discrete Natural Evolution Strategies derives NES for Bernoulli and categorical search distributions over discrete candidate spaces (Amin, 2024). For a Bernoulli variable with probability parameter $$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[f(x)\,\nabla_\theta \log p_\theta(x)\right].$$4,

$$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[f(x)\,\nabla_\theta \log p_\theta(x)\right].$$5

$$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[f(x)\,\nabla_\theta \log p_\theta(x)\right].$$6

(Amin, 2024). The natural direction then simplifies to the centered sufficient statistic $$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[f(x)\,\nabla_\theta \log p_\theta(x)\right].$$7, weighted by fitness.

For a categorical distribution with probability vector $$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[f(x)\,\nabla_\theta \log p_\theta(x)\right].$$8,

$$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[f(x)\,\nabla_\theta \log p_\theta(x)\right].$$9

(Amin, 2024). Under logits $$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$0 with $$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$1, the score becomes

$$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$2

(Amin, 2024). The paper argues that, in discrete settings, the “natural-gradient effect” is often implicit in the standard score update and that explicit Fisher multiplication may be unnecessary or even harmful. In program induction experiments, explicit FIM multiplication led to higher losses and diverging outputs relative to plain score updates across learning rates $$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$3 (Amin, 2024).

This result marks an objective controversy within the broader ProbNES literature. In continuous Gaussian families, explicit or implicit Fisher geometry is central to invariance and stable covariance adaptation (Wierstra et al., 2011). In discrete Bernoulli and categorical families, by contrast, the Fisher inverse is trivial and closely tied to the parameters themselves, which may make explicit preconditioning redundant (Amin, 2024). A plausible implication is that “ProbNES” names a unified probabilistic framework, but the practical role of explicit natural-gradient multiplication depends strongly on the search family.

Hybrid continuous-discrete formulations are also supported. In the program induction task reported in the discrete NES work, discrete holes were optimized with discrete NES and continuous holes with Gaussian NES, showing that mixed-variable problems can be handled by combining family-specific search distributions inside a single probabilistic optimization loop (Amin, 2024).

6. Probabilistic numerics, inference, and broader conceptual extensions

In later literature, “ProbNES” acquires a narrower meaning: Natural Evolutionary Search meets Probabilistic Numerics introduces Probabilistic Natural Evolutionary Strategy Algorithms as NES enhanced with Bayesian quadrature (Osselin et al., 9 Jul 2025). Standard NES estimates

$$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$4

which can be sample-inefficient when evaluations are expensive (Osselin et al., 9 Jul 2025). The probabilistic-numerics extension instead places a Gaussian process prior on $$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$5 and treats

$$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$6

as a quantity to be inferred (Osselin et al., 9 Jul 2025).

Given a GP posterior, Bayesian quadrature yields a posterior mean

$$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$7

and variance

$$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$8

(Osselin et al., 9 Jul 2025). For Gaussian search distributions and an RBF kernel written as a Gaussian density, the kernel mean embeddings and their derivatives are available in closed form, enabling a BQ-based gradient for the NES natural-gradient step (Osselin et al., 9 Jul 2025).

ProbNES in this later sense also adds active sampling by maximizing variance reduction of the integral,

$$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$9

and restricts GP fitting to a local Mahalanobis trust region

$$F(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[ \nabla_\theta \log p_\theta(x)\,\nabla_\theta \log p_\theta(x)^\top \right].$$0

with $$F(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[ \nabla_\theta \log p_\theta(x)\,\nabla_\theta \log p_\theta(x)^\top \right].$$1 (Osselin et al., 9 Jul 2025). The reported result is that ProbNES variants consistently outperform classical NES, vanilla BO, and $$F(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[ \nabla_\theta \log p_\theta(x)\,\nabla_\theta \log p_\theta(x)^\top \right].$$2BO in sample efficiency across benchmark functions, semi-supervised tasks, user-informed hyperparameter tuning, and locomotion tasks, while remaining slower than plain NES but faster than vanilla BO and $$F(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[ \nabla_\theta \log p_\theta(x)\,\nabla_\theta \log p_\theta(x)^\top \right].$$3BO in wall-clock time on the reported examples (Osselin et al., 9 Jul 2025).

The probabilistic-search viewpoint also extends beyond optimization in the narrow sense. NES has been used as a black-box estimator for stochastic variational inference and VAEs by treating the ELBO as a fitness function over model and variational parameters, thereby removing the need for the reparameterization trick (Chen et al., 2023). In variational quantum computation, xNES and SNES were used to optimize parameterized quantum circuits in barren-plateau regimes, with empirical evidence that NES search-direction variance remains non-vanishing while exact gradient variance decays toward zero (Anand et al., 2020). In combinatorial optimization and variational Monte Carlo, NES and quantum NES were unified through Fisher–Rao and Fubini–Study geometry, with neural quantum states providing the probabilistic model $$F(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[ \nabla_\theta \log p_\theta(x)\,\nabla_\theta \log p_\theta(x)^\top \right].$$4 (Zhao et al., 2020). These developments suggest that ProbNES is best understood as a general methodology for optimization and inference on manifolds of search distributions.

7. Empirical behavior, comparisons, and limitations

Across the original NES study, xNES achieved best published performance on various standard benchmarks and competitive performance on others (Wierstra et al., 2011). On BBOB-style suites, xNES was reported as best or competitive on unimodal and multimodal functions, with restart strategies improving reliability on multimodal landscapes (Wierstra et al., 2011). SNES showed strong high-dimensional results, including neuroevolution for non-Markovian double-pole balancing and Lennard-Jones atom clusters, while heavy-tailed Cauchy-NES variants found better local optima on deceptive multimodal landscapes (Wierstra et al., 2011).

eNES was competitive with state-of-the-art evolutionary algorithms on unimodal and multimodal benchmarks, though generally slower than CMA-ES in higher dimensions; it was faster on DiffPow across all tested dimensions and used importance mixing to reduce evaluations by about $$F(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[ \nabla_\theta \log p_\theta(x)\,\nabla_\theta \log p_\theta(x)^\top \right].$$5 (Sun et al., 2012). FM-NES outperformed DX-NES-IC on ridge-structured unconstrained and implicitly constrained benchmarks and also outperformed xNES, CMA-ES, and resampling variants under the settings reported in that study (Nomura et al., 2021). CoNES reported substantially improved sample efficiency over ES, NES, and CMA-ES on $$F(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[ \nabla_\theta \log p_\theta(x)\,\nabla_\theta \log p_\theta(x)^\top \right].$$6-dimensional benchmark functions and faster attainment of target returns on several MuJoCo tasks (Veer et al., 2020). The probabilistic-numerics ProbNES variant reported consistent sample-efficiency gains over classical NES and BO-family baselines across diverse tasks (Osselin et al., 9 Jul 2025).

Comparison with CMA-ES is recurrent in the literature. NES is derived from the natural-gradient principle and uses explicit Fisher preconditioning or local coordinates where the Fisher is identity, while CMA-ES uses evolution paths and additive covariance updates (Wierstra et al., 2011). First-order approximation of the xNES exponential covariance update yields an additive form similar to CMA-ES rank-$$F(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[ \nabla_\theta \log p_\theta(x)\,\nabla_\theta \log p_\theta(x)^\top \right].$$7 updates (Wierstra et al., 2011). CMA-ES remains highly competitive, particularly with small populations and path cumulation, but xNES preserves a state equivalent to the search distribution itself and keeps positive definiteness through multiplicative exponential updates (Wierstra et al., 2011).

Several limitations recur. Full-covariance Gaussian variants incur $$F(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[ \nabla_\theta \log p_\theta(x)\,\nabla_\theta \log p_\theta(x)^\top \right].$$8 cost per iteration because of matrix exponentials, eigendecompositions, or Fisher inversion (Wierstra et al., 2011). Diagonal or low-rank variants improve scalability but lose some invariance or curvature expressivity [(Wierstra et al., 2011); (Sun et al., 2011)]. GP-based probabilistic-numerics variants inherit $$F(\theta) = \mathbb{E}_{x\sim p_\theta}\!\left[ \nabla_\theta \log p_\theta(x)\,\nabla_\theta \log p_\theta(x)^\top \right].$$9 GP costs and sensitivity to kernel misspecification or non-stationarity, partially mitigated by local modeling (Osselin et al., 9 Jul 2025). Discrete NES indicates that explicit Fisher multiplication may be counterproductive in categorical settings (Amin, 2024). Heavy-tailed variants may proceed even when the Fisher is undefined, but this relies on local invariant coordinates rather than classical Fisher geometry in the strictest sense (Wierstra et al., 2011).

These caveats do not alter the central identity of ProbNES. Whether implemented as xNES, eNES, SNES, R1-NES, MiNES, CoNES, discrete NES, or Bayesian-quadrature-enhanced ProbNES, the unifying structure is the same: a parameterized probability distribution over candidate solutions is adapted by score-based estimates of expected fitness, with information geometry governing the update. The main axes of variation are the choice of search family, the treatment of covariance or higher-order structure, the use of rank shaping and variance reduction, and the degree to which probabilistic modeling extends beyond the search distribution to the objective itself.