---
title: Evolution Strategies Smoothing Estimators
url: https://www.emergentmind.com/topics/evolution-strategies-es-smoothing-estimators
type: topic
---

# Evolution Strategies Smoothing Estimators

Evolution Strategies (ES) smoothing estimators are a class of derivative-free optimization techniques based on random perturbation and smoothing of the objective function. They generate search directions by perturbing parameters, evaluating objective responses, and aggregating these results via finite-difference-like or kernel-based estimators. While classical ES typically use Gaussian-distributed perturbations, modern ES smoothing estimators employ a range of perturbation laws, estimator constructions, and variance-reduction mechanisms—shaping the landscape of black-box optimization, policy search in reinforcement learning, and, increasingly, variational inference and meta-optimization.

## 1. Smoothing Kernels and Perturbation Laws

The smoothing kernel in ES determines the probability distribution from which parameter perturbations are sampled. The canonical approach uses an isotropic Gaussian kernel, with the search distribution $q(u) = (2\pi \sigma^2)^{-d/2} \exp(-\|u\|^2/(2\sigma^2))$, smoothing the objective over $\mathbb{R}^d$ [1712.06568]. The gradient of the smoothed objective with respect to parameters $\theta$ is estimated via
\[
\nabla_\theta J_\sigma(\theta) = \frac{1}{\sigma} \mathbb{E}_{u \sim \mathcal N(0, I)} \bigl[ f(\theta + \sigma u) \cdot u \bigr].
\]
This construction underlies both classic Monte Carlo estimators and more advanced structured variants.

Recent work introduces kernels of bounded support, most notably the symmetric triangular distribution. For example, the Triangular-Distribution ES (TD-ES) defines the per-coordinate kernel as
\[
q_\triangle(\varepsilon_k) = 
\begin{cases}
\frac{1}{\sigma_{\mathrm{ES}}} \left(1 - \frac{|\varepsilon_k|}{\sigma_{\mathrm{ES}}}\right), & |\varepsilon_k| \leq \sigma_{\mathrm{ES}} \\
0, & \text{otherwise}
\end{cases}
\]
with $\varepsilon = (\varepsilon_1, ..., \varepsilon_d)$ sampled independently per dimension [2511.09923]. This yields perturbations strictly within a “soft trust region,” controlling parameter excursions and concentrating sample mass near zero.

Some ES variants employ directional or axis-aligned kernels: Directional Gaussian Smoothing (DGS-ES) constructs one-dimensional smoothing operators along each coordinate or an orthogonal basis, enabling nonlocal exploration and facilitating analytic quadrature [2002.03001, 2002.09077].

## 2. Monte Carlo, Antithetic, and Quadrature-Based Gradient Estimation

The classical ES estimator forms a Monte Carlo finite-difference average using i.i.d. perturbations. With $N$ samples,
\[
\widehat{g} = \frac{1}{N \sigma} \sum_{i=1}^N f(\theta + \sigma \varepsilon_i) \varepsilon_i,
\]
where $\varepsilon_i \sim q$. To reduce estimator variance, antithetic sampling evaluates both $\theta+\sigma\varepsilon_i$ and $\theta-\sigma\varepsilon_i$, exploiting the oddness in the objective’s Taylor expansion:
\[
\widehat{g}_{\text{antithetic}} = \frac{1}{2N\sigma} \sum_{i=1}^N [ f(\theta + \sigma \varepsilon_i) - f(\theta - \sigma \varepsilon_i) ] \varepsilon_i.
\]
This halves variance by canceling even-order noise terms [2511.09923, 2304.11153, 1910.05268].

For non-Gaussian or directional smoothing, high-order quadrature rules—especially Gauss-Hermite—yield deterministic, low-variance gradient estimates. In DGS-ES, each one-dimensional integral along coordinate $e_j$ is approximated by [2002.03001, 2002.09077]:
\[
D_{j, \sigma}[f](x) \approx \frac{1}{\sqrt{\pi \sigma}} \sum_{k=1}^M w_k f(x + \sqrt{2\sigma} v_k e_j) (\sqrt{2\sigma} v_k),
\]
where $(v_k, w_k)$ are nodes and weights of the Hermite polynomial of order $M$.

Centering and normalization of reward signals—such as centered-rank transforms—further reduce estimator variance and improve shift- and scale-invariance [2511.09923].

## 3. Bias, Variance, and Locality Properties

The bias-variance properties of ES smoothing estimators stem from the smoothing kernel’s support and the estimation protocol:

- **Bias:** All smoothers convolve $f$ with their kernel, introducing a bias of $\mathcal{O}(\sigma^2)$ under standard $C^2$ regularity [1712.06568, 2511.09923, 2002.09077]. Directional or bounded-support kernels can exert additional bias by localizing sampling to near-linear regions and enforcing maximum per-parameter changes.
- **Variance:** ES estimators based on Gaussian MC samples have variance scaling inversely with the number of samples but growing with problem dimension. Bounded-support kernels (e.g., triangular) and antithetic sampling yield substantial variance reductions—TD-ES achieves an 83.1% reduction in empirical studies compared to standard Gaussian ES [2511.09923]. In DGS-ES, quadrature renders estimator variance negligible except for observation noise [2002.03001].
- **Locality:** Bounded kernels enforce hard exploration limits, effectively instituting a “soft trust region” per update. The triangular law's mode at zero accentuates local, low-variance update steps, leading to robust convergence in the late, high-precision stage of policy refinement.

## 4. Algorithmic Scheduling and Practical Integration

Smoothing estimators are often positioned within broader optimization pipelines, enabling efficient exploration, robust refinement, and compatibility with gradient-based methods.

A representative schedule involves two stages [2511.09923]:
1. **Baseline Pre-Training:** Use an on-policy gradient method (e.g., PPO) until performance plateaus or a preset threshold is met.
2. **ES-Based Refinement:** Switch to TD-ES (or similar ES smoothing estimator), inheriting parameters from PPO and allocating the remainder of environment steps to variance-reduced, local exploration. Parameters are updated by
   \[
   \theta \leftarrow \theta + \alpha_t g_t,
   \]
   with $g_t$ the TD-ES (antithetic, centered-rank) update and $\sigma_{\mathrm{ES}}$ decayed geometrically or adaptively.

Practical implementations select the smoothing scale, step size, and, when using directional kernels, orthonormal bases and quadrature orders for an appropriate efficiency-accuracy trade-off [2002.03001, 2511.09923].

## 5. Theoretical Foundations and Convergence Guarantees

The foundations of ES smoothing estimators rely on kernel convolution, Monte Carlo estimation, and control-variate analysis. Under mild regularity, the smoothed gradient approximates the true gradient with a second-order error in the smoothing radius. For symmetric, finite-moment kernels (e.g., triangular or Gaussian), Taylor expansion guarantees bias is $\mathcal{O}(\sigma^2)$ [2511.09923, 1712.06568, 2304.11153].

Variance reduction is supported theoretically by multiple mechanisms:
- **Antithetic sampling** eliminates leading stochastic noise terms.
- **Centered ranking** matches the control variate effect, ensuring invariance to baseline shifts in the objective.
- **Quadrature-based integration** achieves spectral accuracy in smooth integrands, making bias arbitrarily small with sufficient quadrature order [2002.03001, 2002.09077].
- **Surrogate-direction projection** exploits correlation in descent directions and yields provably improved cosine alignment with the true gradient in both linear and non-linear settings [1910.05268].

For learning-rate decay and step size adaptation, diminishing $\sigma$ and step size sequence $\{\alpha_t\}$ ensures convergence to local optima, with convergence rates governed by the estimator’s variance and the problem’s dimensionality [2511.09923].

## 6. Empirical Benchmarks and Domain Impact

ES smoothing estimators, particularly those with advanced smoothing kernels and variance-reduction schemes, deliver marked improvements in both synthetic and real-world tasks:

- **Robotic Manipulation:** PPO→TD-ES pipelines improve success rates by up to 26.5% over PPO alone, with systematic variance reduction and robust final-stage refinement. In ablations, triangular smoothing achieves the highest mean success and lowest variance, especially in precision tasks [2511.09923].
- **Reinforcement Learning Benchmarks:** DGS-ES converges much faster than vanilla ES or other baselines on MountainCarContinuous, Pendulum, Hopper, and others. Large smoothing radii prove essential for escaping local minima [2002.09077].
- **High-dimensional Black-box Optimization:** Directional quadrature-based ES drastically improves convergence and estimator stability for nonconvex problems in up to $d=2000$ (e.g., Ackley, Rastrigin), and in engineering PDE-constrained applications [2002.03001].
- **Meta-optimization and Long-horizon Learning:** In unrolled optimization graphs, the ES-Single approach exhibits variance independent of unroll length, in contrast to Persistent ES—enabling stable meta-learning in regimes where classical ES, BPTT, or PES struggle [2304.11153].

Hybrid approaches—such as SV-ES, which blends ES smoothing with Stein variational updates—expand the utility of ES smoothing estimators to inference over unnormalized densities, further hybridizing the derivative-free and variational inference toolkits [2410.10390].

## 7. Extensions, Control Variates, and Variance-Reduction Enhancements

Advanced smoothing estimators exploit structural properties of the problem or data to further reduce variance:
- **Structured control variates** combine standard ES and reparameterization-based estimators, optimally blending them via per-coordinate adaptive weights to minimize estimator variance while maintaining unbiasedness [1906.08868].
- **Surrogate directions and memory**: Reuse of past update directions or surrogates, orthogonalization, and projection in high-dimensional settings accelerates alignment with the true gradient and hedges against noisy objective evaluations [1910.05268].

Such methodological advances extend the practical reach of ES smoothing estimators to increasingly high-dimensional, stochastic, and ill-posed optimization tasks—achieving superior efficiency, robustness, and reliability compared to both classical black-box search and naive finite-difference schemes.

Source: https://www.emergentmind.com/topics/evolution-strategies-es-smoothing-estimators