---
title: Mesh Adaptive Direct Search (MADS)
url: https://www.emergentmind.com/topics/mesh-adaptive-direct-search-mads
type: topic
---

# Mesh Adaptive Direct Search (MADS)

Mesh Adaptive Direct Search (MADS) is a rigorous derivative-free optimization (DFO) framework for solving black-box problems where functions are costly, noisy or non-differentiable, and constraints may be present or themselves black-box. MADS operates on a sequence of adaptive, discrete grids ('meshes') and employs a structured pattern of candidate evaluations, systematically refining the search around promising areas while maintaining strong theoretical convergence guarantees in both deterministic and stochastic settings.

## 1. Algorithmic Framework and Mathematical Structure

MADS addresses the generic DFO problem:
\[
\min_{x \in \Omega} f(x)
\]
where $f: \mathbb{R}^n \to \mathbb{R}$ is a black-box function and $\Omega \subseteq \mathbb{R}^n$ encodes feasible solutions. At each iteration $k$, MADS maintains a mesh
\[
\mathcal{M}_k = \{ x^k + \Delta^m_k D^k z : z \in \mathbb{Z}^p \}
\]
with:
- $x^k$: current iterate,
- $\Delta^m_k > 0$: mesh size,
- $D^k$: generating matrix with columns forming a positive spanning set.

Algorithm progression involves two phases:
- **Search** (optional): arbitrary trial points on $\mathcal{M}_k$ suggested by heuristics or surrogate models are evaluated.
- **Poll** (mandatory): a finite set of trial points close to $x^k$ (usually $\{x^k + \Delta^p_k d_i\}$ where $d_i$ are positive spanning directions) is evaluated.

Upon finding an improved solution, the mesh is refined ($\Delta^m_{k+1} = \tau \Delta^m_k$, $\tau>1$) or contracted ($\Delta^m_{k+1} = \tau^{-1} \Delta^m_k$, $\tau>1$) in the case of success or failure, respectively. As $\Delta^m_k \to 0$, the union of used directions becomes dense on the unit sphere, essential for theoretical guarantees [1907.01698].

Table 1: MADS Iteration Components

| Step       | Description                                               | Typical Operation                       |
|------------|----------------------------------------------------------|-----------------------------------------|
| Mesh       | Discrete grid for candidate generation                   | $\mathcal{M}_k = \{x^k + \Delta^m_k D^k z \}$ |
| Search     | User-defined exploration on the mesh                     | Surrogates, random sampling, heuristics |
| Poll       | Systematic trial in positive spanning directions         | Evaluate $x^k + \Delta^p_k d_i$         |
| Mesh update| Refine (success) or contract (failure) search granularity| $\Delta^m_{k+1}$ adaptive               |

## 2. Key Theoretical Properties

The mesh and poll-step structure of MADS ensures that, under mild regularity conditions (e.g., $f$ Lipschitz continuous, directions becoming asymptotically dense), every accumulation point is Clarke-stationary:
\[
0 \in \partial f(\bar{x}) + N_\Omega(\bar{x})
\]
where $\partial$ is Clarke's subdifferential and $N_\Omega$ the normal cone [1907.01698].

For constrained or stochastic optimization, as in StoMADS-PB, probabilistic bounds and progressive barrier mechanisms extend these guarantees almost surely by adapting Lyapunov-type and martingale arguments [2011.04225]. In the stochastic subspace setting (StoDARS), first- and second-order convergence to Clarke-stationary points is established via generalized Hessian constructions, leveraging Johnson–Lindenstrauss projections and subspace randomization [2403.13320].

## 3. Algorithmic Enhancements and Variants

Several MADS variants and enhancements address domain-specific challenges:

- **Ortho-MADS:** Poll directions are constructed via deterministic orthogonal bases (e.g., Householder reflections), rotating through orientations to improve coverage of the search sphere and ensure positive spanning at each iteration [1904.11649].
- **Categorical Handling (HyperNOMAD):** Mixed-type hyperparameter blocks (categorical and continuous) are integrated via ad-hoc neighborhood moves, such as adding/removing network layers or changing optimizers. Polling around current categorical choices explores discrete architectural modifications within the same convergence theory [1907.01698].
- **Hybrid Search Strategies:** Integration of Nelder–Mead local search and Variable Neighborhood Search grants escape from local optima and improves convergence rates, especially for hyperparameter optimization in SVMs [1904.11649].
- **Bilevel and Black-box Constraint Handling:** In bilevel contexts, MADS employs inexact lower-level solvers with error control, maintaining stationary point guarantees with explicit relationship between lower-level precision and achieved optimality [2307.09924].
- **Pareto-based Filtering:** For multi-objective/discrete optimization (e.g., feeder reconfiguration), Pareto filters prune dominated solutions and local polling refines the non-dominated frontier efficiently with minimal simulation calls [2507.16027].

## 4. Representative Applications and Empirical Results

MADS has been applied to diverse domains, particularly where black-box evaluations are expensive:

**Hyperparameter and Neural Architecture Optimization**
- On MNIST and CIFAR-10 with a 100-evaluation budget:
    - HyperNOMAD achieved test accuracies: MNIST $\approx 99.61\%$, CIFAR-10 up to $77.6\%$ from default $28.3\%$, surpassing Random Search and Bayesian TPE, especially in feasibility-constrained architectures [1907.01698].
- On ResNet compression:
    - Out of 25 architectures on ImageNet, NOMAD enabled $>25\%$ MACs reduction with negligible accuracy loss, outperforming TPE [2301.06641].

**Support Vector Machine (SVM) Hyperparameter Tuning**
- Ortho-MADS with dynamics stopping, VNS, and NM hybridization reached or exceeded the best known validation accuracies for 13 UCI datasets using under 100 evaluations, often escaping false local minima [1904.11649].

**Stochastic/Constrained Optimization**
- StoMADS-PB invoked in noisy, constrained black-boxes admits intermediate infeasible iterates, leverages sample-based confidence bounds, and achieves Clarke-stationarity almost surely [2011.04225].
- In large-scale stochastic settings, StoDARS achieves $O(\epsilon^{-2})$ expected complexity for finding first-order stationary points, matching established trust-region methods up to constants, with full second-order extension under $C^{1,1}$ smoothness [2403.13320].

**Engineering and Power Systems**
- In black-box feeder reconfiguration, MADS with bi-objective Pareto filtering realized near-optimal solutions on the IEEE-123 node test feeder with $\sim 10$ simulation calls (vs. $100+$ for heuristics), demonstrating high empirical efficiency in discrete, combinatorial environments [2507.16027].

**Model Predictive Control with Non-Smooth Costs**
- Embedding cost and path-constraint integrals as ODE states enables MADS to directly optimize non-differentiable system trajectories with constraints, as shown for robust rocket apogee control [2106.05025].

## 5. Convergence Analysis and Complexity

MADS convergence is grounded in the density of poll directions and systematic mesh refinement. Under bounded level-set and positive spanning hypotheses, vanishing mesh size implies no descent can be found in any direction, establishing first-order stationarity (or Clarke-criticality in nonsmooth cases). Extensions (e.g., StoDARS) leverage generalized Hessian constructions for second-order analysis, yielding the necessary optimality conditions and expected complexity bounds:

- Deterministic/stochastic MADS: $O(\epsilon^{-2})$ expected function evaluations to reach $\|\nabla f(x)\|\leq\epsilon$ in the $C^1$ case [2403.13320].
- Bilevel adaptation: To achieve $\|\nabla F(x)\|\leq\epsilon$, lower-level oracle precision should scale as $\mathcal{O}(\epsilon^2)$ [2307.09924].

## 6. Implementation Considerations and Practical Guidance

Efficient application of MADS relies on:
- Careful tuning of initial mesh sizes and contraction/expansion factors, often problem-specific.
- Early stopping via the mesh becoming sufficiently fine, rather than fixed evaluation budgets, prevents wasteful allocations in unpromising regions.
- For categorical or mixed-variable search spaces, ad-hoc neighborhood construction is crucial to preserve convergence properties [1907.01698].
- Opportunistic evaluation (stop poll as soon as an improving point is found) reduces evaluation cost.
- In stochastic/black-box constraints, integration of sample-based bounds and progressive barrier methods enables robust performance with partial, noisy observations [2011.04225].

A plausible implication is that the structural guarantees and adaptability of MADS make it especially suitable for applications with high evaluation cost, non-differentiability, and combinatorial or mixed search spaces.

## 7. Extensions, Limitations, and Research Directions

MADS continues to be extended:
- Random-subspace and high-dimensional variants (StoDARS) support scalability to $n \gtrsim 10^4$ dimensions with random Johnson–Lindenstrauss projections [2403.13320].
- Bilevel optimization variants address complex hierarchical applications with controlled lower-level oracle accuracy [2307.09924].
- Integration with advanced surrogate or meta-model strategies inside the optional search phase (e.g., using TPE or other Bayesian optimization as search) is a promising direction.

Limitations include:
- Sequential nature may be less immediately suited to embarrassingly parallel or massively scalable contexts unless the poll/search evaluations are parallelized.
- Convergence rates may slow in high-noise settings if sample-based estimation is not carefully managed.
- Exploiting problem structure (e.g., via custom poll directions or hybridization with local solvers) remains a key area for domain-specific tuning.

Continued research includes augmentation for distributed infrastructure, real-time control, richer mixed-variable encoding, and theoretical investigations of convergence rates under weaker regularity and high-dimensionality.

Source: https://www.emergentmind.com/topics/mesh-adaptive-direct-search-mads