Adaptive Direct Search (ADS)

• Adaptive Direct Search (ADS) is a derivative-free, mesh-free optimization method that uses a punctured-space framework to exclude points too close to prior evaluations.
• It operates via a search–poll–update loop with adaptive scale updates based on strict improvement criteria rather than mesh constraints or sufficient decrease thresholds.
• ADS extends to constrained optimization through extreme and progressive barrier techniques, demonstrating competitive performance on both smooth and nonsmooth blackbox problems.

Adaptive Direct Search (ADS) is a class of derivative-free directional direct-search methods for blackbox optimization. In the recent constrained-optimization formulation, ADS is introduced as a bridge between Mesh Adaptive Direct Search (MADS) and Sufficient Decrease Direct Search (SDDS): unlike MADS, it does not force trial points to lie on a mesh, and unlike SDDS, it does not require a sufficient decrease threshold on the objective value. Instead, it uses a geometric admissibility mechanism based on a “punctured space,” which excludes points that are too close to previously visited or previously successful points, while retaining simple strict-improvement acceptance rules (Audet et al., 30 Jul 2025, Audet et al., 6 Jul 2026). In parallel, earlier literature also uses “adaptive direct search” more broadly to denote the direct-search backbone inherited from the MADS family, especially in hybrid Bayesian, stochastic, and pattern-search settings (Acerbi et al., 2017, Achddou et al., 2022).

1. Definition, scope, and nomenclature

Recent ADS papers define the method by contrast with the two established families of directional direct search: MADS and SDDS. MADS is mesh-based and accepts any feasible trial point with simple decrease; SDDS allows trial points anywhere but accepts only points that satisfy a forcing-function test; ADS removes both restrictions by evaluating trial points anywhere in the space subject to punctured-space admissibility and by accepting strict improvement without a sufficient-decrease threshold (Audet et al., 30 Jul 2025).

Framework	Trial-point placement	Acceptance rule
MADS	Trial points lie on a mesh $\mathbb{M}^k$	Simple decrease
SDDS	Trial points may lie anywhere in the space	Sufficient decrease $f_\Omega(y) < f(\bar x^k)-\rho(\alpha)$
ADS	Trial points may lie anywhere except too close to excluded points in the punctured space	Simple strict improvement plus punctured-space admissibility

The nomenclature is not uniform across the literature. In “Bayesian Adaptive Direct Search,” ADS is not presented as a standalone new algorithm with its own full theory; rather, it refers to the direct-search backbone inherited from the MADS family, especially the poll stage and the success/failure-driven mesh update (Acerbi et al., 2017). Likewise, stochastic pattern-search work for blind resource allocation explicitly treats direct search, also known as pattern search, as an ADS-style scheme based on probing a positive spanning set, accepting sufficient improvement, and shrinking the step size when no descent is found (Achddou et al., 2022). This suggests that “ADS” has both a narrow contemporary meaning, associated with punctured-space mesh-free direct search, and a broader lineage meaning, associated with adaptive direct-search mechanisms descending from MADS and pattern search.

2. Geometric mechanism and basic algorithmic structure

The central geometric device in ADS is the punctured space. In the constrained formulation of ADS, the punctured space at iteration $k$ excludes balls of radius $\delta^k$ around previously visited points:

$$\mathcal{E}^k := \left\{ x \in \mathbb{R}^n : \|x-y\| \geq \delta^k \text{ for all } y \in \mathbb{V}^k \right\}.$$

In ADS-PB, the constrained extension with a Progressive Barrier, the exclusion set is defined instead from previously successful iterates:

$$\mathcal{E}^k := \{x\in\mathbb{R}^n:\ \|x-y\|\ge \delta^k \text{ for all } y\in X_{\text{succ}}^k\}.$$

In both cases, the punctured space replaces the MADS mesh as the mechanism that prevents evaluations from being arbitrarily close to previously explored points (Audet et al., 30 Jul 2025, Audet et al., 6 Jul 2026).

The algorithmic skeleton is a search–poll–update loop. The search step generates a finite set of trial points anywhere in the space, for example by heuristics, surrogates, or other user-defined strategies. A search point is successful if it improves the incumbent and lies in the punctured space. If it improves but lies outside the punctured space, it is still useful: in ADS it becomes the poll center, and the algorithm proceeds to the poll step around that point rather than discarding it (Audet et al., 30 Jul 2025).

The poll step uses normalized directions from a finite set $\mathbb{D}^k$, typically a positive spanning set, and forms a poll set

$$\mathbb{P}^k := \{p^k + \Delta^k v : v \in \mathbb{D}^k\}.$$

Only poll points that belong to the punctured space are evaluated. The poll succeeds if some evaluated point strictly improves the poll center. Otherwise the iteration is unsuccessful, and the incumbent is unchanged or reset to the current poll center according to the variant (Audet et al., 30 Jul 2025).

The adaptive mechanism is carried by the frame size $\Delta^k$ and the exclusion size $\delta^k$. One ADS implementation uses multiplicative updates with $f_\Omega(y) < f(\bar x^k)-\rho(\alpha)$0: on failure, $f_\Omega(y) < f(\bar x^k)-\rho(\alpha)$1; on success, $f_\Omega(y) < f(\bar x^k)-\rho(\alpha)$2. The corresponding exclusion rule is

$f_\Omega(y) < f(\bar x^k)-\rho(\alpha)$3

ADS-PB states the update principle more abstractly: if $f_\Omega(y) < f(\bar x^k)-\rho(\alpha)$4, then $f_\Omega(y) < f(\bar x^k)-\rho(\alpha)$5 and $f_\Omega(y) < f(\bar x^k)-\rho(\alpha)$6 (Audet et al., 30 Jul 2025, Audet et al., 6 Jul 2026).

3. Constraint handling: extreme barriers and progressive barriers

For constrained optimization, the basic ADS paper adopts the extreme barrier formulation

$f_\Omega(y) < f(\bar x^k)-\rho(\alpha)$7

so the algorithm is applied to unconstrained minimization of $f_\Omega(y) < f(\bar x^k)-\rho(\alpha)$8, while only feasible points can be accepted (Audet et al., 30 Jul 2025). This aligns ADS with the classical direct-search treatment of unrelaxable constraints.

ADS-PB extends the framework to quantifiable and relaxable inequality constraints. The problem is

$f_\Omega(y) < f(\bar x^k)-\rho(\alpha)$9

where $k$0 encodes the blackbox domain and the inequality constraints are both quantifiable and relaxable. Constraint violation is aggregated by

$k$1

Thus $k$2 (Audet et al., 6 Jul 2026).

The Progressive Barrier maintains feasible incumbents and infeasible incumbents separately. The best feasible incumbents in the cache are

$k$3

while the infeasible nondominated points are filtered through an infeasible dominance relation and then restricted by a barrier threshold $k$4. For feasible points, domination is purely objective-based: $k$5 iff $k$6. For infeasible points, $k$7 iff

$k$8

The barrier threshold starts at $k$9 and is progressively lowered, so infeasible points can contribute to progress early in the run but become less admissible as the method advances (Audet et al., 6 Jul 2026).

ADS-PB introduces four iteration classes: dominating, improving, unsuccessful, and reframing. Dominating and improving are standard progressive-barrier notions. The reframing class is specific to ADS-PB: it occurs when the search finds a dominating point outside the punctured space, but the poll step fails to find a dominating point. In that case the iteration is neither dominating nor improving, yet the incumbents and local frames are repositioned (Audet et al., 6 Jul 2026). A common misconception is that ADS merely replaces a mesh with continuous freedom; the constrained literature shows that its distinctiveness also lies in how this freedom interacts with barrier logic and with the admissibility geometry defined by the punctured space.

4. Theoretical guarantees and major variants

The ADS convergence analysis follows the standard direct-search pattern of refining subsequences and generalized stationarity. Under the compactness assumption that the lower level set

$\delta^k$0

is compact, ADS proves

$\delta^k$1

As a consequence, infinitely many unsuccessful iterations occur, and there exists a refining subsequence converging to a refined point $\delta^k$2. If $\delta^k$3 is Lipschitz continuous near $\delta^k$4, then for any refining direction $\delta^k$5,

$\delta^k$6

where $\delta^k$7 is the constrained Clarke generalized directional derivative (Audet et al., 30 Jul 2025).

ADS-PB strengthens the vanishing-scale statement. Under the assumption that all evaluated points lie in a compact set $\delta^k$8, it proves

$\delta^k$9

For feasible refining subsequences, if $$\mathcal{E}^k := \left\{ x \in \mathbb{R}^n : \|x-y\| \geq \delta^k \text{ for all } y \in \mathbb{V}^k \right\}.$$0 is Lipschitz near the refined point $$\mathcal{E}^k := \left\{ x \in \mathbb{R}^n : \|x-y\| \geq \delta^k \text{ for all } y \in \mathbb{V}^k \right\}.$$1, then for every refining direction $$\mathcal{E}^k := \left\{ x \in \mathbb{R}^n : \|x-y\| \geq \delta^k \text{ for all } y \in \mathbb{V}^k \right\}.$$2,

$$\mathcal{E}^k := \left\{ x \in \mathbb{R}^n : \|x-y\| \geq \delta^k \text{ for all } y \in \mathbb{V}^k \right\}.$$3

For infeasible refining subsequences, if $$\mathcal{E}^k := \left\{ x \in \mathbb{R}^n : \|x-y\| \geq \delta^k \text{ for all } y \in \mathbb{V}^k \right\}.$$4 is Lipschitz near $$\mathcal{E}^k := \left\{ x \in \mathbb{R}^n : \|x-y\| \geq \delta^k \text{ for all } y \in \mathbb{V}^k \right\}.$$5, then for every refining direction $$\mathcal{E}^k := \left\{ x \in \mathbb{R}^n : \|x-y\| \geq \delta^k \text{ for all } y \in \mathbb{V}^k \right\}.$$6,

$$\mathcal{E}^k := \left\{ x \in \mathbb{R}^n : \|x-y\| \geq \delta^k \text{ for all } y \in \mathbb{V}^k \right\}.$$7

If $$\mathcal{E}^k := \left\{ x \in \mathbb{R}^n : \|x-y\| \geq \delta^k \text{ for all } y \in \mathbb{V}^k \right\}.$$8, then $$\mathcal{E}^k := \left\{ x \in \mathbb{R}^n : \|x-y\| \geq \delta^k \text{ for all } y \in \mathbb{V}^k \right\}.$$9, so $$\mathcal{E}^k := \{x\in\mathbb{R}^n:\ \|x-y\|\ge \delta^k \text{ for all } y\in X_{\text{succ}}^k\}.$$0 is a global minimizer of $$\mathcal{E}^k := \{x\in\mathbb{R}^n:\ \|x-y\|\ge \delta^k \text{ for all } y\in X_{\text{succ}}^k\}.$$1 on $$\mathcal{E}^k := \{x\in\mathbb{R}^n:\ \|x-y\|\ge \delta^k \text{ for all } y\in X_{\text{succ}}^k\}.$$2 (Audet et al., 6 Jul 2026).

In the broader ADS/pattern-search literature, stochastic and constrained variants replace punctured-space admissibility with sufficient-decrease logic and statistical testing. In the blind resource-allocation setting, feasible direct search tests only feasible directions in a polyhedral domain, uses $$\mathcal{E}^k := \{x\in\mathbb{R}^n:\ \|x-y\|\ge \delta^k \text{ for all } y\in X_{\text{succ}}^k\}.$$3, and in the planned-sampling version chooses

$$\mathcal{E}^k := \{x\in\mathbb{R}^n:\ \|x-y\|\ge \delta^k \text{ for all } y\in X_{\text{succ}}^k\}.$$4

Under strong convexity, smoothness, and feasibility/tangent-cone assumptions, the deterministic unconstrained case has finite regret, while the noisy constrained case satisfies

$$\mathcal{E}^k := \{x\in\mathbb{R}^n:\ \|x-y\|\ge \delta^k \text{ for all } y\in X_{\text{succ}}^k\}.$$5

for both FDS-Plan and FDS-Seq (Achddou et al., 2022).

Decentralized direct-search methods provide another extension of the lineage. In decentralized blackbox optimization, search directions again come from positive spanning sets, but each agent maintains a local iterate and communicates only with neighbors. DDS-L proves convergence to stationarity of a penalized objective,

$$\mathcal{E}^k := \{x\in\mathbb{R}^n:\ \|x-y\|\ge \delta^k \text{ for all } y\in X_{\text{succ}}^k\}.$$6

whereas DDS-F proves asymptotic consensus,

$$\mathcal{E}^k := \{x\in\mathbb{R}^n:\ \|x-y\|\ge \delta^k \text{ for all } y\in X_{\text{succ}}^k\}.$$7

without a stationarity guarantee for the original constrained problem (Bergou et al., 5 Apr 2025). StoDARS extends the same direct-search lineage to stochastic random subspaces, preserving the adaptive success/failure update logic while replacing full-space positive spanning sets with reduced-space polling; it proves $$\mathcal{E}^k := \{x\in\mathbb{R}^n:\ \|x-y\|\ge \delta^k \text{ for all } y\in X_{\text{succ}}^k\}.$$8 almost surely, expected $$\mathcal{E}^k := \{x\in\mathbb{R}^n:\ \|x-y\|\ge \delta^k \text{ for all } y\in X_{\text{succ}}^k\}.$$9-type complexity in the smooth case, and convergence to Clarke stationary points with probability one in the nonsmooth case (Dzahini et al., 2024).

5. Hybridization, surrogates, and relation to MADS

ADS is closely connected to MADS, but the relationship depends on whether one means the narrow punctured-space method or the broader direct-search lineage. “Bayesian Adaptive Direct Search” is a hybrid of Bayesian optimization and the MADS direct-search backbone. Its Gaussian-process component provides candidate generation through local acquisition optimization, while the ADS/MADS component provides mesh-based polling, opportunistic acceptance, and a fail-safe local search when the surrogate misleads. The paper states that when the search repeatedly fails, BADS switches to poll, and “the poll stage performs a fail-safe, model-free optimization” (Acerbi et al., 2017).

This hybrid pattern recurs in parallel surrogate-assisted MADS. There the SEARCH step solves a surrogate problem based on LOWESS models, populates a surrogate cache, and then selects a block of promising points for concurrent blackbox evaluation, while the POLL step preserves the convergence-enforcing direct-search structure. The method emphasizes that classical MADS is naturally robust but often sequential in SEARCH, and it re-engineers SEARCH as a parallel candidate generator rather than altering the underlying poll guarantees (Talgorn et al., 2021).

The distinction between ADS and MADS is therefore substantive, not terminological. MADS remains mesh-based, with SEARCH and POLL points projected onto or generated from a mesh, whereas recent ADS papers explicitly remove the mesh and remove sufficient-decrease conditions, replacing both with punctured-space admissibility (Audet et al., 30 Jul 2025, Audet et al., 6 Jul 2026). At the same time, the older literature’s broader use of “adaptive direct search” for the MADS backbone explains why ADS-language appears inside hybrid Bayesian and surrogate-assisted methods that are formally implemented within NOMAD and MADS (Acerbi et al., 2017). This suggests that ADS now names both a specific mesh-free framework and a persistent methodological idea: adaptive, derivative-free local probing with success/failure-driven scale control.

6. Applications and empirical behavior

The newest ADS papers report competitive behavior against both MADS and SDDS. On the Moré–Wild unconstrained test set without a search step, ADS, MADS, and SDDS perform comparably overall, but ADS is reported to be faster on smooth problems and on nonsmooth problems at tolerance $\mathbb{D}^k$0, largely because it does not evaluate poll points that are too close to previously visited points. On 16 constrained CUTEst problems with quadratic search, the paper reports 532 total evaluations for ADS, 625 for MADS, and 560 for SDDS; ADS also saved 27 poll evaluations and had 38% search efficiency, versus 23% for MADS and 43% for SDDS. On SOLAR10 and on the simplified-wing MDO problem, ADS is reported to outperform MADS and SDDS when the quadratic search step is effective and when the optimum lies near the feasible boundary (Audet et al., 30 Jul 2025).

ADS-PB reports a broader constrained benchmark. On 14 analytical constrained problems, producing 780 problem instances, ADS-PB consistently outperforms MADS-PB across the target accuracies $\mathbb{D}^k$1, $\mathbb{D}^k$2, and $\mathbb{D}^k$3. The $\mathbb{D}^k$4 polling strategy outperforms the $\mathbb{D}^k$5 strategy for both methods, and the advantage of ADS-PB becomes more pronounced at tighter tolerances because direct evaluation of good trial points is not degraded by mesh projection. On a styrene production simulation problem and on a solar thermal power plant simulator problem, ADS-PB and MADS-PB dominate the evolutionary baselines, with ADS-PB retaining a consistent edge (Audet et al., 6 Jul 2026).

Many of the best-developed application papers in the same direct-search lineage are still framed as MADS rather than punctured-space ADS. In distribution systems, feeder reconfiguration has been formulated as a black-box bi-objective search over binary switch vectors

$\mathbb{D}^k$6

where $\mathbb{D}^k$7 is active-power loss and $\mathbb{D}^k$8 is a worst-case violation measure. The proposed MADS-based method uses one-bit-flip polling, a Pareto frontier filter, and black-box utility simulation modules, and on the IEEE-123 node test feeder it achieves near-optimal configurations with significantly fewer evaluations than heuristic methods (Zheng et al., 21 Jul 2025).

In deep learning, HyperNOMAD casts hyperparameter optimization as a blackbox problem and solves it with NOMAD’s MADS implementation over real, integer, boolean, and categorical variables. The entire “build network → train → validate → test” pipeline becomes one expensive evaluation, and the method reports results comparable to the current state of the art on MNIST and CIFAR-10 (Lakhmiri et al., 2019). MADS has also been used for neural architecture scaling, where a design vector $\mathbb{D}^k$9 controls depth, width, and resolution. On CIFAR-10, MADS/NOMAD found over 29 configurations with top-1 accuracy $$\mathbb{P}^k := \{p^k + \Delta^k v : v \in \mathbb{D}^k\}.$$0; on the ImageNet/ResNet-50 experiment, 15 of 25 sampled points scored above $$\mathbb{P}^k := \{p^k + \Delta^k v : v \in \mathbb{D}^k\}.$$1 top-1 accuracy, and 2 candidates had at least $$\mathbb{P}^k := \{p^k + \Delta^k v : v \in \mathbb{D}^k\}.$$2 fewer MACS than ResNet-50 while maintaining strong accuracy (Lakhmiri et al., 2023). For mixed-variable constrained blackbox optimization, CatMADS extends the MADS framework with categorical distances, integer and continuous quantitative variables, and a progressive barrier, and data profiles on 32 mixed-variable problems show the best results among the compared solvers (Audet et al., 7 Jun 2025).

A plausible implication is that ADS is currently most visible as part of a broader direct-search ecosystem. The punctured-space ADS papers sharpen the conceptual boundary with MADS and SDDS, while much of the surrounding application literature continues to deploy MADS-based implementations in precisely the expensive, black-box, mixed-variable, and constrained regimes that motivate ADS in the first place.