Neural Global Optimization via Iterative Refinement from Noisy Samples

Abstract: Global optimization of black-box functions from noisy samples is a fundamental challenge in machine learning and scientific computing. Traditional methods such as Bayesian Optimization often converge to local minima on multi-modal functions, while gradient-free methods require many function evaluations. We present a novel neural approach that learns to find global minima through iterative refinement. Our model takes noisy function samples and their fitted spline representation as input, then iteratively refines an initial guess toward the true global minimum. Trained on randomly generated functions with ground truth global minima obtained via exhaustive search, our method achieves a mean error of 8.05 percent on challenging multi-modal test functions, compared to 36.24 percent for the spline initialization, a 28.18 percent improvement. The model successfully finds global minima in 72 percent of test cases with error below 10 percent, demonstrating learned optimization principles rather than mere curve fitting. Our architecture combines encoding of multiple modalities including function values, derivatives, and spline coefficients with iterative position updates, enabling robust global optimization without requiring derivative information or multiple restarts.

• The paper proposes a neural architecture that iteratively refines noisy sample inputs to locate global minima in complex multimodal landscapes.
• The methodology integrates multiple modalities—locations, values, derivatives, and spline coefficients—via a tailored U-Net fusion to capture both local and global function structure.
• Experimental results show the model achieves a mean error of 8.05%, effectively suppressing local minima and outperforming traditional methods under noisy conditions.

Neural Global Optimization via Iterative Refinement from Noisy Samples

Problem Setting and Motivation

Global optimization of black-box, multimodal functions under noisy sampling constraints remains a central challenge in machine learning, scientific computing, and applied engineering. The classical approaches—Bayesian optimization (BO) leveraging Gaussian processes and gradient-free metaheuristics such as CMA-ES—suffer from significant limitations: BO can become trapped in local minima due to surrogate mismatch on highly multimodal landscapes, while evolutionary or heuristic approaches typically require excessive sample complexity, making them impractical for domains with expensive queries or substantial noise.

This work proposes a fundamentally different paradigm by directly training a neural architecture to optimize unknown functions, guided by exhaustively computed global minima as supervision. The intent is to move beyond handcrafted acquisition strategies, instead enabling neural networks to discover optimization heuristics directly from experience, even in the presence of high observational noise and complex multimodality.

Methodology and Core Architecture

The neural model consists of three primary components—MainEncoder, Iterator, and Updater—interacting through iterative refinement. The model receives as input a set of noisy sample locations $\{\mathbf{x}\}$, their corresponding values $\{\mathbf{y}\}$, derivative estimates $\{\mathbf{y}'\}$ extracted from a cubic spline fitted to the noisy observations, and the spline coefficients $\{\mathbf{c}\}$. This multi-modal encoding is crucial for capturing both local and global function structure as well as noise characteristics, differentiating the approach from conventional deep learning optimizers restricted to local information.

MainEncoder

The MainEncoder projects each modality (locations, values, derivatives, spline coefficients) independently through learnable linear transforms with custom cubic activations, followed by a U-Net-style fusion network incorporating skip connections and hierarchical aggregation. Three levels of pooling (global, focus, local) are concatenated and projected into a compact encoding ($\mathbf{e} \in \mathbb{R}^{64}$), forming the functional substrate for optimization trajectory prediction. This multi-scale approach is designed to capture coarse topological structure, unimodal versus multimodal features, and fine-scale local curvature.

The custom cubic activation (StableCubic) is parameterized to avoid gradient instabilities while ensuring high-order expressivity, which is empirically necessary for encoding non-quadratic landscapes.

Iterator and Updater

The Iterator computes the direction and step size for each iteration using a decoupled two-headed MLP. Direction is bounded through $\tanh$ and step size through softplus activation, allowing adaptive, learned annealing schedules analogous to but more flexible than those used in traditional optimization algorithms. Each update adjusts the candidate position $x_{t+1}$.

After each update, the Updater expends the low-dimensional encoding back into per-sample representations through inverse U-Net and modality-specific conditioning based on the new candidate position and step, enabling the representation to adapt dynamically. This cycling of encoding, update, and re-encoding injects persistent state and memory into the optimization process, allowing the network to suppress previously explored local minima and focus exploration adaptively.

Training and Function Generation

Ground truth is obtained via exhaustive grid search on spline-generated multimodal functions, ensuring supervision is for the true global minimum, not just a locally optimal candidate. The loss function combines final position error with monotonic trajectory regularization encouraging steps that do not reverse progress toward the minimum. Functions for training are generated to possess strong multimodality and significant additive Gaussian noise (up to 30% of function range), with penalization for boundary minima or excessive derivative magnitude at the minima, ensuring challenge and diversity in the training set.

Experimental Results

On a set of fifty previously unseen, highly challenging ("NIGHTMARE" level) test functions, the neural model achieves a mean error of 8.05% in the location of the minimum (measured as fraction of domain width), a 28.18% absolute improvement over the best cubic spline initialization baseline (36.24% mean error). Median error falls to 5.73%, with 72% of test cases achieving errors less than 10%. In 88% of cases, the learned model improves upon spline initialization, even under heavy observation noise. The error distribution is heavily skewed toward low error, with few outliers, indicating both robustness to noise and multimodal distractors, as well as a degree of generalization outside the training function class.

Qualitative analysis of iteration trajectories reveals that the model: (1) performs aggressive exploration at early iterations, (2) anneals its step size in the proximity of a minimum, and (3) is capable of escaping poor local minima fostered by noisy spline interpolations, behaviors which are unattainable with standard one-shot regression or hand-designed surrogates under similar constraints.

Critical Evaluation and Implications

The proposed architecture demonstrates that neural models, when trained with exhaustive supervision on challenging function classes, can learn to exploit and generalize optimization heuristics—effectively navigating complex, multimodal, and noisy landscapes in a way that evades the principal traps of both BO and evolutionary heuristics. Key differentiators include:

• Utilization of multi-modal input representations, crucial for integrating local sample evidence and global structural priors.
• Explicit separation and adaptive control of directional search and step size.
• Iterative state refinement, allowing in-trajectory suppression of encountered minima and re-focusing of search.
• Stable, expressive activation functions enabling both smooth and sharp decision boundaries as needed for global search.

Despite its success in one-dimensional settings and on synthetic function classes, the work admits several critical limitations: extension to higher-dimensional spaces faces severe challenges due to the curse of dimensionality; the efficacy of the spline-derived modality encoding is uncertain for functions with low smoothness or discontinuities; and the computational demands of training, while manageable for 1D, may become burdensome as dimensionality increases.

The model’s competitive performance opens up further inquiry into neural approaches for global optimization under limited evaluations, including higher-dimensional extensions, active sampling strategies, uncertainty quantification within the neural search trajectory, and hybridization with existing BO/EA frameworks. There is also significant potential for applications in experimental design and hyperparameter tuning, where observation budgets are limited and function landscapes are poorly behaved.

Conclusion

This work establishes a neural architecture that reliably performs global black-box optimization in the presence of strong noise and multiple minima, guided by iterative refinement from multi-modal, noisy samples. By learning directly from ground truth minima and iteratively reconciling multi-scale evidence, the approach validates the hypothesis that neural networks can internalize optimization strategies more effectively than conventional surrogates or heuristics in resource-constrained, noisy scenarios. Future directions will assess scalability, generalization to broader function classes, and integration with adaptive sampling and uncertainty-aware design.

Reference: "Neural Global Optimization via Iterative Refinement from Noisy Samples" (2604.03614)