Towards Fully Parameter-Free Stochastic Optimization: Grid Search with Self-Bounding Analysis

Abstract: Parameter-free stochastic optimization aims to design algorithms that are agnostic to the underlying problem parameters while still achieving convergence rates competitive with optimally tuned methods. While some parameter-free methods do not require the specific values of the problem parameters, they still rely on prior knowledge, such as the lower or upper bounds of them. We refer to such methods as partially parameter-free''. In this work, we target achievingfully parameter-free'' methods, i.e., the algorithmic inputs do not need to satisfy any unverifiable condition related to the true problem parameters. We propose a powerful and general grid search framework, named \textsc{Grasp}, with a novel self-bounding analysis technique that effectively determines the search ranges of parameters, in contrast to previous work. Our method demonstrates generality in: (i) the non-convex case, where we propose a fully parameter-free method that achieves near-optimal convergence rate, up to logarithmic factors; (ii) the convex case, where our parameter-free methods are competitive with strong performance in terms of acceleration and universality. Finally, we contribute a sharper guarantee for the model ensemble, a final step of the grid search framework, under interpolated variance characterization.

• The paper presents a fully parameter-free stochastic optimization method that eliminates the need for any user-specified parameter bounds.
• It introduces the Grasp framework, which uses grid search with self-bounding analysis to compute verifiable search intervals and ensure near-optimal convergence rates.
• Empirical results on CIFAR-10/ResNet-18 show that the method is robust, performing within 8%-23% of hand-tuned baselines for both convex and non-convex cases.

Fully Parameter-Free Stochastic Optimization via Grid Search with Self-Bounding Analysis

Introduction and Motivation

This paper addresses the central problem of designing fully parameter-free stochastic optimization methods, introducing the Grasp framework (Grid-search with Self-bounding Analysis for Stochastic Parameter-free optimization). The fully parameter-free property is defined such that the optimization algorithm requires no user-given unverifiable bounds or values of problem parameters (such as smoothness constants, noise bounds, initial optimality gap, or distance to optimum), in contrast to much of the literature, which is only partially parameter-free (requiring, for example, upper or lower bounds on such quantities).

The motivation stems from the mismatch between realistic practical scenarios and the theoretical guarantees of most parameter-free approaches: practitioners seldom know either tight or even loose parameter bounds, making existing algorithms dependent on unverifiable inputs. By proposing algorithms that only rely on the given objective function and the oracle budget and are agnostic to all other problem-dependent inputs, the paper closes this gap and aligns rigorous guarantees with practical usability.

Core Contributions

The main contributions are as follows:

• Full Parameter-Free Characterization: The paper formalizes the notion of full parameter-freeness, sharply distinguishing it from partial approaches.
• Self-Bounding Grid Search Framework (Grasp): The proposed framework uses a theoretically grounded self-bounding approach to determine effective (and verifiable) search intervals for grid search, obviating the need for user-specified input ranges. These bounds are derived in a way that a candidate solution with vacuously large parameter values is dominated by trivial baselines, allowing a reduction to a verifiable search space.
• General Applicability: Grasp applies to both non-convex and convex (including Hölder-smooth and universal) stochastic optimization, delivering near-optimal rates (modulo logarithmic factors) without any unverifiable parameter inputs.
• Ensemble Selection under Gap-Dependent Variance: The paper provides sharper high-probability guarantees for the grid candidate ensemble selection step, exploiting problem-dependent noise structure, especially in the interpolation regime, supplanting previous worst-case variance analyses.
• Extensive Theoretical and Experimental Benchmarks: Fully parameter-free variants match the best known (tuned) rates up to polylogarithmic terms and outperform prior partially parameter-free baselines according to the new definitions.

Technical Overview

Problem Setting and Parameter-Freeness

The methods support unconstrained stochastic minimization with access to unbiased gradient and function value oracles, under general (possibly data-dependent) noise structures. The relevant parameters (smoothness, distance to optimum, objective gap, noise bounds, etc.) are assumed inaccessible. Parameter-freeness is defined rigorously: full parameter-freeness means no input to the algorithm needs to satisfy even relaxed conditions (e.g., upper/lower bounds) referencing the unknown parameters; partial parameter-freeness subsumes methods that still depend on unverifiable bounds.

The Grasp Framework: Grid Search with Self-Bounding

Self-Bounding to Determine Parameter Ranges

The crux of Grasp lies in the self-bounding analysis, which, by comparing the target rate (as a function of unknown parameters) to a trivial but always valid baseline (e.g., function value at the initial point), deduces effective, computable upper bounds for search intervals. For example, in $L$-smooth convex minimization, the quadratic dependence of the accelerated rate on the initial distance $d_0$ ensures that too large choices will be dominated by the trivial baseline $\|\nabla \ell(x^0)\| d_0$, and hence the search can be safely restricted to a computed upper bound involving only observable quantities and the oracle budget.

Grid Search and Budget Allocation

Given the computed ranges, the framework discretizes them (usually geometrically), runs base algorithms (e.g., SGD in the non-convex case, UniXGrad for universal convex optimization) at each grid point, and distributes the oracle budget efficiently. Final selection is performed by an ensemble procedure over all candidates.

Ensemble Selection with Gap-Dependent Variance

For candidate selection, the ensemble step compares (possibly noisy) estimates of the objective (or, for non-convex cases, gradient norms), incorporating multiple evaluations to average out noise. The analysis provides a new sharp high-probability guarantee leveraging a variance decomposition that depends on the optimality gap rather than a worst-case bound, yielding improved rates in interpolated (nearly noise-free-at-optimum) regimes.

Non-Convex and Convex Results

Non-Convex Optimization

Grasp-NC, the fully parameter-free non-convex variant, achieves the following with no unverifiable parameter input (other than the budget and confidence level):

$$\mathbb{E}\left[ \|\nabla \ell(x^\text{out}) \|^2 \right] = \widetilde{O} \left( \sqrt{ \frac{L F \Delta^2}{T} } + \frac{L F + \Delta^2 }{T} \right)$$

(up to logarithmic factors), where $L$ (smoothness), $F$ (initial gap), and $\Delta$ (max noise) are all unknown and not required as input. This matches the best-known tuned SGD rates modulo logarithms and surpasses previous partially parameter-free approaches that required (e.g.) lower bounds on $F$ or upper bounds on $L$.

Convex (and Universal/Hölder-Smooth) Optimization

In the convex setting (Grasp-C), two variants are provided:

• Smooth-only acceleration: With no function value lower bound, the method delivers near-optimal accelerated rates (again up to logs) for $L$-smooth objectives.
• Universal/Hölder-smooth adaptation: If a lower bound on the optimal function value is available (often trivial for non-negative losses), the method is fully parameter-free and achieves universal convergence rates for all levels of smoothness:

$d_0$0

where $d_0$1 (Hölder constant), $d_0$2, $d_0$3, and $d_0$4 are all unknown and not required as input (the lower function value input can often be simply set to $d_0$5).

Through a careful analysis of the ensemble error and self-bounding search intervals, the method does not require a lower bound on the distance to optimum or an upper bound on the gradient noise function, in stark contrast to the best-known universal methods such as U-DOG.

Explicit Guarantee for Ensemble Error

A new lemma gives, under mild interpolation-type assumptions, an ensemble error at optimality of $d_0$6 (where $d_0$7 is the number of samples/per candidate, $d_0$8 noise at optimum, and $d_0$9 is the slope of the variance with function gap), which is strictly sharper than previous worst-case bounds when the variance collapses near the minimizer.

Empirical Validation

Experiments on the CIFAR-10/ResNet-18 pipeline validate that the proposed fully parameter-free methods are:

• Practically competitive: Convergence is within 8%-23% of hand-tuned SGD or UniXGrad, despite never requiring any parameter tuning.
• Robust to search parameter settings: Algorithm performance is largely insensitive to input choices for grid fineness or initial sampling budget, indicating the method's practicality and the weak impact of logarithmic overheads.

Implications and Future Directions

This work redefines the landscape of parameter-freeness in stochastic optimization. By closing the gap between practical usage and theoretical guarantees, it enables theoretically sound deployment of stochastic optimization without user-tuned parameter selection, a significant step for robust and scalable machine learning.

Theoretically, the self-bounding analysis is broadly applicable and could impact the design of parameter-free algorithms in other domains (e.g., stochastic control, online learning, reinforcement learning) where parameter-dependence impedes real-world adoption.

Practically, practitioners can expect strong guarantees when deploying Grasp-style grid search with self-bounding, especially in high-variance or hard-to-tune environments.

Future work suggested includes:

• Extension to constrained or composite minimization settings, including accelerated or adaptive variants.
• Reduction of the remaining polylogarithmic factors and improved ensemble selection to further close the gap to optimally-tuned rates.
• Application to more complex oracle models such as those in bandit or reinforcement learning.

Conclusion

This paper establishes, both in theory and experiment, that fully parameter-free stochastic optimization at near-optimal rates is possible via the Grasp grid search and self-bounding analysis. The methodologies synthesize practically relevant search intervals with robust, tight ensemble selection and provide a new baseline for theoretical and algorithmic research in optimization and learning without parameter tuning.

---

Reference:

"Towards Fully Parameter-Free Stochastic Optimization: Grid Search with Self-Bounding Analysis" (2604.16888)