Lower Bounds and Proximally Anchored SGD for Non-Convex Minimization Under Unbounded Variance

Abstract: Analysis of Stochastic Gradient Descent (SGD) and its variants typically relies on the assumption of uniformly bounded variance, a condition that frequently fails in practical non-convex settings, such as neural network training, as well as in several elementary optimization settings. While several relaxations are explored in the literature, the Blum-Gladyshev (BG-0) condition, which permits the variance to grow quadratically with distance has recently been shown to be the weakest condition. However, the study of the oracle complexity of stochastic first-order non-convex optimization under BG-0 has remained underexplored. In this paper, we address this gap and establish information-theoretic lower bounds, proving that finding an $ε$-stationary point requires $Ω(ε^{-6})$ stochastic BG-0 oracle queries for smooth functions and $Ω(ε^{-4})$ queries under mean-square smoothness. These limits demonstrate an unavoidable degradation from classical bounded-variance complexities, i.e., $Ω(ε^{-4})$ and $Ω(ε^{-3})$ for smooth and mean-square smooth cases, respectively. To match these lower bounds, we consider Proximally Anchored STochastic Approximation (PASTA), a unified algorithmic framework that couples Halpern anchoring with Tikhonov regularization to dynamically mitigate the extra variance explosion term permitted by the BG-0 oracle. We prove that PASTA achieves minimax optimal complexities across numerous non-convex regimes, including standard smooth, mean-square smooth, weakly convex, star-convex, and Polyak-Lojasiewicz functions, entirely under an unbounded domain and unbounded stochastic gradients.

• The paper establishes complexity lower bounds for achieving ε-stationarity under the BG-0 condition, demonstrating Ω(ε⁻⁶) queries for L-smooth functions.
• The paper introduces PASTA, an algorithm that couples proximal anchoring, Tikhonov regularization, and variance reduction to attain optimal convergence rates.
• The findings imply that classical SGD is suboptimal in unbounded variance settings, necessitating innovative strategies for non-convex optimization.

Lower Bounds and Proximally Anchored SGD for Non-Convex Minimization Under Unbounded Variance

Problem Formulation and BG-0 Variance Condition

The paper rigorously addresses stochastic non-convex minimization in unconstrained domains, abandoning the classical bounded variance assumption in favor of the Blum-Gladyshev (BG-0) condition. BG-0 posits that the variance of the stochastic gradient estimator is permitted to grow quadratically with the distance from a reference anchor: $$\mathbb{E}[\|\widetilde{\nabla} f(x) - \nabla f(x)\|^2] \le B_v^2\|x - x_0\|^2 + b_v^2$$. This variance model is the weakest viable assumption and is notably applicable to machine learning settings such as neural network training where bounded variance does not hold. Over unbounded domains, this structural variance growth invalidates contractive convergence proofs of standard SGD, requiring new algorithmic and theoretical approaches.

Information-Theoretic Lower Bounds under BG-0

The paper establishes strong complexity lower bounds for achieving $\epsilon$-stationarity in smooth non-convex settings under unbounded variance. For $L$-smooth functions, any algorithm interacting with a BG-0 oracle requires $\Omega(\epsilon^{-6})$ queries; for mean-square smoothness (MSS), the lower bound is $\Omega(\epsilon^{-4})$. These results formally demonstrate that the attainable oracle complexity degrades sharply compared to classical bounded variance regimes ($\Omega(\epsilon^{-4})$ and $\Omega(\epsilon^{-3})$ respectively), owing to the adversarial noise scaling permitted by BG-0. The construction leverages a composite optimization landscape wherein high variance is encountered only after traversing a large distance, causing a severe signal-to-noise ratio collapse for stochastic algorithms.

Proximally Anchored Stochastic Approximation: PASTA Algorithm

To match these lower bounds, the paper introduces PASTA—a unified framework that couples Halpern-style geometric anchoring with Tikhonov regularization and variance reduction. The anchoring parameter $\beta_t$ and step size $\eta_t$ are tightly coupled ($\beta_t = \lambda \eta_t$), making each step equivalent to proximal gradient descent on a strongly convex regularized surrogate $\epsilon$0. The epoch-based structure updates the anchor iteratively, enabling convergence to a true stationary point by sequentially attenuating bias from the regularized objective. Variance reduction (PAGE-style) is incorporated for MSS regimes. Key algorithmic instantiations include:

• Smooth, non-convex functions: Dynamic batching ensures bounded estimator variance, achieving $\epsilon$1 complexity.
• Mean-square smoothness: Recursive variance reduction is used with dynamic large batches and small batch corrections, reaching $\epsilon$2 complexity.
• Weakly convex, star-convex, and PL settings: Structural regularization and epoch-based anchoring automatically enforce optimal rates without domain restrictions.

Geometric Universality and Uncertainty Principle

PASTA demonstrates minimax optimality across geometric regimes: smooth, mean-square smooth, weakly convex, star-convex, and Polyak-Lojasiewicz landscapes. The analysis reveals a geometric-statistical uncertainty principle: removing geometric anchoring necessitates unbounded batch sizes to control drifting variance, whereas anchoring with sufficiently large regularization absorbs the variance without large computation. This conjugacy establishes that non-expansive settings cannot be solved by naïvely small batches and minimal anchoring under BG-0.

Sharp Upper Bounds and Optimality

Formal convergence guarantees are established for each geometric class. The complexity bounds derived match the information-theoretic limits, substantiating that the signal-to-noise degradation under BG-0 is inherent and algorithm-independent:

• Smooth functions: PASTA achieves $\epsilon$3 complexity via dynamic batching; classical optimality is unattainable due to unbounded variance.
• Mean-square smooth functions: Recursive PAGE-style reduction reaches $\epsilon$4.
• PL and star-convex: Quadratic growth enables natural absorption of variance; fast linear convergence, $\epsilon$5, is preserved.
• Weakly convex: Stationarity with respect to the Moreau envelope is attained, with complexity $\epsilon$6.

Theoretical and Practical Implications

The work precisely quantifies the price of relaxing bounded variance, pushing stochastic optimization theory beyond classical constraints. Practically, algorithms employing BG-0-compatible anchoring and variance reduction are required for non-convex learning in unbounded domains; simple SGD or accelerated variants relying on bounded variance are provably suboptimal. Theoretically, future developments should investigate further relaxations (e.g., heavy-tailed noise), adaptive anchoring schedules, and integration with high-order information, always mindful of the structural noise scaling.

Conclusion

The paper resolves the stochastic oracle complexity of non-convex minimization under the weakest variance condition (BG-0), furnishing tight lower bounds and minimax optimal algorithms. By coupling anchoring and regularization, PASTA achieves universal convergence across regimes with unbounded variance, requiring neither bounded domain nor gradient assumptions. These advances clarify both the limitations and effective strategies for stochastic non-convex optimization beyond classical protocols, charting the course for future algorithmic designs under minimal variance assumptions.