Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization

Abstract: Deep learning optimization relies heavily on the assumption of smooth loss landscapes, a condition systematically violated by modern architectures due to non-smooth components such as ReLU activations and quantization operators. In such non-smooth regimes, adaptive optimizers such as Adam suffer from gradient chattering, violent oscillations caused by conflicting signals within the Clarke subdifferential, leading to poor convergence and suboptimal generalization. To address this, we introduce Singularity-aware Adam (S-Adam), a novel optimizer that stabilizes training by dynamically modulating step sizes based on local geometric instability. Our key contribution is the Local Geometric Instability (LGI) metric, a computationally efficient estimator of the Clarke subdifferential diameter derived from the variance of randomized directional derivatives. S-Adam incorporates an adaptive damping mechanism exp(-$λ$$ρ$) that decelerates updates in high-instability regions while preserving fast convergence in smooth basins. We provide a rigorous convergence analysis using differential inclusions, proving that S-Adam converges almost surely to ($δ$,$ε$)-Clarke stationary points at the optimal O(1/$\sqrt(T)$) rate. Empirical evaluations on Quantization-Aware Training (QAT) and high-noise small-batch learning demonstrate that S-Adam consistently outperforms AdamW and Prox-SGD, achieving accuracy gains of up to 6 percent on CIFAR-100 and 3 percent on TinyImageNet while effectively mitigating gradient oscillations.

• The paper introduces the LGI metric, a novel estimator of local non-smoothness using randomized directional derivatives.
• It adapts the Adam optimizer by incorporating a geometry-aware damping factor to mitigate gradient chattering in singular loss landscapes.
• Empirical results demonstrate that S-Adam outperforms conventional optimizers in quantization-aware and high-noise training scenarios.

Singularity-aware Optimization via Randomized Geometric Probing: Stable Non-smooth Optimization with S-Adam

Motivation and Problem Setting

Deep learning optimization traditionally assumes the existence of Lipschitz-continuous gradients, a property that is consistently violated in state-of-the-art architectures incorporating ReLU activations, quantization, and various sparsity-promoting regularizers. These constructs induce regions of non-differentiability—"singularities"—in loss landscapes that invalidate smooth optimization theory and render classical optimizers such as Adam prone to gradient chattering due to the conflicting directions in the Clarke subdifferential. This phenomenon manifests as violent parameter oscillations, degeneration of convergence, and compromised generalization—issues exacerbated in quantization-aware training (QAT) and high-noise small-batch regimes.

Local Geometric Instability (LGI): A Metric for Nonsmoothness

The central technical contribution is the Local Geometric Instability (LGI) metric, a computationally efficient estimator for the diameter of the Clarke subdifferential, operationalized via the variance of directional derivatives sampled in random directions:

Figure 2: The behavior of LGI and random geometric probing are visualized for both smooth (left) and non-smooth (right) functions, indicating sharp nonsmooth regions.

Given a locally Lipschitz function $f:\mathbb{R}^d\rightarrow \mathbb{R}$, the Clarke subdifferential $\partial_C f(x)$ quantifies the local geometric complexity at $x$. The LGI metric,

$$\rho_t(x_t) = \frac{\text{Var}_{u_i} \left[ D_i \right]}{\mathbb{E}[D_i^2] + \epsilon}, \quad D_i = \frac{f(x_t + \delta u_i) - f(x_t)}{\delta}$$

for random directions $u_i$ on the unit sphere, is a normalized probe signal indicating instability: a large value indicates high geometric disorder (i.e., conflicting subgradients), whereas values near zero expose locally smooth (almost Euclidean) regimes.

The LGI has a dual interpretation: in the non-smooth view, it is a proxy for the Clarke subdifferential diameter, and in the stochastic smoothing view, it encodes the relative curvature of a locally smoothed surrogate.

Singularity-aware Adam (S-Adam): Methodology

The S-Adam algorithm introduces an adaptive, geometry-aware damping factor to the Adam update, dynamically decelerating progress in the presence of local nonsmoothness as signaled by a high LGI score. The modified update reads:

$\hat{\eta}_t = \eta_t \exp(-\lambda \rho_t)$

$$w_{t+1} = w_t - \hat{\eta}_t \frac{m_t}{\sqrt{v_t} + \epsilon}$$

where $m_t, v_t$ are standard Adam moment estimates, and $\lambda$ is a damping hyperparameter. The exponential form ensures both non-degeneracy ($\hat{\eta}_t$ is bounded away from zero for any finite $\partial_C f(x)$0) and monotonicity with respect to the instability measure. For a non-smooth optimization step, the geometric brake slows updates as soon as the local geometry is detected to be sharply non-smooth, effectively suppressing the gradient chattering effect.

Convergence Guarantees

A rigorous convergence analysis is presented, leveraging the framework of stochastic differential inclusions suitable for set-valued subdifferential dynamics. The main theorem proves that S-Adam achieves almost sure convergence to the Clarke stationary set at an optimal $\partial_C f(x)$1 rate, under standard assumptions: locally Lipschitz continuity, bounded iterates, and consistent moment estimators. The analysis shows strict Lyapunov descent outside the stationary set, with the geometric brake modulating update velocity but not direction. Notably, in smooth regions (where $\partial_C f(x)$2), S-Adam degenerates to vanilla Adam, maintaining compatibility.

Empirical Analysis

Quantization-Aware Training (QAT) and Non-smooth Landscapes

Empirical studies on QAT benchmarks—CIFAR-100 (2-bit), TinyImageNet and Imagewoof2-160 (4-bit), and ImageNet (8-bit)—demonstrate that S-Adam achieves up to +6 percentage points improvement over AdamW/Prox-SGD in test accuracy, with a negligible computational overhead at low probe counts ($\partial_C f(x)$3):

Figure 4: S-Adam loss curves on CIFAR-100 highlight consistently lower and more stable losses compared to AdamW and Prox-SGD.

Figure 1: Training dynamics with S-Adam on CIFAR-100 maintain accuracy gains during epochs where baselines collapse due to gradient chattering.

These results are robust across datasets, and S-Adam's performance advantage is larger in more singular and unstable loss landscapes (e.g., low-bit quantization), where conventional optimizers become unstable or diverge. Importantly, even with minimal geometric probing ($\partial_C f(x)$4 directions), most of the geometric signal is captured, and further increasing $\partial_C f(x)$5 only modestly improves performance.

High-Noise Small-batch Regimes

In extremely stochastic regimes (ResNet18, batch size $\partial_C f(x)$6), S-Adam outperforms AdamW and Prox-SGD by +4.99 and +24.69 points on CIFAR-100 and Imagewoof2-160 respectively, demonstrating that LGI-triggered damping effectively suppresses noise-induced chattering not as mere gradient clipping, but by discriminating between geometric instability and random gradient noise.

Figure 3: On CIFAR-10 with batch size 4, S-Adam achieves faster and more stable convergence, diverging less than AdamW under high gradient noise.

Algorithmic and Practical Implications

• Generalization: S-Adam's ability to stabilize parameter trajectories in non-smooth, singular regimes provides an explicit form of algorithmic stability, which is theoretically connected to generalization guarantees through uniform stability bounds.
• Cost-Efficiency: The ability to operate with minimal random probes ($\partial_C f(x)$7) ensures practical deployability in large-scale training settings, where wall-clock overhead is a critical constraint.
• Transparent Geometry Adaptivity: Unlike smoothing or surrogate techniques that alter the optimization dynamics (potentially targeting different minima), S-Adam preserves the exact gradient direction and adaptively decelerates only when required by the geometry.

Theoretical Directions and Future Prospects

The LGI signal, as a computational probe, opens the possibility for the design of architecture- and regime-aware optimizers that automatically detect entry into non-smooth regions (e.g., ReLU, quantization, sparse patterns) and adapt optimization strategies accordingly. Similarly, enhanced diagnostics of the LGI landscape could allow for real-time sharpness/generalization estimation without recourse to full-surface analysis. Future development may also include incorporating LGI signals into trust-region or natural gradient frameworks, or coupling with second-order (curvature-aware) methods in regimes where non-smoothness is intermittent or structured.

Conclusion

The S-Adam optimizer introduces principled, stochastic estimates of local non-smoothness via the LGI metric and demonstrably leverages this geometric signal to stabilize and accelerate learning in both synthetic and applied non-smooth regimes. The theoretical guarantee of almost sure convergence to Clarke stationary points at minimax rates, together with strong empirical benchmarks under quantization-aware and small-batch high-noise training, provide robust evidence for the efficacy of geometric adaptivity in deep learning optimization. The methodology meaningfully connects nonsmooth optimization theory with modern scale and complexity, and lays foundational ground for further work on geometry-driven and automatic regime-adaptive optimization in neural networks.