Flat Local Minima

• Flat local minima are regions in loss landscapes with low curvature, ensuring that small parameter perturbations cause minimal increases in loss.
• They are quantified using measures like perturbation metrics, Hessian eigenvalues, and local entropy functionals to assess model robustness.
• Optimization algorithms such as SAM and SWA are designed to locate flat minima, thereby enhancing generalization and stability in deep models.

A flat local minimum is a point in a loss landscape where the objective function remains nearly constant or exhibits low curvature within a neighborhood, such that small perturbations of the parameters do not substantially raise the loss. In contrast to sharp minima, which are associated with high curvature and rapid changes in loss under parameter perturbation, flat minima are broadly recognized as crucial for improving generalization, stability, and robustness across a spectrum of optimization problems. Flatness can be quantified by various measures, including Hessian eigenvalues, perturbation metrics, and non-local “local entropy” functionals. The importance of flat local minima extends from deep learning to mathematical analysis of variational problems and plays a central role in the design and analysis of optimization algorithms, implicit regularization, and generalization theory.

1. Mathematical Definitions and Geometric Formalism

Flatness of a local minimum can be rigorously formalized in multiple, often complementary, ways:

• Perturbation-based sharpness/flatness: For a function $F:\mathbb{R}^d \to \mathbb{R}$, a (local) flat minimum at $x^*$ is one for which the sharpness metric

$$S_\rho(x^*) = \max_{\| \epsilon \|_2 \leq \rho} F(x^* + \epsilon) - F(x^*)$$

is small for suitably chosen $\rho$. This constrains the loss increase within a ball of radius $\rho$ around $x^*$ (Liu et al., 31 Oct 2025).

• Hessian-based flatness: The eigenvalue spectrum of the Hessian, $\nabla^2 F(x^*)$, provides a quantitative measure, with flat minima exhibiting a small maximum eigenvalue ($\lambda_{\max}$) or low trace:

$$\text{Flatness}(x^*) := \operatorname{Tr} [\nabla^2 F(x^*)] \text{ or } \lambda_{\max} [\nabla^2 F(x^*)]$$

Both trace and spectral norm are widely used as proxies for local curvature (Liu et al., 31 Oct 2025, Kaddour et al., 2022, Zhang et al., 5 Jun 2025).

• Neighborhood/Volume-based criteria: The (ε, δ)-flatness definition states that a minimizer $x^*$ is flat if

$$\{ x \mid \|x-x^*\|_2 \leq \delta \} \subseteq \{ x \mid |F(x) - F(x^*)| \leq \epsilon \}$$

for appropriate $(\epsilon, \delta)$ (Kaddour et al., 2022).

• Local entropy (non-local flatness): Flatness may also be quantified by the “local entropy” functional:

$$\mathcal{L}_{\rm LE}(w) = -\frac{1}{\beta} \log \int_{\mathbb{R}^N} \!\! \exp \left( -\beta L(w') - \beta \gamma \frac{1}{2} \| w'-w \|^2 \right) dw'$$

which encodes the volume of the low-loss neighborhood surrounding $w$ (Pittorino et al., 2020, Zhang et al., 2023).

• Variational/Geometric duality: Maximal variation around a point ($\omega_F(x, r) = \sup_{\|y-x\|\leq r} |F(y)-F(x)|$) and the minimal distance to a specified increase in F ($\delta_F(x, \ell)$) form an inverse relationship to capture flatness in non-smooth and non-Euclidean settings (Josz, 14 Sep 2025).

These viewpoints are not mutually exclusive; in high-dimensional models (such as DNNs), they often agree in characterizing the geometry of wide basins that correspond to robust or generalizable solutions.

2. Origins and Structural Causes of Flat Minima

A key source of flat local minima derives from the intrinsic symmetries or degeneracies in parameterizations, especially in overparameterized models:

• Symmetry-induced flat minima: In deep linear networks, positive-dimensional manifolds of stationary points (flat minima) arise due to scaling and rotation symmetries. These flat sets can be artifacts of continuous group invariances and vanish under generic L₂ regularization (Mehta et al., 2018, Josz, 14 Sep 2025).
• Conservation laws and group actions: Conservation laws arising from invariance (e.g., under GL(r) for matrix factorization problems) yield quadratic quantities that remain constant along flat directions. The action of group symmetries can be used to construct explicit flattening flows, or to distinguish “truly” flat minima from degenerate ones (Josz, 14 Sep 2025).
• Optimization dynamics and regularization: Both explicit and implicit regularization (e.g., weight decay, layer-wise noise, or SGD itself) drive the trajectory along the minima manifold toward flatter regions. This has been formalized analytically in the context of layer imbalance in linear networks and more generally via the volume of attraction basins in nonlinear models (Ginsburg, 2020, Mulayoff et al., 2020, Xie et al., 2020).

3. Algorithmic Approaches for Locating Flat Minima

Multiple optimization strategies have been developed or shown to be biased toward flat minima:

• Stochastic Weight Averaging (SWA): Averaging iterates from late-stage SGD trajectories increases the likelihood of locating the center of flat valleys, where the local curvature is minimized (Kaddour et al., 2022, He et al., 2019). SWA does not alter the loss function but exploits the natural lingering of SGD in wide, flat basins.
• Sharpness-Aware Minimization (SAM): SAM explicitly solves

$$\min_{x} \max_{\| \epsilon \| \leq \rho} F(x + \epsilon)$$

using an efficient first-order approximation, thereby biasing solutions toward neighborhoods with uniformly low loss (Liu et al., 31 Oct 2025, Kaddour et al., 2022, Ahn et al., 2023, Pittorino et al., 2020).

• Entropy-based algorithms (Entropy-SGD, Replicated-SGD): These algorithms incorporate local entropy objectives into the optimization process, encouraging convergence to regions with large volumes of low loss (effectively maximizing the “local entropy” defined above) (Pittorino et al., 2020, Zhang et al., 2023).
• Gradient-norm penalties: Penalizing the global gradient norm (as in DP-FedPGN) directs optimization toward points where not only the loss but also its gradient is small, ensuring flatness both locally and in federated (aggregated) contexts (Liu et al., 31 Oct 2025).
• Zeroth-order and random perturbation methods: Zeroth-order algorithms with two-point gradient estimators and explicit random noise can exhibit an inherent bias toward minimizing the trace of the Hessian, thus favoring flat minima even without access to gradients (Zhang et al., 5 Jun 2025, Ahn et al., 2023).
• Lookahead and Interpolation: Strategies that combine fast, large-learning-rate exploration steps with slow, stable interpolation (as in Lookahead and SWAD variants) promote weight diversity and implicitly favor convergence to flat regions of the loss landscape (Zhang et al., 2023).

4. Analytical and Theoretical Properties

Flat minima are associated with several favorable theoretical properties:

• Generalization: There is substantial empirical and theoretical evidence that flat minima correspond to models that generalize better, as these solutions are less sensitive to small parameter perturbations and robust to train-test domain shifts (Kaddour et al., 2022, Ahn et al., 2023, Pittorino et al., 2020). PAC-Bayes analysis and geometric Occam’s Razor principles support the view that flatness regularizes complexity in the functional sense.
• Concentration and escape dynamics: Stochastic gradient descent, due to Hessian-aligned noise, is exponentially biased toward flat minima. The escape time from sharp minima grows rapidly with batch size and decreases with learning rate, quantitatively explaining why large batches may result in sharper minima (Xie et al., 2020).
• Absence of spurious or degenerate flat minima: Under generic regularization, symmetries that induce flat manifolds of stationary points are broken, yielding isolated, nondegenerate minima (Mehta et al., 2018). This eliminates ambiguity in optimization and theoretically guarantees convergence to robust optima.
• Robustness to data and input perturbations: Regularization schemes that flatten parameter-space gradients (e.g., Mirror Gradient, local entropy, or explicit gradient-norm penalties) also suppress input-space sensitivity, producing solutions robust to noise and distributional shifts (Zhong et al., 17 Feb 2024, Liu et al., 31 Oct 2025, Zhang et al., 2023).

5. Empirical Manifestations and Benchmarks

The prevalence of flat minima and the importance of explicit or implicit bias toward them have been validated across diverse architectures and tasks:

Domain	Flatness Optimization (Examples)	Empirical Outcomes
Computer Vision	SAM, SWA, WASAM, Entropy-SGD, Lookahead	Lower Hessian λ_max, improved test accuracy, faster convergence
NLP / Transformers	SAM, DP-FedPGN, Mirror Gradient	Enhanced generalization, higher robustness to noise/shifts
Federated Learning	DP-FedPGN, DP-FedSAM	Reduced privacy-induced degradation, global-wide flatness
Recommender Systems	Mirror Gradient (MG)	Gains in Recall/NDCG, mitigated robustness risks
Matrix Factorization	Conservation law approaches	Characterization of all flat minima, explicit geometric structure
Shallow Models	Belief Propagation, BP-based entropy maximization	Bayes-optimal classifier coincides with widest basin

• Visualization: Loss-landscape slices, Hessian spectrum plots, and volume-based entropy profiles are used to distinguish flat from sharp minima (Kaddour et al., 2022, Zhang et al., 2023, Pittorino et al., 2020).
• Quantitative gains: In benchmarks, flat-minima optimizers yield significant boosts in accuracy (e.g., DP-FedPGN: +13% over DP-FedAvg on CIFAR-10), with consistently lower measures of sharpness and faster convergence (Liu et al., 31 Oct 2025).

6. Open Problems and Future Directions

Despite a unified analytic and empirical case for the value of flat minima, several challenges and questions remain:

• Precise characterization in non-Euclidean and nonsmooth settings: General frameworks (e.g., maximal variation and level-set dualities) extend the notion of flatness beyond Hessian-based approaches, but further theoretical development is needed for highly nonconvex or nondifferentiable landscapes (Josz, 14 Sep 2025).
• Optimal flatness regularization vs. explicit margin control: Flatness does not universally guarantee the best possible generalization. There is ongoing debate regarding the optimality of flatness-based regularization compared to large-margin or data-dependent criteria in specific contexts.
• Scaling of dimension dependency: Zeroth-order flat-minimizing methods exhibit dimension-dependent convergence that may become prohibitive in large-scale models; developing sharper estimators and hybrid gradient schemes is a likely research direction (Zhang et al., 5 Jun 2025).
• Asymmetry and directional flatness: Recent work demonstrates that minima in deep nets are often asymmetric, with sharpness/flatness depending on direction. Weight-averaging and batch-normalization can bias solutions towards flatter basin sides, affecting generalization (He et al., 2019).
• Robustness/generalization trade-offs under privacy constraints: In privacy-preserving federated learning, global flatness is necessary to avoid degradation, but enforcing global-wide flatness remains algorithmically challenging under privacy noise and communication constraints (Liu et al., 31 Oct 2025).

In summary, flat local minima lie at the intersection of geometry, analysis, algorithm design, and statistical learning theory, providing a powerful and unifying concept for designing robust and generalizable learning systems across domains.