Batch Sharpness in Neural Networks

• Batch sharpness is a metric that quantifies the local curvature of the loss landscape by assessing worst-case or average-case loss increases under small, norm-bounded parameter perturbations.
• It reveals how mini-batch size and gradient noise influence the optimization dynamics and regularization effects in neural network training.
• Empirical and theoretical studies demonstrate that using small batches and sharpness-aware algorithms like SAM effectively promotes flatter minima and enhances generalization performance.

Batch sharpness quantifies the local loss landscape curvature encountered by stochastic optimizers at the mini-batch level, serving as a bridge between classical (global) sharpness, generalization theory, and modern sharpness-aware training algorithms. It captures the worst-case or average-case increase in mini-batch loss under small, norm-bounded parameter perturbations and is critical for understanding, controlling, and regularizing solution flatness in large-scale neural network training.

1. Formal Definitions and Notions of Batch Sharpness

Let $L: \mathbb{R}^d \to \mathbb{R}$ be the loss function, $x \in \mathbb{R}^d$ parameters, $\mathcal{B}$ a mini-batch, and $\rho > 0$ the perturbation radius.

Several operational definitions of batch sharpness are prevalent:

• Worst-case (max) sharpness:

$$R_{\max}(x) := \max_{\|v\|_2 \leq 1} [L(x + \rho v) - L(x)]$$

This measures the maximal increase in loss in a $\rho$-ball centered at $x$.

• Ascent-direction sharpness (SAM objective):

$$R_{\text{asc}}(x) := L\left(x + \rho\frac{\nabla L(x)}{\|\nabla L(x)\|_2}\right) - L(x)$$

The practical implementation in SAM (Wen et al., 2022).

• Average-direction sharpness (for generalization):

$$R_{\text{avg}}(x) := \mathbb{E}_{g \sim N(0, I)} [L(x + \rho g / \|g\|_2) - L(x)]$$

Used in stochastic analysis and generalization bounds.

Alternative metrics include:

• Largest Hessian eigenvalue (spectral norm):

$S(x) = \lambda_{\max}(\nabla^2 L(x))$

(Sun et al., 21 Jun 2025, Liao et al., 22 Apr 2026)

• Trace of Hessian:

$x \in \mathbb{R}^d$0

(Ghosh et al., 2023)

• Directional batch curvature (“batch sharpness statistics”):

$x \in \mathbb{R}^d$1

with $x \in \mathbb{R}^d$2 the mini-batch gradient, $x \in \mathbb{R}^d$3 the mini-batch Hessian (Andreyev et al., 15 Apr 2026).

2. Batch Sharpness in Stochastic Optimization and Generalization

Stochastic optimizers (SGD, SGDM, Adam, etc.) traverse the parameter space under mini-batch noise. Gradient variance induced by smaller batches tends to bias the optimizer towards flatter minima with lower sharpness, a phenomenon supported by multiple theoretical and empirical results:

• The expected sharpness under SGD is suppressed below the full-batch “edge of stability” threshold $x \in \mathbb{R}^d$4, with the sharpness gap scaling inversely with batch size:

$x \in \mathbb{R}^d$5

where $x \in \mathbb{R}^d$6 is the gradient noise projected onto the top Hessian eigenvector, and $x \in \mathbb{R}^d$7 and $x \in \mathbb{R}^d$8 are progressive sharpening and self-stabilization rates (Liao et al., 22 Apr 2026).

• Smaller mini-batches inject more stochasticity, which acts as a regularizer for sharpness (i.e., promotes flatter minima, empirically demonstrated in vision, NLP, graph, and autoencoder domains) (Sun et al., 21 Jun 2025, Liu et al., 2022, Ghosh et al., 2023).
• Momentum further modifies the plateau values of batch sharpness, creating two distinct regimes:
  • For small batches: $x \in \mathbb{R}^d$9 (noise-dominated, flatter)
  • For large batches: $\mathcal{B}$0 (deterministic, sharper)
  • (Andreyev et al., 15 Apr 2026).

Empirical observations consistently confirm an inverse relationship between batch size and the sharpness (as measured by the spectral norm or trace of the Hessian) at solutions found by SGD/SGDM (Sun et al., 21 Jun 2025, Ghosh et al., 2023).

3. Batch Sharpness in Sharpness-Aware Minimization Algorithms

SAM and its variants (mSAM, Reweighted-SAM, Friendly-SAM, δ-SAM) directly manipulate batch sharpness via adversarial perturbations:

• SAM: Solves

$\mathcal{B}$1

where the inner maximization is approximated by a step along the batch gradient:

$\mathcal{B}$2

(Wen et al., 2022, Zhou et al., 2021, Li et al., 2024).

• mSAM: Splits each mini-batch into $\mathcal{B}$3 micro-batches and averages adversarial gradients, yielding even flatter minima as $\mathcal{B}$4 decreases (“m-sharpness” phenomenon) (Behdin et al., 2023, Luo et al., 22 Sep 2025).
• Reweighted-SAM, δ-SAM, F-SAM: Enhance batch sharpness regularization by adaptively weighting perturbation directions (e.g., based on per-instance curvature or gradient norm) or by isolating the stochastic (noise) component of the batch gradient (Zhou et al., 2021, Luo et al., 22 Sep 2025, Li et al., 2024).

Theoretical analysis shows full-batch SAM penalizes the largest eigenvalue (worst-case), whereas small-batch or micro-batch variants penalize average-case metrics (e.g., trace of Hessian), often enforcing stronger generalization (Wen et al., 2022, Luo et al., 22 Sep 2025).

Algorithm	Penalized Sharpness Notion	Effect (Flatness)
SGD	None / incidental	Baseline
SAM	$\mathcal{B}$5 (worst-case)	Flatter than SGD
mSAM	Trace$\mathcal{B}$6 / variance-based	Flattest (for $\mathcal{B}$7)
F-SAM, δ-SAM	Instance-adaptive	Similar to mSAM

4. Batch Sharpness, Model Complexity, and Scheduler Effects

Causal and dynamical modeling of training elucidates the role of batch sharpness in broader generalization and optimization metrics:

• HGCNet causal analysis: All batch size effects on generalization are mediated by gradient noise, which in turn controls batch sharpness (largest Hessian eigenvalue); smaller batches cause lower $\mathcal{B}$8 (Sun et al., 21 Jun 2025).
• Scheduler effects: Increasing batch size or decaying learning rate reduces search-direction noise and drives convergence to flatter minima in SAM/GSAM. Empirically, batch ramp-up can reduce sharpness by 10–50× (Harada et al., 2024).
• Algorithmic extensions: Layer-wise LookSAM, dynamic reweighting, and BN-invariant sharpness regularizers efficiently scale batch-sharpness-aware methods to large batch sizes and BatchNorm-equipped nets, always targeting reduction of spectral or scale-invariant curvature statistics (Liu et al., 2022, Yi et al., 2021).

5. Measurement and Empirical Protocols for Batch Sharpness

Batch sharpness is operationalized by several measurement protocols:

• Largest eigenvalue ($\mathcal{B}$9):
  • Compute the Hessian (or an approximation, e.g., via power iteration) on a mini-batch at a checkpointed parameter.
  • Report mean/standard deviation over all mini-batches / epochs / seeds (Sun et al., 21 Jun 2025, Liao et al., 22 Apr 2026).
• Directional batch-sharpness:

$\rho > 0$0

Estimated by sampling fresh mini-batches, computing gradients and Hessian-vector products per batch, averaging directional curvature statistics (Andreyev et al., 15 Apr 2026).

• Trace of the Hessian:

Directly sum Hessian eigenvalues at the minimized solution, favored in certain theoretical analyses (Ghosh et al., 2023).

• Maximal $\rho > 0$1-normed loss difference: Empirically measure worst-case increase in loss under small, adversarially chosen perturbations in parameter space (Harada et al., 2024).

6. Theoretical Mechanisms and Alignment Properties

The sharpness targeted by different training regimes is governed by fundamental geometric mechanisms:

• Gradient-Hessian alignment: In the vicinity of a minimizer, the normalized gradient direction aligns with the top Hessian eigenvector, so SAM’s ascent approximation closely tracks descent on $\rho > 0$2 for full-batch and on $\rho > 0$3 for small-batch (Wen et al., 2022).
• Phase transitions and scaling: In simple models (e.g., deep linear nets, single-neuron autoencoders), expected sharpness scales with key structural quantities (network depth, data difficulty) and is controlled by batch size and step size (Yoo et al., 7 Jun 2025, Ghosh et al., 2023). Practically, increasing batch size or model depth accelerates sharpening unless regularized.

7. Practical Guidelines and Implications

Empirical and theoretical studies consistently recommend optimizing batch sharpness for improved generalization:

• Preference for small batches to achieve flatter minima; this remains effective in domains beyond vision (e.g., graphs, text, SNNs) (Sun et al., 21 Jun 2025, Nicholson, 14 Mar 2026).
• Progressive batch schedules (starting small, then increasing) or joint batch-size and learning-rate scaling to exploit regularization effects while retaining computational efficiency (Harada et al., 2024).
• Monitoring batch sharpness (e.g., via Hessian spectral norm) as a dynamic stability and generalization diagnostic (Andreyev et al., 15 Apr 2026, Liao et al., 22 Apr 2026).
• Utilizing sharpness-aware algorithms (SAM, mSAM, F-SAM, δ-SAM) and their layer-wise/instance-adaptive variants to directly regularize sharpness independent of batch constraints (Behdin et al., 2023, Zhou et al., 2021, Li et al., 2024).

Batch sharpness thus provides a quantifiable, manipulable axis for understanding, predicting, and improving deep learning generalization and optimization dynamics across scales, architectures, and domains.