Batch-Size-Aware Gradient Noise Bound

• The paper establishes a bound showing that the mean-squared gradient error scales as 1/b, with corrections for covariance and non-Euclidean geometry.
• It details how optimizer geometry and curvature-weighted covariance structures modify noise estimates beyond simple scalar variance considerations.
• It motivates adaptive batch selection strategies that balance noise reduction and computational constraints to improve training efficiency.

Searching arXiv for the cited papers and closely related work on batch-size-aware gradient-noise bounds. [arXiv search] Query: "batch-size-aware gradient-noise bound stochastic conditional gradient (Islamov et al., 22 Mar 2026)" A batch-size-aware gradient-noise bound is a quantitative statement that makes the stochastic error of a mini-batch gradient estimator explicit as a function of batch size, and, in some formulations, sequence length, optimizer geometry, curvature, or a fixed compute budget. In the classical unbiased mini-batch setting, the mean-squared gradient error scales as $1/b$; in more structured settings, the bound is modified by finite-population corrections, covariance structure, non-Euclidean dual norms, or heavy-tailed normalization. Recent work has also made the batch variable part of the optimization design itself, using such bounds to derive critical batch sizes, signal-to-noise tests, and adaptive schedules rather than treating batch size as a static throughput parameter (De et al., 2016, Islamov et al., 22 Mar 2026, Naganuma et al., 3 Feb 2026).

1. Formal core and canonical variance laws

At its most basic, the object of interest is the deviation between a mini-batch gradient and the full gradient. If $g_b(x)$ denotes the average gradient over a batch of size $b$, and $\nabla \ell(x)$ the population or empirical full gradient, the standard unbiased model writes

$$e_b(x) = g_b(x) - \nabla \ell(x), \qquad \mathbb{E}[e_b(x)] = 0.$$

Under i.i.d. sampling with bounded second moments, one exact identity is

$$\mathbb{E}\,\|g_b(x)-\nabla \ell(x)\|^2 =\frac{1}{b}\,\mathrm{Tr}\,\mathrm{Var}_{z}\big(\nabla f(x;z)\big),$$

which makes the $1/b$ law explicit (De et al., 2016). A uniform version replaces the data-dependent variance by

$$\mathbb{E}_B\,\|g_b(x)-\nabla \ell(x)\|^2 \le \frac{4L_z^2\,\mathrm{Tr}\,\mathrm{Var}_z(z)}{b},$$

and this directly yields batch-size conditions for maintaining a target signal-to-noise ratio (De et al., 2016).

When sampling is performed without replacement from a finite dataset, the same principle holds but acquires a finite-population correction. One exact covariance formula is

$$\Sigma_b^{\mathrm{wor}}(\theta) = \frac{n-b}{b(n-1)}\big(F(\theta)-g(\theta)g(\theta)^\top\big),$$

where $F(\theta)$ is the empirical Fisher and $g_b(x)$0 the full empirical gradient; with replacement, the factor becomes $g_b(x)$1 (Wu et al., 2019). This distinction matters when $g_b(x)$2 is no longer negligible relative to dataset size, but the leading behavior remains batch-size controlled.

A second basic pattern is relative, rather than absolute, noise control. For example, the norm test

$g_b(x)$3

implies the lower bound

$g_b(x)$4

so that the required batch size grows as the gradient norm shrinks (Ostroukhov et al., 2024). This is the formal basis of many adaptive schemes: near stationarity, keeping noise proportional to signal requires increasing the batch.

2. Beyond scalar variance: covariance structure and effective noise

A major refinement of the concept is that scalar variance alone need not determine optimization behavior. In convex-quadratic analysis of large-batch SGD with injected structured noise, the expected suboptimality obeys

$g_b(x)$5

so the relevant quantity is $g_b(x)$6, not merely $g_b(x)$7 (Wen et al., 2019). Two noise models can therefore have the same scalar variance but different optimization effects.

This point is made concrete by comparing full Fisher and diagonal Fisher perturbations. In the quadratic model, full Fisher-like noise yields $g_b(x)$8, whereas diagonal Fisher-like noise yields $g_b(x)$9, which is smaller when off-diagonal structure is significant (Wen et al., 2019). The paper’s empirical finding that diagonal Fisher noise preserves optimization speed better than full Fisher noise is therefore not explained by variance magnitude alone, but by curvature-weighted covariance structure.

Related formulations express the same idea through signal-to-noise rather than curvature. In large-batch methods based on gradient signal-to-noise ratio, the per-coordinate quantity

$b$0

scales linearly with batch size, and the required batch size for a target GSNR level $b$1 becomes

$b$2

This makes batch selection coordinate- or layer-dependent rather than globally variance-matched (Jiang et al., 2023). A plausible implication is that a single scalar batch-size rule is often too coarse once gradient anisotropy becomes strong.

3. Geometry-aware bounds and non-Euclidean formulations

Recent work extends batch-size-aware bounds beyond Euclidean SGD to algorithms whose descent geometry is defined by a dual norm. In momentum-based stochastic conditional gradient under $b$3-smoothness, norm equivalence, and the variance model

$b$4

the momentum-tracking error satisfies

$b$5

Here, batch size $b$6 and sequence length $b$7 jointly reduce both transient and steady-state stochasticity through the factor $b$8 (Islamov et al., 22 Mar 2026).

The same paper derives the one-step descent law

$b$9

which shows directly how the dual-norm tracking error enters contraction under a $\nabla \ell(x)$0-Kurdyka–Łojasiewicz condition (Islamov et al., 22 Mar 2026). In this setting, the batch-size-aware noise bound is inseparable from geometry: the relevant error lives in $\nabla \ell(x)$1, not in a Euclidean norm alone.

A similar geometric generalization appears for signSGD and specSGD. For signSGD, whose geometry is $\nabla \ell(x)$2 with dual $\nabla \ell(x)$3, the non-Euclidean gradient-noise scale is

$\nabla \ell(x)$4

For specSGD or Muon, whose geometry is spectral with dual nuclear norm, the corresponding scale is

$\nabla \ell(x)$5

These formulas are the non-Euclidean analogues of the Euclidean gradient-noise scale, but measured in the optimizer’s dual norm rather than $\nabla \ell(x)$6 (Naganuma et al., 3 Feb 2026). This suggests that an optimizer-specific notion of “noise level” is required whenever the update rule itself is not Euclidean.

4. Fixed-budget regimes, critical batch sizes, and saturation

Once batch size is coupled to a fixed token or compute budget, the dependence of optimization error on batch size can become non-monotone. For stochastic conditional gradient under the $\nabla \ell(x)$7-KL condition, setting $\nabla \ell(x)$8 yields the fixed-budget error law

$\nabla \ell(x)$9

The three terms define a noise-dominated regime, a saturation regime, and an iteration-starved regime, and the critical effective batch size is

$$e_b(x) = g_b(x) - \nabla \ell(x), \qquad \mathbb{E}[e_b(x)] = 0.$$0

Below $$e_b(x) = g_b(x) - \nabla \ell(x), \qquad \mathbb{E}[e_b(x)] = 0.$$1, increasing $$e_b(x) = g_b(x) - \nabla \ell(x), \qquad \mathbb{E}[e_b(x)] = 0.$$2 reduces noise; beyond $$e_b(x) = g_b(x) - \nabla \ell(x), \qquad \mathbb{E}[e_b(x)] = 0.$$3, further increases reduce the number of updates too aggressively and worsen the bound (Islamov et al., 22 Mar 2026).

A different but related expression appears in empirical training-time modeling for mini-batch SGD. There the observed iteration complexity follows

$$e_b(x) = g_b(x) - \nabla \ell(x), \qquad \mathbb{E}[e_b(x)] = 0.$$4

with a practical critical scale

$$e_b(x) = g_b(x) - \nabla \ell(x), \qquad \mathbb{E}[e_b(x)] = 0.$$5

Once $$e_b(x) = g_b(x) - \nabla \ell(x), \qquad \mathbb{E}[e_b(x)] = 0.$$6, the floor $$e_b(x) = g_b(x) - \nabla \ell(x), \qquad \mathbb{E}[e_b(x)] = 0.$$7 dominates and increasing batch size yields only diminishing returns in update count (Perrone et al., 2019). This is a time-to-train analogue of the saturation regime.

The notion of critical batch size becomes more controversial in LLM training with Adam. Direct measurements on OLMo 1B and 7B found that CBS is near $$e_b(x) = g_b(x) - \nabla \ell(x), \qquad \mathbb{E}[e_b(x)] = 0.$$8 at initialization, increases rapidly at first, and then plateaus around $$e_b(x) = g_b(x) - \nabla \ell(x), \qquad \mathbb{E}[e_b(x)] = 0.$$9 documents, whereas the gradient-noise-scale proxy underestimated CBS by several orders of magnitude throughout training (Merrill et al., 29 May 2025). This does not invalidate gradient-noise bounds; rather, it limits the reliability of particular proxies when their assumptions, such as SGD-style scaling and well-conditioned curvature, do not hold.

5. Adaptive batch selection and scheduling rules

Batch-size-aware gradient-noise bounds are often operationalized as online tests. In big-batch SGD, a sample-variance estimator

$$\mathbb{E}\,\|g_b(x)-\nabla \ell(x)\|^2 =\frac{1}{b}\,\mathrm{Tr}\,\mathrm{Var}_{z}\big(\nabla f(x;z)\big),$$0

supports the acceptance rule

$$\mathbb{E}\,\|g_b(x)-\nabla \ell(x)\|^2 =\frac{1}{b}\,\mathrm{Tr}\,\mathrm{Var}_{z}\big(\nabla f(x;z)\big),$$1

with the batch increased by $$\mathbb{E}\,\|g_b(x)-\nabla \ell(x)\|^2 =\frac{1}{b}\,\mathrm{Tr}\,\mathrm{Var}_{z}\big(\nabla f(x;z)\big),$$2 if the test fails (De et al., 2016). The purpose is to keep the gradient signal-to-noise ratio above a target threshold so that constant or adaptively selected step sizes remain viable.

AdaBatchGrad uses the same principle in expectation-based form. Its norm test implies

$$\mathbb{E}\,\|g_b(x)-\nabla \ell(x)\|^2 =\frac{1}{b}\,\mathrm{Tr}\,\mathrm{Var}_{z}\big(\nabla f(x;z)\big),$$3

while inexact inner-product and orthogonality tests determine when batch growth is required in practice (Ostroukhov et al., 2024). Under exact tests the method attains $$\mathbb{E}\,\|g_b(x)-\nabla \ell(x)\|^2 =\frac{1}{b}\,\mathrm{Tr}\,\mathrm{Var}_{z}\big(\nabla f(x;z)\big),$$4 iteration complexity, whereas under inexact tests it attains

$$\mathbb{E}\,\|g_b(x)-\nabla \ell(x)\|^2 =\frac{1}{b}\,\mathrm{Tr}\,\mathrm{Var}_{z}\big(\nabla f(x;z)\big),$$5

iterations (Ostroukhov et al., 2024). The explicit role of batch size here is to keep the mini-batch gradient sufficiently aligned with the full gradient.

Other adaptive rules replace variance or norm tests by directional proxies. One Transformer method accumulates gradients until the short-step angle starts fluctuating again, stopping when

$$\mathbb{E}\,\|g_b(x)-\nabla \ell(x)\|^2 =\frac{1}{b}\,\mathrm{Tr}\,\mathrm{Var}_{z}\big(\nabla f(x;z)\big),$$6

with default $$\mathbb{E}\,\|g_b(x)-\nabla \ell(x)\|^2 =\frac{1}{b}\,\mathrm{Tr}\,\mathrm{Var}_{z}\big(\nabla f(x;z)\big),$$7 (Xu et al., 2020). Another line of work estimates gradient diversity,

$$\mathbb{E}\,\|g_b(x)-\nabla \ell(x)\|^2 =\frac{1}{b}\,\mathrm{Tr}\,\mathrm{Var}_{z}\big(\nabla f(x;z)\big),$$8

and updates the next epoch’s batch size by

$$\mathbb{E}\,\|g_b(x)-\nabla \ell(x)\|^2 =\frac{1}{b}\,\mathrm{Tr}\,\mathrm{Var}_{z}\big(\nabla f(x;z)\big),$$9

so that batch growth is tied to a theoretically motivated second-moment expression rather than a purely heuristic schedule (Chen et al., 19 Sep 2025).

6. Extensions, caveats, and recurring misconceptions

Several extensions show that the classical $1/b$0 law is not universal without qualification. In DP-SGD, the total gradient variance decomposes into a subsampling-induced term and a noise-induced term. For Poisson subsampling with rate $1/b$1, the subsampling variance decreases with $1/b$2, while the effective DP-noise variance becomes asymptotically invariant to $1/b$3 as the number of iterations grows (Räisä et al., 2024). In that regime, large batches reduce effective total variance because only the subsampling term continues to shrink.

Heavy-tailed settings require more substantial changes. When gradient noise lies in the domain of attraction of an $1/b$4-stable law with $1/b$5, the mini-batch average at $1/b$6 scales as

$1/b$7

and, for $1/b$8, the SGD error satisfies

$1/b$9

Under a constant stepsize $$\mathbb{E}_B\,\|g_b(x)-\nabla \ell(x)\|^2 \le \frac{4L_z^2\,\mathrm{Tr}\,\mathrm{Var}_z(z)}{b},$$0, increasing $$\mathbb{E}_B\,\|g_b(x)-\nabla \ell(x)\|^2 \le \frac{4L_z^2\,\mathrm{Tr}\,\mathrm{Var}_z(z)}{b},$$1 can still drive convergence in probability because $$\mathbb{E}_B\,\|g_b(x)-\nabla \ell(x)\|^2 \le \frac{4L_z^2\,\mathrm{Tr}\,\mathrm{Var}_z(z)}{b},$$2 (Glowacki et al., 8 May 2026). Thus batching remains effective, but the relevant normalization is no longer the finite-variance $$\mathbb{E}_B\,\|g_b(x)-\nabla \ell(x)\|^2 \le \frac{4L_z^2\,\mathrm{Tr}\,\mathrm{Var}_z(z)}{b},$$3 law.

Two misconceptions recur across the literature. One is that batch size affects only variance magnitude; covariance structure and optimizer geometry can matter as much as, or more than, scalar variance (Wen et al., 2019, Naganuma et al., 3 Feb 2026). The other is that the batch size favored by optimization theory automatically matches the one favored by generalization or system throughput. Some results explicitly optimize token efficiency or training time, while others show that generalization may prefer different schedules, even when optimization error is minimized at a particular critical scale (Islamov et al., 22 Mar 2026, Perrone et al., 2019). The concept of a batch-size-aware gradient-noise bound is therefore best viewed not as a single formula, but as a family of problem-dependent relations that connect stochastic error, geometry, and resource allocation.