Scale-Invariant Self-Normalized Bounds

• Scale-invariant self-normalized bounds are estimates whose defining feature is invariance to natural rescaling via data-dependent normalization.
• They apply across probability, PDEs, online learning, and optimization by adapting classical normalization to match the problem’s intrinsic geometry.
• The topic highlights sharp conditions, optimality limits, and structural challenges, especially in higher-dimensional settings where scale invariance is delicate.

Scale-invariant self-normalized bounds are estimates whose controlling quantity is unchanged under the natural rescaling of the problem. Across probability, statistics, PDE, online learning, bandits, and neural-network generalization, the common mechanism is to normalize by an observed quadratic form, empirical norm, information divergence, or critical function-space norm, so that the resulting statistic or bound no longer depends on an external scale parameter. Canonical examples include $S_k/V_n(\beta)$ with $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$, $S_n/\sqrt{[S]_n}$, $N(n)I(X(n);\mu)$, $\|(\lambda I+V_t)^{-1/2}M_t\|$, and elliptic estimates whose constants depend only on scale-critical Lorentz norms such as $\|b-c\|_{L^{n,1}}$ (Fan, 2016, Fan et al., 2017, Garivier, 2013, Martinez-Taboada et al., 5 Nov 2025, Sakellaris, 2019).

1. Canonical forms of self-normalization

In the probabilistic literature, self-normalization usually means dividing a partial sum by a random norm built from the same sample. For independent, symmetric, nondegenerate variables $(\xi_i)_{i=1}^n$, one central construction is

$$V_n(\beta)=\Big(\sum_{i=1}^n |\xi_i|^\beta\Big)^{1/\beta},\qquad \max_{1\le k\le n}\frac{S_k}{V_n(\beta)},\qquad S_k=\sum_{i=1}^k \xi_i,$$

with $\beta>1$. If $\xi_i' = c\xi_i$, then $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$0 and $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$1, so the upper-tail law of the self-normalized statistic is unchanged under global rescaling (Fan, 2016). This is the basic finite-dimensional model of scale invariance.

A closely related Student-type normalization is

$$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$2

for i.i.d. $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$3 with $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$4 and $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$5. The same ratio appears in asymptotic expansions, local limit theorems, and entropic CLTs for self-normalized sums, and remains invariant under $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$6 because numerator and denominator scale by the same factor (Beckedorf et al., 2022). In martingale form, the analogous statistics are

$$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$7

where $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$8 and $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$9. These are invariant under $S_n/\sqrt{[S]_n}$0 for the same reason (Fan et al., 2017).

A distinct but related construction replaces quadratic normalization by information normalization. In a filtered setting with empirical mean $S_n/\sqrt{[S]_n}$1, the self-normalized quantity is

$S_n/\sqrt{[S]_n}$2

where $S_n/\sqrt{[S]_n}$3 is the rate function defined from the conditional log-mgf. With a random number of observations,

$S_n/\sqrt{[S]_n}$4

the relevant statistic becomes $S_n/\sqrt{[S]_n}$5 (Garivier, 2013). Here the normalization is internal to the deviation functional itself rather than an explicit denominator.

These constructions have a common formal pattern: the numerator is a cumulative fluctuation, while the denominator or rate functional is computed from the same path. This suggests that “self-normalized” is best understood as a structural principle rather than a single formula.

2. Non-asymptotic deviation bounds and Gaussian approximation

For independent symmetric variables, the basic maximal deviation inequality is explicit. If $S_n/\sqrt{[S]_n}$6, then

$S_n/\sqrt{[S]_n}$7

where

$S_n/\sqrt{[S]_n}$8

with

$S_n/\sqrt{[S]_n}$9

The range $N(n)I(X(n);\mu)$0 is optimal, and for $N(n)I(X(n);\mu)$1 this yields the simpler sub-Gaussian-type bound

$N(n)I(X(n);\mu)$2

which for $N(n)I(X(n);\mu)$3 becomes $N(n)I(X(n);\mu)$4 (Fan, 2016).

For martingale differences, the Berry–Esseen theory has the same self-normalized flavor. If

$N(n)I(X(n);\mu)$5

then

$N(n)I(X(n);\mu)$6

and the same rate holds for $N(n)I(X(n);\mu)$7. The paper also proves optimality: there exist martingale difference sequences for which the order $N(n)I(X(n);\mu)$8 cannot be improved (Fan et al., 2017).

At the level of refined asymptotics, self-normalized sums admit non-uniform Edgeworth expansions. If $N(n)I(X(n);\mu)$9 are symmetric, non-singular, and $\|(\lambda I+V_t)^{-1/2}M_t\|$0 for some real $\|(\lambda I+V_t)^{-1/2}M_t\|$1, with $\|(\lambda I+V_t)^{-1/2}M_t\|$2, then

$\|(\lambda I+V_t)^{-1/2}M_t\|$3

where $\|(\lambda I+V_t)^{-1/2}M_t\|$4 is the Edgeworth approximation for the cdf of $\|(\lambda I+V_t)^{-1/2}M_t\|$5. Parallel weighted bounds hold for the density and feed into entropic CLTs and total variation convergence (Beckedorf et al., 2022).

A high-dimensional analogue considers the coordinate-wise maximum of self-normalized statistics,

$\|(\lambda I+V_t)^{-1/2}M_t\|$6

With finite third absolute moment, the Gaussian approximation error satisfies

$\|(\lambda I+V_t)^{-1/2}M_t\|$7

so $\|(\lambda I+V_t)^{-1/2}M_t\|$8 as long as $\|(\lambda I+V_t)^{-1/2}M_t\|$9 (Chang et al., 15 Jan 2025). The dependence on standardized coordinates rather than raw $\|b-c\|_{L^{n,1}}$0 again encodes scale invariance.

3. Information-normalized and vector-valued concentration in sequential problems

Information-normalized bounds provide uniform-in-time confidence sequences without explicit variance normalization. In the martingale/exponential-moment setting,

$\|b-c\|_{L^{n,1}}$1

and for a random sample size $\|b-c\|_{L^{n,1}}$2,

$\|b-c\|_{L^{n,1}}$3

For bounded $\|b-c\|_{L^{n,1}}$4-valued variables the rate function becomes the Bernoulli KL divergence $\|b-c\|_{L^{n,1}}$5, giving confidence sets of the form

$\|b-c\|_{L^{n,1}}$6

which directly underlie KL-UCB-type indices in stochastic bandits (Garivier, 2013).

In Hilbert-space form, the self-normalized statistic is

$\|b-c\|_{L^{n,1}}$7

For light tails beyond sub-Gaussianity, a supermartingale construction yields time-uniform bounds of the form

$\|b-c\|_{L^{n,1}}$8

with $\|b-c\|_{L^{n,1}}$9 derived under Bernstein or Bennett assumptions and depending on $(\xi_i)_{i=1}^n$0 or the corresponding information gain $(\xi_i)_{i=1}^n$1 (Martinez-Taboada et al., 5 Nov 2025). Here the normalization is matrix-valued and geometric: the empirical Gram matrix determines both the scale and the shape of the confidence region.

This perspective becomes essential in generalized kernelized bandits. For bounded martingale noise $(\xi_i)_{i=1}^n$2 with predictable heteroscedastic variance $(\xi_i)_{i=1}^n$3, the paper introduces

$(\xi_i)_{i=1}^n$4

and proves a self-normalized Bernstein-like dimension-free inequality for

$(\xi_i)_{i=1}^n$5

This yields regret $(\xi_i)_{i=1}^n$6 for GKB-UCB, matching, up to multiplicative constants and logarithmic terms, the state-of-the-art bounds for both kernelized bandits and generalized linear bandits (Metelli et al., 3 Aug 2025). A plausible implication is that variance-weighted self-normalization is the right replacement for Hoeffding-type normalization whenever the reward model is heteroscedastic.

4. Elliptic PDEs and critical-space normalization

In elliptic theory, scale-invariant self-normalized bounds refer to estimates whose constants depend only on norms that remain unchanged under the natural PDE scaling. The model operator is

$(\xi_i)_{i=1}^n$7

with $(\xi_i)_{i=1}^n$8 measurable, bounded, and uniformly elliptic, $(\xi_i)_{i=1}^n$9 for some $$V_n(\beta)=\Big(\sum_{i=1}^n |\xi_i|^\beta\Big)^{1/\beta},\qquad \max_{1\le k\le n}\frac{S_k}{V_n(\beta)},\qquad S_k=\sum_{i=1}^k \xi_i,$$0, $$V_n(\beta)=\Big(\sum_{i=1}^n |\xi_i|^\beta\Big)^{1/\beta},\qquad \max_{1\le k\le n}\frac{S_k}{V_n(\beta)},\qquad S_k=\sum_{i=1}^k \xi_i,$$1, and either $$V_n(\beta)=\Big(\sum_{i=1}^n |\xi_i|^\beta\Big)^{1/\beta},\qquad \max_{1\le k\le n}\frac{S_k}{V_n(\beta)},\qquad S_k=\sum_{i=1}^k \xi_i,$$2 or $$V_n(\beta)=\Big(\sum_{i=1}^n |\xi_i|^\beta\Big)^{1/\beta},\qquad \max_{1\le k\le n}\frac{S_k}{V_n(\beta)},\qquad S_k=\sum_{i=1}^k \xi_i,$$3 in distributions. The decisive critical hypothesis is

$$V_n(\beta)=\Big(\sum_{i=1}^n |\xi_i|^\beta\Big)^{1/\beta},\qquad \max_{1\le k\le n}\frac{S_k}{V_n(\beta)},\qquad S_k=\sum_{i=1}^k \xi_i,$$4

because under the elliptic scaling $$V_n(\beta)=\Big(\sum_{i=1}^n |\xi_i|^\beta\Big)^{1/\beta},\qquad \max_{1\le k\le n}\frac{S_k}{V_n(\beta)},\qquad S_k=\sum_{i=1}^k \xi_i,$$5,

$$V_n(\beta)=\Big(\sum_{i=1}^n |\xi_i|^\beta\Big)^{1/\beta},\qquad \max_{1\le k\le n}\frac{S_k}{V_n(\beta)},\qquad S_k=\sum_{i=1}^k \xi_i,$$6

In this precise sense, $$V_n(\beta)=\Big(\sum_{i=1}^n |\xi_i|^\beta\Big)^{1/\beta},\qquad \max_{1\le k\le n}\frac{S_k}{V_n(\beta)},\qquad S_k=\sum_{i=1}^k \xi_i,$$7 is the scale-critical Lorentz space for first-order terms (Sakellaris, 2019).

Under these assumptions there exist nonnegative Green’s functions $$V_n(\beta)=\Big(\sum_{i=1}^n |\xi_i|^\beta\Big)^{1/\beta},\qquad \max_{1\le k\le n}\frac{S_k}{V_n(\beta)},\qquad S_k=\sum_{i=1}^k \xi_i,$$8 for $$V_n(\beta)=\Big(\sum_{i=1}^n |\xi_i|^\beta\Big)^{1/\beta},\qquad \max_{1\le k\le n}\frac{S_k}{V_n(\beta)},\qquad S_k=\sum_{i=1}^k \xi_i,$$9 and $\beta>1$0 for $\beta>1$1 such that, for all $\beta>1$2,

$\beta>1$3

and for almost every fixed $\beta>1$4,

$\beta>1$5

with analogous bounds for $\beta>1$6. The constants satisfy

$\beta>1$7

and are independent of the domain size, including $\beta>1$8 and half-spaces (Sakellaris, 2019).

The scaling is explicit. If

$\beta>1$9

then the bound $\xi_i' = c\xi_i$0 becomes

$\xi_i' = c\xi_i$1

with the same constant. The weak-$\xi_i' = c\xi_i$2 and weak-$\xi_i' = c\xi_i$3 norms are likewise invariant at the critical exponents. The paper states that, in this sense, the bounds are self-normalized: changing the spatial scale does not change the size of the constants (Sakellaris, 2019).

The same Green’s function bounds yield scale-invariant maximum principles for subsolutions of

$\xi_i' = c\xi_i$4

For finite $\xi_i' = c\xi_i$5, if $\xi_i' = c\xi_i$6, $\xi_i' = c\xi_i$7, $\xi_i' = c\xi_i$8, and $\xi_i' = c\xi_i$9 is a subsolution, then

$$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$00

A local version on $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$01 gives

$$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$02

with the same scaling invariance (Sakellaris, 2019).

5. Learning-theoretic and optimization viewpoints

In online learning, scale invariance is enforced algorithmically. The normalized online learning algorithms NG and NAG are designed to be independent of feature scales, with regret bounds dependent on the ratio of scales existent in the data rather than the absolute scale (Ross et al., 2013). The setup introduces a scaling adversary that chooses a positive definite matrix $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$03 and restricts inputs by $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$04. The comparator class is

$$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$05

For the minimax optimal diagonal conditioner in hindsight, the regret takes the scale-invariant form

$$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$06

which is unchanged under coordinatewise rescaling because $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$07 and $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$08 transform inversely (Ross et al., 2013). In the streaming case the only additional dependence is through scale ratios $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$09.

In PAC-Bayesian analyses of neural networks, the same idea appears as normalization by parameter geometry. “Normalized flat minima” introduces the objective

$$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$10

where $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$11 are Hessian-diagonal terms and the variances factor into row and column scales. The minimized value is invariant under row–column rescalings that preserve the ReLU network function, and serves as a scale-invariant sharpness term in the PAC-Bayes bound (Tsuzuku et al., 2019).

A related construction replaces raw parameters by “connectivity” coordinates $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$12 defined through

$$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$13

With an isotropic Gaussian prior on $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$14,

$$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$15

and a Gaussian posterior from Bayesian linear regression in the linearized model, the resulting PAC-Bayes-CTK bound depends on the eigenvalues of the Connectivity Tangent Kernel

$$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$16

Because (\mathbf J_\theta(x,\mathcal T(\psi))\operatorname{diag}(\mathcal T(\psi))

\mathbf J_\theta(x,\psi)\operatorname{diag}(\psi)) for function-preserving diagonal scalings $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$17, the prior, posterior, CTK, and KL term are invariant under those scalings (Kim et al., 2022).

Self-normalization also appears in conditional exponential families. In self-normalized log-linear models one constrains or penalizes the variance of the log-normalizer $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$18, and the likelihood gap satisfies

$$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$19

so the controlling quantity is the normalized tolerance $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$20, which is invariant under inverse rescaling of features and parameters that preserves $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$21 (Andreas et al., 2015).

At the optimization level, scale-invariant first-order methods based on linear minimization oracles explicitly normalize directions by the geometry of a norm ball. In matrix optimization with heavy-tailed noise and spectral norm geometry, the paper proves that any scale-invariant first-order method requires

$$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$22

oracle calls, and that a batched Scion method with spectral norm achieves the matching upper bound

$$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$23

With Hessian Lipschitzness, a transported Scion method improves this to

$$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$24

(Zhang et al., 18 May 2026). The controlling constant is a self-normalized martingale factor

$$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$25

which is dimension-free for Frobenius geometry but scales as $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$26 for the nuclear norm, making the price of scale invariance geometry dependent.

6. Sharpness, impossibility, and structural conditions

Several literatures show that scale-invariant self-normalization is a sharp property rather than an automatic consequence of dividing by a random quantity. In elliptic PDE, the condition

$$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$27

is both necessary and optimal for scale-invariant Green’s function bounds. The counterexample

$$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$28

lies in $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$29 for every $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$30 but not in $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$31, and for operators such as $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$32 the Green’s function fails to satisfy the pointwise and weak-type bounds uniformly in the pole even when $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$33 is arbitrarily small (Sakellaris, 2019). The paper’s conclusion is that the Lorentz refinement, not merely scale invariance of the Lebesgue exponent $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$34, is essential.

In probability, self-normalization does not eliminate substantive assumptions. The maximal inequality for $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$35 requires independent, symmetric, nondegenerate variables and $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$36 (Fan, 2016). The non-uniform Edgeworth theory for $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$37 requires symmetry, non-singularity, and moment assumptions such as $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$38 (Beckedorf et al., 2022). The Berry–Esseen bound for self-normalized martingales requires $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$39 for some $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$40 and involves the deviation of the predictable quadratic variation from $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$41 through

$$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$42

(Fan et al., 2017). In high dimension, the best current Berry–Esseen bound for the coordinate-wise maximum of self-normalized sums is slower than the classical $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$43 rate and requires $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$44 (Chang et al., 15 Jan 2025).

The strongest impossibility result concerns vector-valued self-normalized martingales. For

$$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$45

the paper proves that nontrivial scale-invariant upper bounds exist only in dimension $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$46 without further assumptions. In $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$47, there is an $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$48 scale-invariant bound for dyadic martingales and an explicit algorithm with $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$49 doubly-uniform regret in online linear regression. In contrast, for $$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$50, no nontrivial scale-invariant bound can hold in full generality, and sublinear doubly-uniform regret is impossible (Chen et al., 2 May 2026).

That same paper also identifies a recovery mechanism. Under a smoothness condition requiring bounded Radon–Nikodym derivatives of the conditional covariate laws with respect to a fixed base measure, VAW achieves

$$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$51

and the self-normalized martingale satisfies

$$V_n(\beta)=\big(\sum_{i=1}^n |\xi_i|^\beta\big)^{1/\beta}$$52

with high probability (Chen et al., 2 May 2026). This suggests that scale-invariant self-normalized control in higher dimensions requires structural randomness or smoothness in the covariates, not merely algebraic normalization.

Taken together, these results delineate a broad principle. Self-normalization removes nuisance scale only when the normalizer matches the intrinsic geometry of the problem: a critical Lorentz norm in elliptic PDE, an empirical quadratic form or information divergence in probability, a variance-weighted Gram matrix in bandits, or an invariant parameterization in learning and optimization. Where such geometry is absent or incomplete, sharp counterexamples show that scale invariance can fail despite formal normalization.