Empirically Localized Rademacher Complexity

• Empirically localized Rademacher complexity is a data-dependent refinement that restricts the supremum to functions near a reference object defined from the sample.
• It leverages constraints such as empirical L2 balls, parameter-space neighborhoods, or offset penalties to tightly control excess risk and yield fast rates.
• Applications include deep network regularization and improved risk bounds under various noise and entropy conditions, enhancing both theory and practice.

Searching arXiv for papers on local/empirical localized Rademacher complexity and related variants. {} Empirically localized Rademacher complexity is a data-dependent refinement of classical Rademacher complexity in which the supremum is restricted to functions near a reference object defined from the sample. The reference may be a target function $f^*$ through an empirical $L_2$ constraint, an empirical minimizer $\hat w$ through a parameter-space ball, or an implicit quadratic localization induced by an offset process. The central motivation is that the global empirical complexity of a rich class can be too large to yield informative bounds—Yang et al. note that for deep nets it may even equal $1$—whereas localization can produce tighter control of excess risk or generalization and, in favorable cases, rates up to $O(1/n)$ (Yang et al., 2019, Zhivotovskiy et al., 2016, Kanade et al., 2022).

1. Definitions and basic forms

The global empirical Rademacher complexity of a class $\mathcal F$ on a sample $\{x_i\}_{i=1}^n$ is

$$\widehat{\mathcal R}_n(\mathcal F) = \mathbb E_{\sigma_{1:n}} \Biggl[ \sup_{f\in\mathcal F} \frac{1}{n}\sum_{i=1}^n \sigma_i f(x_i) \;\Bigm|\; \{x_i\}_{i=1}^n \Biggr],$$

with i.i.d. Rademacher variables $\sigma_i\in\{-1,+1\}$. Localization replaces the full class by a sample-dependent subset (Lei et al., 2015).

Two standard empirical localizations appear in the literature summarized here. In the norm-based formulation, one fixes a reference $f^*$ and restricts to functions satisfying an empirical $L_2$0 constraint: $L_2$1 An equivalent empirical-norm perspective is written in terms of localized classes

$L_2$2

leading to

$L_2$3

Yang et al. give a distinct parameter-centric empirical localization for deep networks. If $L_2$4 is a network with parameters $L_2$5, and $L_2$6 minimizes the empirical hinge or cross-entropy loss, then

$L_2$7

and the empirical Local Rademacher Complexity centered at $L_2$8 is

$L_2$9

For the hinge-margin loss, the paper writes

$\hat w$0

These formulations share the same principle: the complexity is evaluated not on the entire hypothesis class, but on a neighborhood determined by the empirical learning problem (Yang et al., 2019).

Formulation	Localized set	Reference object
Empirical norm localization	$\hat w$1 or $\hat w$2	$\hat w$3 or the origin
Population norm localization	$\hat w$4	Distribution $\hat w$5
Deep-network parameter localization	$\hat w$6	Empirical minimizer $\hat w$7
Offset localization	Supremum with negative quadratic terms	No explicit radius

2. Fixed points, rates, and the logic of localization

The usual local-Rademacher program is organized around a fixed point. A representative formulation defines

$\hat w$8

after which one derives bounds of the form

$\hat w$9

This formalizes the idea that the statistically relevant scale is the smallest radius at which stochastic fluctuations no longer dominate the radius itself (Zhivotovskiy et al., 2016).

Zhivotovskiy and Hanneke argue that localization need not be expressed only through local Rademacher fixed points. For binary classification under Massart’s bounded-noise condition, they introduce a fixed point of the local empirical entropy,

$1$0

where $1$1 is a worst-case local packing number on the sample. Their upper bound states that for a VC class of dimension $1$2, if $1$3, then for any ERM $1$4 and any $1$5,

$1$6

and with probability at least $1$7,

$1$8

They also provide a matching minimax lower bound under a mild pseudoconvexity condition, and state that the new entropy-based fixed point is never larger, up to constants, than the classical local-Rademacher fixed point. In that sense, the paper positions local Rademacher complexity as one localization device among several, rather than the unique canonical one (Zhivotovskiy et al., 2016).

A recurring significance statement across these developments is that local quantities are meant to exploit the fact that learning procedures select functions in low-variance or near-optimal regions of the class. This is the formal reason that local quantities can yield fast rates while global complexity bounds remain coarse.

3. From empirical localization to generalization bounds

Lei, Ding, and Bi study how to bound true local Rademacher complexity

$1$9

through empirical localization and covering numbers. Assuming the class is uniformly bounded, $O(1/n)$0, they prove

$O(1/n)$1

Here

$O(1/n)$2

is the worst-case empirical covering number. The theorem explicitly reduces the problem of bounding a true localized complexity to two ingredients: an empirical local complexity on an $O(1/n)$3-ball and a covering-number control (Lei et al., 2015).

The paper then derives corollaries under standard entropy conditions. Under poly-log covering growth,

$O(1/n)$4

one obtains bounds that, for $O(1/n)$5, become $O(1/n)$6. Under polynomial entropy with a $O(1/n)$7 factor,

$O(1/n)$8

the resulting local-complexity bounds depend on the regime $O(1/n)$9, $\mathcal F$0, or $\mathcal F$1. The significance of these results is methodological: once a covering-number estimate is available, the paper gives a systematic route to a local-Rademacher bound and then to fast generalization bounds. The summary also states that the resulting complexities are always sub-root functions in the radius $\mathcal F$2, which is why they fit directly into the Bartlett–Mendelson localized concentration framework (Lei et al., 2015).

4. Offset localization and the extension beyond Bernstein conditions

A major development after the classical local-Rademacher framework is offset localization. Kanade, Rebeschini, and Vaškevičius define the offset Rademacher complexity of a class $\mathcal F$3 as

$\mathcal F$4

The negative quadratic terms localize the supremum automatically to functions of small $\mathcal F$5 norm, so the theory does not require an external Bernstein condition (Kanade et al., 2022).

The corresponding estimator-dependent geometric assumption is the offset condition. If $\mathcal F$6 is an estimator and $\mathcal F$7, then $\mathcal F$8 satisfies an $\mathcal F$9 offset condition if, for every $\{x_i\}_{i=1}^n$0, with probability at least $\{x_i\}_{i=1}^n$1,

$\{x_i\}_{i=1}^n$2

This is contrasted directly with the classical Bernstein condition

$\{x_i\}_{i=1}^n$3

The paper emphasizes that the Bernstein condition is distribution-dependent and estimator-independent, whereas the offset condition is estimator-dependent and covers improper and non-convex settings.

Under bounded range $\{x_i\}_{i=1}^n$4, a $\{x_i\}_{i=1}^n$5-Lipschitz loss, and the offset condition, the main theorem gives an exponential-tail excess-risk bound in terms of offset complexity. The paper also states that if $\{x_i\}_{i=1}^n$6 is convex and $\{x_i\}_{i=1}^n$7 is $\{x_i\}_{i=1}^n$8-strongly convex in its first argument, then ERM over $\{x_i\}_{i=1}^n$9 satisfies the deterministic offset condition with the same $$\widehat{\mathcal R}_n(\mathcal F) = \mathbb E_{\sigma_{1:n}} \Biggl[ \sup_{f\in\mathcal F} \frac{1}{n}\sum_{i=1}^n \sigma_i f(x_i) \;\Bigm|\; \{x_i\}_{i=1}^n \Biggr],$$0. For star-shaped classes, the summary states

$$\widehat{\mathcal R}_n(\mathcal F) = \mathbb E_{\sigma_{1:n}} \Biggl[ \sup_{f\in\mathcal F} \frac{1}{n}\sum_{i=1}^n \sigma_i f(x_i) \;\Bigm|\; \{x_i\}_{i=1}^n \Biggr],$$1

so offset-based bounds subsume the classical ones. This suggests a broader notion of empirical localization in which the localization is not specified by a radius in advance but is built into the empirical process itself (Kanade et al., 2022).

5. Deep-network regularization via empirical local complexity

Yang et al. translate empirical Local Rademacher Complexity into an explicit deep-learning regularizer. Their starting point is the observation that for any loss class $$\widehat{\mathcal R}_n(\mathcal F) = \mathbb E_{\sigma_{1:n}} \Biggl[ \sup_{f\in\mathcal F} \frac{1}{n}\sum_{i=1}^n \sigma_i f(x_i) \;\Bigm|\; \{x_i\}_{i=1}^n \Biggr],$$2 bounded in $$\widehat{\mathcal R}_n(\mathcal F) = \mathbb E_{\sigma_{1:n}} \Biggl[ \sup_{f\in\mathcal F} \frac{1}{n}\sum_{i=1}^n \sigma_i f(x_i) \;\Bigm|\; \{x_i\}_{i=1}^n \Biggr],$$3, with probability at least $$\widehat{\mathcal R}_n(\mathcal F) = \mathbb E_{\sigma_{1:n}} \Biggl[ \sup_{f\in\mathcal F} \frac{1}{n}\sum_{i=1}^n \sigma_i f(x_i) \;\Bigm|\; \{x_i\}_{i=1}^n \Biggr],$$4, every $$\widehat{\mathcal R}_n(\mathcal F) = \mathbb E_{\sigma_{1:n}} \Biggl[ \sup_{f\in\mathcal F} \frac{1}{n}\sum_{i=1}^n \sigma_i f(x_i) \;\Bigm|\; \{x_i\}_{i=1}^n \Biggr],$$5 satisfies

$$\widehat{\mathcal R}_n(\mathcal F) = \mathbb E_{\sigma_{1:n}} \Biggl[ \sup_{f\in\mathcal F} \frac{1}{n}\sum_{i=1}^n \sigma_i f(x_i) \;\Bigm|\; \{x_i\}_{i=1}^n \Biggr],$$6

but the global empirical complexity can be large for deep nets. Their proposal is to replace the global term by a local one centered at the empirical minimizer $$\widehat{\mathcal R}_n(\mathcal F) = \mathbb E_{\sigma_{1:n}} \Biggl[ \sup_{f\in\mathcal F} \frac{1}{n}\sum_{i=1}^n \sigma_i f(x_i) \;\Bigm|\; \{x_i\}_{i=1}^n \Biggr],$$7 (Yang et al., 2019).

Assuming $$\widehat{\mathcal R}_n(\mathcal F) = \mathbb E_{\sigma_{1:n}} \Biggl[ \sup_{f\in\mathcal F} \frac{1}{n}\sum_{i=1}^n \sigma_i f(x_i) \;\Bigm|\; \{x_i\}_{i=1}^n \Biggr],$$8 is $$\widehat{\mathcal R}_n(\mathcal F) = \mathbb E_{\sigma_{1:n}} \Biggl[ \sup_{f\in\mathcal F} \frac{1}{n}\sum_{i=1}^n \sigma_i f(x_i) \;\Bigm|\; \{x_i\}_{i=1}^n \Biggr],$$9-Lipschitz in $\sigma_i\in\{-1,+1\}$0, the paper gives uniform upper bounds for the localized deep-net complexity. For hinge-margin loss,

$\sigma_i\in\{-1,+1\}$1

For cross-entropy with $\sigma_i\in\{-1,+1\}$2 classes,

$\sigma_i\in\{-1,+1\}$3

As $\sigma_i\in\{-1,+1\}$4, the additive $\sigma_i\in\{-1,+1\}$5 terms vanish, leading to the practical approximations

$\sigma_i\in\{-1,+1\}$6

and

$\sigma_i\in\{-1,+1\}$7

Training then minimizes

$\sigma_i\in\{-1,+1\}$8

and for the hinge case the paper writes

$\sigma_i\in\{-1,+1\}$9

The minibatch procedure is a Monte Carlo estimate of the Rademacher expectation. For each minibatch of size $f^*$0, one samples Rademacher signs $f^*$1 times, computes

$f^*$2

averages them to obtain $f^*$3, forms the loss

$f^*$4

or the analogous cross-entropy version, and takes one gradient step. Each extra pass over the Rademacher variables costs $f^*$5 operations, multiplied by $f^*$6; the paper states that in practice $f^*$7 is small, for example $f^*$8 to $f^*$9, so the overhead is minor relative to the forward/backward pass. The implementation uses the $L_2$00 bound, so no explicit ball projection is performed.

On CIFAR-10, the paper reports experiments with ResNet-18 and seven DARTS-discovered architectures $L_2$01. The regularizer weight is chosen from $L_2$02 on a 5,000-sample hold-out set, with best $L_2$03. For ResNet-18 with cross-entropy, the baseline test error is approximately $L_2$04 with test loss $L_2$05, while the LRC variant yields a small but consistent reduction in loss and gap, with test loss approximately $L_2$06. For DARTS models $L_2$07–$L_2$08, adding LRC improves or matches accuracy for nearly every model and is reported to be insensitive to $L_2$09. For the ensemble $L_2$10, the paper reports baseline $L_2$11, $L_2$12 with LRC, $L_2$13 with mixup alone, and $L_2$14 with mixup plus LRC, which it describes as state-of-the-art on CIFAR-10 (Yang et al., 2019).

6. Interpretation, limitations, and recurrent points of confusion

A common source of confusion is that “localized Rademacher complexity” is not a single object. The summaries above exhibit at least four distinct constructions: localization by $L_2$15, localization by $L_2$16, localization around $L_2$17 through $L_2$18, and localization around a deep-network empirical minimizer through $L_2$19. Offset localization adds a further variant in which no explicit radius appears, because the quadratic penalty induces the localization internally (Lei et al., 2015, Kanade et al., 2022).

Another recurrent misconception is that faster rates follow from localization alone. The cited works impose different structural assumptions. Classical localized theory is paired with variance–expectation or Bernstein-type conditions; the entropy-based VC analysis of Zhivotovskiy and Hanneke is developed under Massart’s bounded-noise condition; Yang et al.’s deep-network construction assumes local Lipschitz continuity and uses a vanishing-radius approximation; and the offset theory replaces Bernstein with the estimator-dependent offset condition. The rate improvement is therefore conditional on the relevant geometry, noise model, or curvature hypothesis, not merely on the act of restricting the class.

The deep-learning application also has explicit limitations. Yang et al. state that one does not yet have a fully non-vacuous LRC-based generalization bound with explicit constants for very deep nets, that the theory presumes local Lipschitz continuity and a vanishing ball radius while the radius is implicit in practice, and that choosing $L_2$20 still requires cross-validation. Their summary adds that extensions may include combining LRC with adversarial robustness bounds via spectral norms, adaptive radii, or integration with variational-Bayes style PAC-Bayes analysis (Yang et al., 2019).

A final interpretive point is comparative rather than controversial. Zhivotovskiy and Hanneke present local empirical entropy as an alternative localization approach that can yield tighter control than local Rademacher fixed points in VC classes, while Kanade, Rebeschini, and Vaškevičius show that offset localization extends high-probability fast-rate analysis to improper and non-convex estimators. This suggests that empirically localized Rademacher complexity is best viewed as one member of a broader family of localization methods: important, widely applicable, and especially natural for empirical-process analysis, but not exhaustive of the modern localization toolkit (Zhivotovskiy et al., 2016, Kanade et al., 2022).