---
title: Empirically Localized Rademacher Complexity
url: https://www.emergentmind.com/topics/empirically-localized-rademacher-complexity
type: topic
---

# Empirically Localized Rademacher Complexity

Searching arXiv for papers on local/empirical localized Rademacher complexity and related variants.
{}
Empirically localized Rademacher complexity is a data-dependent refinement of classical Rademacher complexity in which the supremum is restricted to functions near a reference object defined from the sample. The reference may be a target function $f^*$ through an empirical $L_2$ constraint, an empirical minimizer $\hat w$ through a parameter-space ball, or an implicit quadratic localization induced by an offset process. The central motivation is that the global empirical complexity of a rich class can be too large to yield informative bounds—Yang et al. note that for deep nets it may even equal $1$—whereas localization can produce tighter control of excess risk or generalization and, in favorable cases, rates up to $O(1/n)$ [1902.00873, 1606.00922, 2202.11461].

## 1. Definitions and basic forms

The global empirical Rademacher complexity of a class $\mathcal F$ on a sample $\{x_i\}_{i=1}^n$ is
\[
\widehat{\mathcal R}_n(\mathcal F)
=
\mathbb E_{\sigma_{1:n}}
\Biggl[
\sup_{f\in\mathcal F}
\frac{1}{n}\sum_{i=1}^n \sigma_i f(x_i)
\;\Bigm|\;
\{x_i\}_{i=1}^n
\Biggr],
\]
with i.i.d. Rademacher variables $\sigma_i\in\{-1,+1\}$. Localization replaces the full class by a sample-dependent subset [1510.01463].

Two standard empirical localizations appear in the literature summarized here. In the norm-based formulation, one fixes a reference $f^*$ and restricts to functions satisfying an empirical $L_2$ constraint:
\[
\hat{\mathfrak R}_n(\mathcal F,r)
=
\mathbb E_\sigma\Biggl[
\sup_{f\in\mathcal F:\,P_n(f-f^*)^2\le r}
\frac1n\sum_{i=1}^n \sigma_i f(x_i)
\Biggr],
\qquad
P_n(f-f^*)^2=\frac1n\sum_{i=1}^n (f(x_i)-f^*(x_i))^2.
\]
An equivalent empirical-norm perspective is written in terms of localized classes
\[
\mathcal F_{n,r}=\{f\in\mathcal F:P_n f^2\le r\},
\]
leading to
\[
{}_\sigma\mathcal R_n(\mathcal F,r)
=
\mathbb E_\sigma
\Bigl[
\sup_{f:\,P_n f^2\le r}
\frac1n\sum_{i=1}^n \sigma_i f(Z_i)
\;\Bigm|\;
Z_1,\dots,Z_n
\Bigr].
\]

Yang et al. give a distinct parameter-centric empirical localization for deep networks. If $h_w$ is a network with parameters $w$, and $\hat w$ minimizes the empirical hinge or cross-entropy loss, then
\[
B(\hat w,r)=\{w:\|w-\hat w\|\le r\},
\]
and the empirical Local Rademacher Complexity centered at $\hat w$ is
\[
\widehat{\mathcal R}_n(\hat w,r)
=
\mathbb E_{\sigma_{1:n}}
\Biggl[
\sup_{w\in B(\hat w,r)}
\frac1n\sum_{i=1}^n
\sigma_i\,\ell(h_w(x_i),y_i)
\;\Bigm|\;
\{(x_i,y_i)\}
\Biggr].
\]
For the hinge-margin loss, the paper writes
\[
\widehat R^{(m)}(\hat w,r)
=
\mathbb E_\sigma
\Biggl[
\sup_{w\in B(\hat w,r)}
\frac1n\sum_{i=1}^n
\sigma_i\,
\Phi\!\Bigl(\frac{m_{h_w,y_i}(x_i)}{\gamma}\Bigr)
\Biggr].
\]
These formulations share the same principle: the complexity is evaluated not on the entire hypothesis class, but on a neighborhood determined by the empirical learning problem [1902.00873].

| Formulation | Localized set | Reference object |
|---|---|---|
| Empirical norm localization | $\{f:P_n(f-f^*)^2\le r\}$ or $\{f:P_n f^2\le r\}$ | $f^*$ or the origin |
| Population norm localization | $\{f:Pf^2\le r\}$ | Distribution $P$ |
| Deep-network parameter localization | $B(\hat w,r)=\{w:\|w-\hat w\|\le r\}$ | Empirical minimizer $\hat w$ |
| Offset localization | Supremum with negative quadratic terms | No explicit radius |

## 2. Fixed points, rates, and the logic of localization

The usual local-Rademacher program is organized around a fixed point. A representative formulation defines
\[
r^*=
\inf\Bigl\{r>0:\;R_n(\mathcal F,\sqrt r)\le r\Bigr\},
\]
after which one derives bounds of the form
\[
R(\hat f)-R(f^*)\lesssim r^*+\frac{\log(1/\delta)}{n}.
\]
This formalizes the idea that the statistically relevant scale is the smallest radius at which stochastic fluctuations no longer dominate the radius itself [1606.00922].

Zhivotovskiy and Hanneke argue that localization need not be expressed only through local Rademacher fixed points. For binary classification under Massart’s bounded-noise condition, they introduce a fixed point of the local empirical entropy,
\[
\gamma^{\loc}_{h,h}(n,\mathcal F)
=
\max\Bigl\{\gamma\in\mathbb N:\;
h\,\gamma\le \log \mathcal M^{\loc}_1(\mathcal F,\gamma,n,h)
\Bigr\},
\]
where $\mathcal M^{\loc}_1$ is a worst-case local packing number on the sample. Their upper bound states that for a VC class of dimension $d$, if $h\in(\sqrt{d/n},1]$, then for any ERM $\hat f$ and any $P\in\mathcal P(h,\mathcal F)$,
\[
\mathbb E[R(\hat f)-R(f^*)]\lesssim \frac{\gamma^{\loc}_{h,h}(n,\mathcal F)}{n},
\]
and with probability at least $1-\delta$,
\[
R(\hat f)-R(f^*)\lesssim
\frac{\gamma^{\loc}_{h,h}(n,\mathcal F)}{n}
+
\frac{\log(1/\delta)}{n h}.
\]
They also provide a matching minimax lower bound under a mild pseudoconvexity condition, and state that the new entropy-based fixed point is never larger, up to constants, than the classical local-Rademacher fixed point. In that sense, the paper positions local Rademacher complexity as one localization device among several, rather than the unique canonical one [1606.00922].

A recurring significance statement across these developments is that local quantities are meant to exploit the fact that learning procedures select functions in low-variance or near-optimal regions of the class. This is the formal reason that local quantities can yield fast rates while global complexity bounds remain coarse.

## 3. From empirical localization to generalization bounds

Lei, Ding, and Bi study how to bound true local Rademacher complexity
\[
\mathcal R_n(\mathcal F,r)
=
\mathbb E_{Z,\sigma}
\Bigl[
\sup_{f\in\mathcal F:\,P f^2\le r}
\frac1n\sum_{i=1}^n \sigma_i f(Z_i)
\Bigr]
\]
through empirical localization and covering numbers. Assuming the class is uniformly bounded, $\|f\|_\infty\le b$, they prove
\[
\mathcal R_n(\mathcal F,r)
\le
\inf_{\varepsilon>0}
\Biggl[
2\,\mathcal R_n\bigl(\{f:P_n f^2\le \varepsilon^2\}\bigr)
+
\frac{8b}{n}\log N\!\Bigl(\frac{\varepsilon}{2},\mathcal F,\|\cdot\|_2\Bigr)
+
\sqrt{\frac{2r}{n}\log N\!\Bigl(\frac{\varepsilon}{2},\mathcal F,\|\cdot\|_2\Bigr)}
\Biggr].
\]
Here
\[
N(\delta,\mathcal F,\|\cdot\|_2)
=
\sup_{P_n}
N\!\bigl(\delta,\mathcal F,\|\cdot\|_{L_2(P_n)}\bigr)
\]
is the worst-case empirical covering number. The theorem explicitly reduces the problem of bounding a true localized complexity to two ingredients: an empirical local complexity on an $\varepsilon^2$-ball and a covering-number control [1510.01463].

The paper then derives corollaries under standard entropy conditions. Under poly-log covering growth,
\[
\log N(\delta,\mathcal F,\|\cdot\|_2)\le d[\log(\gamma/\delta)]^p,
\]
one obtains bounds that, for $r\approx n^{-1}$, become $O((\log n)^p/n)$. Under polynomial entropy with a $\log^2$ factor,
\[
\log N(\delta,\mathcal F,\|\cdot\|_2)\le \gamma\,\delta^{-p}\log^2(2/\delta),
\]
the resulting local-complexity bounds depend on the regime $0<p<2$, $p=2$, or $p>2$. The significance of these results is methodological: once a covering-number estimate is available, the paper gives a systematic route to a local-Rademacher bound and then to fast generalization bounds. The summary also states that the resulting complexities are always sub-root functions in the radius $r$, which is why they fit directly into the Bartlett–Mendelson localized concentration framework [1510.01463].

## 4. Offset localization and the extension beyond Bernstein conditions

A major development after the classical local-Rademacher framework is offset localization. Kanade, Rebeschini, and Vaškevičius define the offset Rademacher complexity of a class $\mathcal H$ as
\[
\mathfrak R_n^{\mathrm{off}}(P_X,\mathcal H,\gamma)
=
\mathbb E_{S_n^X,\sigma}
\Biggl[
\sup_{h\in\mathcal H}
\Bigl\{
\frac1n\sum_{i=1}^n \sigma_i h(X_i)
-
\gamma \frac1n\sum_{i=1}^n h(X_i)^2
-
\gamma \mathbb E_{X\sim P_X}[h(X)^2]
\Bigr\}
\Biggr].
\]
The negative quadratic terms localize the supremum automatically to functions of small $L_2$ norm, so the theory does not require an external Bernstein condition [2202.11461].

The corresponding estimator-dependent geometric assumption is the offset condition. If $\hat f$ is an estimator and $g^*\in\arg\min_{g\in\mathcal G}R(g)$, then $\hat f$ satisfies an $(\mathcal G,\ell,\varepsilon,\gamma)$ offset condition if, for every $\delta\in(0,1)$, with probability at least $1-\delta$,
\[
R_n(\hat f)-R_n(g^*)
\le
-\gamma\sum_{i=1}^n (\hat f(X_i)-g^*(X_i))^2
+
\varepsilon(\delta).
\]
This is contrasted directly with the classical Bernstein condition
\[
\forall f\in\mathcal F,\quad
\mathbb E[f(X)-g^*(X)]^2
\le
\frac1\gamma\,
\mathbb E\!\bigl[\ell(f(X),Y)-\ell(g^*(X),Y)\bigr].
\]
The paper emphasizes that the Bernstein condition is distribution-dependent and estimator-independent, whereas the offset condition is estimator-dependent and covers improper and non-convex settings.

Under bounded range $\mathcal F\subset[-b,b]^\mathcal X$, a $C_b$-Lipschitz loss, and the offset condition, the main theorem gives an exponential-tail excess-risk bound in terms of offset complexity. The paper also states that if $\mathcal F$ is convex and $\ell$ is $\gamma$-strongly convex in its first argument, then ERM over $\mathcal F$ satisfies the deterministic offset condition with the same $\gamma$. For star-shaped classes, the summary states
\[
\mathfrak R_n^{\mathrm{off}}(\mathcal H,\gamma)\le \mathfrak R_n^{\mathrm{loc}}(\mathcal H,\gamma),
\]
so offset-based bounds subsume the classical ones. This suggests a broader notion of empirical localization in which the localization is not specified by a radius in advance but is built into the empirical process itself [2202.11461].

## 5. Deep-network regularization via empirical local complexity

Yang et al. translate empirical Local Rademacher Complexity into an explicit deep-learning regularizer. Their starting point is the observation that for any loss class $\mathcal F$ bounded in $[0,1]$, with probability at least $1-\delta$, every $f\in\mathcal F$ satisfies
\[
\mathbb E[\ell(f(X),Y)]
\le
\frac1n\sum_{i=1}^n \ell(f(x_i),y_i)
+
2\,\widehat{\mathcal R}_n(\mathcal F)
+
3\sqrt{\frac{\ln(2/\delta)}{2n}},
\]
but the global empirical complexity can be large for deep nets. Their proposal is to replace the global term by a local one centered at the empirical minimizer $\hat w$ [1902.00873].

Assuming $h_w$ is $L$-Lipschitz in $w$, the paper gives uniform upper bounds for the localized deep-net complexity. For hinge-margin loss,
\[
\widehat R^{(m)}(\hat w,r)
\le
\frac1\gamma
\Bigl|
\mathbb E_\sigma\Bigl[\frac1n\sum_{i=1}^n \sigma_i\,m_{h_{\hat w},y_i}(x_i)\Bigr]
\Bigr|
+
\frac{3Lr}{\gamma}.
\]
For cross-entropy with $c$ classes,
\[
\widehat R^{(c)}(\hat w,r)
\le
\sqrt{2(c-1)}
\Bigl|
\mathbb E_{\sigma_{ij}}
\Bigl[
\frac1n\sum_{i=1}^n\sum_{j\ne y_i}
\sigma_{ij}\bigl(h_{\hat w,j}(x_i)-h_{\hat w,y_i}(x_i)\bigr)
\Bigr]
\Bigr|
+
2\sqrt{2(c-1)}(c-1)Lr.
\]
As $r\to 0$, the additive $O(r)$ terms vanish, leading to the practical approximations
\[
R_{\mathrm{hinge}}(w)
=
\Bigl|
\frac1n\sum_{i=1}^n \sigma_i\,m_{h_w,y_i}(x_i)
\Bigr|,
\]
and
\[
R_{\mathrm{xent}}(w)
=
\sqrt{2(c-1)}
\Bigl|
\frac1n\sum_{i=1}^n\sum_{j\ne y_i}
\sigma_{ij}\bigl(h_{w,j}(x_i)-h_{w,y_i}(x_i)\bigr)
\Bigr|.
\]
Training then minimizes
\[
\min_w\;
\frac1n\sum_{i=1}^n \ell(h_w(x_i),y_i)
+\lambda R(w),
\]
and for the hinge case the paper writes
\[
\Omega(w)
=
\lambda\,
\mathbb E_\sigma
\Bigl[
\Bigl|
\frac1n\sum_{i=1}^n \sigma_i\,m_{h_w,y_i}(x_i)
\Bigr|
\Bigr].
\]

The minibatch procedure is a Monte Carlo estimate of the Rademacher expectation. For each minibatch of size $B$, one samples Rademacher signs $K$ times, computes
\[
r_k=\frac1B\Bigl|\sum_i \sigma_i\,m_{h_w,y_i}(x_i)\Bigr|,
\]
averages them to obtain $R$, forms the loss
\[
L=
\frac1B\sum_{i=1}^B \Phi\!\Bigl(\frac{m_{h_w,y_i}(x_i)}{\gamma}\Bigr)
+\lambda R,
\]
or the analogous cross-entropy version, and takes one gradient step. Each extra pass over the Rademacher variables costs $O(B)$ operations, multiplied by $K$; the paper states that in practice $K$ is small, for example $K=1$ to $5$, so the overhead is minor relative to the forward/backward pass. The implementation uses the $r\to 0$ bound, so no explicit ball projection is performed.

On CIFAR-10, the paper reports experiments with ResNet-18 and seven DARTS-discovered architectures $M_1,\dots,M_7$. The regularizer weight is chosen from $\lambda\in\{0.1,0.5,1.0\}$ on a 5,000-sample hold-out set, with best $\lambda=0.5$. For ResNet-18 with cross-entropy, the baseline test error is approximately $5.66\%$ with test loss $0.161$, while the LRC variant yields a small but consistent reduction in loss and gap, with test loss approximately $0.123$. For DARTS models $M_1$–$M_6$, adding LRC improves or matches accuracy for nearly every model and is reported to be insensitive to $\lambda\in[0.1,0.7]$. For the ensemble $M_7$, the paper reports baseline $97.42\%$, $97.44\%$ with LRC, $97.89\%$ with mixup alone, and $98.01\%$ with mixup plus LRC, which it describes as state-of-the-art on CIFAR-10 [1902.00873].

## 6. Interpretation, limitations, and recurrent points of confusion

A common source of confusion is that “localized Rademacher complexity” is not a single object. The summaries above exhibit at least four distinct constructions: localization by $Pf^2\le r$, localization by $P_n f^2\le r$, localization around $f^*$ through $P_n(f-f^*)^2\le r$, and localization around a deep-network empirical minimizer through $B(\hat w,r)$. Offset localization adds a further variant in which no explicit radius appears, because the quadratic penalty induces the localization internally [1510.01463, 2202.11461].

Another recurrent misconception is that faster rates follow from localization alone. The cited works impose different structural assumptions. Classical localized theory is paired with variance–expectation or Bernstein-type conditions; the entropy-based VC analysis of Zhivotovskiy and Hanneke is developed under Massart’s bounded-noise condition; Yang et al.’s deep-network construction assumes local Lipschitz continuity and uses a vanishing-radius approximation; and the offset theory replaces Bernstein with the estimator-dependent offset condition. The rate improvement is therefore conditional on the relevant geometry, noise model, or curvature hypothesis, not merely on the act of restricting the class.

The deep-learning application also has explicit limitations. Yang et al. state that one does not yet have a fully non-vacuous LRC-based generalization bound with explicit constants for very deep nets, that the theory presumes local Lipschitz continuity and a vanishing ball radius while the radius is implicit in practice, and that choosing $\lambda$ still requires cross-validation. Their summary adds that extensions may include combining LRC with adversarial robustness bounds via spectral norms, adaptive radii, or integration with variational-Bayes style PAC-Bayes analysis [1902.00873].

A final interpretive point is comparative rather than controversial. Zhivotovskiy and Hanneke present local empirical entropy as an alternative localization approach that can yield tighter control than local Rademacher fixed points in VC classes, while Kanade, Rebeschini, and Vaškevičius show that offset localization extends high-probability fast-rate analysis to improper and non-convex estimators. This suggests that empirically localized Rademacher complexity is best viewed as one member of a broader family of localization methods: important, widely applicable, and especially natural for empirical-process analysis, but not exhaustive of the modern localization toolkit [1606.00922, 2202.11461].

Source: https://www.emergentmind.com/topics/empirically-localized-rademacher-complexity