Bit-Wise Rademacher Complexity

• Bit-wise Rademacher complexity is a measure of how well multi-bit classifiers can fit random noise, linking each bit's VC dimension to overall generalization capacity.
• The framework utilizes the ℓ∞-vector contraction theorem to obtain sharp bounds that scale as O(L√K · maxₖ Radₙ(𝓕|ₖ)) with only logarithmic overhead.
• It provides actionable insights for designing multi-label predictors, ensuring per-bit error rates decay at O(√(d/n)) even as the number of output bits increases.

Bit-wise Rademacher complexity characterizes the ability of classes of multiple binary predictors to fit random noise, providing a quantitative measure of the generalization capacity of multi-bit classification systems. In the context of vector-valued function classes mapping into $\mathbb{R}^K$, bit-wise Rademacher complexity enables sharp generalization bounds for predictors that simultaneously output $K$ binary labels, clarifying how complexity scales with the number of output bits and properties such as VC dimension.

1. Formal Definition and Notation

Given a fixed sample $x_1, \ldots, x_n \in \mathcal{X}$, let $$\mathcal{F} \subseteq \{ f: \mathcal{X} \to \mathbb{R}^K \}$$ be a class of $K$-dimensional vector-valued predictors. For each $k \in \{1, \ldots, K\}$, denote the $k$-th coordinate class as $\mathcal{F}|_k$, the set of all functions formed by projecting each $f \in \mathcal{F}$ onto its $k$-th coordinate. The empirical Rademacher complexity of $K$0 on $K$1 is

$K$2

where $K$3 are i.i.d. Rademacher variables. The worst-case (global) Rademacher complexity is $K$4. For function composition, for a sequence $K$5 of real-valued functions each $K$6-Lipschitz with respect to $K$7 norm, the composed class can be analyzed via Rademacher complexity under specific contraction principles (Foster et al., 2019).

2. $K$8 Vector Contraction and its Application

The principal advance is the $K$9-vector-contraction theorem for Rademacher complexity. For $x_1, \ldots, x_n \in \mathcal{X}$0 and $x_1, \ldots, x_n \in \mathcal{X}$1 $x_1, \ldots, x_n \in \mathcal{X}$2-Lipschitz functions with $x_1, \ldots, x_n \in \mathcal{X}$3 and $x_1, \ldots, x_n \in \mathcal{X}$4 for all $x_1, \ldots, x_n \in \mathcal{X}$5, the following upper bound holds for any $x_1, \ldots, x_n \in \mathcal{X}$6, for an absolute constant $x_1, \ldots, x_n \in \mathcal{X}$7:

$x_1, \ldots, x_n \in \mathcal{X}$8

Neglecting polylogarithmic terms yields the clean statement:

$x_1, \ldots, x_n \in \mathcal{X}$9

This result highlights the scaling behavior with respect to $$\mathcal{F} \subseteq \{ f: \mathcal{X} \to \mathbb{R}^K \}$$0 and shows that the Rademacher complexity of the multi-bit class, after Lipschitz transformation, is controlled by the maximal per-coordinate complexity, up to a factor $$\mathcal{F} \subseteq \{ f: \mathcal{X} \to \mathbb{R}^K \}$$1 and logarithmic factors (Foster et al., 2019).

3. Specialization to Bit-Wise Classifiers

For bit-wise classifiers, each coordinate class $$\mathcal{F} \subseteq \{ f: \mathcal{X} \to \mathbb{R}^K \}$$2 consists of binary predictors. The standard VC/Rademacher theory gives

$$\mathcal{F} \subseteq \{ f: \mathcal{X} \to \mathbb{R}^K \}$$3

This leads to the bound

$$\mathcal{F} \subseteq \{ f: \mathcal{X} \to \mathbb{R}^K \}$$4

If $$\mathcal{F} \subseteq \{ f: \mathcal{X} \to \mathbb{R}^K \}$$5 for all $$\mathcal{F} \subseteq \{ f: \mathcal{X} \to \mathbb{R}^K \}$$6, ignoring log factors:

$$\mathcal{F} \subseteq \{ f: \mathcal{X} \to \mathbb{R}^K \}$$7

This yields generalization guarantees for the average 0–1 bit-wise loss. For example, with probability $$\mathcal{F} \subseteq \{ f: \mathcal{X} \to \mathbb{R}^K \}$$8:

$$\mathcal{F} \subseteq \{ f: \mathcal{X} \to \mathbb{R}^K \}$$9

Even though there are $K$0 bits, all generalize at rate $K$1, implying per-bit error rate decays as $K$2 (Foster et al., 2019).

4. Proof Outline and Vector Contraction Mechanism

The proof strategy proceeds through the following critical methodological elements:

• Dudley chaining bound: The classic Dudley entropy integral provides an upper bound in terms of empirical $K$3 covering numbers:

$K$4

• Exploiting $K$5-Lipschitz property: By the Lipschitz condition, $K$6 covering of the composed class contracts to an $K$7-covering of the vector class, leading to

$K$8

• Combinatorial covering and fat-shattering: The covering number is then bounded using the Rudelson–Vershynin inequality in terms of the fat-shattering dimension, and standard results relate fat-shattering to Rademacher complexity. Together, this machinery produces the $K$9 dependence, with only polylogarithmic slack in $k \in \{1, \ldots, K\}$0 (Foster et al., 2019).

5. Tightness, Constants, and Lower Bounds

The hidden constant $k \in \{1, \ldots, K\}$1 in the contraction bound depends only on the auxiliary parameter $k \in \{1, \ldots, K\}$2 introduced in the covering number argument. Importantly, $k \in \{1, \ldots, K\}$3 does not scale with $k \in \{1, \ldots, K\}$4 or $k \in \{1, \ldots, K\}$5. The polylogarithmic term $k \in \{1, \ldots, K\}$6 arises from the Dudley integral and the Rudelson–Vershynin bound. In applications, the complexity is commonly written as $k \in \{1, \ldots, K\}$7, suppressing polylogarithmic factors.

A matching lower bound (Proposition 1 in (Foster et al., 2019)) establishes that the $k \in \{1, \ldots, K\}$8 dependence is unavoidable if control over the worst-case Rademacher complexity is required. For instance, with $k \in \{1, \ldots, K\}$9 and a specifically constructed class $k$0, the lower bound

$k$1

holds. On the other hand, Maurer's $k$2-vector-contraction yields a $k$3 factor but applies to the empirical Rademacher complexity. There is thus an inherent trade-off between the sharpness of the $k$4-dependence and whether the bound concerns empirical or worst-case Rademacher complexity. The $k$5-vector-contraction provided by Foster & Rakhlin is the first near-optimal $k$6 contraction bound for $k$7-Lipschitz functions, exhibiting only polynomial logarithmic overhead in the sample size (Foster et al., 2019).

6. Implications and Broader Significance

The bit-wise Rademacher complexity framework establishes principled generalization guarantees for multi-label binary classification and related vector prediction problems. A plausible implication is that even as the number of output bits increases, global generalization rates are optimally controlled—up to $k$8—by the hardest bit-wise subproblem. This enables the design and analysis of high-dimensional predictors (e.g., in multi-label learning, error-correcting output codes, or structured prediction) with explicit control over both sample complexity and per-bit performance. Furthermore, the results clarify the boundary between worst-case generalization ($k$9 dependence) and empirical generalization ($\mathcal{F}|_k$0 dependence), guiding model selection and analysis strategies where output dimensionality is large.