Information-Theoretic Learning Curve

• The information-theoretic learning curve is a framework that defines the relationship between data constraints and excess risk using mutual information and rate-distortion measures.
• It quantifies the penalty in prediction error due to finite-rate communication, showing how increased coding rates reduce the degradation in learning performance.
• Applications include distributed learning, nonparametric regression under noise, and modular design in systems where data compression plays a critical role.

An information-theoretic learning curve describes the fundamental relationship between data constraints (such as sample size or compression limitations) and the achievable prediction error, formulated using information-theoretic quantities such as rate-distortion functions, mutual information, or entropy. In the setting where learning occurs from compressed or rate-limited observations, this curve quantifies how information bottlenecks augment or degrade the rate at which excess risk (the gap between the performance of a learned predictor and the best achievable by the class) decays with increasing training samples or communication rate. The information-theoretic perspective introduces additive or multiplicative terms—explicitly characterized by rate-distortion or related functionals—that precisely quantify the penalty imposed by such constraints. This approach provides a unified and rigorous framework to assess learning performance in idealized settings and under physically motivated limitations such as communication rates, especially relevant in distributed, remote, or privacy-sensitive applications.

1. Information-Theoretic Characterization of Excess Risk

The canonical information-theoretic learning curve is established by considering the statistical learning problem in which labels $Y$ must be encoded and transmitted at some finite rate $R$, while inputs $X$ are observed perfectly. Under regularity conditions, for a hypothesis class $\mathcal{F}$ and a loss function $\ell$, the asymptotic generalization error is upper bounded as

$$\limsup_n \mathbb{E}[L(\hat{f}_n,P)] \leq L^*(\cdot,P) + 2\eta(\bar{D}_{Y|X}(R,\cdot)),$$

where $L^*(\cdot,P)$ is the minimal achievable risk under full-information training data, $\eta(\cdot)$ is a penalty function arising from a generalized Lipschitz condition on the loss, and the key term

$$\bar{D}_{Y|X}(R, \cdot) = \sup_{P \in \mathcal{P}} D_{Y|X}(R,P)$$

is the worst-case conditional distortion-rate function describing the minimum expected distortion of $Y$ given $X$ and a code of rate $R$. The inverse relationship of error penalty to rate provides a precise information-theoretic correction to the classical sample complexity curve.

The conditional distortion-rate function $D_{Y|X}(R,P)$ is the inverse of the pointwise conditional rate-distortion function: $$R_{Y|X}(D, P) = \inf \left\{ I(Y; \hat{Y} | X) : \mathbb{E}[\ell(Y, \hat{Y})] \leq D \right\},$$ which quantifies the minimal amount of information needed to represent $Y$ within an average distortion $D$, when $X$ is known.

2. Conditional Rate-Distortion and Loss Penalty Tradeoffs

The expressiveness of the learning curve under information-theoretic constraints is mediated by the conditional distortion-rate function, which maps the finite code rate $R$ into an irreducible penalty in the empirical risk minimization framework. For a fixed rate $R$ and probability law $P$ governing $(X,Y)$,

$$D_{Y|X}(R, P) = \inf \left\{ D : R_{Y|X}(D,P) \leq R \right\}$$

gives the minimal expected loss achievable by any lossy representation of $Y$ (given $X$) at rate $R$. The term $2\eta(\bar{D}_{Y|X}(R, \cdot))$ in the excess risk bound is then a direct translation of this distortion into a bound on the learning performance, modulated by the smoothness of the loss function.

Consequently, the achievable learning curve under rate constraints is never better than the ‘ideal’ learning curve (infinite rate or uncompressed data), but suffers an additive penalty that decays as $R$ increases—manifesting a fundamental separation between statistical estimation error and information-theoretic (compression) error.

3. Nonparametric Regression under Rate Constraints

The nonparametric regression example with additive Gaussian noise provides an explicit instantiation. Given data

$$Y_k = f_0(X_k) + Z_k, \quad Z_k \sim \mathcal{N}(0, \sigma^2),$$

and squared error loss, the conditional distortion-rate function for compressing $Y$ (given $X$) at rate $R$ reduces to

$D_{Y|X}(R,P) = \sigma^2 2^{-2R}.$

Applying the excess risk bound, the learning curve for the (root mean square) prediction loss is then

$$\limsup_n \mathbb{E}[L(\hat{f}_n,P_f)^{1/2}] \leq \sigma + 2\sigma 2^{-R},$$

demonstrating that the deleterious effect of label compression on learning performance decays exponentially in the available coding rate per sample. Even in nonparametric regimes, this calculation shows that sample-compression tradeoffs are sharply characterized by rate-distortion theory, with closed-form degradation rates.

4. Critical Regularity Assumptions and Uniformity

The generality of the information-theoretic learning curve hinges on several technical regularity assumptions:

• ULLN for loss class: The loss class $\{\ell_f : f \in \mathcal{F}\}$ must satisfy a uniform law of large numbers, ensuring consistency of empirical risk minimization in the uncompressed case.
• Loss smoothness: The loss function $\ell$ must admit a generalized Lipschitz modulus $\eta$: for any $f$, $u$, $u'$, $$|\ell(f(x),u) - \ell(f(x),u')| \leq \eta(\ell(u,u'))$$.
• Distributional conditions: The family $\mathcal{P}$ of permissible data distributions must satisfy bounded mutual information $I(X;Y)$ and Dobrushin’s entropy condition, which limits the growth of the covering number with increasing precision.
• Moment or boundedness of $\ell$: Either the loss is bounded, or suitable moment conditions are imposed.

These conditions collectively guarantee that empirical averages converge (so that learning is meaningful), the error in reconstructing $Y$ due to compression is controlled in its effect on the loss, and that robust rate-distortion codes can be constructed uniformly across all allowed distributions.

5. Consequences and Extensions for Learning Curve Theory

• The rate of decay of excess risk is fundamentally altered by finite-rate communication constraints on the outputs, yielding an additive penalty determined by the rate-distortion tradeoff.
• The structural separation in the learning curve—$L^*(\cdot,P)$ (statistical limit) plus $2\eta(\bar{D}_{Y|X}(R,\cdot))$ (compression penalty)—clarifies how increased rate $R$ makes the learning curve approach its ideal (uncompressed) limit.
• In high-rate regimes, the penalty term decays rapidly (e.g., exponentially in the Gaussian example), making source-encoding and learning design modular for many practical distributed or resource-limited learning scenarios.
• The framework is extensible: while the analysis in the paper assumes $X$ is observed perfectly, the same methodology could be adapted for cases of dual compression (compressing both $X$ and $Y$) or more general nonparametric models.

6. Key Mathematical Formulas

Expression	Meaning
$$R_{Y|X}(D,P) = \inf \{ I(Y;\hat{Y}|X): \mathbb{E}[\ell(Y,\hat{Y})] \leq D \}$$	Conditional rate-distortion function
$D_{Y|X}(R,P)$	Inverse: minimum distortion at rate $R$
$$\limsup_n \mathbb{E}[L(\hat{f}_n,P)] \leq L^*(\cdot,P) + 2\eta(\bar{D}_{Y|X}(R,\cdot))$$	Generalization upper bound
For Gaussian regression: $\bar{D}_{Y|X}(R,\cdot) = \sigma^2 2^{-2R}$	Rate-penalty in nonparametric regression
$$\limsup_n \mathbb{E}[L(\hat{f}_n, P_f)^{1/2}] \leq \sigma + 2\sigma 2^{-R}$$	Regression learning curve with compressed labels

7. Implications and Applications

This characterization informs both theory and practical design:

• Distributed or remote learning: The framework enables rigorous performance analysis for learning-from-data settings where training labels or signals must be transmitted at finite rates (e.g., wireless sensor networks, privacy-respecting distributed learning).
• Guideline for architectural modularity: Information-theoretic learning curves justify separating encoder (compression) design and learning design, provided that the resulting distortion-induced penalty is acceptable.
• Foundational for hybrid learning-communication systems: It provides a template for extending classical statistical learning theory with physical or engineering constraints, highlighting when and how rate limitations fundamentally change achievable performance curves.

This approach provides the explicit bridge between information theory and learning theory in quantifying performance under compression constraints, furnishing sharp—and sometimes closed-form—bounds for excess risk as a function of code rate and sample size.