Limits To (Machine) Learning (2512.12735v1)

Abstract: Machine learning (ML) methods are highly flexible, but their ability to approximate the true data-generating process is fundamentally constrained by finite samples. We characterize a universal lower bound, the Limits-to-Learning Gap (LLG), quantifying the unavoidable discrepancy between a model's empirical fit and the population benchmark. Recovering the true population $R^2$, therefore, requires correcting observed predictive performance by this bound. Using a broad set of variables, including excess returns, yields, credit spreads, and valuation ratios, we find that the implied LLGs are large. This indicates that standard ML approaches can substantially understate true predictability in financial data. We also derive LLG-based refinements to the classic Hansen and Jagannathan (1991) bounds, analyze implications for parameter learning in general-equilibrium settings, and show that the LLG provides a natural mechanism for generating excess volatility.

• The paper introduces the Limits-to-Learning Gap (LLG) as a universal lower bound capturing the irreducible estimation error in high-dimensional ML tasks.
• It provides closed-form LLG corrections to adjust out-of-sample R² estimates, revealing substantial signal in data previously seen as weakly predictable.
• Empirical and simulation results confirm that LLG distinguishes statistical noise from true signal, informing optimal model design and resource allocation.

Universal Lower Bounds and Statistical Limits in Machine Learning: A Review of "Limits To (Machine) Learning" (2512.12735)

Theoretical Foundation: Limits-to-Learning Gap (LLG)

"Limits To (Machine) Learning" develops a comprehensive theoretical analysis of the fundamental statistical limitations facing ML models when applied to finite, high-dimensional datasets. The paper introduces the Limits-to-Learning Gap (LLG), a universal lower bound on the discrepancy between the out-of-sample generalization performance of an estimator and the infeasible population performance. Crucially, LLG is determined solely by observable sample features and does not require specification or estimation of the data-generating function.

Formally, for prediction tasks $y_{t+1} = f_t + \varepsilon_{t+1}$, any linear (and many nonlinear) estimator's out-of-sample error is bounded below by the irreducible noise plus an estimation error. The latter term, characterized by the LLG, grows with the statistical complexity (where the number of features $P$ is not negligible compared to sample size $T$), and does not vanish even asymptotically for over-parameterized regimes. This result holds for high-dimensional linear estimators (ridge, kernel ridge regression) and, via NTK theory, for wide deep neural networks.

The closed-form LLG correction allows explicit construction of corrected lower confidence bounds for population $R^2$, revealing situations where observed low $R^2_{\text{OOS}}$ metrics may vastly understate true data predictability due to high estimation error. The paper further develops asymptotic normality and pivotal finite-sample lower bounds for this quantity, bringing rigorous statistical inference to high-dimensional predictive setups.

Empirical Analysis: Financial Time Series and Statistical Illusion of Weak Predictability

Empirical application uses the Welch & Goyal (2008) macro-financial dataset, encompassing returns, yields, spreads, and valuation ratios. The key empirical finding is that LLG corrections are quantitatively large in almost all settings studied—even where state-of-the-art ML estimators indicate low or negative $R^2_{\text{OOS}}$. For instance, the LLG for S&P 500 monthly return prediction implies a lower bound for population $R^2$ around 20%, contrasting with empirical $R^2_{\text{OOS}}$ of 1–2%. Several processed asset pricing variables exhibit LLG-corrected predictabilities in the 10–70% range, even though standard models fail to detect structure. This provides strong support for the claim that "statistical illusions" regarding weak predictability arise from high estimation error intrinsic to complex, finite-sample regimes.

By evaluating nonlinear, feature-selected models (e.g., recursive ridge and advanced regularized regressors), the paper substantiates that in cases with high LLG, out-of-sample predictability is indeed recoverable by more expressive or better-architected algorithms, and that the LLG predicts the attainable $R^2$ frontier. In contrast, negative or near-zero LLG tightly identifies hopeless predictive settings, distinguishing noise from signal absence.

Implications for Asset Pricing, Equilibrium, and Econometric Theory

The analysis yields concrete theoretical advancements for econometric theory and macro-finance. The paper derives LLG-adjusted versions of the Hansen–Jagannathan (1991) bounds, thereby quantifying how parameter uncertainty relaxes asset pricing restrictions. LLG enters into the calculation of the lower bound on admissible stochastic discount factor (SDF) volatility, resolving some of the "excess volatility" and SDF dispersion puzzles commonly attributed to behavioral explanations.

The framework formalizes how rational equilibrium agents in high-dimensional economies also face intrinsic LLG-induced estimation error, leading to excess price volatility (in the spirit of Shiller) without recourse to irrationality or behavioral preferences. This overturns the equivalence between econometrician and agent information sets often assumed in asset pricing and macro models.

Moreover, the connection to ML scaling laws is theoretically elucidated: the empirically observed sharply diminishing returns to data or compute in large-scale ML is attributed to the slow (subclassical) convergence of the LLG with increasing sample size. This clarifies why additional data or complexity may not substantially reduce the generalization gap in overparameterized or high-dimensional regimes.

Methodological Contributions

The principal methodological contributions are:

• Derivation of universal lower bounds (LLG) and practical computation for both linear and nonlinear (NTK-based) estimators
• Provision of confidence intervals and pivotal finite-sample inference for the infeasible $R^2$
• Application to robust, model-free empirical tests for the existence of signal versus limits of learning
• Explicit demonstration that LLG depends only on the design and distribution of features, not on the properties or complexity of the unknown predictive function

Strength of Empirical and Numerical Results

The simulation results, using both synthetic and real data with controlled signal-to-noise manipulation, validate the theoretical predictions. The lower bounds generated by LLG corrections closely track the true infeasible $R^2$, and confidence band calibration is empirically supported. Notably, for financial tasks frequently labeled "unpredictable" by standard methods, the LLG correction establishes that substantial signal is in fact present but unlearnable by extant ML paradigms, challenging canonical interpretations in econometrics and empirical finance.

Theoretical and Practical Implications

For Empirical Researchers

Evaluation of predictive studies in economics, finance, or other high-dimensional sciences should incorporate the LLG to distinguish absence of predictability from limits in statistical inference. Model architecture, variable selection, and sample design should be guided by the statistical complexity and LLG magnitude, rather than raw $R^2$ alone.

For Theory and Policy

Policymakers and theorists drawing conclusions from ML-based forecast studies (e.g., of economic volatility, SDF estimates, risk premia) should account for statistical limits imposed by data and feature dimensionality. The LLG provides a formal, model-agnostic approach to calibrating learning limits that is applicable across disciplines.

Future Directions

The demonstrated sharpness and generality of the LLG suggest several avenues for further research:

• Extension to fully nonparametric function classes and robust estimation settings
• Dynamic statistical complexity analysis across time-varying macro-financial environments
• LLG-guided design of optimal experiment/resource allocation strategies in high-dimensional scientific inference
• Integration into uncertainty quantification for DSGE and other structural econometric models

Conclusion

This work establishes the LLG as a foundational constraint on feasible machine learning, reconciling the apparent failure of ML models in high-dimensional prediction tasks with the statistical limits of finite-sample inference. The resulting framework corrects empirical practice, enriches econometric theory, and provides actionable diagnostics for research design and policy analysis in all settings where model complexity rivals sample size. Incorporating the LLG should become standard in rigorous predictive modeling, particularly in economics and finance, as data dimensionality and model expressiveness further escalate.