How Does Machine Learning Manage Complexity?

Abstract: We provide a computational complexity lens to understand the power of machine learning models, particularly their ability to model complex systems. Machine learning models are often trained on data drawn from sampleable or more complex distributions, a far wider range of distributions than just computable ones. By focusing on computable distributions, machine learning models can better manage complexity via probability. We abstract away from specific learning mechanisms, modeling machine learning as producing P/poly-computable distributions with polynomially-bounded max-entropy. We illustrate how learning computable distributions models complexity by showing that if a machine learning model produces a distribution $μ$ that minimizes error against the distribution generated by a cryptographic pseudorandom generator, then $μ$ must be close to uniform.

• The paper presents a formal model characterizing ML as minimizing KL divergence over P/poly-computable distributions.
• It shows neural networks achieve bounded max entropy through nonuniform advice, enabling probabilistic handling of complexity.
• The analysis highlights limitations in learnability, particularly when training on cryptographic pseudorandom data yielding near-uniform outputs.

Complexity Management in Machine Learning: A Computational Perspective

Abstract and Motivation

The paper "How Does Machine Learning Manage Complexity?" (2604.07233) develops a computational complexity-theoretic model to conceptualize how machine learning systems, particularly neural networks, represent and control complexity. Rather than analyzing specific architectures or algorithmic details, the paper abstracts machine learning as the process of producing $\mathrm{P}/\text{poly}$-computable distributions with polynomially-bounded max entropy. This formalization enables the rigorous study of complexity management in learned probabilistic models. The core assertion is that machine learning, by constraining hypotheses to computable distributions and representing complex phenomena probabilistically, can learn distributions that are as close to uniform as possible when trained on cryptographically pseudorandom data. This approach bridges Kolmogorov complexity, information theory, and cryptographic indistinguishability with practical neural network paradigms.

Minimizing KL Divergence as Managing Complexity

The paper grounds its analysis in minimizing KL divergence ($\mathrm{KL}(D\|\mu)$) between a target distribution $D$ and a learned distribution $\mu$. The justification draws on foundational principles from Occam's Razor and Minimum Description Length (MDL), connecting the complexity of a probabilistic explanation to Kolmogorov complexity and cross-entropy decomposition. Importantly, the minimization of KL divergence is formalized not merely as a practical training objective (e.g., for neural networks), but as a computational principle: the best computable explanation for observed data probabilistically minimizes KL divergence plus the descriptional complexity of the model.

The paper distinguishes between sampleable and computable distributions, emphasizing that computability is necessary for next-token prediction and conditional sampling—the backbone of language modeling and generative tasks.

Neural Networks as $\mathrm{P}/\text{poly}$-Computable Distributions

A central technical claim is that large neural networks instantiate $\mathrm{P}/\text{poly}$-computable distributions with polynomially bounded max entropy. Specifically:

• Computable Distributions: Next-token prediction in transformers corresponds mathematically to computable conditional probability functions. For a prefix $y$, the network computes $f(y) = \Pr_{\mu}[x_{|y|+1}=1 | x_{1..|y|}=y]$, which is a $\mathrm{P}/\text{poly}$-computable function in practice.
• Polynomial-Time: Despite networks being shallow, reasoning models and code execution mechanisms allow the simulation of polynomial-time computations by sequential reasoning or code generation, extending effective computational depth.
• Nonuniformity: The weights learned during training are regarded as nonuniform advice—a formal analogy to circuit complexity, where advice is tailored to the input size. Training is computationally expensive and encodes vast amounts of information nonuniformly.
• Max Entropy: Neural nets naturally produce distributions with bounded max entropy (no output is assigned probability zero), further ensuring the KL divergence is not infinite and all outcomes retain exponentially small probabilities.

These structural properties enable the representation of highly complex behaviors through probability rather than deterministic computation.

Machine Learning and Cryptographic Pseudorandomness

The main theorem establishes that, when training a $\mathrm{P}/\text{poly}$-computable distribution on the output of a cryptographic pseudorandom generator (PRG), the learned distribution $\mathrm{KL}(D\|\mu)$0 cannot distinguish $\mathrm{KL}(D\|\mu)$1 from uniform in an information-theoretic sense. Formally,

• If $\mathrm{KL}(D\|\mu)$2 minimizes $\mathrm{KL}(D\|\mu)$3 over computable distributions with bounded max entropy, and $\mathrm{KL}(D\|\mu)$4 comes from a PRG, then $\mathrm{KL}(D\|\mu)$5 is negligible, where $\mathrm{KL}(D\|\mu)$6 is uniform.
• Without bounded max entropy, statistical difference remains negligible.

This result leverages computational indistinguishability: PRG outputs cannot be distinguished from uniform by polynomial-time algorithms (or $\mathrm{KL}(D\|\mu)$7 circuits). Thus, any computable model trained on PRG data will be virtually uniform information-theoretically, reflecting the inability of machine learning to extract structure from information-theoretically random data.

Strong numerical claim: $\mathrm{KL}(D\|\mu)$8 for all $\mathrm{KL}(D\|\mu)$9, underscoring the information-theoretic closeness.

Theoretical Implications

• Complexity as Randomness: The inability to extract structure from PRG output demonstrates that learning, constrained to computable distributions, manages complexity by probabilistically averaging over unresolvable ambiguity. This is foundational for understanding the limits of learning: for highly complex or cryptographically random sources, all computable models essentially default to probabilistic guessing.
• Role of Nonuniformity: Nonuniform advice (weights) fundamentally enhances learning capacity, encoding prior data and capability. This parallels cryptographic results showing the limits of PAC learning under cryptographic hardness assumptions.
• Probability vs. Determinism: By representing hypotheses as distributions rather than deterministic algorithms, machine learning can model behaviors not feasible for classical computational paradigms.

Practical Implications

• Model Robustness: This formalization explains why neural network models do not (and cannot) guarantee deterministic accuracy—probabilistic outputs are essential for robustness amid complex, data-driven uncertainty.
• Boundaries of Learnability: For data generated by cryptographically secure sources, learning will be limited to uniform guessing; no practical learning algorithm can do better unless the computational assumptions are violated.
• Adversarial/Hard Cases: The analysis suggests limits for adversarial training or model attacks when randomization and complexity coincide.

Speculation on Future Developments

• Refinement of Learning Models: The paper advocates for further abstraction and mathematical formalization of machine learning beyond current technologies.
• Complexity-Theoretic Limits: Resolving whether efficient learning can find the $D$0-computable distribution minimizing KL divergence for all sampleable distributions (without breaking cryptography) remains a key open question.
• Theory of Optiland: The paper opens avenues toward understanding a world in which optimization is broadly feasible, yet cryptography remains secure (the "Optiland" scenario).

Conclusion

"How Does Machine Learning Manage Complexity?" (2604.07233) positions machine learning within the landscape of computational complexity, demonstrating that neural networks enforce computable probabilistic hypotheses with bounded entropy and leverage nonuniformity (trained weights) to manage complexity. When learning from cryptographically random sources, all computable models are forced to output near-uniform distributions. This exposes fundamental boundaries of learnability and motivates new theoretical frameworks for analyzing machine learning capabilities and limitations.