Distribution-independent dimension complexity of SGD-learnable neural networks

Establish an O(TS) common linear representation, and hence a dimension-complexity bound of order TS, for every binary hypothesis class that is learnable with expected error below 1/4 by a fully connected ReLU network with S parameters trained by all-layer SGD for T steps under every input distribution, without imposing antipodal oddness, a bias-free depth-two architecture, or a stricter accuracy condition.

Background

Dimension complexity measures the smallest shared feature dimension in which an entire hypothesis class can be represented by linear predictors. The source question asks whether distribution-independent SGD learnability of a neural network implies a comparable shared linear representation.

The paper proves only a restricted identity-feature result for bias-free depth-two ReLU networks, antipodally odd target classes, and strict high accuracy. It explicitly leaves open obtaining the O(TS) representation under the unrestricted source assumptions.

References

The remaining problem is to obtain an $O(TS)$ common linear representation under the source assumptions, without these three restrictions.

— VALG: An Agentic System for ML Theory Research  (2608.13060 - Zhang et al., 13 Aug 2026) in Section 4, Subsection “Is the Power of Deep Learning over Linear Models Inherently Distribution Dependent?”, Subproblem 1: Learning with SGD over Neural Networks