Dimension reduction for gradient-descent-trained over-parameterized deep networks

Establish whether gradient-descent-trained deep neural-network regression estimates achieve dimension reduction for independent and identically distributed data when the regression function satisfies a hierarchical composition model or when the predictor is supported on a submanifold of Euclidean space, while the network size and number of gradient-descent steps grow no faster than polynomially in the sample size.

Background

The paper contrasts computable gradient-descent-trained neural-network estimates with least-squares estimates, whose empirical-risk minimization is generally nonlinear and non-convex. Although prior work established dimension-reduction rates for least-squares deep neural-network estimators under hierarchical composition or manifold assumptions, the corresponding guarantee for gradient-descent-trained estimators was not known even in the i.i.d. setting.

The unresolved issue concerns avoiding exponential computational growth: the network size and the number of optimization steps should not be exponentially large in the sample size. The present paper instead proves rates for gradient-descent-trained networks under globally smooth regression functions, including dependent exponentially beta-mixing data and manifold-supported predictors, but it does not settle the broader hierarchical-composition question stated here.