Gaussian Multi-index Models

• Gaussian multi-index models are a statistical framework that projects high-dimensional inputs onto a low-dimensional index space for regression.
• They enable effective dimension reduction and consistent subspace estimation through methods like the Response-Conditional Least Squares estimator.
• Empirical and theoretical results confirm minimax optimal rates in both subspace recovery and nonparametric regression under Gaussian assumptions.

Gaussian multi-index models provide a unifying statistical framework for representing high-dimensional regression scenarios where the output depends solely on a low-dimensional projection of the input, enabling practitioners to circumvent the curse of dimensionality through effective dimension reduction. The model assumes the existence of a low-rank "index space" such that the conditional mean of the outcome variable is an unknown function of a linear projection of high-dimensional covariates. Efficient, consistent, and optimally convergent estimation of this index space is essential in statistical learning, dimension reduction, and nonparametric regression, especially under Gaussian distributions.

1. Model Formulation and Problem Setup

The canonical multi-index model is

$Y = g(A^\top X) + \zeta,$

where $X \in \mathbb{R}^D$ is the high-dimensional predictor, $Y \in \mathbb{R}$ is the response, $A \in \mathbb{R}^{D \times d}$ is an unknown full-rank matrix with $d \ll D$, $g: \mathbb{R}^d \to \mathbb{R}$ is an unknown link function, and $\zeta$ is mean-zero noise: $\mathbb{E}[\zeta | X] = 0$.

Objective:

Estimate the index space $\operatorname{Im}(A)$ given $N$ i.i.d. samples $\{(X_i, Y_i)\}_{i=1}^N$, and subsequently fit the link function $g$ for prediction or inference.

The setup is especially tractable and theoretically sharp when $X \sim \mathcal{N}(0, \Sigma)$ (often with $\Sigma = I$), as the model then satisfies the linear conditional mean property that is foundational to dimension reduction methods.

2. Response-Conditional Least Squares (RCLS) Estimator

The paper introduces the Response-Conditional Least Squares (RCLS) estimator, constructed as follows:

1. Partition the Range of $Y$: Divide the real line (or the observed range of $Y$) into $J$ disjoint intervals $\{ \mathcal{R}_{J, \ell} \}_{\ell=1}^J$.
2. Create Level Sets: For each interval, select samples $$\mathcal{X}_{J,\ell} := \{ X_i : Y_i \in \mathcal{R}_{J,\ell} \}$$.
3. Local OLS on Each Level Set:

Within each set $\mathcal{X}_{J,\ell}$, perform ordinary least squares regression to obtain the slope vector $\hat{b}_{J,\ell}$. Specifically:

$$\hat b_{J,\ell} := \hat\Sigma_{J,\ell}^\dagger \frac{1}{|\mathcal{X}_{J,\ell}|} \sum_{X_i \in \mathcal{X}_{J,\ell}}\! (X_i - \bar{X}_{J,\ell})(Y_i - \bar{Y}_{J,\ell}),$$

with sample means and pseudo-inverse $(\cdot)^\dagger$.

1. Aggregate Matrix Formation: Construct the matrix

$$\hat M_J = \sum_{\ell=1}^J \hat\rho_{J, \ell}\, \hat{b}_{J,\ell} \hat{b}_{J,\ell}^\top,$$

where $\hat\rho_{J, \ell} = |\mathcal{X}_{J, \ell}| / N$ (empirical fraction in slice).

1. Index Space Estimation: Obtain the orthoprojector onto the span of the top $d$ eigenvectors of $\hat M_J$, giving the estimator $\hat{A}$ for the index space.

Only a single hyperparameter needs to be set—the number of level sets $J$.

3. Theoretical Guarantees and Statistical Efficiency

Finite Sample Error Bound

Under LCM and sub-Gaussian design (satisfied for Gaussian $X$), the following holds: $\|\hat{P}_J - P_J\|_F \leq C(J) \sqrt{D/N},$ where $\hat{P}_J$ and $P_J$ are empirical and population orthoprojectors onto the index space, $\| \cdot \|_F$ is the Frobenius norm, and $C(J)$ depends on the number of level sets and geometric factors.

Convergence rate is $N^{-1/2}$ (oracle/minimax optimal for subspace estimation).

Generalization Bounds for Regression

If, after estimating the index space, nonparametric regression is performed with kNN or piecewise polynomial estimators on the reduced data, then the total mean squared prediction error is bounded by: $$\mathbb{E}[(\hat{f}(X) - f(X))^2] \lesssim N^{-2s/(2s+d)} + \|\hat{P}-P\|^{\min\{2s,2\}},$$ where $s$ is the link function smoothness and $d$ is the intrinsic dimension. If the subspace estimate is consistent at rate $N^{-1/2}$, the overall rate matches the minimax optimal $d$-dimensional nonparametric regression rate: $N^{-2s/(2s + d)}$.

4. Implementation and Practical Guidance

• Complexity: $O(N D^2)$ due to repeated OLS fits and a single $D \times D$ eigendecomposition.
• Hyperparameter Selection:

Theoretical and empirical guidance is provided for tuning $J$; e.g., choose $J$ to minimize an empirical upper bound on projection error.

• Subspace Dimension Selection:

Determined by inspecting the spectrum of $\hat M_J$ or via cross-validation.

• Extensions:
  • RCLS naturally extends to settings where the projection matrix is sparse by replacing OLS with Lasso.
  • Does not require knowledge or estimation of $g$ or strong smoothness of $g$.

5. Empirical Performance and Comparative Evaluation

Synthetic Experiments

• Models tested in $D = 20$ with $d = 1, 2, 3$.
• Functions include nontrivial nonlinear link functions.
• Metrics: Frobenius norm distance between estimated and true subspace projection.
• Results: RCLS matches or outperforms SIR, SIRII, SAVE, DR, pHd, and demonstrates $N^{-1/2}$ empirical rates.

Real Data (UCI Repository)

• Best predictive performance in multiple real datasets (Airquality, Concrete, Skillcraft, Yacht).
• Requires less hyperparameter tuning and computation than comparators.
• Strong empirical results align with theoretical rate guarantees.

6. Applicability to Gaussian Multi-Index Models and SDR Methods

For Gaussian input variables, both the LCM and constant conditional variance (CCV) conditions required for RCLS hold, guaranteeing correctness and optimal convergence. Specifically:

• RCLS enjoys minimax-optimality, low computational complexity, and is robust to non-Gaussian extensions as long as the LCM assumption holds.
• Requires only LCM (weaker than CCV), making it less restrictive than alternatives like SAVE.
• All theory and practice from the paper apply directly in the Gaussian scenario, which is the best-case setting for RCLS and most SDR methods.

Comparison Table: RCLS Capabilities in Gaussian Multi-Index Models

Aspect	RCLS Capabilities (Gaussian Setting)
Identification	Consistent subspace recovery, $N^{-1/2}$ rate, efficient
Theoretical Bound	$\|\hat{P} - P\| = O(N^{-1/2})$, regression achieves minimax rate
Implementation	Simple (one hyperparameter), fast, with practical tuning guidelines
Empirical Perf.	Matches/exceeds SIR, SAVE, DR, pHd in synthetic/real benchmarks
Gaussian setting	All assumptions met; theory and practice fully applicable

Conclusion

The RCLS estimator provides a computationally and statistically optimal approach for estimating the index space in multi-index regression models, especially under Gaussian designs. It is simple to implement, requires minimal hyperparameter tuning, and achieves minimax rates both in the estimation of the index space and in downstream prediction. This establishes RCLS as a robust, general, and efficient technique for practical high-dimensional regression and supervised dimension reduction tasks where a low-dimensional structure under Gaussian assumptions is expected.