Minimax-optimal rates for sparse additive models over kernel classes via convex programming (1008.3654v2)

Abstract: Sparse additive models are families of $d$-variate functions that have the additive decomposition $f^* = \sum_{j \in S} f^*_j$, where $S$ is an unknown subset of cardinality $s \ll d$. In this paper, we consider the case where each univariate component function $f^*_j$ lies in a reproducing kernel Hilbert space (RKHS), and analyze a method for estimating the unknown function $f^*$ based on kernels combined with $\ell_1$-type convex regularization. Working within a high-dimensional framework that allows both the dimension $d$ and sparsity $s$ to increase with $n$, we derive convergence rates (upper bounds) in the $L^2(\mathbb{P})$ and $L^2(\mathbb{P}_n)$ norms over the class $\MyBigClass$ of sparse additive models with each univariate function $f^*_j$ in the unit ball of a univariate RKHS with bounded kernel function. We complement our upper bounds by deriving minimax lower bounds on the $L^2(\mathbb{P})$ error, thereby showing the optimality of our method. Thus, we obtain optimal minimax rates for many interesting classes of sparse additive models, including polynomials, splines, and Sobolev classes. We also show that if, in contrast to our univariate conditions, the multivariate function class is assumed to be globally bounded, then much faster estimation rates are possible for any sparsity $s = \Omega(\sqrt{n})$, showing that global boundedness is a significant restriction in the high-dimensional setting.

• The paper introduces a convex programming strategy that attains minimax-optimal rates for estimating high-dimensional sparse additive models using reproducing kernel Hilbert spaces.
• It demonstrates that convergence rates scale as Θ((s log d)/n + sνₙ) for both finite-rank and Sobolev-type kernels, offering rigorous error bounds.
• The study avoids restrictive global boundedness by requiring boundedness only for individual univariate functions, guiding more effective model design.

Minimax-Optimal Rates for Sparse Additive Models over Kernel Classes via Convex Programming

This paper provides an in-depth analysis of sparse additive models (SAMs) within a high-dimensional framework, employing reproducing kernel Hilbert spaces (RKHS) for component functions. The authors Raskutti, Wainwright, and Yu tackle the challenge of modeling high-dimensional data by estimating the unknown function $f$ through a convex programming approach that combines kernel methods with $\ell_1$-type regularization.

The paper examines a polynomial-time method to obtain upper bounds on the error rates for estimation within the $L^2(P)$ and empirical $L^2(P_n)$ norms. By analyzing a class of d-variate functions decomposed additively, with each univariate component lying within the unit ball of a univariate RKHS, the authors derive rates of convergence that scale with the sample size $n$, dimension $d$, and sparsity $s$. Notably, the convergence rate is represented by an upper bound of $\Theta(\frac{s \log d}{n} + s\nu_n)$, where $\nu_n$ signifies the optimal rate for estimating a single univariate function in the RKHS. The mentioned rate characterizes the method's performance as minimax optimal by comparing these rates to established minimax lower bounds.

The paper presents strong numerical results, discussing instances where the procedure achieves optimal convergence properties for different kernel classes, including finite rank and Sobolev-type RKHS. These results are shown to hold without imposing the restrictive global boundedness conditions on the multivariate function class, which is often assumed in the classical setting to ensure much faster rates of estimation. Instead, the assumption of boundedness is only required for individual univariate component functions.

A significant addition to the field is the examination of sharp lower bounds on the minimax \emph{L}-error, providing a complete characterization of the achievable rates for both finite-rank kernels and those with polynomially decaying eigenvalues. Such detailed theoretical analysis is crucial for understanding the capabilities and limitations of sparse models, particularly when addressing the curse of dimensionality in non-parametric settings.

Overall, the paper raises important points about the feasibility and optimality of using SAM estimates in high-dimensional spaces while avoiding overly restrictive conditions. It mathematically solidifies the notion that convex programming frameworks, equipped with appropriate regularization, can effectively manage complex, high-dimensional data.

The potential implications of this paper are vast, especially for machine learning and statistics, where sparse models and kernel methods are frequently employed. The emphasis on minimax rates not only enhances theoretical understanding but provides guidelines for practitioners to choose and optimize modeling strategies in applied statistical problems.

Furthermore, the discussion highlights areas for future exploration, such as extending the analysis to correlated design points or considering hierarchical model decompositions. This sets a promising agenda for further research, driving advancements in the accurate and efficient estimation of complex data structures in high-dimensional spaces.