2000 character limit reached
Golden Ratio-Based Sufficient Dimension Reduction
Published 25 Oct 2024 in cs.LG and stat.ME | (2410.19300v2)
Abstract: Many machine learning applications deal with high dimensional data. To make computations feasible and learning more efficient, it is often desirable to reduce the dimensionality of the input variables by finding linear combinations of the predictors that can retain as much original information as possible in the relationship between the response and the original predictors. We propose a neural network based sufficient dimension reduction method that not only identifies the structural dimension effectively, but also estimates the central space well. It takes advantages of approximation capabilities of neural networks for functions in Barron classes and leads to reduced computation cost compared to other dimension reduction methods in the literature. Additionally, the framework can be extended to fit practical dimension reduction, making the methodology more applicable in practical settings.
- R. Bellman and R. Kalaba, “On adaptive control processes,” IRE Transactions on Automatic Control, vol. 4, no. 2, pp. 1–9, 1959.
- B. Li, Sufficient dimension reduction: Methods and applications with R. Chapman and Hall/CRC, 2018.
- K.-C. Li, “Sliced inverse regression for dimension reduction,” Journal of the American Statistical Association, vol. 86, no. 414, pp. 316–327, 1991.
- Y. Xia, H. Tong, W. K. Li, and L.-X. Zhu, “An adaptive estimation of dimension reduction space,” Journal of the Royal Statistical Society Series B: Statistical Methodology, vol. 64, no. 3, pp. 363–410, 2002.
- R. D. Cook and S. Weisberg, “Sliced inverse regression for dimension reduction: Comment,” Journal of the American Statistical Association, vol. 86, no. 414, pp. 328–332, 1991.
- Q. Wang and X. Yin, “Aggregate inverse mean estimation for sufficient dimension reduction,” Technometrics, pp. 1–10, 2020.
- L.-P. Zhu, L.-X. Zhu, and Z.-H. Feng, “Dimension reduction in regressions through cumulative slicing estimation,” Journal of the American Statistical Association, vol. 105, no. 492, pp. 1455–1466, 2010.
- R. D. Cook and X. Zhang, “Fused estimators of the central subspace in sufficient dimension reduction,” Journal of the American Statistical Association, vol. 109, no. 506, pp. 815–827, 2014.
- Z. Cai, R. Li, and L. Zhu, “Online sufficient dimension reduction through sliced inverse regression.,” J. Mach. Learn. Res., vol. 21, no. 10, pp. 1–25, 2020.
- Q. Lin, Z. Zhao, and J. S. Liu, “On consistency and sparsity for sliced inverse regression in high dimensions,” Ann. Statist., 2018.
- Q. Wang, X. Yin, B. Li, and Z. Tang, “On aggregate dimension reduction,” Statistica Sinica, pp. 1027–1048, 2020.
- A. Artemiou, Y. Dong, and S. J. Shin, “Real-time sufficient dimension reduction through principal least squares support vector machines,” Pattern Recognition, vol. 112, p. 107768, 2021.
- A.-N. Soale and Y. Dong, “On expectile-assisted inverse regression estimation for sufficient dimension reduction,” Journal of Statistical Planning and Inference, vol. 213, pp. 80–92, 2021.
- W. Luo, “On efficient dimension reduction with respect to the interaction between two response variables,” Journal of the Royal Statistical Society Series B: Statistical Methodology, vol. 84, no. 2, pp. 269–294, 2022.
- Q. Wang and Y. Xue, “A structured covariance ensemble for sufficient dimension reduction,” Advances in Data Analysis and Classification, pp. 1–24, 2022.
- E. Pircalabelu and A. Artemiou, “High-dimensional sufficient dimension reduction through principal projections,” Electronic Journal of Statistics, vol. 16, no. 1, pp. 1804–1830, 2022.
- G. Zhang, J. Zhang, and J. Hinkle, “Learning nonlinear level sets for dimensionality reduction in function approximation,” Advances in Neural Information Processing Systems, vol. 32, 2019.
- Y. Teng, Z. Wang, L. Ju, A. Gruber, and G. Zhang, “Level set learning with pseudoreversible neural networks for nonlinear dimension reduction in function approximation,” SIAM Journal on Scientific Computing, vol. 45, no. 3, pp. A1148–A1171, 2023.
- C. Meng, J. Yu, J. Zhang, P. Ma, and W. Zhong, “Sufficient dimension reduction for classification using principal optimal transport direction,” Advances in Neural Information Processing Systems, vol. 33, pp. 4015–4028, 2020.
- S. Liang, Y. Sun, and F. Liang, “Nonlinear sufficient dimension reduction with a stochastic neural network,” Advances in Neural Information Processing Systems, vol. 35, pp. 27360–27373, 2022.
- D. Kapla, L. Fertl, and E. Bura, “Fusing sufficient dimension reduction with neural networks,” Computational Statistics & Data Analysis, vol. 168, p. 107390, 2022.
- A. R. Barron, “Approximation and estimation bounds for artificial neural networks,” Machine Learning, vol. 14, no. 1, pp. 115–133, 1994.
- J. M. Klusowski and A. R. Barron, “Approximation by combinations of relu and squared relu ridge functions with ℓ1superscriptℓ1\ell^{1}roman_ℓ start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT and ℓ0superscriptℓ0\ell^{0}roman_ℓ start_POSTSUPERSCRIPT 0 end_POSTSUPERSCRIPT controls,” IEEE Transactions on Information Theory, vol. 64, no. 12, pp. 7649–7656, 2018.
- R. D. Cook, “Using dimension-reduction subspaces to identify important inputs in models of physical systems,” in Proceedings of the section on Physical and Engineering Sciences, pp. 18–25, 1994.
- A. R. Barron, “Universal approximation bounds for superpositions of a sigmoidal function,” IEEE Transactions on Information theory, vol. 39, no. 3, pp. 930–945, 1993.
- J. Schmidhuber, “Deep learning in neural networks: An overview,” Neural networks, vol. 61, pp. 85–117, 2015.
- M. Kohler and A. Krzyżak, “Nonparametric regression based on hierarchical interaction models,” IEEE Transactions on Information Theory, vol. 63, no. 3, pp. 1620–1630, 2016.
- B. Bauer and M. Kohler, “On deep learning as a remedy for the curse of dimensionality in nonparametric regression,” The Annals of Statistics, vol. 47, no. 4, pp. 2261–2285, 2019.
- M. B. Stinchcombe, “Neural network approximation of continuous functionals and continuous functions on compactifications,” Neural Networks, vol. 12, no. 3, pp. 467–477, 1999.
- H. N. Mhaskar and T. Poggio, “Deep vs. shallow networks: An approximation theory perspective,” Analysis and Applications, vol. 14, no. 06, pp. 829–848, 2016.
- T. De Ryck, S. Lanthaler, and S. Mishra, “On the approximation of functions by tanh neural networks,” Neural Networks, vol. 143, pp. 732–750, 2021.
- A. V. Knyazev and M. E. Argentati, “Principal angles between subspaces in an a-based scalar product: algorithms and perturbation estimates,” SIAM Journal on Scientific Computing, vol. 23, no. 6, pp. 2008–2040, 2002.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.