Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
175 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Deep Feature Screening: Feature Selection for Ultra High-Dimensional Data via Deep Neural Networks (2204.01682v3)

Published 4 Apr 2022 in stat.ML and cs.LG

Abstract: The applications of traditional statistical feature selection methods to high-dimension, low sample-size data often struggle and encounter challenging problems, such as overfitting, curse of dimensionality, computational infeasibility, and strong model assumption. In this paper, we propose a novel two-step nonparametric approach called Deep Feature Screening (DeepFS) that can overcome these problems and identify significant features with high precision for ultra high-dimensional, low-sample-size data. This approach first extracts a low-dimensional representation of input data and then applies feature screening based on multivariate rank distance correlation recently developed by Deb and Sen (2021). This approach combines the strengths of both deep neural networks and feature screening, and thereby has the following appealing features in addition to its ability of handling ultra high-dimensional data with small number of samples: (1) it is model free and distribution free; (2) it can be used for both supervised and unsupervised feature selection; and (3) it is capable of recovering the original input data. The superiority of DeepFS is demonstrated via extensive simulation studies and real data analyses.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Concrete autoencoders for differentiable feature selection and reconstruction. arXiv preprint arXiv:1901.09346.
  2. On the approximability of minimizing nonzero variables or unsatisfied relations in linear systems. Theoretical Computer Science, 209(1):237–260.
  3. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
  4. Predictable features elimination: An unsupervised approach to feature selection. In International Conference on Machine Learning, Optimization, and Data Science, pages 399–412. Springer.
  5. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(10):281–305.
  6. A survey on feature selection methods. Computers & Electrical Engineering, 40(1):16–28. 40th-year commemorative issue.
  7. Variable-length representation for ec-based feature selection in high-dimensional data. In International conference on the applications of evolutionary computation (Part of EvoStar), pages 325–340. Springer.
  8. Multivariate rank-based distribution-free nonparametric testing using measure transportation. Journal of the American Statistical Association, pages 1–16.
  9. Ding, C. H. (2003). Unsupervised feature selection via two-way ordering in gene expression analysis. Bioinformatics, 19(10):1259–1266.
  10. Sure independence screening for ultrahigh dimensional feature space. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 70(5):849–911.
  11. Ultrahigh dimensional feature selection: beyond the linear model. The Journal of Machine Learning Research, 10:2013–2038.
  12. Deep neural networks for estimation and inference. Econometrica, 89(1):181–213.
  13. Graph autoencoder-based unsupervised feature selection with broad and local data structure preservation. Neurocomputing, 312:310–323.
  14. Radical inverse quasi-random point sequence, algorithm 247. Commun. ACM, 7(12):701.
  15. Autoencoder inspired unsupervised feature selection. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2941–2945.
  16. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.
  17. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144.
  18. Drug discovery with explainable artificial intelligence. Nature Machine Intelligence, 2(10):573–584.
  19. A new local search based hybrid genetic algorithm for feature selection. Neurocomputing, 74(17):2914–2928.
  20. A survey of feature selection and feature extraction techniques in machine learning. In 2014 Science and Information Conference, pages 372–378.
  21. Feature selection: a literature review. SmartCR, 4(3):211–229.
  22. Lassonet: A neural network with feature sparsity. Journal of Machine Learning Research, 22(127):1–29.
  23. Li, K. (2022). Variable selection for nonlinear cox regression model via deep learning. arXiv preprint arXiv:2211.09287.
  24. Calibrating multi-dimensional complex ode from noisy data via deep neural networks. arXiv preprint arXiv:2106.03591.
  25. Semiparametric regression for spatial data via deep learning. arXiv preprint arXiv:2301.03747.
  26. Feature screening via distance correlation learning. Journal of the American Statistical Association, 107(499):1129–1139.
  27. Deep feature selection: theory and application to identify enhancers and promoters. Journal of Computational Biology, 23(5):322–336.
  28. Deep neural networks for high dimension, low sample size data. In IJCAI, pages 2287–2293.
  29. Optimal nonparametric inference via deep neural network. Journal of Mathematical Analysis and Applications, 505(2):125561.
  30. On deep instrumental variables estimate.
  31. A survey on feature selection. Procedia Computer Science, 91:919–926. Promoting Business Analytics and Quantitative Management of Technology: 4th International Conference on Information Technology and Quantitative Management (ITQM 2016).
  32. Variational relevant sample-feature machine: a fully bayesian approach for embedded feature selection. Neurocomputing, 241:181–190.
  33. Deep feature selection using a teacher-student network. Neurocomputing, 383:396–408.
  34. Incremental relevance sample-feature machine: A fast marginal likelihood maximization approach for joint feature selection and classification. Pattern Recognition, 60:835–848.
  35. Unsupervised feature selection by regularized matrix factorization. Neurocomputing, 273:593–610.
  36. A review of feature selection techniques in bioinformatics. Bioinformatics, 23(19):2507–2517.
  37. Group sparse regularization for deep neural networks. Neurocomputing, 241:81–89.
  38. Schmidt-Hieber, J. (2020). Nonparametric regression using deep neural networks with relu activation function. The Annals of Statistics, 48(4):1875–1897.
  39. Fsnet: Feature selection network on high-dimensional biological data. arXiv preprint arXiv:2001.08322.
  40. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25.
  41. Sobol’, I. M. (1967). On the distribution of points in a cube and the approximate evaluation of integrals. Zhurnal Vychislitel’noi Matematiki i Matematicheskoi Fiziki, 7(4):784–802.
  42. A review of unsupervised feature selection methods. Artificial Intelligence Review, 53(2):907–948.
  43. Deep-fs: A feature selection algorithm for deep boltzmann machines. Neurocomputing, 322:22–37.
  44. Novel Unsupervised Feature Filtering of Biological Data. Bioinformatics, 22(14):e507–e513.
  45. Estimation of the mean function of functional data via deep neural networks. Stat, 10(1):e393.
  46. Genome-wide association analysis by lasso penalized logistic regression. Bioinformatics, 25(6):714–721.
  47. Ultra high-dimensional nonlinear feature selection for big biological data. IEEE Transactions on Knowledge and Data Engineering, 30(7):1352–1365.
  48. Prioritizing genetic variants in GWAS with lasso using permutation-assisted tuning. Bioinformatics, 36(12):3811–3817.
  49. Distribution-free and model-free multivariate feature screening via multivariate rank distance correlation. Journal of Multivariate Analysis, page 105081.
  50. Co-regularized unsupervised feature selection. Neurocomputing, 275:2855–2863.
Citations (28)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now