Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Contrast Based Feature Selection Algorithm for High-dimensional Data set in Machine Learning (2401.07482v1)

Published 15 Jan 2024 in cs.LG

Abstract: Feature selection is an important process in machine learning and knowledge discovery. By selecting the most informative features and eliminating irrelevant ones, the performance of learning algorithms can be improved and the extraction of meaningful patterns and insights from data can be facilitated. However, most existing feature selection methods, when applied to large datasets, encountered the bottleneck of high computation costs. To address this problem, we propose a novel filter feature selection method, ContrastFS, which selects discriminative features based on the discrepancies features shown between different classes. We introduce a dimensionless quantity as a surrogate representation to summarize the distributional individuality of certain classes, based on this quantity we evaluate features and study the correlation among them. We validate effectiveness and efficiency of our approach on several widely studied benchmark datasets, results show that the new method performs favorably with negligible computation in comparison with other state-of-the-art feature selection methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (41)
  1. Concrete autoencoders: Differentiable feature selection and reconstruction. In International Conference on Machine Learning, pp. 444–453. PMLR, 2019.
  2. Bellman, R. Dynamic Programming. Princeton University Press, Princeton, 1957.
  3. Feature selection in image analysis: A survey. Artificial Intelligence Review, 53(4):2905–2931, April 2020.
  4. Conditional Likelihood Maximisation: A Unifying Framework for Information Theoretic Feature Selection. Journal of Machine Learning Research, 13(2):27–66, 2012.
  5. Feature selection in machine learning: A new perspective. Neurocomputing, 300:70–79, July 2018.
  6. A survey on feature selection methods. Computers & Electrical Engineering, 40(1):16–28, January 2014.
  7. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pp.  785–794, New York, NY, USA, August 2016. Association for Computing Machinery. ISBN 978-1-4503-4232-2.
  8. Block HSIC Lasso: Model-free biomarker detection for ultra-high dimensional data. Bioinformatics, 35(14):i427–i435, July 2019.
  9. Similarity of feature selection methods: An empirical study across data intensive classification tasks. Expert Systems with Applications, 42(10):4632–4642, June 2015.
  10. Pattern Classification. Wiley, New York, 2nd ed edition, 2000. ISBN 978-0-471-05669-0.
  11. Statistical challenges with high dimensionality: Feature selection in knowledge discovery. In 25th International Congress of Mathematicians, ICM 2006, 2006.
  12. Fleuret, F. Fast Binary Feature Selection with Conditional Mutual Information. Journal of Machine Learning Research, 5:1531–1555, December 2004.
  13. Deep Learning. Adaptive Computation and Machine Learning. The MIT Press, Cambridge, Massachusetts, 2016. ISBN 978-0-262-03561-3.
  14. An introduction to variable and feature selection. Journal of machine learning research, 3(Mar):1157–1182, 2003.
  15. Feature selection for machine learning: Comparing a correlation-based filter approach to the wrapper. In FLAIRS Conference, volume 1999, pp.  235–239, 1999.
  16. A survey on feature selection approaches for clustering. Artificial Intelligence Review, 53(6):4519–4545, August 2020.
  17. Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(8):1226–1238, August 2005.
  18. Statistical challenges of high-dimensional data. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 367(1906):4237–4253, November 2009.
  19. Wrappers for feature subset selection. Artificial Intelligence, 97(1-2):273–324, December 1997.
  20. Feature selection in enterprise analytics: A demonstration using an R-based data analytics system. Proceedings of the VLDB Endowment, 6(12):1306–1309, August 2013.
  21. LassoNet: A Neural Network with Feature Sparsity. Journal of Machine Learning Research, 22(127), June 2021.
  22. Challenges of Feature Selection for Big Data Analytics. IEEE Intelligent Systems and Their Applications, 32(2):9–15, March 2017.
  23. Feature Selection: A Data Perspective. ACM Computing Surveys, 50(6):1–45, November 2018.
  24. Recent advances in feature selection and its applications. Knowledge and Information Systems, 53(3):551–577, December 2017.
  25. Information-Theoretic Feature Selection in Microarray Data Using Variable Complementarity. IEEE Journal of Selected Topics in Signal Processing, 2(3):261–274, June 2008.
  26. Trace ratio criterion for feature selection. In AAAI, volume 2, pp.  671–676, 2008.
  27. Efficient and Robust Feature Selection via Joint \mathscrl2,1-Norms Minimization. In Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010.
  28. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12(85):2825–2830, 2011.
  29. Ruppert, D. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Journal of the American Statistical Association, 99(466):567–567, June 2004.
  30. A review of feature selection techniques in bioinformatics. Bioinformatics, 23(19):2507–2517, October 2007.
  31. Robust Feature Selection Using Ensemble Feature Selection Techniques. In Daelemans, W., Goethals, B., and Morik, K. (eds.), Machine Learning and Knowledge Discovery in Databases, volume 5212, pp. 313–325. Springer Berlin Heidelberg, Berlin, Heidelberg, 2008. ISBN 978-3-540-87480-5 978-3-540-87481-2.
  32. A survey on feature selection methods for mixed data. Artificial Intelligence Review, 55(4):2821–2846, April 2022.
  33. Feature Selection via Dependence Maximization. Journal of Machine Learning Research, 13(5), 2012.
  34. Variable selection with false discovery rate control in deep neural networks. Nature Machine Intelligence, 3(5):426–433, March 2021.
  35. Tibshirani, R. Regression Shrinkage and Selection Via the Lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267–288, January 1996.
  36. Wasserman, L. All of Statistics: A Concise Course in Statistical Inference. Springer Texts in Statistics. Springer, New York Berlin Heidelberg, corr. 2. print., [repr.] edition, 2010. ISBN 978-1-4419-2322-6.
  37. High-Dimensional Feature Selection by Feature-Wise Kernelized Lasso. Neural Computation, 26(1):185–207, January 2014.
  38. Data Visualization and Feature Selection: New Algorithms for Nongaussian Data. In Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999.
  39. L2, 1-norm regularized discriminative feature selection for unsupervised. In Twenty-Second International Joint Conference on Artificial Intelligence, 2011.
  40. A Comparative Study on Feature Selection in Text Categorization. In Proceedings of the Fourteenth International Conference on Machine Learning, ICML ’97, pp.  412–420, San Francisco, CA, USA, July 1997. Morgan Kaufmann Publishers Inc. ISBN 978-1-55860-486-5.
  41. On Similarity Preserving Feature Selection. IEEE Transactions on Knowledge and Data Engineering, 25(3):619–632, March 2013.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now