A Review of Feature Selection Methods Based on Mutual Information (1509.07577v1)

Abstract: In this work we present a review of the state of the art of information theoretic feature selection methods. The concepts of feature relevance, redundance and complementarity (synergy) are clearly defined, as well as Markov blanket. The problem of optimal feature selection is defined. A unifying theoretical framework is described, which can retrofit successful heuristic criteria, indicating the approximations made by each method. A number of open problems in the field are presented.

• The paper presents a comprehensive review of MI-based feature selection methods, emphasizing the trade-off between relevance and redundancy.
• It outlines a unified framework that reinterprets heuristic methods like mRMR and JMI as approximations to optimization problems based on conditional mutual information.
• The authors highlight open challenges such as scaling efficiency, refining MI-classification error bounds, and integrating causal discovery in feature selection.

A Review of Feature Selection Methods Based on Mutual Information

Jorge R. Vergara and Pablo A. Estévez present a comprehensive review of feature selection methods, particularly those based on mutual information (MI). The paper is pivotal for researchers engaged in the development of machine learning models, offering an overview of the advancements in information-theoretic criteria for feature selection over the past two decades.

Overview

The paper begins by clearly defining the problem of feature selection, emphasizing the goal of selecting the smallest subset of features that achieves a specified generalization error. The review covers three main groups of feature selection methods: wrappers, embedded methods, and filters. The focus is on filter methods that utilize MI for their relative robustness against overfitting and computational efficiency.

Definitions and Background

The authors provide essential definitions of mutual information, entropy, and conditional mutual information, emphasizing their relevance to feature selection. MI stands out for its capacity to measure any kind of relationship between random variables, including nonlinear ones, and its invariance under invertible transformations.

Evolution of Feature Selection Methods

Vergara and Estévez trace the development of MI-based feature selection methods from Battiti’s pioneering work, which introduced greedy selection to control the trade-off between relevancy and redundancy. The paper explores various criteria for feature evaluation, from single features to subsets, listing several prominent methods such as mRMR and CMIM.

Relevance, Redundancy, and Complementarity

A significant portion of the paper is devoted to discussing the properties of features, categorized into relevant, redundant, and complementary. The authors critique existing definitions of relevance, pointing out limitations such as the curse of dimensionality and the overly restrictive nature of strong relevance criteria. They advocate for more nuanced definitions that account for weakly relevant but non-redundant features, with an in-depth discussion on the Markov blanket and total correlation as measures of redundancy.

Optimal Feature Subset

The paper reviews criteria for defining the optimal feature subset. It examines sufficient feature subsets that preserve the information about the class variable and connects this definition with practical search strategies like sequential forward selection (SFS) and sequential backward elimination (SBE). Notably, the authors highlight that the quest for optimality is constrained by the need to efficiently estimate probability distributions or mutual information in high-dimensional spaces.

Unified Framework for MI Feature Selection

Vergara and Estévez review and extend the unifying framework proposed by Brown et al., which reinterprets common heuristic feature selection methods as low-order approximations to an optimization problem based on conditional likelihood or conditional mutual information. The authors derive and analyze several well-known methods such as MIFS, mRMR, and JMI under this framework, providing new insights into their limitations and theoretical underpinnings.

Open Problems and Future Directions

The paper identifies several open challenges in the field:

1. Further Development of Unifying Frameworks: Enhancing the connection between heuristic criteria and principled approaches like the Markov blanket.
2. Scaling Efficiency: Addressing computational challenges posed by high-dimensional datasets.
3. Mutual Information and Classification Error: Establishing tighter bounds and relationships between MI and Bayes error rates for feature subsets.
4. Finite Sample Effects: Understanding how finite data samples impact statistical criteria and MI estimation.
5. Causal Discovery: Investigating the interplay between feature selection and causal inference.
6. New Measures of Dependence: Exploring alternatives to MI that address its estimation and normalization challenges.

Implications and Conclusion

The review by Vergara and Estévez is a crucial resource for researchers aiming to improve feature selection methodologies in machine learning. By systematically organizing the state-of-the-art and identifying key directions for future research, the paper not only enhances our theoretical understanding but also drives practical innovations in handling complex, high-dimensional datasets. As challenges in information-theoretic feature selection are addressed, we can expect advances in model accuracy, efficiency, and interpretability, contributing to robust machine learning systems.