---
title: Minimum Redundancy Maximum Relevance
url: https://www.emergentmind.com/topics/minimum-redundancy-maximum-relevance-mrmr-51754298-6844-4751-861c-3c6e4a93925b
type: topic
---

# Minimum Redundancy Maximum Relevance

Minimum Redundancy Maximum Relevance (mRMR) is a foundational information-theoretic approach for feature selection that aims to identify compact subsets of features which maximize predictive relevance for a target variable while simultaneously minimizing redundancy among themselves. mRMR is widely used across machine learning domains, prominently in genomics, biomedical data analysis, remote sensing, functional data analysis, high-dimensional benchmarking, and interpretable model composition. Its operational core is jointly maximizing the mutual information between selected features and the response (maximum relevance), while penalizing or minimizing the aggregate mutual information among selected features (minimum redundancy).

## 1. Mathematical Formulation and Selection Objective

mRMR formalizes feature selection via the simultaneous optimization of two criteria over subsets $S$ of $m$ features from a candidate pool $F$:

- **Relevance**:
  \[
  D(S) = \frac{1}{|S|}\sum_{f_i\in S} I(f_i; c)
  \]
  where $I(f_i; c)$ is the mutual information between feature $f_i$ and the class or target variable $c$.
  
- **Redundancy**:
  \[
  R(S) = \frac{1}{|S|^2}\sum_{f_i,f_j\in S} I(f_i; f_j)
  \]
  quantifying the average pairwise mutual information among features in $S$.

mRMR then optimizes a scalar objective over candidate sets $S$:
\[
S^* = \operatorname{arg\,max}_{S, |S|=m}\left[D(S) - R(S)\right]
\]
alternatively, some variants employ the quotient $D(S)/R(S)$ to yield the "mutual information quotient" (MIQ) [2403.19014, 2605.25773].

For computational tractability, mRMR is generally implemented via a sequential forward-selection strategy. At each step, the next feature $f$ to add to $S$ is chosen by:
\[
\Delta(f) = I(f; c) - \frac{1}{|S|}\sum_{g\in S}I(f; g)
\]
and $f^* = \operatorname{arg\,max}_f \Delta(f)$ is greedily added to $S$ [2403.19014, 2308.09791, 2603.28417, 2210.16496].

## 2. Algorithmic Realizations, Scalability, and Extensions

### Greedy Forward Selection

The canonical procedure iteratively builds $S$ via:

1. Initialization: $S=\emptyset$.
2. Add the unselected feature $f$ with the highest $I(f; c)$.
3. At each iteration, select $f^*$ maximizing $I(f; c) - (1/|S|)\sum_{g\in S}I(f; g)$.
4. Repeat until $|S|=m$.

This linear-in-features, quadratic-in-selected-size process is tractable for moderate $|F|$ but becomes bottlenecked in ultra-high-dimensional settings due to $O(n^2)$ mutual information computations [2308.09791, 1709.02327, 2208.09901].

### Distributed and Scalable Implementations

To address high dimensionality, distributed mRMR variants leverage MapReduce and Spark primitives—either via row-wise layouts (efficient for tall datasets) or column-wise/broadcast layouts (for wide/short applications) [1709.02327, 2208.09901]. These implementations cache entropy and MI computations, enable partitioned aggregation, and reduce communication overhead, providing dramatic runtime reductions (up to 97% and 4–6× speedups compared to naïve versions).

### Non-convex, Penalized, and Global Optima

Recent innovations include the continuous penalized mRMR (SmRMR), which solves a convex or nonconvex regularized minimization of an mRMR-inspired loss (incorporating, e.g., SCAD or MCP penalties for sparsity) [2508.18901]. Additionally, polyhedral relaxations yield provably optimal mixed-integer linear programming (MILP) formulations, enabling globally optimal mRMR feature sets for hundreds of features [2508.16123].

## 3. Mutual Information Estimation and Alternative Association Measures

Estimation of $I(X; Y)$ is data- and variable-type dependent:

- **Discrete variables**: Use empirical plug-in/histogram estimates.
- **Continuous variables**: Discretize into bins or use k-nearest-neighbor density estimators (KSG estimator, PCA-corrected KSG for continuous responses) [2605.25773, 2006.12919].
- **Alternative measures**: Distance correlation and related association statistics substitute for MI in some contexts (e.g., functional data analysis), offering tuning-free, smoothing-free, and nonlinear dependence capture [1507.03496, 2006.12919]. Such variants can achieve higher accuracy and select fewer features than MI-based mRMR.

## 4. Integration with Hybrid and Wrapper Methods

mRMR is often employed as a filter stage, followed by or in combination with wrapper-based (model-dependent) feature selection:

- **Hybrid frameworks**: Combine mRMR with classifier-guided elimination (e.g., SVM-RFE). A convex combination of SVM weights and mRMR scores boosts predictive accuracy and yields more stable, interpretable subsets [2404.12610].
- **Metaheuristic wrappers**: mRMR is paired with population-based optimizers (e.g., Binary Horse Herd Optimization) to restrict the wrapper search space, achieving efficient gene selection and improved accuracy [2308.09791].
- **Multi-stage selection**: Two-stage or staged filter methods (e.g., pre-pruning with maximum information gain, followed by mRMR, then a wrapper) significantly reduce computational cost and redundancy [2210.16496].

## 5. Applications and Empirical Performance

mRMR has been applied across domains:

- **Biomedical and genomics**: mRMR underpins robust gene selection with quantifiable improvements in SVM/RF accuracy and sharp reductions in dimensionality [2308.09791, 2210.16496].
- **Emotion and signal recognition**: In VR-based emotion recognition, mRMR reduces a 175-feature pupillometry representation to a critical 50-dimensional embedding, increasing classification accuracy from 85% to 98.8% [2403.19014].
- **Benchmarking LLMs**: In LLM evaluation, mRMR-selected question subsets minimize RMSE and maximize rank correlation (Kendall's $\tau$, Spearman's $\rho$), outperforming AnchorPoints and IRT-based approaches and yielding much higher stability across random seeds [2605.25773].
- **Remote sensing**: For hyperspectral imaging, mRMR, as part of a hybrid feature selection pipeline, achieves high accuracy with far fewer bands compared to information-gain alone or simple univariate filters [2210.16496].

**Empirical findings**:

- mRMR-based feature sets often reach or exceed the performance of full feature sets, with a much smaller subset (e.g., 7–11 features achieving better accuracy than 33 in power system transient stability assessment [1903.01907]; 14 selected from >11,000 radiomics descriptors maintaining cross-vendor AUC for SVM/RF models [2407.18060]).
- In large-scale benchmarking and biomedical applications, mRMR and its scalable variants enable feasible, reproducible large-$n$, large-$p$ analyses with state-of-the-art classification and regression performance [2605.25773, 2212.06143].
- New univariate clustering-based variants, such as KGroups, approximate classical mRMR's performance while being two to three orders of magnitude faster, facilitating hyperparameter tuning and rapid prototyping [2603.28417].

## 6. Limitations, Modifications, and Recent Advances

### Limitations

- **Computational cost**: Classical mRMR is $O(n^2)$ and infeasible for very large $n$ without distributed or approximate computation [2208.09901].
- **Pairwise heuristics**: True joint mutual information with the target is not optimized—greedy, pairwise reduction may miss higher-order, synergetic dependencies [2212.06143].
- **Estimation sensitivity**: MI estimation for continuous data is subject to binning choice, kernel bandwidths, and sample size constraints [2404.12610].

### Modifications

- **Tradeoff tuning**: Weighted/formulated improvements (e.g., using a trade-off parameter $\alpha$ in $J_\alpha(S) = \alpha\, D(S, c) - (1-\alpha)\, R(S)$ [1903.01907]), enable fine-grained balancing of relevance and redundancy.
- **Augmentation with unique relevance**: MRwMR-BUR integrates a "unique relevance" (UR) term---the conditional MI given all other features---to further emphasize non-redundant, indispensable predictors. This yields consistently smaller feature sets and 2–5% accuracy gains [2212.06143].
- **Penalized and FDR-controlled**: SmRMR applies nonconvex penalties (SCAD, MCP) and interfaces with model-X knockoff filtering to achieve feature selection with statistical false discovery rate control, supporting both theoretical guarantees and empirical competitiveness with HSIC-LASSO [2508.18901].

### New Directions

- MILP-based global optimization of the mRMR criterion enables provably optimal feature selection for moderately large sets ($m \leq 800$) [2508.16123].
- Distance correlation and kernel-based association statistics can replace MI to yield tuning-free, smoothing-free, and unbiased estimators, especially effective for functional and highly correlated data [1507.03496, 2006.12919].

## 7. Summary Table: Core mRMR Objective and Key Formulas

| Component                  | Mathematical Expression                                                                                              | Key Property                                                   |
|----------------------------|----------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------|
| Relevance                  | $D(S) = \frac{1}{|S|}\sum_{f_i \in S} I(f_i; c)$                                                                    | MI between feature and target                                 |
| Redundancy                 | $R(S) = \frac{1}{|S|^2}\sum_{f_i,f_j\in S} I(f_i; f_j)$                                                             | Average MI among features                                     |
| Difference criterion       | $S^* = \operatorname{arg\,max}_S [D(S) - R(S)]$                                                                     | Classical "MID" scoring                                       |
| Greedy update (new $f$)    | $\Delta(f) = I(f; c) - \frac{1}{|S|}\sum_{g\in S}I(f; g)$                                                           | One-step incremental update                                   |
| Quotient criterion         | $S^* = \operatorname{arg\,max}_S [D(S)/R(S)]$                                                                       | Alternative to "MID"                                          |
| Penalized SmRMR objective  | $\mathcal{L}(\theta)=\frac12\,D_v(Y, Y) - \sum_k\theta_k D_v(X_k, Y) + \frac12 \sum_{k,\ell}\theta_k\theta_\ell D_v(X_k, X_\ell) + \sum_k (\lambda_n, \theta_k)$ | Penalized estimation (continuous weights)                     |
| Unique relevance (UR, BUR) | $\mathrm{UR}(X_k) = I(X_k; Y \mid \Omega\setminus{\{X_k\}})$                                                        | Conditional MI given all other features [2212.06143]          |

## References

- [2403.19014] Thelxinoë: Recognizing Human Emotions Using Pupillometry and Machine Learning
- [2308.09791] Efficient High-Dimensional Gene Selection Based on Binary Horse Herd Optimization Algorithm
- [1903.01907] Feature Selection for Transient Stability Assessment Based on Improved Maximal Relevance and Minimal Redundancy Criterion
- [2605.25773] Efficient Benchmarking Is Just Feature Selection and Multiple Regression
- [1709.02327] Feature Selection in High-Dimensional Dataset Using MapReduce
- [2208.09901] Scalable mRMR Feature Selection to Handle High Dimensional Datasets
- [1507.03496] The mRMR Variable Selection Method: A Comparative Study for Functional Data
- [2404.12610] Corporate Financial Distress Prediction: Based on Multi-source Data and Feature Selection 
- [2212.06143] Improving Mutual Information Based Feature Selection by Boosting Unique Relevance
- [2603.28417] KGroups: A Versatile Univariate Max-Relevance Min-Redundancy Feature Selection Algorithm for High-dimensional Biological Data
- [2508.16123] Optimal Data Reduction Under Information-Theoretic Criteria
- [2508.18901] Sparse Minimum Redundancy Maximum Relevance for Feature Selection
- [2210.16496] Hybridization of Filter and Wrapper Approaches for the Dimensionality Reduction and Classification of Hyperspectral Images
- [2006.12919] Distance Correlation Sure Independence Screening for Accelerated Feature Selection in Parkinson's Disease Vocal Data
- [2112.10369] Feature Selection for Efficient Local-to-Global Bayesian Network Structure Learning
- [1911.06193] Predicting Indian Stock Market Using the Psycho-linguistic Features of Financial News
- [2407.18060] Cross-Vendor Reproducibility of Radiomics-Based Machine Learning Models for Computer-aided Diagnosis

## Conclusion

mRMR provides a theoretically principled, empirically validated, and highly extensible framework for feature selection in high-dimensional learning, unifying mutual-information-based relevance with redundancy penalization. Its modern developments—distributed implementation, penalized relaxation, alternative association measures, and unique-relevance boosting—address computational, inferential, and practical limitations, ensuring ongoing relevance for large-data, multi-source, and interpretable modeling scenarios.

Source: https://www.emergentmind.com/topics/minimum-redundancy-maximum-relevance-mrmr-51754298-6844-4751-861c-3c6e4a93925b