---
title: Weighted and Penalized mRMR
url: https://www.emergentmind.com/topics/weighted-and-penalized-mrmr
type: topic
---

# Weighted and Penalized mRMR

Weighted and penalized minimum Redundancy Maximum Relevance (mRMR), formalized in the SmRMR framework, denotes a class of feature selection methodologies that unifies continuous, weight-based redundancy-aware variable screening with sparsity-inducing nonconvex penalization, and formal false discovery rate (FDR) control through a multi-stage knockoff procedure. SmRMR is motivated by the need for scalable, model-free selection of relevant features in ultra-high-dimensional datasets, particularly where classical mRMR is computationally prohibitive and unable to provide explicit statistical error control [2508.18901].

## 1. Continuous Weighted mRMR Objective

SmRMR generalizes the classic discrete mRMR approach by introducing a continuous optimization framework with feature-wise nonnegative weights. Let $X_1,\ldots,X_p$ denote $p$ features and $Y$ the target. Define an association measure $D(\cdot,\cdot)$ that is nonnegative and zero iff the arguments are independent (e.g., Hilbert-Schmidt Independence Criterion (HSIC) or Projection Correlation). The model constructs two matrices:
- $R_{yX} \in \mathbb{R}^p$ with $(R_{yX})_j = D(X_j, Y)$ (feature-response associations)
- $R_{XX} \in \mathbb{R}^{p \times p}$ with $(R_{XX})_{j,k} = D(X_j, X_k)$ (feature-feature associations)

The continuous, unpenalized mRMR objective is
$$
\max_{w \geq 0} Q(w) := w^\top R_{yX} - \frac{1}{2} w^\top R_{XX} w,
$$
which corresponds to minimizing
$$
\mathcal{L}_0(w) := -w^\top R_{yX} + \frac{1}{2} w^\top R_{XX} w.
$$
Feature weights $w = (w_1,\ldots,w_p)^\top \in \mathbb{R}_+^p$ directly capture each variable's overall redundancy-aware utility.

## 2. Nonconvex Penalization and Sparse Optimization

To induce sparsity (i.e., automatic exclusion of inactive features), SmRMR augments $\mathcal{L}_0(w)$ with a componentwise penalty:
$$
\min_{w \geq 0} \mathcal{L}(w) := \mathcal{L}_0(w) + \sum_{j=1}^p P_\lambda(w_j),
$$
where $P_\lambda$ is a nonconvex regularizer such as:
- **SCAD** (Smoothly Clipped Absolute Deviation; param $a>2$)
- **MCP** (Minimax Concave Penalty; param $b>0$)

The explicit forms are:

| Penalty | $P_\lambda(x)$ | Derivative $\partial P_\lambda(x)$ |
|---------|----------------|------------------------------------|
| SCAD    | $\lambda x$, $0\leq x\leq\lambda$<br>$( -x^2 + 2a\lambda x - \lambda^2 )/2(a-1)$, $\lambda < x \leq a\lambda$<br>$(a+1)\lambda^2/2$, $x>a\lambda$ | $\lambda\,\mathbf{1}_{x\leq\lambda} + \left( (a\lambda - x)_+ / (a-1)\right) \mathbf{1}_{x>\lambda}$ |
| MCP     | $\lambda x - x^2/(2b)$, $0\leq x\leq b\lambda$<br>$b\lambda^2/2$, $x>b\lambda$              | $(\lambda - x/b)_+$                                  |

Asymptotically, $a\rightarrow\infty$ or $b\rightarrow\infty$ recovers LASSO. Nonconvex penalties sharpen zeroing of truly inactive coefficients, with the sparsity pattern in $w$ providing the support of relevant features.

## 3. Numerical Algorithm: Local Linear Approximation

SmRMR leverages the Local Linear Approximation (LLA) algorithm to address the nonconvexity of SCAD/MCP:
1. Initialize $w^{(0)}$ by solving the nonnegative LASSO (i.e., $P_\lambda(x)=\lambda x$).
2. Iterate for $s=1,\dots,M$:
    - (a) Compute $v_j = \partial P_\lambda(w_j^{(s-1)})$,
    - (b) Solve the convex weighted-LASSO problem $\min_{w\geq 0} \mathcal{L}_0(w) + \sum_j v_j w_j$ (via coordinate descent or QP),
    - (c) If $\|w^{(s)}-w^{(s-1)}\|_2 < \epsilon$, terminate.
3. Return $w^{(s)}$.

Typically, $M=2$ suffices for convergence. This iterative reweighting adapts the penalty based on the current coefficient estimates, promoting sharper recovery of the feature support.

## 4. Multi-Stage Knockoff+ Filter for FDR Control

SmRMR integrates a two-stage knockoff pipeline to ensure explicit control of the false discovery rate at a user-specified level $\alpha$:
**Stage 1: Screening.**
- Randomly split the data $(n_0, n_1)$.
- Apply penalized mRMR to $n_0$ samples; retain a working set $S_0$ of size $s_0$ with $2s_0 < n_1$.

**Stage 2: Knockoff+ Selection.**
- Generate model-X knockoff variables $\widetilde X_j$ for $j \in S_0$ (e.g., using the equi-correlated construction).
- On $n_1$ samples, solve penalized mRMR over all $X_{S_0}$ and $\widetilde X_{S_0}$, yielding weights $\widehat w_j$, $\widetilde w_j$.
- Compute $W_j = \widehat w_j - \widetilde w_j$.
- Define the knockoff threshold at FDR level $\alpha$:
$$
T(\alpha) = \min\left\{ t>0: \frac{1 + \#\{j: W_j\leq -t\}}{\max\{1,\#\{j:W_j\geq t\}\}} \leq \alpha \right\}
$$
- Select $\widehat S(\alpha) = \{ j \in S_0 : W_j \geq T(\alpha)\}$. Theoretical guarantees (Thm 3.3) ensure $E[\mathrm{FDP}\mid\text{all true features}\in S_0] \leq \alpha$.

## 5. Oracle Screening Properties and Theoretical Guarantees

Consider a sequence of problems with $p_n$ features, true active set size $s_n$, and oracle pseudo-truth $\theta_n^0$ (with $\theta_{n,j}^0\neq 0$ for $j\leq s_n$, $=0$ otherwise). Key assumptions include uniqueness of $\theta_n^0$, bounded eigenvalues of $R_{XX}$, and regularity of the penalty:
- $a_n := \max_{j\in S_n}\partial_2 P_\lambda(\theta_{n,j}^0)=O(n^{-1/2})$
- $b_n := \max_{j\in S_n} \partial_{22}^2 P_\lambda(\theta_{n,j}^0)\to 0$
- Minimum signal: $\min_{j\in S_n} \theta_{n,j}^0/\lambda_n \to \infty$

Under these, Theorem 3.1 shows existence of a local minimizer $\hat\theta_n$ with
$$
\|\hat\theta_n - \theta_n^0\|_2 = O_p\left( \sqrt{p_n s_n^2\log p_n} (n^{-1/2} + a_n) \right).
$$
Theorem 3.2 (sparsistency) asserts that, for SCAD/MCP, if $\lambda_n\to 0$ and $\sqrt{n/(p_ns_n^2\log p_n)} \lambda_n \to \infty$,
$$
P\left( \hat\theta_{n,j}=0 \text{ for all } j>s_n \right) \to 1,
$$
i.e., all inactive features are correctly excluded as $n\to\infty$ [2508.18901]. This provides formal oracle-screening guarantees in high-dimensional regimes.

## 6. FDR Threshold Choice and Output Guarantees

The FDR level $\alpha$ is a direct control lever for the model complexity. Practical choices are $\alpha \in [0.2, 0.4]$; lower values yield sparser selections and lower FDP but may reduce power. If no variables are selected for a conservative $\alpha$, a variant “SmRMR$_2$” procedure increments $\alpha$ until at least one selection is made, trading a marginal FDR increase for guaranteed nonempty sets. Cross-validation for $\lambda_n$ is performed only during the knockoff stage, not at initial screening.

## 7. Empirical Comparisons and Implementation Considerations

SmRMR with SCAD/MCP matches the predictive accuracy of HSIC-LASSO while selecting $2$–$4\times$ fewer features and maintaining FDR below $\alpha$—a more conservative selection than HSIC-LASSO, which often has higher FDP. Classical discrete mRMR is impractical for large $p$ and less effective in redundancy control. On GWAS and high-dimensional biological datasets, SmRMR yields similar or better test performance than HSIC-LASSO with fewer selected SNPs/genes, simplifying interpretation.

Key implementation considerations:
- Each mRMR fit incurs $O(n^2p)$ complexity (kernel or $V$-statistic construction); block HSIC or projection correlation can accelerate screening.
- Knockoff requires $n_1>2s_0$, necessitating the two-stage split; data recycling may increase power.
- Projection correlation (PC$^2$) may be preferable over HSIC for heavy-tailed or non-Euclidean data.

Code is publicly available at https://github.com/PeterJackNaylor/SmRMR [2508.18901].

Source: https://www.emergentmind.com/topics/weighted-and-penalized-mrmr