Mean Decrease Accuracy (MDA) Explained

• MDA is a method that quantifies variable importance by assessing prediction accuracy loss after permuting each predictor in models like random forests.
• It is also applied in photonic-assisted frequency estimation, where averaging errors across multiple Nyquist zones significantly reduces quantization deviations.
• Advancements such as Sobol-MDA offer consistent importance rankings and efficient computation even in high-dimensional, correlated settings.

Mean Decrease Accuracy (MDA) denotes a family of accuracy- or error-based statistics widely used for quantifying variable importance in random forests and for frequency estimation precision improvement in photonic-assisted measurement systems. The term encompasses distinct methodologies in disparate domains—most prominently, variable importance orderings in machine learning via permutation-based loss comparison, and precision enhancement in spectral analysis via multiorder error averaging. MDA's implementation details, interpretation, and statistical properties can therefore differ substantially depending on context.

1. MDA in Random Forests: Definition and Implementation Variants

The original conception of MDA in random forests attributes importance to each covariate by evaluating the reduction in predictive accuracy upon permuting (or otherwise noising) the variable under consideration. In regression, this typically involves measuring the increase in mean squared error; in classification, it reflects the rise in misclassification rate. Canonical implementations include:

• Train/Test MDA: Deploys independent (holdout) test sets, contrasting error rates pre- and post-permutation using

$$\widehat{\textrm{MDA}^{(TT)}(X^{(j)})} = \frac{1}{n} \sum_{i=1}^n \big\{ (Y'_i - m_{M,n}(X'_i))^2 - (Y'_i - m_{M,n}(X'_{i,\pi_j}))^2 \big\}$$

• Breiman–Cutler MDA (BC-MDA): Leverages out‐of‐bag (OOB) samples per tree, computing the difference in their prediction errors before and after permuting the covariate within those samples.
• Ishwaran–Kogalur MDA (IK-MDA): Aggregates the OOB error over the entire forest before and after permutation and requires the number of trees to grow with sample size.

Distinct mainstream software platforms such as randomForest (R), ranger, randomForestSRC, and scikit-learn implement these variants (or their normalized forms), so "MDA" as reported in practical studies can refer to non-identical operationalizations (Bénard et al., 2021). This diversity is consequential: distinct implementations have non-equivalent statistical properties in the large-sample limit.

2. Asymptotic Behavior and Statistical Properties

Rigorous analysis demonstrates that permutation-based MDA formulations, while conceptually related, do not converge to a unified "variable importance" functional as $n \to \infty$—even assuming consistency of the underlying random forests (Bénard et al., 2021). Specifically:

• Train/Test and BC-MDA:

$$\widehat{\textrm{MDA}^{(TT)}(X^{(j)})} \xrightarrow{L^1} \mathbb{E}\big[(m(X) - m(X_{(\pi_j)}))^2\big]$$

where $m(X_{(\pi_j)})$ is the predicted value with $X^{(j)}$ permuted.

• IK-MDA:

$$\widehat{\textrm{MDA}^{(IK)}(X^{(j)})} \xrightarrow{L^1} \mathbb{E}\big[(m(X) - \mathbb{E}[m(X) \mid X^{(-j)}])^2\big]$$

Although originally motivated as measures of the unique predictive role of each covariate, these limits instead sum multiple contributions, not all of which reflect the isolated effect of the variable.

3. Decomposition and the Problem of Covariate Dependence

The theoretical limit of most permutation-based MDA measures decomposes into three additive nonnegative terms (Bénard et al., 2021):

Term	Mathematical Form	Interpretation
$MDA_1^{\star (j)}$	$V[Y] \cdot ST^{(j)}$	Total Sobol index; explained variance uniquely due to $X^{(j)}$
$MDA_2^{\star (j)}$	$V[Y] \cdot ST_{mg}^{(j)}$	Marginal total Sobol; ignores conditional dependence
$MDA_3^{\star (j)}$	$E[(E[m(X)|X^{(-j)}] - E[m(X_{\pi_j})|X^{(-j)}])^2]$	Contribution from covariate dependence

The dependence structure among covariates directly affects $MDA_2^{\star (j)}$ and $MDA_3^{\star (j)}$. In highly correlated settings, the third term, absent in classical variance-based importance models, may become dominant. This can yield “inflated” apparent importance for variables with redundant or minimal true explanatory contribution, especially at high correlation coefficients (e.g., $\rho > \frac{\sqrt{2}}{2}$). Consequently, permutation-based MDA fails to consistently target the true variable influence in the presence of dependence.

4. Sobol-MDA: Consistent Importance in the Correlated Case

To address these limitations, Sobol-MDA measures variable importance by directly estimating the total Sobol index—specifically, the expected decrease in explained variance upon "removing" a variable. Rather than relying on permutation to break associations, Sobol-MDA projects the forest’s structure onto the subspace orthogonal to $X^{(j)}$: for each sample, tree traversal “forks” at splits on $X^{(j)}$, sending the observation to both child nodes and averaging terminal predictions. The projected terminal cells, $A_n^{(-j)}(X^{(-j)},\Theta)$, thus span only the remaining predictors.

The out-of-bag (OOB) projected estimate becomes:

$$m_{M,n}^{(-j,\mathrm{OOB})}(X_i^{(-j)}) = \frac{1}{|\Lambda_{n,i}|} \sum_{\ell \in \Lambda_{n,i}} m_n^{(-j)}(X_i^{(-j)},\Theta_{\ell})$$

and the normalized Sobol-MDA is:

$$\widehat{\textrm{S-MDA}_{M,n}(X^{(j)})} = \frac{1}{\hat{\sigma}_Y^2}\frac{1}{n}\sum_{i=1}^n \big\{(Y_i - m_{M,n}^{(-j,\mathrm{OOB})}(X_i^{(-j)}))^2 - (Y_i - m_{M,n}^{(\mathrm{OOB})}(X_i))^2\big\}$$

This estimator is consistent, converging in probability to the desired total Sobol index (i.e., only $MDA_1^{\star (j)}$), regardless of predictor dependencies. The methodology extends the “projected-CART” paradigm, imposing only mild regularity constraints.

5. Empirical Comparison and Computational Considerations

Systematic experiments on both synthetic and real datasets confirm that classical MDA variants (BC-MDA, IK-MDA, brute-force retraining) can substantially misestimate importance under correlated features, while Sobol-MDA yields rankings faithfully matching true Sobol index–based ground truth (Bénard et al., 2021). In challenging scenarios (e.g., regression with strong interaction and correlation among inputs), only Sobol-MDA and exhaustive retraining approaches produce reliable importance rankings. However, retraining introduces greater statistical variance and computational overhead.

Sobol-MDA’s computational complexity is $O(M n\, \log^3 n)$ (with $M$ trees and $n$ samples), nearly linear in data and effectively independent of the predictor count $p$, substantially outperforming $O(pM n\, \log^3 n)$ brute-force methods in high dimensions. Open-source R and C++ implementations over “ranger” are available.

6. MDA in Photonic-Assisted Frequency Estimation

A distinct application of Multiorder Deviation Average (MDA) arises in frequency estimation precision improvement via photonic-assisted presampling (Gao et al., 2019). In this context, MDA refers to a technique for reducing quantization- and rounding-induced measurement deviation in FFT-based frequency detection:

• The input signal is represented as $f_{in} = (m + \delta) f_{res}$, with $f_{res}$ the FFT frequency resolution.
• Frequency measurement through FFT introduces a potential rounding error up to $\pm 0.5 f_{res}$.
• Photonic presampling spreads the signal across multiple Nyquist zones, generating replicated spectrally-offset measurements.
• For each zone indexed by $n$, the measured frequency $$f_{in}^{(n)} = [m + \delta - n(a + \delta_c)] f_{res}$$ incurs a deviation $$\Delta f_n = ([rmod(\delta - n\delta_c)] - rmod(\delta - n\delta_c)) f_{res}$$.
• The MDA technique averages these deviations across $N$ zones:

$$f_{avg} = \frac{1}{N} \sum_n f_{in}^{(n)} = f_{in} + \Delta f_{avg}$$

where

$$\Delta f_{avg} = \frac{1}{N} \sum_n ([rmod(\delta - n\delta_c)] - rmod(\delta - n\delta_c)) f_{res}$$

• This averaging cancels out irregular rounding errors, yielding a deviation many times smaller than any individual $\Delta f_n$. For instance, the maximum deviation may be reduced tenfold, and, when combined with DSP refinements, root mean squared error reductions by factors above 800 have been demonstrated.

This photonic-assisted MDA is compatible with FFT-based digital estimation algorithms and is suitable for ultra-wideband, high-stability applications such as radar, LIDAR, and spectrum sensing. By spatially distributing FFT errors and averaging, the approach mitigates both spectrum leakage and the "picket fence" effect without introducing algorithmic complexity.

7. Summary Table of MDA Interpretations

Context	Mechanism	Targeted Quantity
Random Forests	Permutation or projection (Sobol-MDA)	Mean loss increase or Sobol index
Photonic-Assisted Sensing	Multi-Nyquist averaging (MDA)	Reduced frequency estimation error

MDA thus represents variable- or error-importance measures tailored to their domain: as a permutation-based association metric in random forests, and as a multiorder error suppression mechanism in photonic sampling systems. In both cases, domain-specific enhancements—such as Sobol-MDA for correlated predictors, or multiorder averaging in presampled signals—substantially ameliorate the limitations of first-generation MDA estimates.