Mean False Error (MFE)

• Mean false error (MFE) is the expected number of false rejections in a family of tests, providing a clear average-case error measure in large-scale experiments.
• The Bonferroni procedure controls PFER by setting a stringent threshold on p-values, ensuring nonparametric control even under dependence.
• MFE is closely related to metrics like PCER, FWER, and FDR, with its stability and low variance offering practical advantages in high-dimensional testing settings.

The mean false error (MFE), also known as the per-family error rate (PFER), is a fundamental metric in multiple hypothesis testing. It quantifies the expected number of false rejections—incorrectly rejecting true null hypotheses—within a family of simultaneous tests. MFE is formally defined as the expected value of the number of false discoveries, providing a clear average-case measure of error in large-scale testing settings, such as those found in genomics and microarray experiments (0709.0366).

1. Formal Definition and Notation

Let $m$ denote the total number of hypotheses tested, $m_0$ the true nulls, $V$ the number of false rejections, and $R$ the total number of rejections. The mean false error (MFE) is defined as

$\text{PFER} = \mathbb{E}[V].$

This expectation is taken over the joint distribution of test statistics or $p$-values. The terminology "mean false error" (MFE) is entirely synonymous with PFER in this context (0709.0366).

2. Bonferroni Procedure for PFER Control

The Bonferroni procedure provides a direct approach to controlling the PFER at a user-specified level $\alpha$. It rejects hypothesis $i$ if its $p$-value satisfies $P_i \leq \alpha/m$. Without any assumptions on the dependence structure among $p$-values, the following bound holds:

$$\mathbb{E}[V] = \sum_{i \in T} \mathrm{Pr}\{\text{reject}\ H_i\} \leq m_0 \cdot (\alpha/m) \leq \alpha.$$

If all true null $p$-values are uniformly distributed, $\mathrm{Pr}\{P_i \leq \alpha/m\} = \alpha/m$, leading to

$$\text{PFER} = \mathbb{E}[V] = (m_0/m)\alpha \leq \alpha.$$

This property illustrates that the Bonferroni rule offers strong and uniform control of the expected number of false discoveries, regardless of dependence among $p$-values (0709.0366).

3. Relationship to Other Error Metrics

The MFE/PFER is interconnected with several other widely used error rates:

Metric	Definition	Relationship to PFER
PCER (Per-comparison)	$\mathbb{E}[V]/m$	$\text{PFER} = m \cdot \text{PCER}$
FWER (Family-wise)	$\mathrm{Pr}\{V \geq 1\}$	$\text{FWER} \leq \text{PFER}$ (by Markov)
FDR (False discovery)	$\mathbb{E}[V/R]$	FDR is the expectation of the proportion, not the mean number

FWER controls the probability of any false discovery, while PFER controls the expected number, often resulting in numerically close but conceptually distinct interpretations, especially when expected error counts are low. FDR, in contrast, is concerned with the average proportion of false discoveries among all rejections (0709.0366).

4. Variance and Stability Characteristics

The Bonferroni rule not only tightly bounds the expected number of false rejections but also confers superior stability properties:

• For Bonferroni/PFER, $\mathrm{Var}(V) \leq \mathbb{E}[V]$ as $V$ is a sum of indicators.
• Simulation studies demonstrate that, for matched mean power, the Bonferroni rule yields substantially smaller standard deviation in both the number of true discoveries ($\mathrm{SD}(V)$) and the total number of rejections ($\mathrm{SD}(R)$) relative to the Benjamini–Hochberg (BH) method.
• The variability of both $V$ and $R$ increases significantly under moderate pairwise correlation among $p$-values (e.g., $\rho=0.4$), but Bonferroni remains more stable than BH, particularly in high-dimensional settings such as microarray analysis (0709.0366).

5. Impact of Dependence Among p-values

Bonferroni/PFER control is nonparametric: the extremal bound $\mathbb{E}[V] \leq (m_0/m) y$ holds for any dependence structure among $p$-values. In contrast, the BH/FDR procedure requires the positive regression dependence on a subset (PRDS) property for rigorous FDR control; with arbitrary dependence, FDR bounds must be relaxed by a harmonic factor. This distinction is critical in applied settings where correlations among test statistics are the rule rather than the exception (0709.0366).

6. Simulation-based Assessment and Empirical Results

In simulations using $m=1,255$ hypotheses and both independent and exchangeably correlated $p$-values, two critical experimental designs were considered:

• "Equalized FDR": Adjusting $\alpha_{\text{Bon}}$ and $q_{\text{BH}}$ so that their true FDRs coincide, then comparing true discoveries and their standard deviations.
• "Equalized PFER": Matching PFER values and comparing performance.

Findings included near-identical mean power between Bonferroni and BH procedures when FDR is matched, but markedly smaller $\mathrm{SD}(V)$ and $\mathrm{SD}(R)$ for Bonferroni, indicating less variability and greater stability. Scatterplots of $R_{\text{Bonf}}$ vs. $R_{\text{BH}}$ showed high correlation (>0.98), yet Bonferroni exhibited lower variability. These results underscore the consistent stability advantage of the Bonferroni rule, particularly as dependence increases (0709.0366).

7. Practical Recommendations in Large-Scale Testing

• In large-scale settings, such as microarray studies, where the permissible average number of false positives per experiment is known (e.g., $y \approx 1$–5 for $10^4$ genes), the Bonferroni procedure with threshold $y/m$ ensures $\mathbb{E}[V] \leq y$.
• If the bound is required on the proportion of false positives, the BH method at level $q$ is appropriate, though it confers more variability, especially under dependence.
• Bonferroni/PFER is frequently simpler to communicate (“on average we get at most $y$ false genes”) and its nonparametric control is robust to any dependence structure.
• For optimization of stability, it is feasible to scan a grid of $y$ values and empirically assess $\mathrm{SD}(R)$ or $\mathrm{SD}(V)$ (e.g., via permutation or bootstrap), choosing a threshold corresponding to minimal variance.
• In domains valuing reproducibility, such as genomics, a small PFER (e.g., 1 or 2) can provide a balance between statistical power and stability (0709.0366).