---
title: Friedman & Nemenyi Tests Overview
url: https://www.emergentmind.com/topics/friedman-and-nemenyi-tests
type: topic
---

# Friedman & Nemenyi Tests Overview

The Friedman and Nemenyi tests are core nonparametric methodologies for analyzing and comparing multiple treatments or algorithms under a randomized complete block design. Their primary application is the statistical comparison of several methods across multiple data sets, with particular relevance in areas such as machine learning, psychology, and medicine. After an initial omnibus hypothesis test (Friedman), post-hoc analyses such as the Nemenyi mean-ranks test are commonly employed to determine sources of significant differences. However, the dependence of Nemenyi post-hoc inferences on the entire set of treatments and accompanying paradoxes have recently prompted scrutiny and recommendations for alternative pairwise procedures.

## 1. The Friedman Test: Omnibus Nonparametric Comparison

The Friedman test is used to detect differences among $m$ algorithms (treatments) evaluated on $N$ datasets (blocks). For each dataset $j$, the outcomes $X_{ij}$ ($i=1,...,m$) are ranked, yielding $R_{ij}$; average ranks replace raw performance scores, with ties handled via average-ranking. Each algorithm’s sum of ranks is $R_i = \sum_{j=1}^N R_{ij}$, with mean rank $\overline{R}_i = R_i/N$. The null hypothesis $H_0$ asserts “all $m$ algorithms perform equivalently,” that is, $X_{1j} \overset{d}{=} X_{2j} \overset{d}{=} \cdots \overset{d}{=} X_{mj}$ for all $j$.

The Friedman statistic is
$$
\chi^2_F = \frac{12N}{m(m+1)}\sum_{i=1}^m \overline{R}_i^2 - 3N(m+1).
$$
Alternatively, in terms of $R_i$,
$$
S = \frac{12}{N m (m+1)}\sum_{i=1}^m (R_i - N(m+1)/2)^2.
$$
For large $N$, $\chi^2_F$ is approximately $\chi^2$-distributed with $m-1$ degrees of freedom. In practice, the test serves as a robust, nonparametric alternative to repeated-measures ANOVA for arbitrary, not necessarily normal, data [1505.02288, 2202.09131].

## 2. Nemenyi Mean-Ranks Post-hoc Test

Upon rejection of the Friedman test’s omnibus null, the Nemenyi test is traditionally used for all $C = m(m-1)/2$ algorithm pairs. For any algorithms $i$, $j$, the mean-rank difference $|\overline{R}_i - \overline{R}_j|$ forms the test statistic, evaluated versus a critical difference (CD):
$$
CD = q_{\alpha; m, \infty} \sqrt{ \frac{m(m+1)}{6N} },
$$
where $q_{\alpha; m, \infty}$ is the upper $\alpha$–quantile of the Studentized range distribution with $m$ treatments and infinite degrees of freedom. Algorithms $i$ and $j$ are declared significantly different at family-wise level $\alpha$ if $|\overline{R}_i - \overline{R}_j| > CD$. This controls the family-wise error rate across all $m(m-1)/2$ comparisons and is operationally analogous to Tukey-Kramer procedures for parametric ANOVA [1505.02288].

## 3. Critique of the Mean-Ranks Test and Its Dependence on Algorithm Pool

A foundational critique, detailed by Benavoli, Corani, and Mangili (2016) [1505.02288], is that Nemenyi’s mean-ranks test produces decisions for any pair $(A, B)$ contingent on the presence, absence, and relative ordering of all other algorithms involved in the ranking. This can lead to paradoxical scenarios:

- In an experiment where $A$ and $B$ each win on half the cases, two-algorithm tests (sign, Wilcoxon, t-test) yield nonsignificance. However, introducing additional poor-performing algorithms can inflate $|\overline{R}_A-\overline{R}_B|$ sufficiently to exceed $CD$ and declare a significant difference.
- On real-world datasets, the decision on a given pair can flip between significant and non-significant solely due to the composition of the algorithm pool. For example, in UCI data, algorithm pair $C_2$ vs $C_4$ was shown to change significance status depending on which other classifiers were included [1505.02288, Table 4].

This pool-dependence means mean-ranks tests cannot guarantee control of maximum Type I error when equivalent algorithms are present, as also discussed by Fligner & Killeen (1984).

## 4. Alternative Two-Algorithm Post-hoc Procedures

To address the pool-dependence flaw, tests evaluating only the paired performances of $A$ and $B$ are recommended. These include:

**a) Sign Test:** For each dataset $j$, set $d_j = +1$ if $A_j > B_j$, $-1$ if $A_j < B_j$, $0$ if tie. $S =$ number of $+1$s among non-ties. Under $H_0$, $S \sim \text{Binomial}(n, 1/2)$. Large $n$ allows normal approximation:
$$
z = \frac{S - n/2}{\sqrt{n/4}}
$$
to compare to standard normal quantiles.

**b) Wilcoxon Signed-Rank Test:** For dataset $j$, compute $\delta_j = X_{Aj} - X_{Bj}$; discard ties. Rank $|\delta_j|$ among nonzero values, sum ranks $r_j$ for positive $\delta_j$. Under $H_0$ (symmetric differences),
$$
z = \frac{T - n(n+1)/4}{\sqrt{n(n+1)(2n+1)/24}}
$$
for large $n$. Both tests require family-wise correction (e.g., Bonferroni, Holm) over all $m(m-1)/2$ pairs [1505.02288].

## 5. Recent Developments: S-Statistics and Graphical Interpretation

Recent work [2202.09131] proposes a graphical “S-plot” approach that simultaneously provides the global Friedman test and local post-hoc indications with drastically fewer comparisons. For $G$ treatments and $B$ blocks, each treatment $g$ has a score
$$
S_g = \frac{(R_g - \bar{R})^2}{ B G (G+1) /12 }
$$
where $\bar{R} = B(G+1)/2$ is the expected rank sum under $H_0$. The sum $F = \sum_{g=1}^G S_g$ recovers the classical Friedman statistic.

The distribution of $S_g$ is well-approximated via gamma moments matching:
- $E[S_g]$, $\text{Var}(S_g)$, and third moment $M_3(S_g)$ derived from those of $F$.
- Fitted Gamma($a$, $\beta$) with $a=4/\gamma_1(S_g)^2$, $\beta = a/E[S_g]$, matching mean and skewness.
- The threshold $DL = Q_{\text{Gamma}}(1-\alpha_{PT}; a, \beta)$ provides Bonferroni-adjusted familywise Type I error.

The S-plot visualizes each $S_g$; treatments with $S_g > DL$ are significant contributors to rejection. This reduces testing from $G(G-1)/2$ to $G$ with controlled error rates and delivers immediate interpretive insight [2202.09131].

## 6. Empirical Validation and Practical Recommendations

Simulation studies have demonstrated that both the classical Friedman and S-statistic procedures maintain empirical Type I error within Bradley's robustness bounds across a range of $G$ and $B$ for both normal and exponential data, with accuracy improving as $B$ increases. Real-data applications (e.g., class size effects on children’s questions, per Gibbons & Chakraborti) confirm that the S-plot precisely identifies the dominant treatments responsible for global rejection, reducing the reliance on multiple pairwise post-hoc tables [2202.09131].

**Practical guidelines:**
- Apply the Friedman test as the omnibus procedure for multiple-treatment, multiple-block designs.
- Avoid the classical Nemenyi mean-ranks test; its results for a pair may depend irrationally on other treatments present.
- Prefer pairwise comparisons based exclusively on two-algorithm tests (Wilcoxon signed-rank if symmetry plausible, else sign test), with appropriate correction for multiple comparisons [1505.02288].
- Consider global-to-local visualization approaches such as S-plots for succinct interpretability and error control.

## 7. Summary Table: Properties and Critique

| Method                   | Pairwise Test Pool Dependence | Familywise Error Control | Number of Comparisons |
|--------------------------|------------------------------|-------------------------|----------------------|
| Friedman + Nemenyi       | Yes (dependent)              | Yes (nominal)           | $O(m^2)$             |
| Friedman + Pairwise (Wilcoxon/Sign) | No (independent)     | Yes (Bonferroni/Holm)   | $O(m^2)$             |
| S-Statistic/S-Plot [Editor’s term] | No (per-treatment)   | Yes (Gamma approx, Bonferroni) | $O(m)$            |

The core limitation of the mean-ranks test is its statistical dependence on the composition of the entire set of algorithms, which undermines its relevance for pairwise inference. Alternative approaches leveraging either pairwise-only tests or S-statistical visualizations achieve more interpretable, pool-independent, and statistically valid post-hoc inference [1505.02288, 2202.09131].

Source: https://www.emergentmind.com/topics/friedman-and-nemenyi-tests