Stop Using the Wilcoxon Test: Myth, Misconception and Misuse in IR Research

Abstract: In benchmarking of Information Retrieval systems, the Wilcoxon signed-rank test is often treated as a safer alternative to the t-test. This belief is fueled by textbooks and recommendations that portray Wilcoxon as the proper non-parametric alternative because metric scores are not normally distributed. We argue that this narrative is misleading and harmful. A careful review of Statistics textbooks reveals inconsistencies and omissions in how the assumptions underlying these tests are presented, fostering confusion that has propagated into IR research. As a result, Wilcoxon has been routinely misapplied for decades, creating a false sense of safety against a threat that was never there to begin with, while introducing another one so severe that it virtually guarantees the test will break down and mislead researchers. Through a combination of systematic literature review, analysis and empirical demonstrations with TREC data, we show how and why the Wilcoxon test easily loses control of its Type I error rate in IR settings. We conclude that the continued use of Wilcoxon in IR evaluation is unjustified and that abandoning it would improve the methodological soundness of our field.

• The paper demonstrates that the Wilcoxon test's symmetry assumption is systematically violated in IR metrics, leading to inflated Type I error rates.
• It employs extensive textbook reviews and empirical simulations to expose misconceptions about non-parametric safety compared to the t-test.
• The findings advise abandoning the Wilcoxon test in favor of methods better suited to IR data characteristics and practical sample sizes.

Reassessing Statistical Significance Testing in Information Retrieval: The Case Against the Wilcoxon Signed-Rank Test

Introduction

This work rigorously investigates the widespread practice of using the Wilcoxon signed-rank test as a "safe" non-parametric alternative to the paired $t$-test in Information Retrieval (IR) evaluation tasks. It challenges engrained methodological myths, misconceptions, and misuses—specifically, the belief that Wilcoxon is more reliable when normality assumptions are dubious for metric scores. It marshals both a systematic textbook review and empirical evidence to demonstrate how Wilcoxon's underlying symmetry assumption is systematically ignored across IR research, severely undermining Type I error control. Conversely, the $t$-test, often maligned when data depart from normality, remains robust in IR regimes due to the Central Limit Theorem (CLT) and practical sample sizes.

Systematic Review of Statistics Textbooks

The paper presents an extensive analysis of 25 widely used statistics texts. It reveals a striking lack of coherence and clarity regarding assumptions, appropriate contexts, and limitations of significance tests—especially the Wilcoxon signed-rank test. The overwhelming majority of sources frame Wilcoxon primarily as a "non-parametric" alternative to the $t$-test, rarely articulating its vulnerability to the symmetry assumption. The review identifies three key sources of confusion:

• Myth: That non-parametric methods are assumption-free or universally safer.
• Misconception: That the $t$-test requires normality of the metric scores themselves rather than approximate normality of their means.
• Misuse: That Wilcoxon is reliably safer in non-normal data, whereas its performance breaks down acutely in asymmetric settings, which are prevalent in IR.

The concept of "non-parametric" is shown to be imprecisely defined in the education and practice of statistics, with terms like "distribution-free" inappropriately conveying generality. Roadmap tables and decision trees in textbooks propagate these oversimplifications, encouraging IR researchers to substitute Wilcoxon for $t$ simply on the basis of observed non-normality.

Statistical Foundations and Pitfalls

The technical core delineates the assumptions, derivations, and practical functioning of both the $t$-test and the Wilcoxon signed-rank test.

• The $t$-test, for paired comparisons, tests $H_0:\mu_D=0$ via the sample mean of per-topic differences, relying on normality of $\overline{D}$, not $D$ itself—an assumption validated in moderate to large $t$0 regimes by the CLT.
• The Wilcoxon test is not, per se, robust to non-normality; it is strictly robust to violations of normality only if $t$1 is symmetric about zero. When $t$2 is asymmetric—a routine characteristic of IR metric differences—Wilcoxon’s null distribution is misspecified, and Type I error inflates drastically as $t$3 increases.

The empirical evidence is decisive: in simulations, Wilcoxon’s false positive rate approaches certainty in presence of only modest asymmetry, while the $t$4-test's error remains near nominal levels. These deviations are not edge cases but structurally endemic to effectiveness metrics in IR due to the bounded, discrete, and often skewed nature of their distributions.

Empirical Analysis on IR Data

Through large-scale simulations using TREC data and parametric families (drawn to span observed and hypothetical departures from normality and symmetry), the analysis demonstrates:

• IR metric score differences routinely display significant asymmetry and heavy tails.
• For sufficiently large $t$5 (typical in IR test collections), the $t$6-test's Type I error remains controlled across a wide range of non-normality and moderate asymmetry.
• The Wilcoxon test, however, becomes progressively anti-conservative as asymmetry increases, even to moderate extents encountered in standard IR experimentation. This effect is compounded, not mitigated, as sample size increases.

These findings decisively refute the notion that Wilcoxon is safer in real IR evaluation settings and instead show its misuse introduces systematic methodological error.

Implications for Research Practice

The practical and theoretical implications for IR and broader empirical evaluation domains are direct and far-reaching:

• Experimental IR practice must retire Wilcoxon for routine differential effectiveness testing except in the rare case where symmetry of differences is established and justifiable.
• Routine normality diagnoses (based on boundedness, discreteness, or visual tails) should not, in themselves, motivate rejection of the paired $t$7-test in favor of "non-parametric" alternatives.
• IR research—and any empirical discipline characterized by ground-truth differences with realistic distributional complexities—needs to adopt a more critical perspective on textbook guidance and avoid overreliance on field-imported statistical folk wisdom.
• The findings also suggest more robust alternatives, including resampling-based procedures, may be preferable, but these too require analysis under realistic distributional violations.

The study also highlights the urgent need for IR (and related fields) to move beyond contrast-based, one-factor-at-a-time significance tests toward explicit, multifactorial modeling approaches. Linear models, mixed-effects models, and resampling frameworks permit transparent accounting for the actual structure and sources of experimental variance.

Conclusions

This work provides a comprehensive critique of the use of the Wilcoxon signed-rank test in IR evaluation. It synthesizes theoretical, historical, and empirical arguments to show that Wilcoxon's symmetry requirement—not met by most IR metrics—renders it unreliable and even dangerous for significance testing. The paired $t$8-test, contrary to prevailing mythology, is nearly always preferable in realistic IR contexts due to the CLT and proves robust to departures from normality provided by practical sample sizes.

The discipline must abandon Wilcoxon in favor of tests whose assumptions are met or whose properties under violation are well-understood. This requires renewed attention to statistical foundations, empirical validation matching actual data characteristics, and movement toward comprehensive modeling. Continued adherence to the myth of non-parametric safety only perpetuates unreliability and confounds scientific progress in IR evaluation.

Strong Empirical Results:

• Systematic inflation of Type I error in Wilcoxon under asymmetry; error rate approaching 1 at modest skewness and moderate $t$9.
• $t$0-test maintains nominal Type I error even for marked non-normality and heavy tails—critical for standard IR experimental designs.

Contradictory Claims:

• The widespread belief (and textbook teaching) that Wilcoxon is safer than the $t$1-test in non-normal data is shown to be not just incorrect but counterproductive for IR research.

Outlook

Future research should extend such diagnostic, distributional, and simulation-based approach to resampling procedures and complex modeling strategies, ensuring that statistical inference in IR is both reliable and defensibly linked to the properties of evaluation metrics and data. Systematic reviews like this should become a standard prerequisite before statistical methods are canonized in computational research practice.

Citation: For an in-depth treatment, refer to "Stop Using the Wilcoxon Test: Myth, Misconception and Misuse in IR Research" (2604.25349).