Worst-Case Randomization Test

• Worst-case randomization test is a statistical procedure that ensures finite-sample Type I error control by calibrating p-values to the most challenging null configurations.
• It is applied in settings such as missing outcomes and latent principal strata, where traditional exact tests are inadequate, by optimizing over all feasible imputations.
• The method leverages invariance principles and studentization to achieve either exact finite-sample or asymptotically valid inference under weak null hypotheses.

A worst-case randomization test is a randomization-inference procedure calibrated to the least favorable case compatible with the null hypothesis, the observed data, and the design. In the literature summarized here, the phrase “worst-case” is used in several related senses: uniform finite-sample Type I error control over all null distributions satisfying a randomization hypothesis; maximization of a randomization $p$-value over all feasible imputations of missing outcomes or latent strata; and, when exact invariance is unavailable, studentized procedures that remain exact under a sharper invariant null and asymptotically valid under a broader weak null (Li et al., 1 Jul 2025, Dutz et al., 8 Dec 2025, Ritzwoller et al., 2024).

1. Randomization testing and the finite-sample worst-case criterion

In its classical form, a randomization test is built from observed data $X\in\mathcal X$, a null hypothesis $H_0:P\in\Omega_0$, a finite group $\mathcal G$ of measurable transformations $g:\mathcal X\to\mathcal X$, and a statistic $T:\mathcal X\to\mathbb R$. Let $M=|\mathcal G|$, let $$T^{(1)}(x;\mathcal G)\le \cdots \le T^{(M)}(x;\mathcal G)$$ denote the ordered orbit values $\{T(gx):g\in\mathcal G\}$, and define $k=M-\lfloor M\alpha\rfloor$. With randomized tie-breaking, the standard test is

$X\in\mathcal X$0

where $X\in\mathcal X$1, $X\in\mathcal X$2 is the number of orbit values strictly greater than $X\in\mathcal X$3, and $X\in\mathcal X$4 is the number equal to it (Dutz et al., 8 Dec 2025, Ritzwoller et al., 2024).

The decisive finite-sample condition is the randomization hypothesis: for every $X\in\mathcal X$5, $X\in\mathcal X$6 whenever $X\in\mathcal X$7. Under that invariance, the test has exact size

$X\in\mathcal X$8

and the conservative orbit-tail $X\in\mathcal X$9-value

$H_0:P\in\Omega_0$0

satisfies

$H_0:P\in\Omega_0$1

This is the strongest worst-case guarantee in the subject: no asymptotics, no average-case qualification, and no restriction to “nice” null distributions. The guarantee must hold for every $H_0:P\in\Omega_0$2, hence in the worst case over the null class (Ritzwoller et al., 2024, Dutz et al., 8 Dec 2025).

2. Invariance as the boundary of exact worst-case validity

Recent impossibility results make the boundary of exact finite-sample worst-case validity explicit. A null hypothesis $H_0:P\in\Omega_0$3 admits a randomization test if and only if there exists a bijective measurable function $H_0:P\in\Omega_0$4 such that

$H_0:P\in\Omega_0$5

Equivalently, exact worst-case randomization inference is possible only when the null itself is an invariance class under some nontrivial transformation (Dutz et al., 8 Dec 2025).

This characterization rules out many nulls of direct scientific interest. In particular, the one-sample null of mean zero does not admit a valid finite-sample randomization test in general. Sign-change tests are exact under symmetry around zero, not under mean zero alone; and the impossibility result is stronger than the failure of sign flips, because it states that there is no other randomization transformation that restores exact finite-sample validity uniformly over all mean-zero distributions. The same negative conclusion is established for other nulls, including fixed moments, fixed quantiles, and fixed variance, in both finite-support and continuous-support settings (Dutz et al., 8 Dec 2025).

The positive cases are correspondingly narrow. On finite support, admissible nulls are those that impose equal-mass identities across distinct multisets of support points. Under linear group actions, the nontrivial exact cases collapse essentially to symmetry about zero or subsets of Gaussian models. This gives a precise formulation of a longstanding intuition: exact randomization tests are not generic tools for arbitrary weak nulls, but procedures whose finite-sample validity is tied to a genuine group-invariance structure (Dutz et al., 8 Dec 2025).

3. Worst-case $H_0:P\in\Omega_0$6-values under attrition and missing outcomes

A different use of “worst-case randomization test” arises when the assignment mechanism is known but realized outcomes are missing. In randomized experiments with attrition, Fisher randomization tests lose sharpness because missing realized outcomes prevent full imputation of potential outcomes even under a sharp treatment-effect null. The proposed remedy is to treat missing quantities as unknown but constrained by the observed data, the null hypothesis, and assumptions on the missingness mechanism, and then to use the largest randomization $H_0:P\in\Omega_0$7-value over all feasible imputations (Li et al., 1 Jul 2025).

Formally, for a sharp null $H_0:P\in\Omega_0$8, a feasible set $H_0:P\in\Omega_0$9 of null-compatible imputations, and a rank statistic $\mathcal G$0, the worst-case $\mathcal G$1-value is

$\mathcal G$2

Because the rank statistics used are distribution-free and effect-increasing, this becomes

$\mathcal G$3

The optimization is therefore over feasible latent outcomes or composite outcomes, not over the assignment mechanism, which remains the completely randomized experiment (Li et al., 1 Jul 2025).

Under general missingness, the extremal imputation follows partial-identification logic: missing treated outcomes are pushed to the lower support bound and missing control outcomes to the upper support bound. In the paper’s order-only formulation this is written with $\mathcal G$4 and $\mathcal G$5, and for outcome-only statistics the worst-case control potential outcomes are

$\mathcal G$6

Under monotone missingness, the feasible set sharpens, and the paper introduces the composite control potential outcome

$\mathcal G$7

which allows the statistic to incorporate both outcomes and missingness indicators. For monotone positive missingness, $\mathcal G$8, the recommended choice is $\mathcal G$9; for monotone negative missingness, $g:\mathcal X\to\mathcal X$0, the recommended choice is $g:\mathcal X\to\mathcal X$1. The same framework extends to non-sharp nulls about quantiles of individual treatment effects by adding a second optimization over null-compatible effect vectors $g:\mathcal X\to\mathcal X$2 (Li et al., 1 Jul 2025).

The role of assumptions is sharply differentiated. Under general or monotone missingness, “worst-case” means worst over latent outcomes or composite outcomes consistent with the null and missingness restrictions. Under missing at random, the paper does not optimize over missing outcomes; instead, after conditioning on the observed subset, treatment assignment among observed units is again a completely randomized experiment, so standard randomization inference on the observed subset is valid (Li et al., 1 Jul 2025).

4. Worst-case randomization over latent principal strata

A third construction arises in randomized experiments with attrition under monotone reporting when the target estimand is the always-reporter average treatment effect,

$g:\mathcal X\to\mathcal X$3

where

$g:\mathcal X\to\mathcal X$4

is the latent set of units whose outcomes would be observed under either treatment status. Under the monotonicity assumption

$g:\mathcal X\to\mathcal X$5

always-reporter status is only partially observed: control reporters must be always-reporters, treated nonreporters must be never-reporters, and treated reporters remain ambiguous (Chang et al., 26 Mar 2026).

The inferential obstacle is therefore not merely missing outcomes but latent principal-stratum membership. Let $g:\mathcal X\to\mathcal X$6 be the set of always-reporter vectors $g:\mathcal X\to\mathcal X$7 compatible with observed assignment $g:\mathcal X\to\mathcal X$8, reporting $g:\mathcal X\to\mathcal X$9, and monotonicity. For a test statistic $T:\mathcal X\to\mathbb R$0, the randomization $T:\mathcal X\to\mathbb R$1-value for a fixed admissible $T:\mathcal X\to\mathbb R$2 is

$T:\mathcal X\to\mathbb R$3

and the worst-case $T:\mathcal X\to\mathbb R$4-value is

$T:\mathcal X\to\mathbb R$5

This is a Berger–Boos style construction over latent principal-stratum configurations: reject only if the null fails for every admissible always-reporter assignment (Chang et al., 26 Mar 2026).

The paper studies a studentized Hájek-type statistic

$T:\mathcal X\to\mathbb R$6

with variance estimator

$T:\mathcal X\to\mathbb R$7

and

$T:\mathcal X\to\mathbb R$8

It also adds chi-square-type components based on balance in the number of always-reporters across treatment arms, producing $T:\mathcal X\to\mathbb R$9 and $M=|\mathcal G|$0 (Chang et al., 26 Mar 2026).

The validity statement is two-layered. Under the sharp null $M=|\mathcal G|$1 for all $M=|\mathcal G|$2,

$M=|\mathcal G|$3

Under the weak null of zero AR-ATE,

$M=|\mathcal G|$4

Thus the test is finite-sample valid for the sharp null and asymptotically valid for the weak null, with conservativeness driven by latent-stratum uncertainty (Chang et al., 26 Mar 2026).

5. Weak nulls, studentization, and asymptotic robustness

When the scientific null is weak rather than sharp, the contemporary literature does not generally claim finite-sample worst-case validity. Instead, it uses studentization and asymptotic pivotality to obtain procedures that are exact on a structured invariant submodel and asymptotically valid more broadly. A review of randomization inference organizes this architecture into three layers: exact finite-sample validity under the randomization hypothesis, conservative or asymptotically valid inference when exact invariance fails, and restoration of validity by studentization when naive randomization can fail even asymptotically (Ritzwoller et al., 2024).

For randomized experiments, one route is to impute missing potential outcomes under a compatible sharp null that implies the weak null and to calibrate Fisher randomization inference with a studentized quadratic statistic. For a general contrast $M=|\mathcal G|$5, the recommended statistic is

$M=|\mathcal G|$6

where $M=|\mathcal G|$7 is a diagonal conservative covariance estimator. The resulting Fisher randomization test is finite-sample exact under the imputing sharp null and conservatively controls large-sample Type I errors under the weak null, precisely because the randomization reference distribution asymptotically stochastically dominates the true sampling distribution under heterogeneity (Wu et al., 2018).

A more general asymptotic theory replaces design-based rerandomization by randomization through algebraic groups acting on the data. For a functional $M=|\mathcal G|$8, an original studentized statistic

$M=|\mathcal G|$9

is paired with a randomized analogue

$$T^{(1)}(x;\mathcal G)\le \cdots \le T^{(M)}(x;\mathcal G)$$0

Under Hadamard differentiability, a conditional weak convergence theorem for randomized empirical processes, and consistent studentization, the test based on the conditional quantile of $$T^{(1)}(x;\mathcal G)\le \cdots \le T^{(M)}(x;\mathcal G)$$1 has asymptotic level $$T^{(1)}(x;\mathcal G)\le \cdots \le T^{(M)}(x;\mathcal G)$$2 under the weak null and finite-sample exactness under the smaller group-invariant null $$T^{(1)}(x;\mathcal G)\le \cdots \le T^{(M)}(x;\mathcal G)$$3. The paper is explicit that this is pointwise asymptotic validity, not nonasymptotic worst-case control over the whole weak-null class (Dobler, 2019).

Incomplete paired data provide a concrete statistical example of the same compromise. For matched pairs with components missing completely at random, the proposed statistic is

$$T^{(1)}(x;\mathcal G)\le \cdots \le T^{(M)}(x;\mathcal G)$$4

combining a paired $$T^{(1)}(x;\mathcal G)\le \cdots \le T^{(M)}(x;\mathcal G)$$5-component with a Welch-type component. The associated randomization test separately permutes within complete pairs and permutes the pooled incomplete sample. It is asymptotically level $$T^{(1)}(x;\mathcal G)\le \cdots \le T^{(M)}(x;\mathcal G)$$6 and consistent under broad nonparametric conditions, including heteroscedasticity and skewness, and it is finitely exact in the special case that the pair differences are $$T^{(1)}(x;\mathcal G)\le \cdots \le T^{(M)}(x;\mathcal G)$$7-symmetric and the two incomplete populations are equal in distribution. The paper explicitly does not claim universal worst-case finite-sample validity over arbitrary distributions; it illustrates instead the principle that exactness comes from invariance, while studentization can restore asymptotic validity when only a weak mean-null is available (Amro et al., 2016).

6. Taxonomy, computation, and limitations

The resulting literature separates several distinct objects that are all called “worst-case” in practice.

Framework	Worst-case object	Validity statement
Group-invariant randomization test	Uniform control over all $$T^{(1)}(x;\mathcal G)\le \cdots \le T^{(M)}(x;\mathcal G)$$8	Exact finite-sample size $$T^{(1)}(x;\mathcal G)\le \cdots \le T^{(M)}(x;\mathcal G)$$9 or conservative finite-sample control
Attrition with missing outcomes	Supremum over feasible imputations or composite outcomes	Valid under general or monotone missingness; no worst-case step under MAR
AR-ATE with monotone reporting	Supremum over admissible always-reporter vectors $\{T(gx):g\in\mathcal G\}$0	Finite-sample valid under the sharp null; asymptotically valid under the weak null
Studentized weak-null randomization	No finite-sample worst-case optimization; robustness through asymptotic pivotality	Exact on invariant subnulls; asymptotically valid or conservative on broader weak nulls

Computationally, worst-case randomization testing is tractable only because the optimization can often be reduced. In the attrition framework, distribution-free rank statistics and the effect-increasing property turn the search for the largest $\{T(gx):g\in\mathcal G\}$1-value into minimization of the realized rank statistic, yielding closed-form extremal imputations. In the always-reporter framework, discrete outcomes allow reduction to count vectors $\{T(gx):g\in\mathcal G\}$2, and for fixed support size the number of distinct vectors is polynomial in $\{T(gx):g\in\mathcal G\}$3; continuous outcomes require interval decomposition and mixed-integer optimization over matching variables (Li et al., 1 Jul 2025, Chang et al., 26 Mar 2026).

Several limitations are structural rather than technical. First, exact finite-sample worst-case validity is not broadly available: if the null does not satisfy the randomization hypothesis, one should not expect any randomization test to deliver exact finite-sample control uniformly over the null class. Second, in missing-data settings, “worst-case” does not refer to optimization over the assignment mechanism; the assignment law remains fixed by design, while the optimization ranges over latent outcomes, composite outcomes, or latent principal strata. Third, worst-case procedures are often conservative, and the conservativeness can be substantial when the feasible latent set is large. A plausible implication is that the term “worst-case randomization test” should be read less as the name of a single method than as a design principle: exact finite-sample validity when genuine invariance exists, and otherwise a least-favorable or studentized calibration that preserves validity under the broadest defensible assumptions (Dutz et al., 8 Dec 2025, Ritzwoller et al., 2024).