---
title: Anytime-Validity & Type I Error Control
url: https://www.emergentmind.com/topics/anytime-validity-and-type-i-error-control
type: topic
---

# Anytime-Validity & Type I Error Control

Anytime-validity and Type I error control are central concepts in sequential and online statistical inference, where inferential validity must persist regardless of when analysis stops or resumes. Contemporary frameworks employ e-values, test martingales, and adaptive spending mechanisms to guarantee global error bounds—such as familywise error rate (FWER) or marginal Type I error—under arbitrary stopping rules and dependence structures. This article presents the mathematical foundations, construction principles, representative methodologies, and practical regimes of anytime-valid inference and robust Type I error control in modern research.

## 1. Mathematical Foundations of Anytime Validity

Let $(X_t)_{t\geq1}$ denote a possibly infinite data sequence with associated filtrations $(\mathcal{F}_t)$. An inferential procedure (test or confidence interval) is said to be **anytime valid** at level $\alpha$ for a parameter $\theta$ if
\[
\Pr\left(\exists\,t\geq1: \text{Reject }H_0\text{ or } \theta \notin CS_t\right)\leq \alpha
\]
where $CS_t$ is a sequence of random sets (confidence intervals/bands/sequences). This ensures that the Type I error control persists across all possible interim and final analyses, including those informed by arbitrary, data-adaptive stopping rules [2302.10108][2011.03567][2203.09785].

The theoretical pillar is the **test martingale** (or e-process), a nonnegative $(\mathcal{F}_t)$-adapted process $(M_t)$, typically constructed such that for all $P \in H_0$,
\[
\mathbb{E}_P[M_\tau] \le 1
\]
for any (possibly random) stopping time $\tau$. Ville's maximal inequality then yields
\[
\Pr_P\left(\sup_{t\geq1} M_t \geq 1/\alpha\right) \leq \alpha
\]
so the procedure maintains exact or asymptotic Type I error.

## 2. Construction of E-Processes and E-Variables

The construction of e-processes and test martingales underlies almost all modern anytime-valid methods. An **e-variable** is a nonnegative statistic $E$ satisfying $\mathbb{E}_P[E]\leq 1$ for all $P\in H_0$. A sequence $(E_t)$ forms an *e-process* or *test martingale* if $M_t = \prod_{i=1}^t E_i$ is a nonnegative $(\mathcal{F}_t)$-martingale or supermartingale under each $P\in H_0$ [2203.09785][2310.01153].

Key properties:
- **Optional Stopping**: For any (possibly data-adaptive) stopping time $\tau$, $\mathbb{E}_P[M_\tau]\leq 1$ for all $P\in H_0$.
- **Uniformity**: Markov's inequality yields time-uniform guarantees: $\Pr_P(\exists t: M_t\geq 1/\alpha)\leq \alpha$.
- **Confidence Sequences**: Invert the acceptance region ($M_t(\theta)<1/\alpha$) to construct $(1-\alpha)$ confidence sets $CS_t$ satisfying $\Pr(\theta\in CS_t\;\forall t)\ge 1-\alpha$ [2203.09785][2310.01153][2011.03567].

This framework generalizes classical likelihood-ratio based tests, accommodates composite nulls (via e-mixtures and reverse information projections), and enables both parametric and nonparametric robustification [2408.14015].

## 3. Representative Methodologies

### a. Online FWER Control under Dependence

For online multiple testing with *dependent* test statistics, a framework based on **consistent weights** $\xi_{i,n}$ replaces discrete candidate sets [2401.09559]. For sequential hypotheses $H_1,H_2,\dots$, if $V_n$ is the number of false rejections among the first $N$ tests,
\[
\limsup_{n\to\infty}\mathrm{FWER}_n(N) \leq \alpha
\]
is guaranteed by enforcing the adaptive wealth-spending constraint:
\[
\sum_{i=1}^K \alpha_{i,n} (1-\lambda_i) \xi_{i,n} \leq \alpha\quad \forall K, n.
\]
This holds asymptotically even under arbitrary dependencies, and remains valid under arbitrary stops or extensions.

### b. Likelihood Ratio and Mixture Martingales

For fixed or composite hypotheses in parametric or nonparametric settings:
- Likelihood-ratio martingales form classic e-processes; for i.i.d. $(X_t)$, $L_t = \prod_{i=1}^t \frac{dQ}{dP}(X_i)$ is a nonnegative $P$-martingale.
- Mixture martingales (e.g., Dirichlet-multinomial and Bayesian alternatives) extend this to composite nulls or alternatives [2011.03567][2410.09282].
- Bayesian mixtures allow robust power control and adaptive inference via prior tuning; all maintain anytime-valid Type I error by Ville's bound.
- In contaminated or model-uncertain regimes, robust truncation (Huber-style) of LRs yields supermartingales that preserve validity against adversarial or $\epsilon$-contaminated distributions [2408.14015].

### c. Regression, A/B Testing, and Survival Analysis

Anytime-valid confidence sequences and tests have been constructed for:
- Mean-difference or lift statistics in A/B testing [2302.10108], where a nonnegative self-normalized martingale establishes time-uniform bounds.
- Linear regression and regression-adjusted causal inference, where closed-form e-processes for $F$-tests (under Gaussian linear models), robustified quadratic forms, and model-free ATE estimands yield confidence sequences with coverage holding uniformly for all samples [2210.08589].
- Logrank tests and Cox regression in survival analysis, where each event's likelihood-ratio increment forms a step of the test martingale; the product yields an AV (anytime-valid) logrank test and associated confidence sequences [2011.06931].

## 4. Error Control: Theoretical Guarantees and Simulation Evidence

The core theoretical guarantee is that, under the null, the boundary crossing probability for acceptance thresholds is bounded by the nominal level uniformly over all (possibly adaptive) times and under arbitrary stopping:
\[
P\left(\exists\,t:\;M_t \geq 1/\alpha\right) \leq \alpha
\]
or, for FWER:
\[
\limsup_{n\to\infty}\mathrm{FWER}_n(N) \leq \alpha
\]
as shown in exact or asymptotic limits [2401.09559][2203.09785][2011.03567][2310.01153].

Extensive simulation studies demonstrate:
- In online multiple testing, continuous-spending and graph-based procedures maintain empirical FWER at the nominal level across a wide range of scenarios, including dependent tests (AR(1) correlation), platform trial designs, and various proportions of true alternatives [2401.09559].
- In A/B tests, empirical Type I error remains at or below nominal under continuous monitoring; standard fixed-horizon protocols are severely anti-conservative under peeking or optional stopping [2302.10108].
- Under adaptive or contaminated regimes, robust e-processes prevent explosive Type I error, where classical likelihood-based tests fail [2408.14015].

## 5. Adaptivity, Optional Stopping, and Optional Continuation

A distinctive strength of anytime-valid methodology is its **robustness to adaptive design**. For any $(M_t)$ e-process, test levels and inferential boundaries remain valid:
- Under arbitrary, possibly data-dependent stopping times.
- After resuming/continuing a test beyond the original planned sample size.
- When combining evidence from independent sequential tests (by product of e-values or resetting significance levels as future thresholds), maintaining overall procedure validity [2501.03982][2011.06931].

For example, any fixed-$N$ classical test can be "sequentialized" by constructing the Doob martingale of its test function; the process then gives exactly the fixed-horizon power at $N$, and matches the original underlying test at $N$ [2501.03982].

## 6. Classes of Procedures and Their Domains of Application

| Domain                              | Test Statistic/CS Construction             | Reference            |
|--------------------------------------|-------------------------------------------|----------------------|
| Online multiple testing (FWER)       | Consistent weights, adaptive spending     | [2401.09559]         |
| A/B testing, mean/lift               | Self-normalized martingale, CLT           | [2302.10108]         |
| Linear regression, causal inference  | Sequential $F$-test, robust e-process     | [2210.08589]         |
| Categorical/multinomial testing      | Mixture martingale (Dirichlet-Bayes)      | [2011.03567]         |
| Contamination-robust testing         | Huber-truncated LR e-process              | [2408.14015]         |
| Ranked, nonparametric independence   | Sequentially-binned Bayes-factor marting. | [2305.13818]         |
| Survival analysis, Cox regression    | Likelihood-ratio event martingales        | [2011.06931]         |

Each entry reflects an instantiation of the abstract e-process/e-variable paradigm to distinct statistical modalities.

## 7. Limitations and Ongoing Research

Contemporary anytime-valid approaches achieve robustness under general stopping and dependence, but several open problems remain:
- Not all dependency structures or adaptive strategies permit exact power characterization; some procedures guarantee only asymptotic FWER or power [2401.09559].
- Most frameworks implement "adaptivity" (adaptive spending) but not full "discarding" (as in ADDIS–Spending) [2401.09559].
- Power loss relative to fixed-horizon procedures at a pre-specified $N$ can be modest but nonzero, especially for extremely conservative procedures or when minimal detectable effect is underestimated [2302.10108].

Ongoing research directions address integrating discarding, optimal e-variable design, and minimax power under more challenging data corruption and dependency regimes.

---

**References:**  
[2011.03567], [2203.09785], [2210.08589], [2302.10108], [2305.13818], [2310.01153], [2401.09559], [2408.14015], [2410.09282], [2501.03982], [2011.06931].

Source: https://www.emergentmind.com/topics/anytime-validity-and-type-i-error-control