---
title: Valid in Every Hypothesis (VEH)
url: https://www.emergentmind.com/topics/valid-in-every-hypothesis-veh
type: topic
---

# Valid in Every Hypothesis (VEH)

Valid in Every Hypothesis (VEH) is a uniform validity requirement that appears most prominently in modern hypothesis testing with e-values and e-processes. In that setting, VEH means that the defining validity inequality holds for every distribution in a null set, not merely on average over the null or only at a pre-fixed sample size. This makes VEH central to composite-null testing, optional stopping, and anytime-valid inference, including testing-by-betting constructions based on non-negative martingales or supermartingales [2410.23614]. The same acronym is also used in a distinct sense in kinodynamic motion planning, where a trajectory is required to be feasible in every hypothesis of a robot’s recent world-model history [2509.26339].

## 1. Definition and formal statistical meaning

For a composite null hypothesis represented by a set of distributions \(\mathcal{P}\), an e-variable \(E\) is VEH if
\[
\mathbb{E}^{P}[E] \leq 1 \quad \text{for all } P \in \mathcal{P}.
\]
In the formulation used for e-values, this is the basic definition of uniform validity under composite nulls [2410.23614]. For simple nulls, where \(\mathcal{P}=\{P\}\), the definition reduces to the classical one-distribution case.

Parallel definitions extend the same principle to tests and p-variables. A test \(\phi\) is VEH if, for all \(P \in \mathcal{P}\),
\[
\mathbb{E}^{P}[\phi] \leq \alpha,
\]
and a p-variable is VEH if, for all \(P \in \mathcal{P}\),
\[
\mathbb{P}^{P}(p \leq \alpha) \leq \alpha.
\]
For e-variables, Markov’s inequality yields the basic rejection guarantee
\[
\mathbb{P}^{P}(E \geq 1/\alpha) \leq \alpha,
\]
again for every \(P \in \mathcal{P}\) [2410.23614].

In sequential settings, the same idea becomes a stopping-time statement: an e-process \(E_t\) is VEH when
\[
\mathbb{E}^{P}[E_\tau] \leq 1
\]
for every \(P \in \mathcal{P}\) and any stopping time \(\tau\) [2410.23614]. In supervised prediction, an equivalent operational form is called uniform validity; it permits simultaneous control even if the asserted event is data-dependent or adversarially chosen [2112.10234].

VEH is therefore stronger than validity statements tied to a single fixed analysis plan. One formulation states explicitly that this is stronger than typical p-value-based validity, which is only for the whole null hypothesis as a set, not at the level of individual distributions [2109.11761].

## 2. E-values, e-processes, and optional stopping

An e-value is a non-negative random variable \(E\) such that, under a null hypothesis \(\mathcal{H}\),
\[
\mathbb{E}_{P}[E] \leq 1 \qquad \forall P \in \mathcal{H}.
\]
In sequential settings with filtration \((\mathcal{F}_t)\) and lag \(h\), the adapted version is
\[
\mathbb{E}_{P}[E_t \mid \mathcal{F}_{t-h}] \leq 1 \qquad \forall P \in \mathcal{H},\, \forall t,
\]
which is the basic e-value condition used in sequential forecast calibration [2109.11761].

For \(h=1\), e-values combine multiplicatively:
\[
e_t = \prod_{i=1}^t E_i.
\]
For \(h>1\), sequential validity is retained through a U-statistic average,
\[
e_T = \frac{1}{h} \sum_{k=1}^{h} \prod_{l \in I_k(T)} E_l,
\]
where
\[
I_k(T) = \{k + h s : s=0,\dots,\lfloor (T-k)/h \rfloor\}.
\]
Large e-values provide evidence against the null, and the multiplicative inverse of an e-value is a conservative p-value [2109.11761].

The testing-by-betting formulation gives a particularly explicit VEH mechanism. In deep anytime-valid testing, for each batch \(B_t\) and network \(g_\theta\), the betting score is
\[
S_t = \prod_{j=1}^b \bigl(1 + g_{\theta_{t-1}}(Z_{(t-1)b + j})\bigr),
\]
the wealth update is
\[
W_t = W_{t-1} \times S_t,\qquad W_0=1,
\]
and the procedure rejects \(H_0\) at the first \(t\) such that
\[
W_t \geq 1/\alpha.
\]
Under \(H_0\), \(\{W_t\}_{t\ge0}\) is a non-negative martingale, so Ville’s inequality gives
\[
\mathbb{P}_{H_0}\!\left(\exists t \geq 0 : W_t \geq 1/\alpha\right) \leq \alpha.
\]
This yields finite-sample type I error control under continuous monitoring and optional stopping; the paper identifies this strong form of validity as the VEH principle [2310.19384].

The core consequence is that a practitioner does not need to pre-specify a sample size. Evidence can be monitored continuously, and rejection remains valid at the first threshold crossing.

## 3. VEH in deep anytime-valid nonparametric testing

A general nonparametric framework expresses the null through two operators \(T_1\) and \(T_2\) acting on the data distribution \(P_Z\):
\[
H_0: T_1(Z) \stackrel{d}{=} T_2(Z)
\quad \text{vs.} \quad
H_1: T_1(Z) \stackrel{d}{\neq} T_2(Z).
\]
This operator-action formulation is used to unify two-sample testing, independence testing, conditional independence, group invariance, and adversarial robustness in machine learning [2310.19384].

A distinctive feature of the framework is that the payoff function and the betting strategy are not decoupled. Instead, both are learned jointly with a deep neural network by maximizing log-wealth:
\[
\theta_t \in \arg\max_{\theta \in \Theta}
\sum_{l=1}^t \sum_{Z \in B_l} \log(1 + g_\theta(Z)).
\]
The stopping time
\[
\gamma = \inf\{ t \geq 1 : W_t \geq 1/\alpha \}
\]
is a level-\(\alpha\) test of power one, with
\[
\mathbb{P}_{H_0}(\gamma < \infty) \leq \alpha,
\qquad
\mathbb{P}_{H_1}(\gamma < \infty) = 1,
\]
provided the learning procedure is sufficiently consistent [2310.19384].

The framework is presented as a unified machinery for designing and implementing anytime-valid tests across classical and modern tasks. Reported experiments include synthetic and real data such as adversarial CIFAR-10, MNIST rotation, and the Blob dataset, with comparisons to E-C2ST, Seq-IT, MMD, and S-C2ST. The reported findings are tighter or faster type I error control under continuous monitoring, better sample efficiency through early stopping, and competitive or superior power when equipped with powerful DNNs [2310.19384].

A common misunderstanding is to equate these guarantees with ordinary fixed-sample validity. The stated guarantee is stronger: the probability of a false rejection never exceeds \(\alpha\), no matter how the data are monitored or when the test is stopped [2310.19384].

## 4. Post-hoc significance levels, admissibility, and the necessity of e-values

VEH has also become central in post-hoc hypothesis testing, where the significance level may be chosen after seeing and analyzing the data. In this literature, the analyst or adversary selects a loss function, or equivalently a significance level, through a measurable mapping \(B:\mathcal{X}\to\mathcal{B}\). The adversary set \(\Gamma\) specifies which such mappings are allowed. Two extreme cases are \(\Gamma=U\), all possible measurable mappings, and \(\Gamma=C\), only constant mappings [2508.00770].

Within this framework, \(\Gamma=U\)-admissibility corresponds to VEH: the procedure remains type-I risk safe for arbitrary post-hoc, data-dependent choices of the loss or significance level [2508.00770]. The central characterization is that admissible VEH procedures must be based on e-values. For a test family \(\delta\), the associated e-variable is
\[
E_\delta(X) = \sup_{b\in\mathcal{B}} L_b(0,1)\delta(X,b).
\]
Canonical representations are
\[
\delta(X,b) = \min\left\{1, \frac{E_\delta(X)}{L_b(0,1)}\right\}
\]
for randomized tests and
\[
\psi(X,b) = 1\{E_\psi(X) \geq L_b(0,1)\}
\]
for binary tests. Theorems summarized in the paper state that, under the stated conditions, \(U\)-admissible tests are exactly those defined by sharp e-values [2508.00770].

A related 2026 treatment asks how classical type-I error should be extended to data-dependent levels and concludes that the correct extension is control of the expected distortion ratio,
\[
\mathbb{E}^P \left[
\frac{
P(\text{test falsely rejects } H \mid \widetilde \alpha)
}{
\widetilde \alpha
}
\right] \leq 1.
\]
This criterion is characterized by three axioms [2605.28429]:

- **Nesting classical validity**: if \(\widetilde{\alpha}\) is constant, the extension coincides with classical validity.
- **Preserving classical validity**: the data-dependent rule must not violate fixed-threshold error control.
- **Monotonicity**: making the rejection claim weaker cannot invalidate the procedure.

The same work connects this criterion directly to VEH by showing that the numerical representation of the generalized test is an e-value satisfying
\[
\mathbb{E}^P[\mathcal{E}] \leq 1
\]
for every \(P\) in the null [2605.28429].

Taken together, these results establish a precise boundary: for the most flexible post-hoc guarantee, e-value-based procedures are not merely convenient but necessary.

## 5. Extensions across forecasting, prediction, and watermarking

VEH has been instantiated in several domains beyond abstract hypothesis testing. The common structure is a validity inequality that holds for every null distribution or every member of a null family, often together with optional stopping.

| Setting | VEH condition | Consequence |
|---|---|---|
| Forecast calibration [2109.11761] | \(\mathbb{E}_P[E_t \mid \mathcal{F}_{t-h}] \leq 1\) for all \(P \in \mathcal{H}\) | Sequential goodness-of-fit testing with optional stopping |
| Probability forecast performance [2103.08402] | \(\mathbb{E}_{\mathbb{Q}}[e_T] \leq 1\) for all \(\mathbb{Q} \in \mathcal{H}_S\) | Finite-sample validity without assumptions on the data generating processes |
| Supervised prediction [2112.10234] | \(\sup_{P \in \mathcal{P}} P\{\pi_{X_{n+1}}^n(Y_{n+1}) \leq \alpha\} \leq \alpha\) | Uniform validity for probabilistic predictors; valid predictors must be imprecise |
| Statistical watermarking [2602.17608] | \(\sup_{q \in Q(p_0,\delta)} \mathbb{E}_{v\sim q,\,s\sim p_0}[e(v,s)] \leq 1\) | Anytime-valid watermark detection via a test supermartingale |

In sequential forecast calibration, e-values are constructed from PITs or ranks, and rejection occurs when the cumulative e-process crosses \(1/\alpha\); the same framework is also described as more generally applicable for sequential goodness-of-fit testing [2109.11761]. For forecast comparison using proper scoring rules, the resulting e-values are valid in finite samples without any assumptions on the data generating processes, and optional stopping is explicitly allowed [2103.08402].

In supervised learning, the corresponding notion is a valid probabilistic predictor. The papers state that valid probabilistic predictors must be imprecise and that uniform validity yields frequentist error-rate control for derived tests and prediction sets [2112.10234]. In this setting, VEH is formulated through plausibility contours and the probability of under-rating the true future observation.

In statistical watermarking for large language models, Anchored E-Watermarking constructs a test supermartingale for detection. The detector uses an anchor distribution \(p_0\), requires uniform e-value validity over \(q \in Q(p_0,\delta)\), and monitors
\[
E_t = \prod_{i=1}^t e^*(v^i,s^i)
\]
with rejection at the first \(t\) such that \(E_t \geq 1/\alpha\). Reported evaluations state that the framework reduces the average token budget required for detection by 13–15% relative to state-of-the-art baselines [2602.17608].

These examples show that VEH is not tied to a single test construction. It is a criterion for validity under composite nulls, sequential monitoring, or model uncertainty, and different domains instantiate that criterion with different e-values and stopping rules.

## 6. Distinct usage in kinodynamic motion planning

Outside statistical inference, VEH is also the name of a conservative planning paradigm for mobile robots navigating across inconsistent world models. Here, each “hypothesis” is one temporally sampled instance of the environment representation, and a path is considered feasible only if it is valid in every hypothesis [2509.26339].

The baseline formulation extends A*/ARA* over the KEASL lattice so that an edge is expanded only if it is collision-free in every stored world hypothesis. The key validity check is
\[
\text{valid}(n') := \bigwedge_{i=1}^{k} \text{valid in hypothesis } i.
\]
This makes VEH maximally conservative: it effectively treats the union of obstacles across the recent history as obstacles for planning [2509.26339].

The same paper contrasts VEH with PEH, GEH, and GEGRH. PEH invokes a sub-search for every node expansion that crosses a divergence point in the world models. GEH and GEGRH defer the sub-search until after an edge expands into the goal region, and GEGRH additionally revises the graph based on divergent nodes in each world [2509.26339].

Field experiments on 221 planning problems in an off-road environment found that PEH and GEH were impractical for deployment, while GEGRH achieved lower-cost trajectories and faster average planning times than VEH. The reported average planning times were \(0.26 \pm 0.03\) seconds for VEH versus \(0.22 \pm 0.02\) for GEGRH with two hypotheses, and \(0.41 \pm 0.04\) versus \(0.29 \pm 0.03\) with three hypotheses. The corresponding path durations were \(38.56 \pm 2.06\) versus \(36.00 \pm 1.87\) for two hypotheses and \(44.87 \pm 2.91\) versus \(36.47 \pm 1.98\) for three hypotheses [2509.26339].

This robotic meaning differs from the statistical one in object and mechanism, but not in its basic intuition. In both cases, VEH is a worst-case validity requirement across a family of hypotheses: null distributions in statistics, world models in navigation.

Source: https://www.emergentmind.com/topics/valid-in-every-hypothesis-veh