---
title: 'Familywise Evidence: Continuous Multiplicity Control'
url: https://www.emergentmind.com/topics/familywise-evidence
type: topic
---

# Familywise Evidence: Continuous Multiplicity Control

Familywise evidence is a continuous-evidence generalization of classical familywise error control. In this framework, the object of interest is not merely whether at least one true null hypothesis is falsely rejected, but the strongest evidence assigned to any true hypothesis in an entire hypothesis class after observing the data. The concept is introduced through the E-measure, a measure-like extension of the E-value from single hypotheses to structured classes of hypotheses, and it is designed so that post hoc inspection over a family of hypotheses remains valid in a familywise sense [2604.20788].

## 1. Definition and statistical target

Classical familywise inference is organized around the familywise error rate (FWER), the probability of rejecting at least one true null hypothesis. In a general multiple-testing setup with hypothesis family \(\mathcal H\), true subfamily \(\mathcal T(M)\), false subfamily \(\mathcal F(M)\), and final rejection set \(\mathcal R_\infty\), strong FWER control is written as
\[
\mathrm P_M\bigl(\mathcal R_\infty\subseteq \mathcal F(M)\bigr)\ge 1-\alpha
\]
for every \(M\in\mathbb M\) [1211.3313]. This criterion protects the integrity of simultaneous claims by requiring that, with probability at least \(1-\alpha\), all reported rejections are false nulls.

Familywise evidence replaces this binary event by a continuous statistic. For a single hypothesis \(H\), an E-variable is a nonnegative measurable statistic \(x\mapsto \e(H\mid x)\), valid when
\[
\Ex^P[\e(H \mid X)] \le 1, \quad \textnormal{for every } P \in H.
\]
Larger values indicate stronger evidence against \(H\). For an E-function kernel \(\e\), the familywise evidence at truth \(P\) is
\[
\sup_{H \in \H : H \ni P} \e(H \mid X),
\]
and familywise evidence is controlled if
\[
\sup_{P \in \P} \Ex^P\left[\sup_{H \in \H : H \ni P} \e(H \mid X) \right] \leq 1.
\]
This is the exact continuous-evidence analogue of FWER [2604.20788].

The reduction to the binary case is explicit. If \(\e\) is \(\{0,1/\alpha\}\)-valued, then thresholding at \(1/\alpha\) produces a rejection rule, and the familywise evidence criterion becomes classical FWER control at level \(\alpha\) [2604.20788]. Familywise evidence is therefore not merely analogous to FWER; it contains FWER as a special case.

## 2. E-measures and the logical structure of evidence

The formal apparatus begins with an E-function, a map
\[
\e : \H \to [0,\infty]
\]
satisfying \(\e(\emptyset)=\infty\). This encodes the Impossibility axiom: logically impossible hypotheses receive maximal evidence. The Implication axiom requires that if \(H \subseteq H'\), then the more specific hypothesis \(H\) cannot receive less evidence against it than \(H'\). This yields an E-capacity:
\[
H \subseteq H' \implies \e(H') \le \e(H), \quad H,H' \in \H.
\]

An E-measure strengthens this by the Closure axiom:
\[
\e\left(\bigcup_{H \in \S} H\right) = \inf_{H \in \S} \e(H)
\]
for every \(\S\subseteq\H\). The evidence against a union is therefore the weakest evidence against any of its constituent parts. On the reciprocal scale \(p=1/\e\), this becomes a maxitive set function rather than an additive one. The paper’s characterization is that E-measures are “measure-like,” but closed under infimums instead of addition [2604.20788].

This structure depends on the hypothesis class \(\H\). The class is assumed union-closed, and the strongest multiplicity results require intersection-closure. Under intersection-closure, every \(P\in\P\) has a least hypothesis
\[
H_P = \bigcap\{H \in \H : P \in H\},
\]
and every \(H\in\H\) has the canonical representation
\[
H = \bigcup_{P \in H} H_P.
\]
In that case the closure of an E-capacity is
\[
\overline{\e}(H)=\inf_{P \in H}\e(H_P), \quad H \in \H.
\]
The entire E-measure is thus determined by its values on least hypotheses [2604.20788].

The paper further shows that E-measures are the only non-dominated such objects if the hypothesis class is closed under intersections [2604.20788]. This makes the least-hypothesis representation the central structural device behind familywise evidence.

## 3. Least true hypotheses, closure, and multiplicity

The pivotal theorem states that, on an intersection-closed \((\P,\H)\), validity of an E-capacity kernel is equivalent to familywise evidence control [2604.20788]. The decisive identity is
\[
\sup_{H \ni P}\e(H \mid x)=\e(H_P \mid x).
\]
Because every true hypothesis containing \(P\) must also contain the least true hypothesis \(H_P\), antitonicity implies that the maximum evidence against any true hypothesis is attained at \(H_P\).

This has a distinctive multiplicity consequence. Under intersection-closure, plain hypothesis-wise validity already implies post hoc control of the strongest evidence against any true hypothesis in the family. The paper states this sharply: E-measures control familywise evidence and false evidence rate without multiplicity correction if the hypothesis class is intersection-closed [2604.20788]. The reason is structural rather than Bonferroni-type: the search over all true hypotheses collapses to a single least true hypothesis.

Thresholding an E-measure at level \(1/\alpha\) yields a closed testing procedure [2604.20788]. This places familywise evidence in direct relation to classical closure-based FWER logic. In the fixed-sample binary setting, closed testing itself can be viewed as a sequentially rejective procedure on the closure of \(\mathcal H\), where a hypothesis becomes testable only after all stricter implying hypotheses have been rejected [1211.3313]. The continuous E-measure formulation retains that logical organization, but it reports a full evidence map \(H\mapsto \e(H\mid x)\) rather than only reject/non-reject outcomes.

This suggests a reinterpretation of multiplicity: in logically structured classes, multiplicity is not handled by shrinking each individual claim through a universal penalty, but by exploiting the order induced by implication and intersection. In the E-measure framework, that order is encoded directly in the object that represents evidence.

## 4. False evidence rate and general false-evidence functionals

Familywise evidence is the max-type criterion. The corresponding average-type criterion is the false evidence rate (FER), introduced as a continuous-evidence analogue of false discovery rate [2604.20788].

Given a finite selection rule \(S:\X\to 2^\H\), the false evidence proportion at truth \(P\) is
\[
\textnormal{FEP}_P^S(x) := \frac{1}{|S(x)| \vee 1} \sum_{H \in S(x) : H \ni P} \e(H \mid x),
\]
and the false evidence rate is
\[
\sup_{P \in \P}\Ex^P[\textnormal{FEP}_P^S(X)].
\]
The corresponding false selection proportion is
\[
\textnormal{FSP}_P^S(x) := \frac{|S(x) \cap \{H : H \ni P\}|}{|S(x)| \vee 1}.
\]

The key bound is
\[
\textnormal{FEP}_P^S(x) \leq \textnormal{FSP}_P^S(x)\,\e(H_P \mid x) \leq \e(H_P \mid x),
\]
again showing that the least true hypothesis controls the full familywise behavior [2604.20788]. Two corollaries follow. First, for a fixed selection rule, controlling \(\Ex^P[\textnormal{FSP}_P^S(X)\e(H_P\mid X)]\) suffices for FER control. Second, on intersection-closed classes, validity is equivalent to uniform FER control across all selection rules [2604.20788].

The paper also embeds familywise evidence and FER in a broader framework based on functionals \(\Phi_P:[0,\infty]^\H\to[0,\infty]\). Validity under \((\Phi_P)\) means
\[
\Ex^P\left[\Phi_P(\e(\,\cdot \mid X))\right] \le 1 \quad \textnormal{for every } P \in \P.
\]
When \(\Phi_P\) is local, positively homogeneous, and monotonic, one has
\[
\Phi_P(\e) \le \e(H_P)\Phi_P(1_P).
\]
Taking \(\Phi_P(\eta)=\sup_{H\ni P}\eta(H)\) recovers familywise evidence; taking
\[
\Phi_P^A(\eta):=\frac{1}{|A|\vee 1}\sum_{H\in A:H\ni P}\eta(H)
\]
recovers FER [2604.20788]. The same least-hypothesis mechanism therefore governs a broad class of false-evidence criteria.

## 5. Sequential, adaptive, and online relatives of familywise evidence

Familywise evidence is formulated as a continuous generalization of familywise error control, and the surrounding literature clarifies how family-level validity behaves in binary sequential and adaptive settings. In fixed-sample multiple testing, the sequential rejection principle states that strong FWER control follows from a monotonicity condition on the rejection rule together with a single-step safety condition in the configuration where all already rejected hypotheses are false [1211.3313]. For streaming data, the rejection principle is extended by allowing the rejection function to depend on previously rejected hypotheses, previously accepted hypotheses, and current sampling state, thereby handling the specifically sequential distinction between “accepted,” “rejected,” and “still active” [1307.7667].

Adaptive data analysis introduces a different difficulty: the analyst may change the testing strategy after inspecting the data. The interactive FWER test addresses this by masking each \(p\)-value into a revealed part \(g(P_i)\) and a hidden bit \(h(P_i)\), then stopping when
\[
\widehat{\mathrm{FWER}_t = 1-(1-p_*)^{|\mathcal R_t^-|+1}
\]
falls below \(\alpha\) [2002.08545]. The formal guarantee is FWER control under independence of null \(p\)-values and independence between nulls and non-nulls. The procedure is explicitly designed so that a human or computer program may adaptively guide the testing strategy while retaining familywise validity [2002.08545].

Online testing introduces an a priori unbounded sequence of hypotheses. Online Sidak,
\[
\alpha_i := 1-(1-\alpha)^{\gamma_i},
\]
online fallback,
\[
\alpha_i := \gamma_i \alpha + \sum_{k=1}^{i-1} w_{k,i} R_k \alpha_k,
\]
and ADDIS-Spending,
\[
\alpha_i := \alpha(\tau_i-\lambda_i)\gamma_{t(i)},
\]
all provide strong familywise guarantees under their respective independence or local-dependence assumptions [1910.04900]. The paper proves these procedures via PFER control, using the implication \(FWER\le PFER\) [1910.04900].

These developments remain binary rather than continuous, but they exhibit the same underlying concern: family-level validity must survive post hoc choice, sequential updating, and structural dependence. A plausible implication is that familywise evidence extends this logic from rejection events to full evidence surfaces, while preserving the family-level guarantee through the least true hypothesis.

## 6. Scope, extensions, and limitations

The strongest familywise-evidence results require that \((\P,\H)\) be intersection-closed, that \(\e\) be an E-capacity kernel or its closure to an E-measure, and that the relevant suprema and infima be measurable [2604.20788]. Intersection-closure is not a cosmetic assumption: it is what guarantees existence of \(H_P\) and makes the identity
\[
\sup_{H \ni P}\e(H \mid x)=\e(H_P \mid x)
\]
available. Without least hypotheses, the argument that ordinary validity already controls post hoc familywise evidence does not go through in the same way.

The framework also extends beyond standard hypothesis testing. In decision-theoretic form, the paper proves a uniform E-consequence bound
\[
\Ex^P\left[\sup_{d \in \D} \e(L_{P^*}(d) \succsim \ell(d) \mid X)\right] \leq 1, \quad P \in H_\ell,
\]
which is structurally analogous to familywise evidence but indexed by decisions rather than hypotheses [2604.20788]. The same paper also advocates updating from E-prior to E-posterior and extending the framework to predictive E-measures [2604.20788].

By contrast, related binary-control frameworks often require stronger probabilistic assumptions. Step-up procedures based on Simes’ inequality need independence or certain positive dependence structures [1211.3313]. Interactive masking procedures assume null \(p\)-values are mutually independent and independent of the non-nulls [2002.08545]. Online adaptive-discarding procedures require independence or local dependence and, for the strongest guarantees, uniformly conservative null \(p\)-values [1910.04900]. These comparisons do not weaken the familywise-evidence construction; rather, they clarify that its “no multiplicity correction” phenomenon is specifically structural and tied to intersection-closed hypothesis classes.

In summary, familywise evidence reframes multiplicity from a problem of adjusting isolated \(p\)-values to a problem of representing evidence coherently across a logically structured class of hypotheses. Its central object is the E-measure, its central mechanism is the least true hypothesis \(H_P\), and its central claim is that, on intersection-closed classes, ordinary validity already yields post hoc familywise protection in a continuous-evidence sense [2604.20788].

Source: https://www.emergentmind.com/topics/familywise-evidence