AdaFilter-AdaBon: Adaptive Multiple Testing

• The paper refines the AdaFilter-Bon framework by introducing a post-filter null proportion estimator to mitigate conservativeness in detecting replicated signals.
• It employs filtering of partial conjunction p-values to identify features replicated across studies while controlling the generalized k-FWER.
• The method demonstrates enhanced power with asymptotic error guarantees and scalable O(m) computational complexity for high-dimensional analyses.

Searching arXiv for the cited papers and closely related work to ground the article. AdaFilter-AdaBon is an adaptive multiple-testing procedure for detecting replicated signals across several studies while controlling a generalized family-wise error rate, the $k$-FWER. It builds directly on the AdaFilter-Bon method of Wang et al. (2022), and improves its power by estimating the proportion of true nulls after filtering (Tran, 21 Aug 2025). In the literature, the name “AdaFilter” is also used for an unrelated adaptive fine-tuning method in deep transfer learning; that 2019 convolutional-filter method is distinct from the statistical AdaFilter line introduced for partial conjunction testing and should not be conflated with AdaFilter-AdaBon (Guo et al., 2019).

1. Historical lineage and nomenclature

AdaFilter-AdaBon belongs to a line of methods for partial conjunction (PC) hypotheses, where the scientific objective is to identify signals that replicate across multiple studies rather than signals that are merely significant in an aggregate meta-analysis. The earlier AdaFilter framework introduced adaptive filtering procedures for PC hypotheses and developed two principal variants: AdaFilter Bonferroni, often referred to informally as “AdaBon,” for FWER/PFER control, and AdaFilter BH for FDR control (Wang et al., 2016). AdaFilter-AdaBon is a later adaptive refinement of the Bonferroni branch: it retains the same filtering logic as AdaFilter-Bon, but modifies the threshold by incorporating a post-filter estimate of the null proportion (Tran, 21 Aug 2025).

The statistical use of “AdaFilter” is unrelated to the computer-vision method “AdaFilter: Adaptive Filter Fine-tuning for Deep Transfer Learning,” which is an adaptive filter-level fine-tuning framework for deep transfer learning and operates on duplicated convolutional filters and recurrent gating in ResNet-50-style architectures (Guo et al., 2019). That naming overlap is purely terminological. In the multiple-testing literature, “AdaBon” denotes the Bonferroni-style adaptive filtering procedure derived from the original AdaFilter framework (Wang et al., 2016), whereas “AdaFilter-AdaBon” denotes the later procedure that estimates the post-filter null proportion to mitigate conservativeness (Tran, 21 Aug 2025).

A common misconception is therefore to read “AdaFilter-AdaBon” as a hybrid of deep-learning fine-tuning and statistical error control. The record in the cited papers supports the opposite conclusion: AdaFilter-AdaBon is a multiple-testing procedure for replicability analysis, while the 2019 CNN AdaFilter is a transfer-learning algorithm with no direct methodological connection (Tran, 21 Aug 2025).

2. Statistical setting: partial conjunction, replicability, and $k$-FWER

The method considers a meta-analysis of $n \ge 2$ comparable studies, each measuring the same $m$ features. For feature $i$ in study $j$, there is a null hypothesis $H_{ij}$ and a corresponding valid $p$-value $P_{ij}$, meaning

$$\Pr(P_{ij} \le t) \le t \quad \text{for all } t \in [0,1] \text{ when } H_{ij} \text{ is true}.$$

Across studies, the vectors $k$0 are assumed independent, while within each study there can be dependence across features (Tran, 21 Aug 2025).

Replicability is encoded by a number $k$1. A feature is “replicated at level $k$2” if at least $k$3 of its study-specific nulls are false. The corresponding PC null is

$k$4

with the alternative that at least $k$5 of the $k$6 component nulls are false (Tran, 21 Aug 2025). Rejecting $k$7 therefore means declaring feature $k$8 replicated in at least $k$9 studies.

A key structural property is nesting: $n \ge 2$0 This nesting is central to the filtering logic, because failure to replicate at level $n \ge 2$1 precludes replication at level $n \ge 2$2 (Tran, 21 Aug 2025).

For testing, the method uses the Bonferroni PC $n \ge 2$3-value. If $n \ge 2$4 are the order statistics of the study-specific $n \ge 2$5-values, then

$n \ge 2$6

This is the choice used inside AdaFilter-Bon and AdaFilter-AdaBon (Tran, 21 Aug 2025). The focus is not the ordinary FWER alone, but the generalized $n \ge 2$7-family-wise error rate,

$n \ge 2$8

which allows up to $n \ge 2$9 false rejections. When $m$0, this coincides with FWER (Tran, 21 Aug 2025).

The motivation is the classical multiplicity burden of high-dimensional replicability analysis. Even when one has valid PC $m$1-values, testing $m$2 hypotheses with FWER or $m$3-FWER control leads to very stringent per-hypothesis thresholds. This motivates procedures that reduce the multiplicity burden without sacrificing error control, and filtering is the specific device used here (Tran, 21 Aug 2025).

3. Construction of AdaFilter-AdaBon

AdaFilter-AdaBon keeps the filtering structure of AdaFilter-Bon but adapts the rejection threshold using an estimate of the post-filter null proportion (Tran, 21 Aug 2025). For a fixed replicability level $m$4, it defines two quantities for each feature $m$5: $m$6 the testing PC $m$7-value, and

$m$8

the filtering $m$9-value, obtained from the PC null at level $i$0 (Tran, 21 Aug 2025). Because $i$1, one has $i$2.

In AdaFilter-Bon, the rejection threshold is

$i$3

so the multiplicity factor is the number retained by the filter rather than the full $i$4 (Tran, 21 Aug 2025). The later AdaFilter-AdaBon procedure modifies this by estimating the proportion of true nulls among the retained features. Fix a tuning parameter $i$5. The estimator is

$i$6

Its numerator counts retained features whose PC $i$7-value is at least $i$8, and its denominator normalizes by the retained set size and the factor $i$9 (Tran, 21 Aug 2025).

The conceptual AdaFilter-AdaBon threshold is

$j$0

Plugging in the estimator yields the operational definition: $j$1 The rejection rule is then simply: reject $j$2 if $j$3 (Tran, 21 Aug 2025).

The paper also gives an implementable finite grid

$j$4

and defines

$j$5

A key theorem states that the rejection sets coincide: $j$6 The paper states that one can compute $j$7 in $j$8 time by evaluating the left-hand side at each candidate in $j$9 (Tran, 21 Aug 2025).

This construction preserves the original AdaFilter idea—screen with $H_{ij}$0, test with $H_{ij}$1—but replaces the fixed post-filter multiplicity correction of AdaFilter-Bon with a multiplicity correction scaled by an estimated post-filter null proportion. This suggests a direct mechanism for reducing conservativeness when filtering preferentially retains alternatives.

4. Theoretical guarantees and regularity conditions

The main theoretical result is asymptotic $H_{ij}$2-FWER control under weak dependence assumptions (Tran, 21 Aug 2025). The paper defines $H_{ij}$3 as the number of true PC nulls and $H_{ij}$4 as the number of alternatives, and assumes almost-sure convergence of the empirical distributions of $H_{ij}$5 and $H_{ij}$6 under both the null and the alternative. It also assumes

$H_{ij}$7

These are presented as typical empirical-process convergence assumptions encompassing many weak-dependence structures, including finite blocks and mixing processes (Tran, 21 Aug 2025).

A central inequality is

$H_{ij}$8

which reflects a conditional validity lemma: under the PC null and independence of the study $H_{ij}$9-values,

$p$0

This is the asymptotic analogue of the conditional validity phenomenon that already underpinned the original AdaFilter theory (Wang et al., 2016).

Let

$p$1

be the rejection set and

$p$2

the number of false discoveries. The main theorem states that if Assumption 1 holds, if $p$3 for some $p$4, and if there exists $p$5 such that

$p$6

with probability $p$7, then

$p$8

(Tran, 21 Aug 2025).

The paper’s interpretation is that the estimator $p$9 is constructed so that, under the null, the proportion of retained features with $P_{ij}$0 is at least $P_{ij}$1 in expectation. Intuitively, this makes $P_{ij}$2 a conservative estimate of $P_{ij}$3 in large samples, so using it in the threshold tightens the constraint when necessary rather than weakening it unsafely (Tran, 21 Aug 2025).

The guarantees are explicitly asymptotic. The paper states that no explicit finite-sample guarantees are provided, although simulations show good empirical control across a range of settings and correlations (Tran, 21 Aug 2025). This is an important qualification: AdaFilter-AdaBon is theoretically rigorous, but its formal guarantee is not a finite-sample exact $P_{ij}$4-FWER theorem in the style of the original AdaFilter Bonferroni result under full independence (Wang et al., 2016).

5. Power improvement, simulations, and computational profile

The motivation for AdaFilter-AdaBon is the conservativeness of AdaFilter-Bon. The paper gives the bound

$P_{ij}$5

for AdaFilter-Bon under independent and valid $P_{ij}$6 (Tran, 21 Aug 2025). Because filtering preferentially retains hypotheses with signal, the post-filter null proportion $P_{ij}$7 can be much smaller than $P_{ij}$8, so AdaFilter-Bon may control at a level far below the nominal $P_{ij}$9. AdaFilter-AdaBon is designed to mitigate exactly this conservativeness.

The paper’s heuristic comparison is explicit. Under AdaFilter-Bon, the effective threshold is roughly

$$\Pr(P_{ij} \le t) \le t \quad \text{for all } t \in [0,1] \text{ when } H_{ij} \text{ is true}.$$0

where $$\Pr(P_{ij} \le t) \le t \quad \text{for all } t \in [0,1] \text{ when } H_{ij} \text{ is true}.$$1 is the number retained by the filter. Under AdaFilter-AdaBon, the effective threshold is roughly

$$\Pr(P_{ij} \le t) \le t \quad \text{for all } t \in [0,1] \text{ when } H_{ij} \text{ is true}.$$2

If $$\Pr(P_{ij} \le t) \le t \quad \text{for all } t \in [0,1] \text{ when } H_{ij} \text{ is true}.$$3 correctly estimates the post-filter null proportion, the expected number of false rejections becomes approximately $$\Pr(P_{ij} \le t) \le t \quad \text{for all } t \in [0,1] \text{ when } H_{ij} \text{ is true}.$$4, and the threshold is larger by approximately a factor of $$\Pr(P_{ij} \le t) \le t \quad \text{for all } t \in [0,1] \text{ when } H_{ij} \text{ is true}.$$5 (Tran, 21 Aug 2025). This suggests that the method is specifically advantageous when filtering removes a substantial fraction of true nulls.

The reported simulations use $$\Pr(P_{ij} \le t) \le t \quad \text{for all } t \in [0,1] \text{ when } H_{ij} \text{ is true}.$$6 features and $$\Pr(P_{ij} \le t) \le t \quad \text{for all } t \in [0,1] \text{ when } H_{ij} \text{ is true}.$$7 studies, with blockwise equicorrelated Gaussian noise within each study, $$\Pr(P_{ij} \le t) \le t \quad \text{for all } t \in [0,1] \text{ when } H_{ij} \text{ is true}.$$8, signal density $$\Pr(P_{ij} \le t) \le t \quad \text{for all } t \in [0,1] \text{ when } H_{ij} \text{ is true}.$$9, replicability levels $k$00, and target FWER ($k$01) at $k$02 (Tran, 21 Aug 2025). Compared methods include AdaFilter-AdaBon ($k$03), AdaFilter-Bon, standard Bonferroni and Hochberg applied to Fisher PC $k$04-values, and adaptive Bonferroni and adaptive Hochberg applied to Fisher PC $k$05-values.

The stated findings are that AdaFilter-AdaBon maintains FWER below $k$06 across all settings; its FWER is systematically higher than AdaFilter-Bon but still comfortably below target; and it yields notably higher TPR than AdaFilter-Bon in most settings, especially for $k$07 and moderate-to-high signal density $k$08 (Tran, 21 Aug 2025). For very stringent replication ($k$09) and extremely sparse signals ($k$10), AdaFilter-Bon matches AdaFilter-AdaBon’s power. Additional simulations for $k$11 and $k$12 show that all methods are very conservative, while AdaFilter-AdaBon is the least conservative among them and has substantially higher power, especially when $k$13 (Tran, 21 Aug 2025).

On implementation, the paper states that sorting the study $k$14-values per feature is $k$15 per feature, but because $k$16 is small, the overall complexity is $k$17 in practice; constructing $k$18 is $k$19; and evaluating the adaptive statistic can be made linear by cumulative counts (Tran, 21 Aug 2025). The method therefore scales linearly in $k$20 up to large numbers of features.

6. Relation to AdaFilter, AdaBon, and adjacent procedures

The original AdaFilter formulation introduced the filtering and selection statistics

$k$21

and used them to define the AdaFilter Bonferroni threshold

$k$22

for FWER/PFER control (Wang et al., 2016). In that sense, AdaFilter-AdaBon is not a new filtering architecture but a modification of the calibration step: the term $k$23 remains, but it is multiplied by an adaptive estimate of the post-filter null proportion (Tran, 21 Aug 2025).

This relation is best understood in three layers. First, AdaFilter is the general framework for adaptive filtering in PC testing (Wang et al., 2016). Second, AdaBon or AdaFilter Bonferroni is the Bonferroni-style instantiation of that framework (Wang et al., 2016). Third, AdaFilter-AdaBon is the later procedure that augments AdaFilter-Bon with post-filter null-proportion estimation to improve power while preserving asymptotic $k$24-FWER control (Tran, 21 Aug 2025).

The paper also positions AdaFilter-AdaBon relative to standard Bonferroni, Holm, and Hochberg procedures for PC hypotheses. Those methods treat the PC $k$25-values like ordinary single-study $k$26-values and penalize all $k$27 hypotheses equally; they do not use filtering and do not use null-proportion estimation; and they are therefore described as severely conservative in high-dimensional replicability analysis (Tran, 21 Aug 2025). In the broader replicability literature, two-stage selection-and-testing procedures and FDR-based partial conjunction methods address related questions, but they do not estimate post-filter null proportions in the same way (Tran, 21 Aug 2025).

The paper further notes that AdaFilter-AdaBon is defined using Bonferroni PC $k$28-values. If one prefers more powerful combining functions such as Fisher, one cannot directly plug them into this specific algorithm; adapting the method to other combining functions would require new theory (Tran, 21 Aug 2025). This is a substantive methodological boundary rather than a mere implementation detail.

Finally, the 2025 paper outlines extensions beyond $k$29-FWER. It states that AdaFilter-AdaBon can be augmented to control the false exceedance rate (FDX) and the false discovery rate (FDR) asymptotically through a second threshold $k$30 and a Genovese–Wasserman-style bound (Tran, 21 Aug 2025). A plausible implication is that the adaptive-filtering-plus-post-filter-estimation principle is broader than the specific $k$31-FWER instantiation, although the developed theory in the paper is centered on replicability analysis with Bonferroni PC $k$32-values.

7. Interpretation, scope, and limitations

A rejection by AdaFilter-AdaBon means that the corresponding feature is declared replicated in at least $k$33 studies, with the global error metric controlled at the specified $k$34-FWER level in the asymptotic sense established by the paper (Tran, 21 Aug 2025). This is a stronger statement than ordinary meta-analytic significance, because the null explicitly concerns the number of studies in which the effect is non-null.

The method is expected to be most beneficial when there are many features, when a nontrivial fraction of features are truly replicated at level $k$35, when the replication requirement $k$36 is modest, and when dependence across features is moderate rather than pathological (Tran, 21 Aug 2025). This follows the paper’s discussion that filtering is especially informative when $k$37 replication serves as a meaningful screen for $k$38-level replication.

Several limitations are explicit. The asymptotic theory allows certain forms of weak dependence across features, but very strong or complex dependence structures may violate the assumptions (Tran, 21 Aug 2025). The guarantees are asymptotic as $k$39, so for small $k$40 the estimator $k$41 may be noisy and the adaptive threshold may have less predictable behavior (Tran, 21 Aug 2025). The formal theory is developed for Bonferroni PC $k$42-values, and the asymptotic $k$43-FWER theorem assumes that $k$44 grows linearly with $k$45, even though simulations examine fixed small $k$46 such as $k$47, $k$48, and $k$49 (Tran, 21 Aug 2025).

These limitations do not negate the method’s contribution; they define its scope. In the statistical literature, AdaFilter-AdaBon is best understood as an adaptive refinement of AdaFilter-Bon for replicated-signal detection under partial conjunction testing, with asymptotic $k$50-FWER control and empirically higher power than the original AdaFilter-Bon (Tran, 21 Aug 2025). In contrast, the identically named but unrelated deep-learning AdaFilter remains a transfer-learning procedure on convolutional networks and has no role in the statistical construction of AdaFilter-AdaBon (Guo et al., 2019).