Sequential multiple testing with multiple hypotheses and prior information on the hypothesis configuration

Abstract: In this work, we study the problem of testing the marginal distributions of multiple independent, sequentially observed data streams, where for each stream there are multiple candidate hypotheses to select from, in the presence of prior information on the unknown hypothesis configuration. The goal is to understand the benefit of such information and to design a sequential testing procedure that effectively leverages it. We start with arbitrary prior information and specialize to concrete examples, including known number or known lower bound on the number of streams following each hypothesis, and the presence of exclusive hypotheses. The designed procedure is three-fold: (i) reliable, i.e., controlling all types of familywise error probabilities below arbitrary user-specified levels, (ii) computationally efficient, i.e., focusing on minimal sets of alternative hypothesis configurations in making decisions, and (iii) asymptotically optimal, i.e., achieving the minimum expected sample size among all reliable procedures asymptotically as the error levels go to zero. Numerical studies are presented for illustration.

• The paper demonstrates that incorporating prior configuration information yields asymptotically optimal procedures by establishing a universal lower bound on sample size.
• It introduces minimal adjacent alternative sets that significantly reduce computational complexity in multi-hypothesis testing problems.
• Numerical studies confirm effective familywise error control and linear scaling of expected sample size with threshold levels across different prior settings.

Sequential Multiple Testing with Multiple Hypotheses and Prior Information: A Formal Summary

Problem Formulation and Motivation

The paper "Sequential multiple testing with multiple hypotheses and prior information on the hypothesis configuration" (2606.00839) addresses a generalization of sequential multiple testing in settings where each data stream is associated with more than two candidate hypotheses and prior information regarding the global hypothesis configuration is available. In such scenarios, controlling familywise error probabilities while minimizing average sample size poses significant technical challenges, particularly due to the intricate combinatorial structures induced by multiple hypotheses and potential prior information such as constraints on the numbers of streams following each hypothesis.

The motivation for this work arises from diverse application domains, including multi-channel signal detection, multi-sensor anomaly identification, and multi-endpoint clinical trials, where practical constraints often introduce prior information, e.g., known or bounded numbers of signal streams.

Universal Lower Bound and Asymptotic Optimality

A universal lower bound for the expected sample size under familywise error constraints is established. Specifically, for any error tolerance levels $(\alpha_{i,j})_{i \neq j}$ and prior information $A$ about allowed configurations, the asymptotic lower bound is determined by the worst-case evidence accumulation rate (Kullback-Leibler divergence) against each possible error type, tightly matching required log-evidence for error control. For configuration $H$, the lower bound reads as:

$$L_H(\alpha, A) \gtrsim \max_{i \neq j} \frac{|\log \alpha_{j,i}|}{I_{i,j}(H, A)},$$

where $I_{i,j}(H, A)$ denotes the minimal KL divergence between $H$ and alternative configurations that induce a type-$(j,i)$ error.

This result provides an operational benchmark for asymptotic optimality: any sequential procedure controlling all familywise errors below specified levels cannot, in expectation, outperform the above rate.

Procedure Design and Computational Efficiency

The naive approach would require full enumeration over all possible alternative hypothesis configurations consistent with prior information, which is exponentially complex. The paper introduces a principled reduction by defining minimal adjacent alternative sets $\widetilde{Alt}_{i,j}(H, A)$ corresponding to configurations that differ from $H$ in the smallest possible way relevant to each type of error. This enables significant computational savings, especially under strong prior information.

The proposed procedure operates as follows:

• At each time step, identify the maximum-likelihood configuration across streams.
• For each error type $(i, j)$, compare supporting evidence (difference in log-likelihoods) between the current maximum-likelihood configuration and all minimal adjacent alternatives.
• If all evidences exceed corresponding thresholds and the configuration adheres to prior constraints, stop and declare decisions.

Crucially, the threshold selection is shown to achieve familywise error control via Ville's inequality and union bounds; thresholds scale logarithmically with error tolerances and the cardinality of minimal alternatives.

Specialization to Prior Information Scenarios

The framework is specialized to four canonical scenarios:

• No Prior Information: All configurations considered; alternatives differ from the current by single stream modifications.
• Known Exact Numbers: Stream partitions by hypothesis are fixed; errors manifest not just in pairwise exchanges but also cyclic relabelings, demanding control over all cyclic patterns.
• Known Lower Bounds: Only minimal numbers per hypothesis guaranteed; the combinatorial structure of minimal alternatives involves chains and cycles, depending on tightness of bounds.
• Exclusive Hypotheses: Certain hypotheses cannot co-occur; minimal alternatives often involve moving all streams from one exclusive type before another can be assigned.

Each scenario yields closed-form expressions for the minimal KL rates $A$0 and for the stopping rule, with substantial reduction in complexity when prior information is strong.

Numerical Studies

Empirical studies confirm theoretical predictions:

• Expected Sample Size: Simulations show linear scaling with thresholds and convergence of expected sample size to asymptotic rates set by the minimal KL divergences, directly reflecting prior information strength.

Figure 1: Expected sample sizes (left) and scaling ratios (right) versus equal thresholds for various prior information regimes.

• Error Control: Monte Carlo importance sampling is adapted for rare-event estimation of misclassification probabilities, demonstrating reliability of the error control and rapid decay of relative estimation error.

Figure 2: Negative base-$A$1 logarithm of actual error probabilities (left) and relative estimation error (right) for the proposed procedure and variants, illustrating the effectiveness and necessity of all stopping criteria.

The numerical results highlight the gain from strong prior information in reducing average sample size and establishing sharper evidence thresholds. Importantly, neglecting cyclic error constraints in known-number settings leads to catastrophic loss of error control, reinforcing the necessity of all stopping criteria.

Extension to Composite Hypotheses

The methodology is extended to composite (parametrized) hypotheses. KL rates are replaced by minimal rates over parameter spaces. The error control and asymptotic optimality theory carry over under appropriate regularity and convergence assumptions. Threshold calibration, now depending on maximal log-likelihood over unknown parameter sets, can be handled via universal large-deviation bounds and mixture martingales.

Practical and Theoretical Implications

From a practical standpoint, the paper establishes a scalable testing framework for sequential anomaly detection, signal classification, and clinical data analysis in settings involving complex hypothesis structures and prior constraints. Theoretical contributions include

• Generalization of sequential testing optimality theory from binary to multi-hypothesis configurations,
• Characterization of the role of prior information in sample complexity and computational efficiency,
• Identification of cyclic error structures as an essential ingredient in reliable decision-making.

Potential future directions highlighted include: second-order sample size analysis, generalized error metrics (e.g., FDR), asynchronous decision-making, and incorporation of sampling constraints or stream dependencies.

Conclusion

This paper rigorously extends sequential multiple testing to multi-hypothesis settings with prior information on configuration, providing both fundamental limits for expected sample size and computationally efficient, asymptotically optimal procedures that exploit minimal adjacent alternatives and cyclic error structures. The empirical and theoretical analysis confirms the significance of prior information for performance, and the practical utility of the approach across statistical signal processing and data science domains.