Finite Resources False Discovery Rate Control in Structured Hypothesis Spaces

Abstract: Scientific discovery relies on large-scale hypothesis testing. However, the capacity to identify true discoveries while controlling false discovery faces major challenges: obtaining relevant reference data (the null distribution) is resource-intensive, leaving finite-data uncertainty, and the procedure should account for the inherent structure in the hypothesis space, when such structure exists. Here, we present a framework for controlling the false discovery rate both when each hypothesis is evidenced only by a finite count of null draws, leaving its p-value uncertain, and when the hypothesis space carries arbitrary structure, requiring only that the structure be represented through a suitable reproducing kernel. We present two decision rules that are both robust to structural mis-specification, yet offer a distinct trade-off between exact FDR control and statistical power. The first rule guarantees exact FDR control; the second maximizes power by adapting mirror-statistic control into count space, utilizing an analytical framework to assess FDR control when exact mirror symmetry is relaxed. Furthermore, the tractability gained by the RKHS framework allows us to directly investigate finite-data uncertainties, which we leverage to suggest a policy for the efficient allocation of null distribution samples.

• The paper introduces a framework for FDR control using finite null draws and RKHS-based structure to rigorously balance error control with discovery power.
• It develops two decision rules, including a mirror-reflection method, that provide exact or bounded FDR guarantees even under finite-sample constraints.
• The study also proposes an adaptive null-sample allocation strategy that optimizes resource use and enhances detection power in high-dimensional multiple testing scenarios.

Finite-Resource False Discovery Rate Control in Structured Hypothesis Spaces

Introduction and Motivation

Large-scale hypothesis testing in scientific contexts, such as genomics or machine learning benchmarks, requires accurate identification of true signals while rigorously controlling the false discovery rate (FDR). Conventional FDR procedures, reliant on precise p-value estimation under $H_0$ for each hypothesis, are often cost-prohibitive when only a limited number of reference (null) samples can be obtained per test. Moreover, hypothesis spaces are typically structured (e.g., by spatial, hierarchical, or graph relationships), and utilizing this structure can enhance inference, yet most FDR procedures either ignore it or assume incorrectly that p-values are precisely known.

This work introduces a comprehensive framework for FDR control in the simultaneous presence of two fundamental constraints:

• Finite null draws per hypothesis (microsampling regime), where the evidence for each hypothesis is a discrete count of null draws as or more extreme than the observed statistic, rather than a fully resolved p-value.
• Arbitrarily structured hypothesis spaces, represented via reproducing kernel Hilbert spaces (RKHS), enabling information sharing across related hypotheses.

The framework advances both the statistical theory—by characterizing FDR and power guarantees under these constraints—and the methodology, by deriving computationally tractable estimators and adaptive allocation strategies for null-sampling resources. Two principal decision rules are developed, trading off FDR exactness and power, with a rigorous analysis of their operating characteristics.

Generative Model and Likelihood Specification

Each hypothesis $i$ is defined by its location $loc_i$ in a domain $\mathcal{X}$, admitting a symmetric positive-definite kernel $K$. The model hierarchy is:

1. Structural prior: The null probability $\Pr(H_{0,i}) = \alpha(loc_i)$, where $\alpha(\cdot)$ is an unknown function in the RKHS.
2. Latent $p$-value layer:

$$p_i^* \mid \theta_i = 0 \sim \text{Uniform}[0,1],\quad p_i^* \mid \theta_i = 1 \sim f_1(p)$$

1. Observation: Only the count $k_i$ of at least-as-extreme null draws out of $i$0 is observed, not the latent $i$1.

To propagate finite-data uncertainty, the FDR framework marginalizes the unobserved $i$2-value via the Beta-Binomial conjugacy: $i$3 for alternative $i$4, $i$5. This yields closed-form marginal likelihoods per hypothesis and circumvents the inadequacy of plug-in $i$6-value approaches, particularly in the low $i$7 regime.

Estimation and Spatial Model Fitting

The structural prior $i$8 is estimated as the regularized MLE in RKHS: $i$9 where global parameters $loc_i$0 are consistently estimated from marginal counts, independently of structure. The representer theorem ensures finite-dimensional tractability. Natural Gradient Descent is employed, obviating explicit inversion of the kernel matrix.

Decision Rules and FDR Control

Rule 1 (Model-Free):

A two-stage approach: - Use the spatial $loc_i$1 as a gate for preselection. - Apply Sun-Cai's running average rule to the marginal (non-spatial) $loc_i$2 within the gated set.

Guarantee:

Exact FDR control at the nominal level for any spatial structure or mis-specification, though the approach can be conservative—leaving statistical power on the table (see Theorem \ref{thm:rule1_fdr} in the paper).

Rule 2 (Mirror Statistics in Count Space):

Generalizes the knockoff/mirror approach to the finite-resource, structured regime: - Reject hypotheses directly according to $loc_i$3. - Implement a mirror-reflection $loc_i$4 for each $loc_i$5, using step-up thresholds as in the Barber–Candès procedure.

Guarantee:

Under exact mirror symmetry, FDR control is exact. In practice, data-dependent $loc_i$6 breaks symmetry, so relaxation is quantified in terms of a single-flip influence $loc_i$7, and FDR is controlled up to this slack (Theorem~\ref{thm:rule2_fdr}). The procedure controls the error to within grid scale, and in the infinite data limit coincides with the optimal Bayes rule. Critically, Rule 2 uniformly dominates Rule 1 in power whenever structure is informative and properly specified.

Empirical Evaluation and Numerical Results

Rule 2 achieves power close to the oracle (fully specified) solution, while maintaining FDR within the specified budget—even under strong misspecification and low null-draw regimes. Rule 1 is conservative, especially under structural misspecification.

Figure 1: FDR and power as a function of null-draws per hypothesis, under various structural signal and smoothness configurations; the dashed line denotes the target FDR level.

Active null-sampling allocation, driven by explicit uncertainty quantification, decisively accelerates decision-making relative to uniform or random baselines, maximizes discovery power per budget, and converges quickly to the oracle parameterization.

Efficient Null Sample Allocation

The framework leverages tractable Hessian-based estimates of per-hypothesis uncertainty to design adaptive null-calibration allocation strategies. The policy classifies hypotheses as 'decided', 'ambiguous', or 'hopeless' (irreducible uncertainty due to isolation or uninformative $loc_i$8-values), and prioritizes further null draws where they will most rapidly reduce decision ambiguity.

High-level operational flow:

(Figure 2)

Figure 2: Block diagram for the priority-driven null sample allocation policy, dynamically allocating budget based on local and spatial uncertainty decomposition.

This approach systematically outperforms uniform allocation against all key criteria: fraction of decisively classified hypotheses, true discovery rate (power), and explicit FDR control.

Theoretical and Practical Implications

Theoretical:

• Count-based FDR inference yields discretization error $loc_i$9 for the local FDR statistic, outperforming conventional plug-in approaches in the finite-sample regime.
• The RKHS machinery enables nonparametric pooling across arbitrary structure, generalizing prior graph- and spatial-smoothing methods far beyond classical total variation or nearest-neighbor penalty approaches.
• FDR relaxation under broken mirror symmetry is rigorously bounded and can be made arbitrarily tight by regularization and increased sampling.

Practical:

• The method dramatically reduces the required number of calibration null samples per hypothesis, scaling hypothesis testing to previously intractable domains (e.g., high-dimensional genomics, model evaluation benchmarks, etc).
• The adaptive allocation policy proposes a new paradigm in resource-constrained multiple testing: recognize and exploit variance floors and spatial coupling.
• In the context of LLM evaluation benchmarks with intrinsic discreteness and no structure, the count-based approaches enable discoveries impossible for conventional FDR, even for small calibration budgets.

Conclusion

This work delivers a statistically rigorous, computationally tractable, and empirically validated framework for FDR control under finite calibration resources and structured hypothesis spaces. The analytical framework tightly characterizes the control–power tradeoff, quantifies relaxation under realistic model violations, and provides practical procedures for resource allocation and uncertainty quantification. The intersection with RKHS theory opens new directions for structured multiple testing in high-dimensional and non-Euclidean domains, with immediate relevance for scientific discovery pipelines and competitive benchmark evaluation.

---

Figure 1: Sensitivity of FDR control and power across null-draw budget and structure smoothness; robust FDR is maintained for both decision rules, while baseline methods lose control in the finite-resource regime.