Notip: Permutation-based fMRI Inference

• Notip is a permutation-based post hoc inference method for neuroimaging that provides reliable lower bounds on the true discovery proportion of active voxels.
• It leverages nonparametric calibration of spatial dependence through permutation to achieve robust inference for both large and small activation clusters.
• Notip enables drill-down fMRI analyses by adaptively learning threshold sequences, overcoming the spatial specificity limitations of classical cluster-extent methods.

Notip is a post hoc inference procedure for neuroimaging data, designed as a permutation-based extension of All-Resolutions Inference (ARI). Notip provides valid lower confidence bounds on the true discovery proportion (TDP) of active voxels within any subset, addressing the spatial specificity paradox that limits classical cluster-extent thresholding in functional MRI (fMRI) analysis. Originating as an alternative to both analytic ARI and parametric permutation-based ARI ("pARI"), Notip learns the spatial dependence structure of fMRI data nonparametrically, enabling robust and informative inference especially for small and moderate-sized activation clusters (Peyrouset et al., 4 Nov 2025).

1. Theoretical Foundations: All-Resolutions Inference (ARI)

ARI is a nonparametric simultaneous inference framework that, given a family of hypotheses (e.g., voxelwise nulls in fMRI), defines for any subset $S$ the true discovery proportion: $$\mathrm{TDP}(S) = 1 - \frac{|S \cap \mathcal{H}_0|}{|S|}$$ where $\mathcal{H}_0$ denotes the unknown set of true nulls. The central ARI guarantee is the existence of a function $\overline{\mathrm{TDP}}_\alpha(S)$ such that for all $S$: $$P\left(\forall S \subseteq \mathcal{H}:\ \mathrm{TDP}(S) \geq \overline{\mathrm{TDP}}_\alpha(S)\right) \geq 1-\alpha$$ This delivers simultaneous $1-\alpha$ lower bounds across all (possibly data-driven) subsets, thus enabling valid post hoc scientific statements about any region of interest (Peyrouset et al., 4 Nov 2025).

Traditionally, ARI leverages the Simes or Hommel inequalities to compute analytic threshold families for controlling the joint error rate (JER) under independence or positive regression dependency on a subset (PRDS). The lower bound is constructed as

$$\overline{\mathrm{TDP}}^t(S)=\frac{1}{|S|} \max_{k=1\ldots K}\left[1-k+\sum_{i \in S} \mathbf{1}\{p_i < t_k\}\right]$$

for a nondecreasing sequence $t=(t_k)_{k=1}^K$ controlling the JER.

2. Rationale for Permutation-Based Extensions

fMRI data exhibit strong, spatially nontrivial dependence structures that violate the weak dependence assumptions of Simes and PRDS. This can lead analytic ARI bounds to be conservative, reducing power especially at finer spatial resolutions (i.e., "drill-down" settings).

Permutation-based methods address this by explicitly calibrating threshold sequences $t$ under the empirical null, capturing dependencies inherent to the data. In this context, both Notip and pARI generate custom thresholds, but differ fundamentally in their approach to template construction and calibration (Peyrouset et al., 4 Nov 2025).

3. Construction and Algorithmic Details of Notip

Notip forgoes parametric forms for threshold templates. Its procedure is as follows:

• Step 1: Generate $B$ null datasets by permutation (or sign-flipping) of the data under the global null hypothesis.
• Step 2: For each permutation, compute the ordered null p-values $p_{(1)}, p_{(2)}, \ldots, p_{(m)}$.
• Step 3: For $k$ up to a designated maximum $K$ (e.g., $K = 2\% \cdot m$), define $t_k$ as the empirical $(1-\alpha)$ quantile of the $k$-th smallest null p-value across all permutations:

$$t_k = \text{quantile}_{1-\alpha}\bigl\{ p_{(k)}^{(b)}: b=1,...,B \bigr\}$$

• Step 4: Compute, for any subset $S$, $\overline{\mathrm{TDP}}^t(S)$ via the same max formula as in analytic ARI, now using the data-driven $t_k$.

This guarantees $\mathrm{JER}(t) \le \alpha$ nonparametrically, yielding exact error control under the null distribution as instantiated by the data (Peyrouset et al., 4 Nov 2025).

Pseudocode for Notip threshold generation (Editor’s term):

def notip_thresholds(null_pval_matrix, alpha, K): # null_pval_matrix: shape B x m, each row = p-values under one permuted null thresholds = [] for k in range(1, K+1): k_smallest = np.sort(null_pval_matrix, axis=1)[:, k-1] t_k = np.quantile(k_smallest, 1-alpha) thresholds.append(t_k) return thresholds

4. Comparative Properties: Notip versus pARI

Both Notip and pARI achieve valid JER control, but differ in template generation:

• pARI: Uses parametric threshold families $$t_k^\delta(\lambda) = [\lambda (k-\delta)/(m-\delta)]\ \mathbf{1}_{k>\delta}$$, with the lower bound being zero for sets smaller than the cutoff $\delta$. The value of $\lambda$ is chosen via permutation testing.
• Notip: Constructs an entirely empirical data-driven template, unencumbered by explicit minimal set size (no $\delta$ cutoff). This enables Notip to provide informative bounds for smaller clusters where pARI is inoperative (i.e., yields zero lower bounds).

Systematic reanalysis shows:

Regime	Notip Performance	pARI Performance
Large clusters ($|S| \gg 1$)	Competitive, stable	Highest TDP bounds
Small clusters ($|S| \lesssim \delta$)	Robust, informative bounds	Often zero bounds; sometimes inferior to analytic ARI

For drill-down analyses, Notip preserves simultaneous error control as the threshold is adaptively raised, yielding interpretable bounds in successively refined regions. In contrast, pARI's minimal cluster size threshold leads to non-informative (zero) lower bounds in such settings, limiting spatial resolution for exploratory post hoc inference (Peyrouset et al., 4 Nov 2025).

5. Performance Regimes and Practical Implications

A large-scale comparison on Neurovault datasets (Peyrouset et al., 4 Nov 2025) indicates:

• For very large voxel clusters, pARI with $\delta>0$ typically yields the highest lower confidence bounds on TDP.
• Notip is uniformly more powerful than analytic ARI and remains sensitive for both large and, crucially, smaller clusters. Notip never imposes a hard cutoff, facilitating “drill-down” exploration within larger activation areas, i.e., subregions defined post hoc.
• There is no universally optimal method (“no free lunch”): Notip and pARI excel in complementary regimes.

A plausible implication is that Notip serves the needs of investigators requiring both high power for large/maximal clusters and robust post hoc inference on finer spatial domains.

6. Computational and Implementation Considerations

• Computational Cost: The main cost in Notip is the generation of $B$ permuted null maps and, for each, the computation and storage of ordered p-values up to $K$ ($K \ll m$ in practice). Empirical quantile computation is efficiently vectorized.
• Simultaneous Validity: Notip's construction ensures valid inference over all cluster sizes and thresholds, supporting arbitrarily many sequential, interactive, or nested queries (“double dipping”) without error inflation.
• Practical Use: Notip is suitable for pipelines needing robust inference across a range of spatial resolutions, including exploratory analyses and large-scale meta-analysis in neuroimaging.

7. Limitations and Guidance

• Resolution-Cluster Trade-off: Notip's TDP bounds may be less tight than pARI for the very largest clusters; conversely, it avoids power loss and outright zero bounds on small clusters.
• Empirical Calibration: The method’s fidelity depends on realistic null permutations/sign-flips—careful permutation schemes are necessary in complex experimental designs.
• Choice of K: The upper limit $K$ should be chosen conservatively (e.g., $K=2\%\,m$) to balance resolution with computational tractability.

In summary, Notip offers a nonparametric, permutation-calibrated, all-resolutions inference framework that robustly supports lower confidence bounds on TDP for any post hoc subset in fMRI, and can be systematically preferred in analyses requiring flexibility and spatial adaptability (Peyrouset et al., 4 Nov 2025).