Topological Differential Testing (TDT)

• Topological Differential Testing is defined as using algebraic-topological invariants—such as Euler characteristic curves, persistence diagrams, and Betti functions—to summarize data structure and test distributional differences.
• It leverages permutation testing and vectorized representations to improve statistical power and computational efficiency over classical methods.
• TDT extends to software consensus analysis by employing weighted Dowker complexes for identifying inconsistency-inducing inputs and facilitating interpretable root cause analysis.

Topological Differential Testing (TDT) encompasses a family of statistical and algorithmic methodologies leveraging computational topology—in particular, tools such as Euler characteristic curves (ECC), persistence diagrams, Betti functions, and simplicial complexes—to detect and characterize differences between datasets, algorithms, or software behaviors. Rooted in topological data analysis (TDA), TDT provides robust, dimension-agnostic hypothesis testing frameworks and novel approaches to consensus extraction in settings where classical methods are insufficient or inapplicable (Dłotko et al., 2022, Islambekov et al., 2023, Moon et al., 2020, Ambrose et al., 2020).

1. Theoretical Foundations and Core Constructs

At the heart of TDT is the use of algebraic-topological invariants to represent and summarize the geometric and combinatorial structure of data. The key constructions include:

• Euler Characteristic Curve (ECC): Given a finite point cloud $X = \{x_1,\ldots,x_n\} \subset \mathbb{R}^d$, the ECC is defined as $ECC_X(r) = \chi(C_r(X))$, where $C_r(X)$ is the Čech complex at scale $r$. The ECC encodes the alternating count of simplices—vertices, edges, faces, etc.—as a function of $r$, reducing complex multivariate geometry to a structured, one-dimensional summary.
• Persistence Diagrams and Betti Functions: Persistence diagrams summarize topological features—connected components, loops, voids—across scales. The Betti function $\beta_k^D(t)$ counts the active $k$-dimensional homology classes at scale $t$. Integrated Betti vectors and persistence images map these functions into finite-dimensional feature vectors for statistical analysis (Islambekov et al., 2023, Moon et al., 2020).
• Simplicial Complexes for Consensus Analysis: In algorithmic testing, the Dowker complex encodes the pattern of accept/reject behavior of multiple programs across a set of inputs as a weighted simplicial complex, facilitating the localization of inconsistency-inducing inputs and implicit “de facto specifications” (Ambrose et al., 2020).

These topological constructs are designed for stability under perturbation, enabling robust statistics and interpretable summaries in arbitrary dimensions and across disparate problem domains.

2. Topology-Driven Statistical Testing Procedures

TDT primarily addresses two classes of inference tasks: goodness-of-fit (GoF) and two-sample (differential) testing. The principal procedures are as follows:

• ECC-Based GoF and Two-Sample Testing (“TopoTests”): For a sample $X \sim G$ and a null hypothesis $H_0: G$ is Euler-equivalent to $F$, the test statistic is

$$T_1(X; F) = \Delta_n = \sup_{0 \le r \le T} n^{-1/2} \left| ECC_X(r) - \mu_F(n,r) \right|,$$

where $\mu_F(n,r)$ is the expected ECC under $F$. For two samples $X \sim F$ and $Y \sim G$, the two-sample statistic is

$$T_2(X,Y) = \sup_{0 \le r \le T} \left| m^{-1} ECC_X(r) - n^{-1} ECC_Y(r) \right|.$$

Type I error control and exponentially vanishing type II error are established via asymptotic Gaussian process theory and concentration inequalities (Dłotko et al., 2022).

• Vectorized Persistence Function Testing: Persistence diagrams are vectorized either via integrated Betti functions (Islambekov et al., 2023) or persistence images (Moon et al., 2020). Two families of test statistics are prominent:
  • Permutation Test with Diagram Distances: Baseline approaches use pairwise Wasserstein or bottleneck distances among diagrams, with null distribution estimated via group-label permutations.
  • Permutation Test with Vectorized Summaries: Replacing expensive diagram distances with fast $\ell_1$ (or $\ell_2$) vector distances on Betti or image vectors preserves stability and enables large-scale inference.
• Maximal-Mixing Permutation Schemes: Empirically, permutation tests gain power by restricting to permutations that maximally disrupt original grouping—i.e., maximal Hamming distance—while maintaining exchangeability and theoretical validity (Islambekov et al., 2023).
• Two-Stage Multiple Testing for Persistence Images: Filtering uninformative coordinates in the vectorized persistence representations (e.g., via variance thresholds), followed by multiple $t$-tests and Benjamini-Hochberg FDR control, produces interpretable, feature-resolved $p$-value maps (Moon et al., 2020).

3. Topological Analysis of Software and Classifier Consensus

A distinct line of TDT applies algebraic topology for extracting consensus specifications among multiple programs or classifiers without formal oracles:

• Weighted Dowker Complexes: The accept/reject relation $R \subseteq [m] \times [n]$ for $m$ programs and $n$ inputs is projected onto the power set $2^{[m]}$ to construct weight functions $w(\sigma)$, counting inputs where $\sigma$ acts as the accepting set. The simplicial complex $\mathcal{K}$ comprises all non-vanishing faces.
• Homology and Sheaf Theoretic Analysis: $H_0$ (components) identifies clusters with mutual agreement; $H_1$ and higher encode systematic cycles of disagreement. Sheaf constructions and persistent barcodes detect monotonicity violations and long-lived inconsistencies in acceptance pattern distributions (Ambrose et al., 2020).
• Algorithmic Pipeline: Simplicial and sheaf-theoretic algorithms identify minimal sets of inconsistent inputs, rank them by inconsistency score, and extract maximal consensus subcomplexes. Complexity is exponential in $m$ but tractable for small program sets.

4. Computational Methods and Algorithmic Aspects

TDT frameworks employ computational topology libraries (e.g., GUDHI, Dionysus) for efficient complex construction and ECC/diagram evaluation:

• ECC Construction: Alpha complexes scale as $O(n \cdot \mathrm{const}^d)$ in the number of points $n$ and exponentially in dimension $d$; Vietoris-Rips is often used for persistence diagram computation and is exponential in $n$.
• Permutation Testing: Monte Carlo approximations are used with $M+m$ or $K$ permutations for one-sample and two-sample test variants, respectively.
• Statistical and Computational Trade-offs: Vectorized approaches (Betti vectors, persistence images) provide $O(N^2 d)$ run time as opposed to $O(N^2 \cdot \mathrm{cost}(\text{Wasserstein}))$ in classical permutation tests, enabling practical application to data sets with hundreds or thousands of samples (Islambekov et al., 2023, Moon et al., 2020).

Variant	Summary Type	Test Statistic	Complexity
ECC-based	ECC (curve)	$\sup_r$-norm between ECCs	$O((M+m)C(n,d))$
Betti-based	Betti vector (grid)	Within-group $\ell_1$ on Betti vectors	$O(N^2 d)$
PI-based	Persistence image (grid)	Coordinate-wise $t$-test, FDR adjustment	$O(mN)$

5. Empirical Results and Practical Performance

• One-Dimensional Tests: ECC-based GoF tests match or surpass Kolmogorov–Smirnov in power, especially for heavy-tailed or contaminated distributions (Dłotko et al., 2022).
• Higher Dimensions: In $d=2,3,5$, ECC tests achieve 10–20% higher power than multivariate KS extensions; Betti-vector permutation tests exhibit similar or improved sensitivity over Wasserstein-based diagram tests (Dłotko et al., 2022, Islambekov et al., 2023).
• Computational Gains: Vectorized approaches yield an order-of-magnitude speedup; Betti-vector tests achieve $>10\times$ lower run time per permutation compared to Wasserstein distance computations, with highly correlated $p$-values (Islambekov et al., 2023).
• Feature Localization: Two-stage PI-based TDT localizes significant shape or size ranges in real-world data (e.g., porous media, instrument timbre), enhancing interpretability over aggregate $p$-values (Moon et al., 2020).
• Consensus Modeling in Software: Weighted Dowker complexes successfully identify edge-case and adversarial inputs in parser testbeds (Govdocs1, SafeDocs hackathons), supporting practical root cause analysis (Ambrose et al., 2020).

6. Limitations, Interpretability, and Extensions

• Limitations: ECC-based TDT is invariant under isometries in $\mathbb{R}^d$ and is blind to differences between distributions with identical ECC expectations. Pathological non-identifiability cases exist but are empirically rare (Dłotko et al., 2022).
• Extensions: Research directions include replacing ECC with more discriminative topological functionals (Betti curves, persistent Betti numbers), augmenting permutation schemes (max-mixing), enhancing concentration bounds, and adapting TDT for non-Euclidean domains or functional data (Dłotko et al., 2022, Islambekov et al., 2023).
• Interpretability: Vectorized and coordinate-based approaches provide direct localization of differential features (e.g., which birth–persistence regions or size bins are significant), enabling in-depth shape analysis and data-driven specification extraction (Moon et al., 2020, Ambrose et al., 2020).

7. Broader Impact and Research Landscape

Topological Differential Testing unifies diverse methodologies for robust statistical inference, feature localization, and consensus extraction in high, moderate, and even non-Euclidean dimensions. It is applicable to classical statistical problems (GoF, two-sample), software reliability and reverse engineering, time-series analysis, and high-throughput scientific domains. Algorithmic and statistical advances in vectorized summaries and permutation methods enable broad practical adoption. The theory and applications have been consolidated by several research groups, notably by Bobrowski, Adler, Islambekov, Pathirana, Ambrose, Huntsman, Robinson, and collaborators (Dłotko et al., 2022, Islambekov et al., 2023, Moon et al., 2020, Ambrose et al., 2020).