Span-Wise Aggregation Methods

• Span-wise aggregation is a technique that pools contiguous data intervals in ordered datasets, such as genomic markers or time series, to amplify weak signals.
• Kernel and window designs, including flat and Epanechnikov kernels, are used to balance locality, bias, and variance in aggregated analyses.
• Permutation-based null distributions and spill-aware methods ensure robust error control and efficient processing in high-dimensional and streaming data scenarios.

Span-wise aggregation refers to the process of pooling or combining data, statistics, or similarities over contiguous intervals (“spans”) in ordered data such as time series, chromosomes, or streams. Its core principle is to amplify weak signals or efficiently process large windows by aggregating information across locally coherent regions. Span-wise aggregation encompasses a range of methodologies in genetic association studies, statistical independence testing, and large-scale data analytics.

1. Span-wise Aggregation in Statistical Genomics

Span-wise aggregation is a fundamental approach for association testing in contexts where underlying biological signals, such as copy-number variants (CNVs), span contiguous regions rather than isolated markers. In genotyping arrays, the signal at any single marker is typically weak; pooling marker-level statistics across contiguous spans can enhance statistical power and spatial resolution.

Let $m$ be the number of markers at genomic positions $\ell_1,\ldots,\ell_m$, and $p_j$ be the $p$-value from a univariate test at marker $j$. Span-wise aggregation computes a smoothed test statistic $T(b,h)$ at candidate location $b$ by weighted summation: $T(b,h) = \sum_{j=1}^m K_h(\ell_j,\ell_b)\, t_j$ where $t_j = f(p_j)$ is a monotone transform of $p_j$ and $K_h(\ell_j,\ell_b)$ is a kernel function decaying with $|\ell_j-\ell_b|$ at rate governed by bandwidth $h$. Aggregating over spans boosts sensitivity to regions where true CNVs reside, even when their boundaries are unknown (Li et al., 2012).

2. Kernel and Window Design for Span-wise Aggregation

Kernel functions effectuate the local pooling in span-wise aggregation and critically determine the trade-offs between locality, bias, and variance. Common choices include:

• Flat (Uniform) Kernel: $K_h(\ell_j,\ell_b) = 1$ if $|\ell_j-\ell_b| \leq h$; $0$ otherwise.
• Epanechnikov Kernel: $$K_h(\ell_j,\ell_b) = \tfrac{3}{4}(1-(\frac{\ell_j-\ell_b}{h})^2)$$ for $|\ell_j-\ell_b| \le h$.

Bandwidth $h$ can be constant in physical units (“constant width”) or adaptive (“constant marker”) so that exactly $k$ neighboring markers are included. Constant-marker kernels stabilize null variance and are preferable when marker density is variable. The kernel’s shape (flat vs. Epanechnikov) determines the weight decay at window boundaries, with non-flat kernels reducing edge bias for true signals narrower than $h$ (Li et al., 2012).

3. Permutation-Based Null Distributions and Error Control

Chromosomal marker data feature spatial correlation, which undermines independence-based null approximations. Accurate Family-Wise Error Rate (FWER) control is achieved by permuting the phenotype labels $y$, thus preserving marker correlation structure in the intensity matrix $X$. The empirical distribution of the global scan statistic $T_{\text{max}}(h) = \max_b T(b,h)$ under phenotype permutation forms the reference for significance testing. For each marker, rejection regions are based on the permutation quantiles, ensuring control of the FWER at the desired $\alpha$ level under the global null, a property not shared by Monte Carlo nulls or naive marker exchangeability (Li et al., 2012).

4. Span-wise Aggregation in Serial Independence Testing

Weighted aggregation over temporal spans generalizes to independence testing for high-dimensional or non-Euclidean time series. The WISE procedure forms a span-wise sum: $$T_n = \frac{1}{\sum_{\ell=1}^L w_\ell} \sum_{\ell=1}^L w_\ell \sum_{i=1}^{n-\ell} K(X_i, X_{i+\ell})$$ where $K$ is a symmetric similarity function and $w_\ell$ are lag-dependent weights. Choice of $w_\ell$ can be tailored for long-memory, periodicity, or mixed alternatives. Analytic forms for the mean and variance under permutation enable $Z$-statistic computation without resampling. The procedure exhibits asymptotic normality and achieves power consistency in both classical large-$n$ and High-Dimension-Low-Sample-Size (HDLSS) regimes (Zhu et al., 6 Sep 2025).

5. Span-wise Aggregation for Sliding-Window Analytics in Large-Scale Data

In database engines and streaming analytics, span-wise (“windowed”) aggregation is a central feature, e.g., computing moving MIN/MAX or average over very large windows. For extremal aggregates (MIN/MAX), naïve in-memory approaches require $O(W)$ space for window width $W$. When data exceed memory, state must be spilled to disk.

Efficient span-wise aggregation in such settings utilizes a spill-aware deque across disk pages, with each page holding a chunk of the monotonic deque and an in-memory summary $(\text{first\_value}, \text{last\_value})$ per page. Insertion and eviction operations use $O(\log(W/B))$ CPU operations, with amortized $O(1)$ I/O per tuple when summary arrays reside entirely in memory, where $B$ is the page size. This method is robust to data order and achieves near-in-memory throughput for window widths up to $10^7$ (Shi et al., 2020).

6. Theoretical Properties and Power Considerations

Span-wise aggregation methods generally achieve enhanced statistical power over single-point or minimal pooling approaches. The kernel-based approach for CNVs controls FWER via permutation and achieves peak power when the kernel width matches the true CNV length; efficiency is robust to moderate bandwidth misspecification. In temporal/spatial independence testing, span-wise aggregation with analytic nulls and carefully aligned weight functions allows sensitivity to nonlinear, periodic, and HDLSS alternatives. For windowed analytics in data streams, the proposed spill-aware strategies minimize disk I/O without the complexity overhead of re-implementing the full stateful logic out-of-core (Li et al., 2012, Zhu et al., 6 Sep 2025, Shi et al., 2020).

7. Extensions, Limitations, and Empirical Findings

Empirical studies demonstrate that span-wise aggregation approaches outperform competing frameworks, particularly in genomics for small or common CNVs, and in independence testing under high-dimensional or non-Euclidean regimes. Extensions include adaptation to functional, matrix, or distributional data via choice of $S(X_i, X_j)$ in independence testing, and generalization to advanced sliding window aggregates (e.g., top-$k$) in streaming environments. Limitations arise with non-monotonic aggregate requirements or when bandwidth selection is suboptimal. Performance is maximal when in-memory summaries suffice; further gains are possible through parallelization or adaptive pre-fetching in adverse input distributions (Li et al., 2012, Zhu et al., 6 Sep 2025, Shi et al., 2020).