Large-scale semi-supervised learning with online spectral graph sparsification

Abstract: We introduce Sparse-HFS, a scalable algorithm that can compute solutions to SSL problems using only O(n polylog(n)) space and O(m polylog(n)) time.

• The paper introduces Sparse-HFS, combining online spectral graph sparsification with stable label propagation to enable scalable semi-supervised learning.
• It leverages the Kelner-Levin algorithm and efficient SDD solvers to maintain a spectral sparsifier in O(n•polylog(n)) space with fast amortized edge updates.
• Empirical results show Sparse-HFS achieves near-optimal classification using only about 10% of the original graph edges, significantly reducing computational costs.

Large-scale Semi-Supervised Learning with Online Spectral Graph Sparsification

Introduction and Problem Setting

The paper "Large-scale semi-supervised learning with online spectral graph sparsification" (2604.26550) addresses the scalability bottlenecks of graph-based semi-supervised learning (SSL) in the context of massive, densely connected similarity graphs. Label propagation and Laplacian regularization techniques such as harmonic function solution (HFS) yield strong transductive performance but are computationally prohibitive, exhibiting $O(n^2)$ space and $O(n^3)$ time complexity, where $n$ is the number of nodes. This work focuses on regimes where the graph cannot be stored or processed in full, particularly the semi-streaming setting where the available space is constrained to $O(n\,\mathrm{polylog}(n))$ and the edge stream must be processed incrementally.

Sparse-HFS: Algorithmic Approach

The authors propose Sparse-HFS, a scalable algorithm combining online spectral graph sparsification with stable HFS. The online sparsification component is based on the Kelner-Levin algorithm, which maintains an incremental spectral sparsifier $H$ from the edge stream, guaranteeing

$$(1-\varepsilon)\, \mathbf{x}^T L_G \mathbf{x} \leq \mathbf{x}^T L_H \mathbf{x} \leq (1+\varepsilon)\, \mathbf{x}^T L_G \mathbf{x}$$

for all $\mathbf{x}$, where $L_G$ and $L_H$ are the Laplacians of the full and sparsified graphs, and $\varepsilon$ controls the approximation precision. The critical technical tool is the efficient estimation and sampling of edge effective resistances via SDD linear system solvers, as in [koutis_solving_2011], to maintain the graph sketch in $O(n^3)$0 space, updating it in $O(n^3)$1 amortized time per edge.

Sparse-HFS applies the stable HFS procedure on the current sparsified graph $O(n^3)$2. The empirical risk minimization is posed as:

$O(n^3)$3

where $O(n^3)$4 selects the labeled nodes, and the admissible hypothesis space $O(n^3)$5 improves numerical and theoretical properties. The solution requires solving a symmetric, diagonally dominant system on the sparse Laplacian.

Theoretical Guarantees

The analysis leverages algorithmic stability theory to demonstrate that the generalization error bound of Sparse-HFS closely matches that of the original stable HFS (when run on the full graph), with only constant-factor deterioration due to sparsification. Explicitly, for fixed $O(n^3)$6, the convergence rate with respect to the number of labeled samples $O(n^3)$7 is unchanged; the only penalty is introduced as a factor scaling the overall rate but not its order:

$O(n^3)$8

where the first term reflects the error induced by the spectral sparsifier and the second term follows from the stability analysis. The detailed proof shows how the spectral approximation propagates through the Laplacian regularized system and thus into the generalization bound, establishing the robustness of stable-HFS under graph approximation.

Empirical Validation

The empirical component evaluates Sparse-HFS and classic stable-HFS on a synthetic dataset ($O(n^3)$9), constructed to require a highly-connected graph for effective label propagation. For each $n$0 in $n$1-nearest neighbor construction ($n$2 ranging up to $n$3), both algorithms are compared in terms of classification accuracy. With a small labeled set $n$4 (two labeled nodes per class), both methods exhibit subpar performance until the graph connectivity is sufficient to join the clusters. Post-threshold, the performance of the spectral approximation (Sparse-HFS) and the full graph solution are statistically similar near the optimum.

Figure 1: (a) Structure of the synthetic dataset requiring dense connectivity. (b) Generalization error $n$5 of Sparse-HFS (approximate) versus stable-HFS (exact) as connectivity increases; both methods align closely post-threshold. (c) Edge count ratio $n$6, indicating the drastic sparsity induced by the online spectral sparsifier (around 10% of original edges suffice near optimum).

Notably, at peak accuracy, the sparsifier contains only $n$710% of the edges relative to the original dense $n$8-NN graph, but incurs negligible loss in classification performance. This demonstrates that the empirical accuracy-loss tradeoff of aggressive graph sparsification is essentially bounded, supporting the theoretical results.

Implications, Limitations, and Future Prospects

Sparse-HFS enables the scaling of graph-based SSL to datasets with millions of nodes and exceedingly dense affinity matrices without the prohibitive space and time costs of conventional approaches. The integration of graph sketching and fast SDD solvers establishes a template for semi-supervised learning in streaming or resource-constrained settings, relevant for large-scale web graphs, sensor topologies, and other domains with high intrinsic node counts and limited label supervision.

The algorithm operates in the insertion-only streaming context; extending these results to the fully dynamic scenario (with edge deletions) would require fundamentally different techniques, possibly based on linear sketching or more advanced dynamic sparsification [kapralov_single_2014]. Moreover, sharper analyses might further reduce the constant factors in the generalization and stability bounds.

The approach is compatible with ongoing research in spectral bandit algorithms [kocak2014wspectral], scalable Laplacian learning [spielman_spectral_2011], and general large-scale transductive learning. Future research directions include efficient multi-class extensions, application to manifold learning, and combinations with deep representational methods.

Conclusion

This work solves a critical bottleneck in large-scale graph-based SSL by designing, analyzing, and empirically validating an online graph-sparsification-based framework (Sparse-HFS) that guarantees effective label propagation in $n$9 space. Theoretical analysis confirms that approximation does not compromise the statistical rates of the classical solution, and experiments on dense graphs demonstrate that only a small fraction of the original edge set suffices for near-optimal classification. The results position spectral sparsification and streaming sketching as fundamental tools for tractable, scalable semi-supervised learning in practical, large-graph regimes.