SAOT: Self-Supervised Continual Graph Learning with Structure-Aware Optimal Transport

Abstract: Self-supervised Continual Graph Learning (CGL) aims to successively learn from a graph sequence with different tasks without label supervision - a paradigm that has attracted widespread attention. Most existing self-supervised CGL methods rely on instance-level consistency objectives that enforce stability of individual node (or node-pair) embeddings. Due to optimizing nodes in isolation, these methods fail to maintain global relational structure, causing inter-node correspondences to progressively distort under continual learning. To this end, we propose a novel Structure-Aware Optimal Transport (SAOT) framework that explicitly captures and preserves relational structure within graph representations across sequential tasks. Specifically, SAOT leverages optimal transport theory to capture global inter-node correspondences, thereby facilitating and enhancing graph representation learning. Simultaneously, SAOT incorporates a cross-task knowledge distillation mechanism to preserve the previous structural knowledge. Extensive experiments on four CGL benchmark datasets demonstrate that SAOT outperforms existing self-supervised baselines. In particular, SAOT achieves significant performance gains, improving average accuracy by up to 5% on CoraFull-CL and over 15% on Products-CL compared with state-of-the-art methods in the Class-IL setting.

• The paper introduces SAOT, a novel framework that preserves global relational structure in graph learning through optimal transport alignment.
• It integrates cross-task plan-level knowledge distillation and selective replay to minimize forgetting while maintaining high accuracy.
• Experiments on multiple benchmarks show significant improvements in accuracy and robustness, scaling effectively to real-world dynamic graphs.

Summary of "SAOT: Self-Supervised Continual Graph Learning with Structure-Aware Optimal Transport" (2607.00377)

Motivation and Background

Self-supervised continual graph learning (CGL) targets learning from a non-stationary stream of graph-structured data without label supervision. Real-world graphs—spanning citation networks, recommender systems, and biomolecular interaction networks—are inherently dynamic, yielding persistent distributional shift and class expansion. Prior self-supervised CGL methods, predominately instance-level objectives enforcing stability at the node or node-pair level, inadequately preserve higher-order relational dependencies. This shortcoming manifests as progressive and often severe "structural drift," degrading inter-node correspondences and undermining performance on prior tasks.

The need to robustly capture and preserve global graph structure in representations motivates the research. Moreover, practical constraints rule out full-batch retraining or large-scale supervised memory, making strongly self-supervised, structure-aware solutions imperative.

Methodology

The proposed SAOT (Structure-Aware Optimal Transport) framework integrates optimal transport theory and knowledge distillation for sequential, self-supervised learning on graph streams. The methodology is composed of the following key modules:

1. Graph Representation Learning via Structure-Aware Optimal Transport:

For each task, the encoder learns node representations by aligning global optimal transport plans between the graph structure and its embedding space. The Gromov-Wasserstein distance is leveraged for its efficacy in modeling relational dependencies, capturing both node attributes and edge relationships. By contrasting transport plans in the graph and representation domain, the encoder is explicitly guided to preserve relational structure, not just local feature consistency.

2. Cross-Task Knowledge Distillation:

To counter structural drift as new tasks arrive, SAOT employs a transport-plan-level distillation mechanism. Distillation aligns the optimal plan from the previous encoder (the "teacher") to the current representations, regularizing the relational transformations between sequential tasks. This plan-level approach is shown to be superior to point-wise feature matching.

3. Replay Buffer (Memory) and Efficiency Considerations:

Selective replay of historical subgraphs is used for further stability, though ablation and sensitivity analysis reveal SAOT's relative independence from explicit memory—performance remains robust even with minimal buffer capacity.

The training objective combines the OT alignment loss and distillation regularization, with critical trade-off parameters $\alpha$ (intra-task structure alignment) and $\beta$ (cross-task distillation) tuned for best performance.

Experimental Results

Four large-scale benchmark datasets were used (CoraFull-CL, Arxiv-CL, Reddit-CL, Products-CL), evaluated under both Class-Incremental (Class-IL) and Task-Incremental (Task-IL) settings. Comparison was made against a wide suite of supervised and self-supervised CGL baselines, including TRACE, RieGrace, and several contrastive and non-contrastive GNN models.

Key numerical results:

• On Products-CL (Class-IL), SAOT yields over 15% improvement in average accuracy (AP) relative to previous self-supervised state-of-the-art.
• On CoraFull-CL (Class-IL), SAOT improves accuracy by ~5% and nearly eliminates forgetting.
• On Reddit-CL (Task-IL), SAOT achieves up to 99.5% AP with negligible forgetting, matching or exceeding strong fully-supervised and task-specific baselines.
• On Arxiv-CL, despite strong semantic similarity, SAOT sustains higher accuracy and lower forgetting than other SSCL baselines.

Ablation studies demonstrate that removing either the transport-based relational alignment or the cross-task plan-level distillation significantly degrades both accuracy and stability; point-wise objectives (e.g., cosine, MSE) are shown to be insufficient for effective relational regularization.

Buffer size sensitivity analysis confirms that SAOT's structure-aware alignment, not memory replay, is primarily responsible for preventing catastrophic forgetting.

In terms of computational complexity, the use of the Sinkhorn algorithm with adaptive node sampling ensures reasonable runtime for large graphs.

Implications and Theoretical Significance

Practical Implications:

• SAOT is highly effective for real-world deployment in streaming scenarios where annotating labels or storing massive historical data is infeasible.
• The framework offers favorable trade-offs between robustness to forgetting and resource efficiency, scaling to multi-million node graphs with minimal explicit memory.

Theoretical Advancements:

• The introduction of transport-plan-level distillation advances the paradigm of knowledge preservation in self-supervised CGL, addressing not just local discrimination but global relational structure.
• Results underscore the limitations of instance-level consistency objectives in graph streams and validate the need for structure-aware regularization.

Forward-looking Impact:

• The explicit modeling and preservation of global relational information may become a necessary component for continual learning in highly structured or evolving domains.
• Further development may investigate structure-aware OT in heterogeneous/multirelational graphs, integration with parameter isolation, and principled buffer management schemes.
• SAOT's ideas can inform more general continual learning algorithms beyond graphs, especially those involving manifold-aligned or relationally grounded data.

Conclusion

SAOT establishes a novel and effective framework for self-supervised continual graph learning. By leveraging structure-aware optimal transport for both intra-task alignment and cross-task structural distillation, it robustly preserves global relational knowledge under severe non-stationarity. Empirically, SAOT achieves state-of-the-art self-supervised performance and, in many settings, approaches or surpasses supervised methods—even in highly challenging class-incremental regimes on large graphs. This work substantiates the need for principled, structure-aware objectives in continual graph representation learning and sets new technical baselines for the field.