Scaling Transformers to very large graphs

Determine how to adapt the Transformer architecture used for graph representation learning to scale effectively to large graphs with potentially millions of nodes while addressing the quadratic time and memory complexity of all-to-all self-attention.

Background

The paper motivates graph transformers as a remedy for limitations of message-passing GNNs in modeling long-range dependencies, but notes that standard full self-attention incurs quadratic complexity in the number of nodes, which prevents application to graphs beyond a few thousand nodes.

Although prior work proposes linearized attention, engineered attention patterns, or subsampling, these approaches can degrade performance or restrict expressive power. The authors introduce k-MIP attention as one approach to improve scalability, but explicitly acknowledge that the broader challenge of adapting Transformers to truly large graphs remains open.

References

However, this comes at the cost of quadratic complexity, and it remains an open question how to adapt the Transformer architecture to scale effectively to large graphs with potentially millions of nodes.

— k-Maximum Inner Product Attention for Graph Transformers and the Expressive Power of GraphGPS The Expressive Power of GraphGPS  (2604.03815 - Schouwer et al., 4 Apr 2026) in Section 1 (Introduction)

Moving toward hardware-relevant deployments raises two immediate questions: how performance degrades under finite-shot estimation of observables, and how feature generation can be scaled to much larger graphs.

— On the Expressive Power of the Transverse-Field Ising Model for Graph Learning  (2608.17750 - Djellabi et al., 18 Aug 2026) in Section 4, Discussion, paragraph “Practical constraints and limitations”

Thus, a critical open problem is to explore efficient alternatives, such as kernel-approximation-based variants of linear attention mechanisms or sparsification strategies for DHGs, as a prerequisite for investigating the scaling laws of DHG representation learning.

— Dynamic Heterogeneous Graph Representation Learning: A Survey  (2609.04779 - Liu et al., 4 Sep 2026) in Section 6, Conclusion and Future Directions, paragraph “Efficiency”