Graph-Aware Transformer (GRAT)

• Graph-Aware Transformer (GRAT) is a family of Transformer architectures that explicitly integrate graph structure into self-attention through mechanisms like edge-conditioned attention.
• It has been applied across domains such as NLP, vision, and traffic forecasting by leveraging graph-induced sparsity, spectral biases, and graph-aware tokenization.
• Key innovations include explicit edge conditioning, adaptive graph tokenization, and hybrid architectures that combine Transformer and graph convolution principles for enhanced graph modeling.

Graph-Aware Transformer (GRAT) denotes a class of Transformer architectures in which graph structure is not merely an implicit by-product of dense self-attention but an explicit modeling primitive. In the strict sense of the original “Graph-Aware Transformer: Is Attention All Graphs Need?”, GRAT is a Transformer-based encoder–decoder that can encode and decode whole graphs end-to-end, with self-attention directly conditioned on edge information (Yoo et al., 2020). Subsequent work has extended the same design logic across graph classification and generation, semantic role labeling, image parsing, semantic segmentation, recommendation, fraud detection, traffic forecasting, fairness-aware node classification, and pose estimation. Taken together, these systems show that “graph-aware” can mean explicit edge-conditioned attention, graph-induced sparsity, spectral or path-based attention biases, graph-derived tokenization, or hybrid graph-convolution/Transformer compositions (Mohammadshahi et al., 2021, Zhang et al., 2022, Wu et al., 2023, Chen et al., 2024, Tian et al., 2023, Park et al., 2019, Luo et al., 2024, Zhou et al., 2024, Aouaidjia et al., 2 May 2025).

1. Definition and conceptual scope

The original GRAT formulation was introduced as a general encoder–decoder architecture for graph data, targeting both graph representation and graph generation. Its central claim was that Transformer machinery could be adapted to graphs by making self-attention explicitly aware of edge types and by replacing sequence-oriented decoding with an auto-regressive graph decoder based on a two-path approach consisting of sub-graph encoding path and node-and-edge generation path (Yoo et al., 2020). In that formulation, nodes are treated as tokens, edges are explicitly typed, and whole-graph inputs and outputs are handled in end-to-end fashion.

Later work broadened the practical meaning of the term. Some models preserved standard Transformer blocks but injected graph relations into attention logits, as in Syntax-aware Graph-to-Graph Transformer for semantic role labeling, where dependency labels and directions are embedded and used as a soft bias on self-attention (Mohammadshahi et al., 2021). Other models used graph structure more aggressively: Graph-Segmenter explicitly constructs graphs over windows and pixels and applies sparse graph aggregation rather than full all-to-all attention (Wu et al., 2023), while Graph Reasoning Transformer projects image patches into a latent graph space, performs graph reasoning, and maps the result back to token space (Zhang et al., 2022). This suggests that GRAT is better understood as a design family than as a single canonical module.

A common misconception is that any Transformer operating on graph-structured data is already graph-aware. The literature distinguishes more sharply. Graph-Segmenter explicitly notes that its “Graph Transformer” is not a vanilla ViT with an attention mask, but a combination of graph relation construction, sparsified adjacency, and graph convolution integrated into a transformer-like block (Wu et al., 2023). FairGT makes a similar distinction from another direction: it argues that graph Transformers generally require dedicated structural topology encoding and graph-aware attention strategies rather than direct transplantation of sequence Transformers (Luo et al., 2024).

2. Core mechanisms of graph awareness

One major mechanism is edge-conditioned attention. In the original GRAT, edge types are passed through a small MLP to generate a scale and bias, and attention is modulated as

$$\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{\Gamma \odot (QK^\top)+B}{\sqrt{d_k}}\right)V,$$

so each query–key interaction is scaled and shifted by the corresponding edge features (Yoo et al., 2020). This makes edge information first-class in the attention score rather than leaving it implicit in node embeddings alone.

A second mechanism is relation-embedding bias. SynG2G-Tr encodes each token pair $(i,j)$ with a one-hot relation vector $r_{ij}$ over labeled dependency directions and injects it into the attention logit as

$$e_{ij}=\frac{1}{\sqrt d}\left[q_i k_j^\top + q_i (r_{ij}^*)^\top + r_{ij}^* k_j^\top\right].$$

The graph therefore acts as a soft bias: syntactic structure influences attention patterns without becoming a hard mask, and values remain graph-free for efficiency (Mohammadshahi et al., 2021).

A third mechanism is explicit graph construction and sparse aggregation. Graph-Segmenter defines nodes at two granularities—windows globally and pixels locally—and computes similarity-based relations

$$r_{i,j}=\frac{\mathbf{x}_i\cdot \mathbf{x}_j}{\|\mathbf{x}_i\cdot \mathbf{x}_j\|}, \qquad |r_{i,j}| \le 1,$$

followed by thresholded neighborhood aggregation

$$\mathbf{x}_i^{(l+1)}=\sum_{\mathbf{x}_j^{(l)}\in\delta(\mathbf{x}_i^{(l)})}\mathbb{I}(r_{i,j}>\theta)\cdot \mathbf{x}_j^{(l)}.$$

Here graph awareness is semantic and adaptive rather than fixed by grid adjacency; connectivity is induced by feature similarity and then sparsified by a learnable or selected threshold $\theta$ (Wu et al., 2023).

A fourth mechanism is spectral and path-based structural bias. SIGformer defines a signed Laplacian

$$\mathbf{L}=\frac{1}{1-\alpha}(\mathbf{L}^+ - \alpha \mathbf{L}^-)$$

and uses the low-frequency eigenspace to construct a spectral positional bias

$$\mathbf{P}_s^{(l)}=\theta^{(l)}\tilde{\mathbf{H}}^\top\tilde{\mathbf{H}},$$

while signed path patterns yield a second structural attention channel $\mathbf{P}_p^{(l)}$. Attention is then averaged across a content-plus-spectral branch and a path-only branch. This gives graph awareness through signed global structure and signed path semantics rather than through message passing (Chen et al., 2024).

A fifth mechanism is fairness-aware structural encoding. FairGT selects adjacency eigenvectors with the largest-magnitude eigenvalues, concatenates them to node features, and constructs multi-hop node features on a sensitive-feature complete graph via

$(i,j)$0

The Transformer then operates over per-node hop sequences, so fairness constraints enter through the construction of graph encodings rather than through an explicit adversarial or regularized attention loss (Luo et al., 2024).

A sixth mechanism is graph-aware tokenization. Tokenphormer does not modify attention logits at all; instead it builds multiple tokens per node, including walk-tokens from mixed random walks, SGPM-tokens from self-supervised graph-document pretraining, and hop-tokens from $(i,j)$1. The Transformer mixes these structure-bearing tokens using standard self-attention, making token construction rather than attention bias the primary graph-aware component (Zhou et al., 2024).

3. Architectural patterns across domains

Across the literature, GRAT architectures fall into several recurring patterns. Some are whole-graph encoder–decoders, as in the original GRAT for molecule property prediction and graph generation (Yoo et al., 2020). Some are graph-to-graph or graph-to-sequence transducers, as in SynG2G-Tr for semantic role labeling and the graph-to-sequence NMT Graph-Transformer, which treats self-attention as latent multigraph construction over subgraphs of different orders (Mohammadshahi et al., 2021, Duan et al., 2020). Some are vision systems with latent or explicit graphs, such as GReaT, which projects image patches into graph nodes for global reasoning, and Graph-Segmenter, which constructs graphs over windows and within-window pixels (Zhang et al., 2022, Wu et al., 2023). Others are spatio-temporal graph-aware Transformers, including ST-GRAT for road speed forecasting, STA-GT for transaction fraud detection, and the temporal Body Aware Transformer stacked on graph-order-aware spatial encoding for 3D pose estimation (Park et al., 2019, Tian et al., 2023, Aouaidjia et al., 2 May 2025).

The following papers illustrate this spread of design choices.

Paper	Domain	Principal graph-aware mechanism
“Graph-Aware Transformer: Is Attention All Graphs Need?” (Yoo et al., 2020)	Molecular graphs	Edge-conditioned self-attention and graph autoregressive decoding
“Syntax-Aware Graph-to-Graph Transformer for Semantic Role Labelling” (Mohammadshahi et al., 2021)	NLP / SRL	Dependency-relation embeddings in attention logits
“Graph Reasoning Transformer for Image Parsing” (Zhang et al., 2022)	Vision	Token-to-node projection, graph reasoning, node-to-token redistribution
“Graph-Segmenter” (Wu et al., 2023)	Semantic segmentation	Explicit similarity graphs over windows and pixels, sparse aggregation
“SIGformer” (Chen et al., 2024)	Recommendation	Signed spectral and signed path positional encodings
“FairGT” (Luo et al., 2024)	Fair node classification	Adjacency-spectrum topology encoding and sensitive-aware multi-hop tokens
“Tokenphormer” (Zhou et al., 2024)	Node classification	Structure-aware multi-token representation learning
“TorchGT” (Zhang et al., 2024)	Large-scale systems	Graph-sparse dual-interleaved attention and graph-parallel training

A plausible implication is that the most stable architectural distinction is not between “Transformer” and “graph model,” but between where graph structure enters: into tokenization, into positional encodings, into attention logits, into sparsity patterns, or into a separate graph reasoning block placed before or beside attention.

4. Representative systems and empirical profile

The original GRAT reported state-of-the-art performance on 4 regression tasks in the QM9 benchmark, achieved a multi-task standardized MAE of $(i,j)$2 without pretraining, improved to $(i,j)$3 with pp-GRAT in multi-task mode, and reached $(i,j)$4 in single-task mode. In reaction outcome prediction on USPTO, it obtained $(i,j)$5 Top-1 accuracy, essentially on par with Molecular Transformer at $(i,j)$6 and below the $(i,j)$7 figure that used augmentation and checkpoint averaging (Yoo et al., 2020).

SynG2G-Tr showed that graph-biased attention can materially improve semantic role labeling. On CoNLL-2005, the BERT-based model reached $(i,j)$8 F1 for end-to-end and $(i,j)$9 for given-predicate evaluation on WSJ/Brown. On CoNLL-2009, it reported $r_{ij}$0 end-to-end and $r_{ij}$1 given-predicate, with $r_{ij}$2 when trained with gold syntax and tested with predicted syntax in the reported setting (Mohammadshahi et al., 2021).

Graph-Segmenter provides a clear vision-specific instance of sparse graph-aware reasoning. On Cityscapes validation it improved Swin-L* from $r_{ij}$3 to $r_{ij}$4 mIoU; on Cityscapes test it reached $r_{ij}$5 mIoU, exceeding Swin-L* at $r_{ij}$6 and SETR-PUP at $r_{ij}$7. On ADE20K it reported $r_{ij}$8 validation mIoU and $r_{ij}$9 test, and on PASCAL Context it obtained $$e_{ij}=\frac{1}{\sqrt d}\left[q_i k_j^\top + q_i (r_{ij}^*)^\top + r_{ij}^* k_j^\top\right].$$0 mIoU with UperNet (Swin-L) and $$e_{ij}=\frac{1}{\sqrt d}\left[q_i k_j^\top + q_i (r_{ij}^*)^\top + r_{ij}^* k_j^\top\right].$$1 when paired with CAR in the reported comparison (Wu et al., 2023).

ST-GRAT showed the utility of graph-aware spatial attention with temporal attention in traffic forecasting. On METR-LA it reported MAE $$e_{ij}=\frac{1}{\sqrt d}\left[q_i k_j^\top + q_i (r_{ij}^*)^\top + r_{ij}^* k_j^\top\right].$$2 at 15 minutes, $$e_{ij}=\frac{1}{\sqrt d}\left[q_i k_j^\top + q_i (r_{ij}^*)^\top + r_{ij}^* k_j^\top\right].$$3 at 30 minutes, and average MAE $$e_{ij}=\frac{1}{\sqrt d}\left[q_i k_j^\top + q_i (r_{ij}^*)^\top + r_{ij}^* k_j^\top\right].$$4, outperforming several baselines and showing especially strong behavior during impeded intervals where traffic speeds rapidly change (Park et al., 2019). STA-GT extended the same general principle to fraud detection: on the private PR01 splits and the TC public dataset it consistently led in Recall and F1, with examples such as PR1 Recall $$e_{ij}=\frac{1}{\sqrt d}\left[q_i k_j^\top + q_i (r_{ij}^*)^\top + r_{ij}^* k_j^\top\right].$$5, F1 $$e_{ij}=\frac{1}{\sqrt d}\left[q_i k_j^\top + q_i (r_{ij}^*)^\top + r_{ij}^* k_j^\top\right].$$6, AUC $$e_{ij}=\frac{1}{\sqrt d}\left[q_i k_j^\top + q_i (r_{ij}^*)^\top + r_{ij}^* k_j^\top\right].$$7, and TC12 Recall $$e_{ij}=\frac{1}{\sqrt d}\left[q_i k_j^\top + q_i (r_{ij}^*)^\top + r_{ij}^* k_j^\top\right].$$8, F1 $$e_{ij}=\frac{1}{\sqrt d}\left[q_i k_j^\top + q_i (r_{ij}^*)^\top + r_{ij}^* k_j^\top\right].$$9, AUC $$r_{i,j}=\frac{\mathbf{x}_i\cdot \mathbf{x}_j}{\|\mathbf{x}_i\cdot \mathbf{x}_j\|}, \qquad |r_{i,j}| \le 1,$$0 (Tian et al., 2023).

FairGT showed that graph-aware Transformer design can be optimized for fairness rather than only predictive power. It reported, for example, NBA accuracy $$r_{i,j}=\frac{\mathbf{x}_i\cdot \mathbf{x}_j}{\|\mathbf{x}_i\cdot \mathbf{x}_j\|}, \qquad |r_{i,j}| \le 1,$$1 with $$r_{i,j}=\frac{\mathbf{x}_i\cdot \mathbf{x}_j}{\|\mathbf{x}_i\cdot \mathbf{x}_j\|}, \qquad |r_{i,j}| \le 1,$$2, Bail $$r_{i,j}=\frac{\mathbf{x}_i\cdot \mathbf{x}_j}{\|\mathbf{x}_i\cdot \mathbf{x}_j\|}, \qquad |r_{i,j}| \le 1,$$3 with $$r_{i,j}=\frac{\mathbf{x}_i\cdot \mathbf{x}_j}{\|\mathbf{x}_i\cdot \mathbf{x}_j\|}, \qquad |r_{i,j}| \le 1,$$4, German $$r_{i,j}=\frac{\mathbf{x}_i\cdot \mathbf{x}_j}{\|\mathbf{x}_i\cdot \mathbf{x}_j\|}, \qquad |r_{i,j}| \le 1,$$5 with $$r_{i,j}=\frac{\mathbf{x}_i\cdot \mathbf{x}_j}{\|\mathbf{x}_i\cdot \mathbf{x}_j\|}, \qquad |r_{i,j}| \le 1,$$6, Credit $$r_{i,j}=\frac{\mathbf{x}_i\cdot \mathbf{x}_j}{\|\mathbf{x}_i\cdot \mathbf{x}_j\|}, \qquad |r_{i,j}| \le 1,$$7 with $$r_{i,j}=\frac{\mathbf{x}_i\cdot \mathbf{x}_j}{\|\mathbf{x}_i\cdot \mathbf{x}_j\|}, \qquad |r_{i,j}| \le 1,$$8, and Income $$r_{i,j}=\frac{\mathbf{x}_i\cdot \mathbf{x}_j}{\|\mathbf{x}_i\cdot \mathbf{x}_j\|}, \qquad |r_{i,j}| \le 1,$$9 with $$\mathbf{x}_i^{(l+1)}=\sum_{\mathbf{x}_j^{(l)}\in\delta(\mathbf{x}_i^{(l)})}\mathbb{I}(r_{i,j}>\theta)\cdot \mathbf{x}_j^{(l)}.$$0, outperforming both existing graph Transformers and fairness-aware graph learning baselines on the reported fairness metric (Luo et al., 2024).

Tokenphormer and EGTAS show two different routes to strengthening GRATs. Tokenphormer reported state-of-the-art node classification performance through structure-aware multi-token construction rather than graph-biased logits (Zhou et al., 2024). EGTAS, by contrast, treated graph topology-aware Transformer design itself as a search problem and achieved the best reported accuracy on all seven node-classification datasets listed in its comparison, as well as strong graph-level results, including $$\mathbf{x}_i^{(l+1)}=\sum_{\mathbf{x}_j^{(l)}\in\delta(\mathbf{x}_i^{(l)})}\mathbb{I}(r_{i,j}>\theta)\cdot \mathbf{x}_j^{(l)}.$$1 AUC on ogbg-molhiv (Wang et al., 2024).

5. Training objectives, efficiency, and systems design

GRAT models do not share a single optimization objective. The original GRAT uses MAE or L1-style regression losses for graph property prediction and cross-entropy losses for node and edge generation in graph decoding (Yoo et al., 2020). SynG2G-Tr uses cross-entropy over semantic role labels for predicate–argument pairs (Mohammadshahi et al., 2021). Graph-Segmenter uses standard per-pixel cross-entropy and reports mIoU, explicitly without additional boundary annotations or boundary-specific loss terms (Wu et al., 2023). ST-GRAT is trained for regression with MAE, RMSE, and MAPE as evaluation metrics (Park et al., 2019). SIGformer uses a sign-aware BPR loss with separate positive and negative terms,

$$\mathbf{x}_i^{(l+1)}=\sum_{\mathbf{x}_j^{(l)}\in\delta(\mathbf{x}_i^{(l)})}\mathbb{I}(r_{i,j}>\theta)\cdot \mathbf{x}_j^{(l)}.$$2

showing that graph-aware attention can be paired with ranking objectives rather than token-level likelihoods (Chen et al., 2024).

Efficiency is equally heterogeneous. Graph-Segmenter adds parameters but only modest compute: compared with Swin-L, parameters rise from $$\mathbf{x}_i^{(l+1)}=\sum_{\mathbf{x}_j^{(l)}\in\delta(\mathbf{x}_i^{(l)})}\mathbb{I}(r_{i,j}>\theta)\cdot \mathbf{x}_j^{(l)}.$$3M to $$\mathbf{x}_i^{(l+1)}=\sum_{\mathbf{x}_j^{(l)}\in\delta(\mathbf{x}_i^{(l)})}\mathbb{I}(r_{i,j}>\theta)\cdot \mathbf{x}_j^{(l)}.$$4M while GMac increases from $$\mathbf{x}_i^{(l+1)}=\sum_{\mathbf{x}_j^{(l)}\in\delta(\mathbf{x}_i^{(l)})}\mathbb{I}(r_{i,j}>\theta)\cdot \mathbf{x}_j^{(l)}.$$5 to $$\mathbf{x}_i^{(l+1)}=\sum_{\mathbf{x}_j^{(l)}\in\delta(\mathbf{x}_i^{(l)})}\mathbb{I}(r_{i,j}>\theta)\cdot \mathbf{x}_j^{(l)}.$$6, i.e., about $$\mathbf{x}_i^{(l+1)}=\sum_{\mathbf{x}_j^{(l)}\in\delta(\mathbf{x}_i^{(l)})}\mathbb{I}(r_{i,j}>\theta)\cdot \mathbf{x}_j^{(l)}.$$7 parameters and $$\mathbf{x}_i^{(l+1)}=\sum_{\mathbf{x}_j^{(l)}\in\delta(\mathbf{x}_i^{(l)})}\mathbb{I}(r_{i,j}>\theta)\cdot \mathbf{x}_j^{(l)}.$$8 FLOPs (Wu et al., 2023). SynG2G-Tr retains the $$\mathbf{x}_i^{(l+1)}=\sum_{\mathbf{x}_j^{(l)}\in\delta(\mathbf{x}_i^{(l)})}\mathbb{I}(r_{i,j}>\theta)\cdot \mathbf{x}_j^{(l)}.$$9 dense-attention backbone but adds graph-specific terms with $\theta$0 cost per layer because the dependency graph is a tree and NONE relations are ignored in the relation-dependent terms (Mohammadshahi et al., 2021). SIGformer explicitly targets large sparse signed bipartite graphs by using random-walk-based sampling, yielding an update cost $\theta$1 rather than quadratic full-graph attention (Chen et al., 2024).

At system scale, TorchGT reframed graph-aware attention as a systems problem. It introduced Dual-interleaved Attention, Cluster-aware Graph Parallelism, and Elastic Computation Reformation, reported training speedups of up to $\theta$2, and supported graph sequence lengths of up to $\theta$3M (Zhang et al., 2024). This is significant because many algorithmic graph Transformers remained difficult to deploy on real-world graphs involving millions of nodes. TorchGT shows that large-scale GRAT training depends not only on attention formulation but also on graph-aware parallelism and kernel-level handling of irregular sparsity.

Automatic Graph Topology-Aware Transformer extends this efficiency perspective to model design space exploration. EGTAS jointly searches macro-level topology and micro-level graph-aware strategies through surrogate-assisted evolutionary search, rather than assuming that manual composition of PE, AM, GNN blocks, and skip topology is sufficient (Wang et al., 2024). This suggests that for GRATs, architecture search may be as important as any single hand-designed graph bias.

6. Limitations, methodological tensions, and future directions

A central limitation is lack of a single canonical definition. Some GRATs alter attention logits directly (Yoo et al., 2020, Mohammadshahi et al., 2021), some replace dense attention with graph sparsity or graph convolution (Wu et al., 2023, Zhang et al., 2022), some engineer graph-aware tokens while leaving attention unchanged (Zhou et al., 2024), and some insert graph structure chiefly through spectral encodings (Chen et al., 2024, Luo et al., 2024). This diversity suggests conceptual breadth, but it also makes comparison difficult because “graph awareness” can refer to fundamentally different inductive biases.

Scalability remains another persistent fault line. The original GRAT inherits quadratic attention in the number of nodes (Yoo et al., 2020). Graph-Segmenter mitigates this with window partitioning and thresholded sparse neighborhoods, but it still computes similarity matrices and reports a sizeable parameter increase; it also inherits window dependence and uses a hard non-differentiable threshold $\theta$4 (Wu et al., 2023). TorchGT demonstrates that graph-aware sparsity can be made practical at million-node scale, but its benefits depend on graph sparsity, clustering structure, and hardware-aware reordering (Zhang et al., 2024).

Another tension concerns soft versus hard structural bias. SynG2G-Tr argues for a soft bias that encourages but does not enforce syntax-following attention (Mohammadshahi et al., 2021). Graph-Segmenter and ST-GRAT use more explicit structural restrictions: thresholded graph neighborhoods in the former and neighbor-restricted, diffusion-biased spatial attention with sentinel vectors in the latter (Wu et al., 2023, Park et al., 2019). Neither position has emerged as universally dominant. A plausible interpretation is that soft biases suit tasks where off-graph interactions remain useful, whereas hard graph constraints suit settings where topology has stronger semantic status.

Fairness-aware and signed-graph variants expose a different challenge: graph awareness can amplify undesirable structural regularities if sensitive or signed relations are not modeled carefully. FairGT requires known sensitive attributes and shows marked sensitivity to the number of selected eigenvectors and Transformer layers (Luo et al., 2024). SIGformer likewise shows that the semantics of negative feedback are not uniform and must be controlled through $\theta$5 and $\theta$6 in both the signed Laplacian and the loss (Chen et al., 2024). These models indicate that graph-aware attention is not automatically benign; the choice of graph structure can be a normative design decision as much as a representational one.

Future directions are already visible in the literature. Graph-Segmenter explicitly points to learned sparsity, adaptive multi-scale graphs, alternative edge features, cross-modal graphs, and 3D extensions (Wu et al., 2023). TorchGT points toward more scalable graph-specific training systems (Zhang et al., 2024). EGTAS suggests that automated search over topology and graph-aware strategies is likely to remain important (Wang et al., 2024). Taken together, the field points toward a broader view of GRAT as a modular recipe: choose graph-aware tokens, graph-aware positional or spectral encodings, graph-aware attention or sparsity, and topology-aware depth organization according to the structure of the target domain rather than according to a single fixed Transformer template.