Graph2Token: Tokenizing Graphs for Transformers

• Graph2Token is a framework that transforms graph structures into token sequences, enabling Transformers to process nodes, edges, and subgraphs effectively.
• It organizes tokenization into granularity levels—node, hop, composite, quantized, and serialized—to reveal structural details and optimize performance.
• Empirical systems like TokenGT, NAGphormer, and NTFormer demonstrate that Graph2Token methods can achieve expressivity comparable to specialized graph neural networks.

Searching arXiv for papers on Graph2Token and related graph tokenization frameworks. Graph2Token denotes a family of graph-to-token mappings that convert a graph into tokenized inputs consumable by Transformers or LLMs. In the contemporary graph-learning literature, the output may be a real-valued token matrix $$P_n:\mathcal{G}_n\to\mathbb{R}^{N_{\mathrm{tok}}(n)\times d_{\mathrm{tok}}(n)}$$, a discrete sequence $T=(t_1,\dots,t_L)$, a tuple of codebook indices produced by vector quantization, or a small set of continuous prompt vectors inserted directly into an LLM. The common objective is to expose graph structure, attributes, and multiscale context through a sequence-model interface. A separate algebraic-graph-theoretic usage of “token graph” refers instead to the $k$-token graph $F_k(G)$, the symmetric $k$-th power of a graph, which is conceptually distinct from learned graph tokenizers (Bechler-Speicher et al., 21 May 2026, Yu et al., 2 Jan 2025, Perozzi et al., 2024, Reyes et al., 2023).

1. Formalization and design space

A general Graph2Token pipeline can be factorized into graph decomposition, tokenization, and sequence construction. In the formulation summarized in the LLM4graph perspective, graph elements such as nodes, edges, and subgraphs are mapped by tokenization functions $\phi_v$, $\phi_e$, and $\phi_S$, then ordered into a sequence $T$. In a more general transformer-theoretic formulation, a graph tokenization is a possibly non-learnable map $$P_n:\mathcal{G}_n\to\mathbb{R}^{N_{\mathrm{tok}}(n)\times d_{\mathrm{tok}}(n)}$$, so the token interface need not be discrete in the NLP sense (Yu et al., 2 Jan 2025, Bechler-Speicher et al., 21 May 2026).

The survey on Graph Transformers divides graph tokenization into four granularity levels: node-level tokenization, edge-level tokenization, subgraph-level tokenization, and hop-level tokenization. That taxonomy is useful because later systems differ less in the existence of a token interface than in what graph object becomes a token and how much structural information is made explicit before attention is applied (Yuan et al., 23 Feb 2025).

Granularity	Token construction	Representative instantiations
Node+edge	One token per node and per edge	TokenGT (Kim et al., 2022)
Hop-level	$T=(t_1,\dots,t_L)$0	NAGphormer (Chen et al., 2022), NAGphormer+ (Chen et al., 2023)
Composite node-centric	Multiple sequences from topology and attribute views	NTFormer (Chen et al., 2024), SwapGT (Chen et al., 12 Feb 2025)
Quantized or pooled	Codebook indices or pooled soft tokens	GQT (Wang et al., 2024), GraphToken (Perozzi et al., 2024), Graph2Token for molecules (Wang et al., 5 Mar 2025)
Serialized discrete	Reversible graph sequence followed by BPE	GraphTokenizer (Guo et al., 11 Mar 2026)

This design space also separates three token modalities. Some methods emit explicit discrete IDs from learned vocabularies or codebooks; some serialize graphs into symbol sequences and then apply a text tokenizer; others produce continuous “soft tokens” in the embedding space of a frozen LLM. This suggests that Graph2Token is best understood as an interface layer rather than a single architecture.

2. Tokenized graph Transformers

A canonical edge-level Graph2Token system is TokenGT, which treats each node and each edge as an independent token. Given node feature $T=(t_1,\dots,t_L)$1 and edge feature $T=(t_1,\dots,t_L)$2, TokenGT defines

$T=(t_1,\dots,t_L)$3

where $T=(t_1,\dots,t_L)$4 are fixed orthonormal node identifiers and $T=(t_1,\dots,t_L)$5 are learned type identifiers. The packed token matrix is then linearly projected and processed by a standard Transformer with a prepended $T=(t_1,\dots,t_L)$6 token. The paper proves that, with appropriate token embeddings, a pure Transformer on these tokens is at least as expressive as a 2-IGN and can approximate any 2-IGN to arbitrary precision; on PCQM4Mv2, TokenGT with Laplacian node identifiers attained validation MAE $T=(t_1,\dots,t_L)$7, while a linear-cost Performer variant achieved $T=(t_1,\dots,t_L)$8 (Kim et al., 2022).

Hop-level tokenization was systematized by NAGphormer through Hop2Token. For each node $T=(t_1,\dots,t_L)$9, the model precomputes

$k$0

and forms the sequence $k$1, where $k$2 is the $k$3-th row of $k$4. The resulting per-node token sequence is short, enabling mini-batch training with per-node attention rather than full-graph attention. NAGphormer reports time complexity $k$5 and outperforms scalable GNNs on large graphs; on Amazon2M, it reached $k$6 accuracy, exceeded PPRGo, GraphSAINT, and GRAND+, and required $k$7 s and $k$8 MB per epoch in the reported efficiency comparison (Chen et al., 2022). The neighborhood-augmentation variant adds Global Neighborhood Augmentation and Local Neighborhood Augmentation on the hop-token sequences and reports further gains, including $k$9 on Amazon2M and state of the art on $F_k(G)$0 small benchmarks in the summarized experiments (Chen et al., 2023).

Later node-tokenized graph Transformers broadened the token generator itself. NTFormer’s Node2Par maps each node $F_k(G)$1 to four sequences: $F_k(G)$2, $F_k(G)$3, $F_k(G)$4, and $F_k(G)$5, combining topology-view and attribute-view neighborhood tokens with topology-view and attribute-view sampled node tokens. These are encoded by plain Transformer blocks without graph-specific modifications, and the resulting representations are fused by learned weights. The reported gains include $F_k(G)$6 on Squirrel, $F_k(G)$7 on Roman-empire, and $F_k(G)$8 on Reddit (Chen et al., 2024). SwapGT extends the same node-tokenized regime with a token-swapping augmentation that replaces a token by a randomly sampled neighbor of that token in a k-NN similarity graph, then constrains consistency by a center alignment loss. Across eight datasets and two train/validation/test regimes, the summary reports uniform improvements over classical GNNs and recent graph Transformers by approximately $F_k(G)$9–$k$0 points (Chen et al., 12 Feb 2025).

3. Discrete tokenizers: quantization, serialization, and indexing

Quantization-based Graph2Token systems explicitly learn a discrete vocabulary for graphs. GQT, “Graph Quantized Tokenizer,” first trains a multi-layer GNN encoder under multi-task self-supervision, then quantizes node or subgraph embeddings with vector quantization and Residual Vector Quantization (RVQ). For each embedding $k$1, nearest-neighbor assignment in a codebook $k$2 yields $k$3, and the RVQ hierarchy recursively quantizes the residual,

$k$4

so the discrete token is the tuple $k$5. Tokenizer training minimizes $k$6, combines semantic edges and PPR during sequence construction, and feeds the resulting token sequence into a standard TransformerEncoder. The detailed summary reports memory reductions of $k$7 for a $k$8-node graph with $k$9-dimensional features, $\phi_v$0 on Physics, and $\phi_v$1 on ogbn-products with $\phi_v$2; it also reports wins on $\phi_v$3 medium-scale homophilous datasets, $\phi_v$4 heterophilous datasets, and strong large-scale results including $\phi_v$5 ROC-AUC on ogbn-proteins and $\phi_v$6 on ogbn-products (Wang et al., 2024).

A later hierarchical quantization framework, implemented in the QUIET framework and also referred to as Graph2Token, keeps the encoder frozen and introduces a self-weighted gating mechanism over multiple RVQ levels. The tokenizer adds a VQ-VAE commitment loss, a balance loss to discourage uneven codeword usage, and a diversity loss between adjacent levels. For each node, a small gating MLP computes logits $\phi_v$7, transforms them into weights $\phi_v$8 by temperature-scaled softmax, and forms the final token embedding $\phi_v$9. In the reported experiments, Graph2Token improves Corafull node classification by approximately $\phi_e$0 over GQT under the same codebook size, increases Cora link-prediction MRR from $\phi_e$1 to $\phi_e$2, reduces required epochs by $\phi_e$3–$\phi_e$4, and yields token sequences $\phi_e$5–$\phi_e$6 shorter than raw GNN embeddings (Xiang et al., 14 Oct 2025).

A distinct discrete route is reversible graph serialization followed by subword tokenization. “Graph Tokenization for Bridging Graphs and Transformers” constructs deterministic edge-covering walks guided by global frequencies $\phi_e$7 of labeled edge-context patterns $\phi_e$8, then trains a BPE vocabulary over the serialized sequences. The tokenizer is reversible up to isomorphism, and BPE is trained by repeated merges of the most frequent adjacent pair. On 14 benchmarks, GT+GTE achieved $\phi_e$9 ROC-AUC on OGBG-molhiv, $\phi_S$0 accuracy on Mutag, and $\phi_S$1 MAE on QM9; on ZINC, raw reversible Eulerian sequences of approximately $\phi_S$2 tokens were reduced to approximately $\phi_S$3 tokens after BPE, with reported $\phi_S$4 compression and approximately $\phi_S$5 faster per-epoch training than the no-BPE variant (Guo et al., 11 Mar 2026).

Graph2Token can also serve indexing rather than prediction. CoRGII learns dense node embeddings with a shared GNN, maps them through sigmoid MLP heads into $\phi_S$6-bit soft codes with $\phi_S$7, hard-thresholds them into binary codes, and interprets the $\phi_S$8 possible codes as a latent vocabulary for an inverted index. It then augments classical inverted-index scoring with a learned impact network $\phi_S$9 and supports Hamming and co-occurrence multi-probing. The summary reports that CoRGII strictly outperforms baselines in space–accuracy tradeoff on four TU datasets; on COX2, it reaches MAP approximately $T$0 at $T$1, and on PTC-FR it reaches MAP approximately $T$2 while retrieving only about $T$3 of the corpus (Chakraborty et al., 26 Oct 2025).

4. Graph2Token as an interface to LLMs

Graph2Token methods for LLMs often replace discrete graph vocabularies with continuous soft prompts. GraphToken learns an encoder $T$4 that maps an attributed graph $T$5 to $T$6, where $T$7 is a fixed-length sequence of continuous token embeddings in the LLM token space. Internally, it uses a lightweight GNN body and projection heads to produce node-, edge-, and graph-level prompt vectors, which are then spliced directly into the frozen LLM input. On the GraphQA benchmark, the reported gains include node-count $T$8, cycle-check $T$9, edge-existence $$P_n:\mathcal{G}_n\to\mathbb{R}^{N_{\mathrm{tok}}(n)\times d_{\mathrm{tok}}(n)}$$0, shortest-path $$P_n:\mathcal{G}_n\to\mathbb{R}^{N_{\mathrm{tok}}(n)\times d_{\mathrm{tok}}(n)}$$1, and an increase on triangle-counting from $$P_n:\mathcal{G}_n\to\mathbb{R}^{N_{\mathrm{tok}}(n)\times d_{\mathrm{tok}}(n)}$$2 zero-shot and $$P_n:\mathcal{G}_n\to\mathbb{R}^{N_{\mathrm{tok}}(n)\times d_{\mathrm{tok}}(n)}$$3 soft-prompt to $$P_n:\mathcal{G}_n\to\mathbb{R}^{N_{\mathrm{tok}}(n)\times d_{\mathrm{tok}}(n)}$$4, described as up to $$P_n:\mathcal{G}_n\to\mathbb{R}^{N_{\mathrm{tok}}(n)\times d_{\mathrm{tok}}(n)}$$5 absolute accuracy on triangle-counting; overall average accuracy rises from approximately $$P_n:\mathcal{G}_n\to\mathbb{R}^{N_{\mathrm{tok}}(n)\times d_{\mathrm{tok}}(n)}$$6 for text-only baselines to approximately $$P_n:\mathcal{G}_n\to\mathbb{R}^{N_{\mathrm{tok}}(n)\times d_{\mathrm{tok}}(n)}$$7 (Perozzi et al., 2024).

In molecular few-shot learning, Graph2Token is used to align graph representations with an LLM’s native vocabulary without fine-tuning the LLM backbone. The method constructs a molecule–text paired dataset from CHEBI, HMDB, and PubChem IUPAC names, yielding approximately $$P_n:\mathcal{G}_n\to\mathbb{R}^{N_{\mathrm{tok}}(n)\times d_{\mathrm{tok}}(n)}$$8 K unique molecules with short descriptions of average $$P_n:\mathcal{G}_n\to\mathbb{R}^{N_{\mathrm{tok}}(n)\times d_{\mathrm{tok}}(n)}$$9 tokens. A 5-layer GIN encoder produces a graph vector $T=(t_1,\dots,t_L)$00, BERT-base yields a text vector $T=(t_1,\dots,t_L)$01, and a CLIP-style InfoNCE objective aligns them. A cross-attention retrieval module then maps $T=(t_1,\dots,t_L)$02 into the LLM token-embedding space, producing a vector $T=(t_1,\dots,t_L)$03 that replaces a special token $T=(t_1,\dots,t_L)$04 in Llama3-8B. IUPAC names are placed at the start of the prompt. The reported few-shot results include classification averages of $T=(t_1,\dots,t_L)$05 and $T=(t_1,\dots,t_L)$06 in the $T=(t_1,\dots,t_L)$07 and $T=(t_1,\dots,t_L)$08 settings for the TOX21 transfer protocol, regression averages of $T=(t_1,\dots,t_L)$09 and $T=(t_1,\dots,t_L)$10, and parameter efficiency of $T=(t_1,\dots,t_L)$11 M tuned parameters, or $T=(t_1,\dots,t_L)$12 of total parameters (Wang et al., 5 Mar 2025).

The single-token bottleneck of graph-to-LLM interfaces has also been studied directly. “Is One Token All It Takes?” formulates pooling as $T=(t_1,\dots,t_L)$13 and evaluates Top-k, SAGPool, DiffPool, MinCutPool, and Virtual Node Pooling (VNPool). The paper reports that pooling can be unstable under soft-prompt tuning, while LoRA stabilizes specific hierarchical projections, especially VNPool and pruning methods; VNPool is shown to be structurally equivalent to a single-layer Perceiver IO encoder. On WebQSP with $T=(t_1,\dots,t_L)$14 tokens and LoRA $T=(t_1,\dots,t_L)$15, mean pooling with one token attains $T=(t_1,\dots,t_L)$16 Hit@1, all-tokens reaches $T=(t_1,\dots,t_L)$17, and VNPool reaches $T=(t_1,\dots,t_L)$18 (Grover et al., 1 Apr 2026).

The LLM4graph perspective organizes these systems around four modality-gap challenges: alignment, position, multi-level semantics, and context. Alignment concerns projection into the LLM token space; position concerns replacing linear token order by graph positional encodings; multi-level semantics concerns the mismatch between text’s long-range dependencies and graphs’ multi-hop structure; context concerns the limited corpus-level context available in a single graph. In that framing, Graph2Token is one of two major transformation paradigms alongside Graph2text (Yu et al., 2 Jan 2025).

5. Expressivity, trade-offs, and a terminological ambiguity

A central theoretical result is that Graph2Token is not merely an implementation detail. TokenGT proves that a standard Transformer operating on node and edge tokens with suitable embeddings can simulate the equivariant basis structure of 2-IGNs. The proof proceeds through a construction in which, with $T=(t_1,\dots,t_L)$19 heads, one can approximate a single $T=(t_1,\dots,t_L)$20 layer, and stacking such layers yields approximation of any 2-IGN. The survey accordingly identifies edge-level tokenization as a route to 2-WL power, in contrast to node-only tokens without structural positional encoding, which are at best 1-WL expressive (Kim et al., 2022, Yuan et al., 23 Feb 2025).

“Lost in Tokenization” shifts the emphasis from expressivity within a fixed tokenizer to trade-offs across tokenizers. It studies three core Graph2Token maps: spectral tokenization, random-walk tokenization, and adjacency tokenization. The paper proves depth separations: $T=(t_1,\dots,t_L)$21-closed-walk detection is available in $T=(t_1,\dots,t_L)$22 depth under random-walk tokenization but requires $T=(t_1,\dots,t_L)$23 depth with adjacency tokens; graph connectivity can be decided with $T=(t_1,\dots,t_L)$24 under full spectral tokens but requires $T=(t_1,\dots,t_L)$25 layers with adjacency tokens. It also proves that random-walk tokenization is lossy for any walk length, that full spectral tokenization is lossless up to isomorphism but ill-conditioned for local tasks such as edge prediction, and that limited-depth transformers generally cannot translate between tokenization families. In the empirical section, local tasks favor adjacency tokens, global molecular tasks favor spectral tokens, and concatenating multiple tokenizations often gives the best overall performance (Bechler-Speicher et al., 21 May 2026).

The term “token graph” has an older and unrelated meaning in algebraic graph theory. There, the $T=(t_1,\dots,t_L)$26-token graph $T=(t_1,\dots,t_L)$27 has as vertices the $T=(t_1,\dots,t_L)$28-subsets of $T=(t_1,\dots,t_L)$29, and two such subsets are adjacent when their symmetric difference is an edge of $T=(t_1,\dots,t_L)$30. This construction satisfies $T=(t_1,\dots,t_L)$31 and $T=(t_1,\dots,t_L)$32, the Johnson graph. The literature cited here studies the adjacency and Laplacian spectra of $T=(t_1,\dots,t_L)$33, exact spectral-radius results for walk-regular graphs, quotient-matrix analyses for $T=(t_1,\dots,t_L)$34 when $T=(t_1,\dots,t_L)$35 is distance-regular, and a proposed generalization of Aldous’ spectral-gap conjecture (Reyes et al., 2023). A common misconception is therefore terminological: Graph2Token in machine learning and $T=(t_1,\dots,t_L)$36-token graphs in combinatorics are not the same object.

6. Empirical regularities, limitations, and future directions

Several regularities recur across the literature. Hop-level and neighborhood-token methods enable mini-batch training and large-graph scalability, as seen in NAGphormer and its augmentation-based extension (Chen et al., 2022, Chen et al., 2023). Quantization-based interfaces can trade continuous embeddings for compact discrete codes while retaining downstream performance and substantially reducing memory, as in GQT and QUIET (Wang et al., 2024, Xiang et al., 14 Oct 2025). LLM-oriented Graph2Token systems often preserve the LLM backbone and concentrate learning in small encoders, tokenizers, or low-rank adapters, as in GraphToken, molecular Graph2Token, and LoRA-stabilized multi-token pooling (Perozzi et al., 2024, Wang et al., 5 Mar 2025, Grover et al., 1 Apr 2026). Theoretical and empirical work further indicate that no single tokenization family dominates across all tasks; different tasks favor different structural views (Bechler-Speicher et al., 21 May 2026).

The literature also rejects several simplifying assumptions. Graph2Token is not inherently discrete: it may produce codebook indices, symbol sequences, or continuous prompt vectors. It is not tied to one granularity: tokens may represent nodes, edges, hop neighborhoods, subgraphs, whole graphs, or pooled latent slots. Nor is tokenization a neutral preprocessing step. One theory paper states explicitly that the graph-to-token map is a fundamental component of transformer expressivity, and several empirical studies report that the choice of token sequence generator substantially changes performance even when the Transformer backbone remains vanilla (Bechler-Speicher et al., 21 May 2026, Chen et al., 2024, Chen et al., 12 Feb 2025).

Open problems are correspondingly diverse. The survey identifies token explosion and the $T=(t_1,\dots,t_L)$37 burden of self-attention for edge-level tokenization on dense graphs, motivating hierarchical token pruning and linearized attention (Yuan et al., 23 Feb 2025). GraphTokenizer notes that continuous node and edge features must be quantized, that BPE can obscure node-level or edge-level targets, and that serialization becomes a CPU bottleneck at OGB scale (Guo et al., 11 Mar 2026). The hierarchical quantization literature highlights sensitivity to hyperparameters such as $T=(t_1,\dots,t_L)$38 and suggests non-Euclidean quantizers and large-scale unsupervised pre-training of codebooks (Xiang et al., 14 Oct 2025). Molecular Graph2Token points to dependence on CHEBI and HMDB coverage and proposes adding corpora such as ChEMBL and PubMed abstracts, while also noting that multitoken graph vocabularies may help large or complex molecules (Wang et al., 5 Mar 2025). The GraphQA pooling study reports instability of dense clustering operators and benchmark saturation, with FandE scores $T=(t_1,\dots,t_L)$39 on ExplaGraphs and $T=(t_1,\dots,t_L)$40 on WebQSP (Grover et al., 1 Apr 2026). This suggests that future Graph2Token research will be shaped as much by benchmark design and tokenizer objectives as by Transformer scale.