Graph Tokenization Framework

• Graph tokenization framework is a set of methods that convert irregular graph structures into fixed tokens, bridging the gap between graphs and sequence models.
• It employs various approaches such as node2token, pairwise, and holistic tokenization to capture local and global graph features using techniques like residual quantization and specialized vocabularies.
• The framework enables seamless integration with Transformers and LLMs by aligning graph tokens with text tokens, preserving structural, positional, and semantic information for improved model performance.

Graph tokenization framework denotes a class of methods that converts graph-structured data into discrete tokens, token sequences, or token embeddings so that sequence models can operate on graphs through a token interface rather than through raw adjacency. In the LLM4graph taxonomy, this corresponds to the Graph2token paradigm, which transforms graphs into tokens or token embeddings through pre-trained or graph-based models; it is less human-readable than Graph2text, but more fine-grained and can more fully fuse structural and semantic information (Yu et al., 2 Jan 2025). Across recent work, the same idea appears as dedicated entity tokens for knowledge graphs, special tool tokens for dependency-aware planning, hierarchical and quantized node tokens for graph transformers, reversible graph serialization followed by BPE, and sparse latent tokens for graph retrieval (Su et al., 26 Feb 2026).

1. Conceptual foundations

Graph tokenization addresses a basic mismatch between graph-structured data and token-native computation. Graphs are irregular, permutation-invariant, and often sparse in text-like context, whereas Transformers and LLMs are optimized for sequences of tokens. Tokenization therefore acts as the interface that determines which structural facts are already visible at input time and which must be reconstructed by the model during computation. One formal treatment defines a tokenization as a map

$$\mathcal P_n:\mathcal G_n \to \mathbb R^{N_{\mathrm{tok}(n)}\times d_{\mathrm{tok}(n)}},$$

with the token count and token dimension fixed by the chosen representation (Bechler-Speicher et al., 21 May 2026).

A second formalization factorizes the tokenizer as

$\Phi = T \circ f,$

where $f$ is a graph serialization function and $T$ is a sequence tokenizer learned on the serialized corpus, such as BPE (Guo et al., 11 Mar 2026). This suggests that graph tokenization is not a single algorithmic recipe, but a design space spanning serialization, quantization, vocabulary construction, embedding alignment, and downstream decoding.

The distinction between Graph2text and Graph2token is central in this landscape. Graph2text transforms graph structures into readable text, whereas Graph2token transforms graphs into tokens or token embeddings through pre-trained or graph-based models. The former is more interpretable because it uses human-readable descriptions; the latter is more directly aligned with the computational substrate of token-based models and is repeatedly described as preserving structural, positional, semantic, and contextual information at token granularity (Yu et al., 2 Jan 2025).

2. Token units and representational granularity

Within Graph2token, token granularity determines what graph information is preserved. The survey literature distinguishes four classes: Node2token, Pairwise nodes2token, Group-aware nodes2token, and Holistic nodes2token. Node2token maps individual nodes into tokens, usually carrying local neighborhood, attribute, and positional information. Pairwise nodes2token represents node pairs as tokens and is suited to edge-centric or relation-centric tasks. Group-aware nodes2token converts communities, subgraphs, or motifs into tokens. Holistic nodes2token transforms the whole graph into a sequence or a global token representation for graph-level tasks (Yu et al., 2 Jan 2025).

This taxonomy has been specialized in several domains. In molecular graph-language alignment, HIGHT uses a hierarchy of atom, motif, and molecular tokens, arguing that flat node-centric tokenization overlooks higher-order molecular structures and contributes to hallucination (Chen et al., 2024). In masked graph modeling for molecules, SimSGT organizes tokenizers by node, edge, motif, pretrained GNN-based, and simple GNN-based granularity, and treats the tokenizer as the component that defines what the model must reconstruct (Liu et al., 2023). In node classification, Tokenphormer represents each target node with a multi-token set consisting of walk-tokens, an SGPM-token, and hop-tokens, thereby separating path-level, global-contextual, and dense local information (Zhou et al., 2024).

Temporal and patch-based settings extend the same principle. Todyformer first partitions a continuous-time dynamic graph into non-overlapping temporal patches, then applies an MPNN-based tokenizer inside each patch before packing the resulting node tokens into Transformer-ready sequences (Biparva et al., 2024). In recommendation, MoToRec makes an important distinction: the graph itself is not tokenized; rather, the multimodal semantic content used by the graph is converted into discrete semantic units before being injected into the recommender graph (Liu et al., 11 Feb 2026). A plausible implication is that graph tokenization frameworks can target either graph topology directly or graph-conditioned semantic content, provided the downstream system consumes a discrete token interface.

3. Construction mechanisms

A recurrent pipeline for graph tokenization consists of graph decomposition, tokenization, sequence construction, integration with LLMs, and task-specific fine-tuning (Yu et al., 2 Jan 2025). What varies across frameworks is the mechanism used at the tokenization stage.

One major family uses learned quantization. GQT trains a graph tokenizer offline with a GNN encoder, Residual Vector Quantization, and multi-task graph self-supervised learning, then freezes the tokenizer and feeds the resulting discrete token sequences to a vanilla Transformer (Wang et al., 2024). QUIET also uses multi-level residual vector quantization, but adds a self-weighted mechanism that dynamically reweights hierarchical information across quantization levels for task-adaptive aggregation, while keeping the graph encoder frozen (Xiang et al., 14 Oct 2025). MoToRec uses a sparsely-regularized Residual Quantized Variational Autoencoder to turn multimodal item content into discrete semantic codes, and KGTB uses entity-specific codebooks to quantize users, POIs, categories, and regions into structural IDs supervised by KG reconstruction (Liu et al., 11 Feb 2026, Sun et al., 15 Sep 2025).

A second family expands the model vocabulary with graph-native symbols. GRAFT maps each tool node to a dedicated special token and learns directed tool dependencies in the representation space, so that tool planning becomes sequence generation over a controlled vocabulary rather than free-form text generation (Gao et al., 12 May 2026). KGT registers every entity and relation in a knowledge graph as a special token, thereby making KG elements first-class tokens rather than subword aggregates (Su et al., 26 Feb 2026). NT-LLM takes a different route: it selects anchor nodes and represents each node by its vector of shortest-path distances to those anchors, then learns a mapping from these structural signatures into an LLM-compatible embedding space (Ji et al., 2024).

A third family relies on graph serialization followed by discrete sequence tokenization. “Graph Tokenization for Bridging Graphs and Transformers” constructs a reversible graph-to-sequence interface, favors edge-covering traversals such as frequency-guided Eulerian traversal, and then learns higher-level tokens with BPE so that frequently occurring graph substructures become mergeable units (Guo et al., 11 Mar 2026). CORGII uses contextual dense graph representations, converts node representations into sparse binary codes by thresholding, interprets those codes as tokens in a latent vocabulary, and indexes corpus graphs through inverted lists (Chakraborty et al., 26 Oct 2025).

Across these mechanisms, token identity is rarely treated as a raw surrogate for a node identifier. Instead, token construction typically embeds a chosen structural view: local neighborhoods, pairwise relations, motif membership, hierarchy level, relative position, graph-wide context, or dataset-level substructure frequency. This is why the survey literature characterizes the process not as mere serialization, but as graph tokenization (Yu et al., 2 Jan 2025).

4. Integration with Transformers and LLMs

Once constructed, graph tokens must be aligned with the computational interface of the downstream model. The standard LLM pipeline consists of tokenization, embedding, and generation; when the graph input is already tokenized or embedded, parts of the initial pipeline may be bypassed or adjusted, and graph token embeddings are often aligned to text token space before being fed into the LLM (Yu et al., 2 Jan 2025).

Several integration patterns recur. Prefix-based and soft-prompt mechanisms map graph tokens into the LLM embedding space and prepend them to language instructions. This is the dominant pattern in Graph-Tokenizing LLMs, where graph embeddings are projected into graph tokens and used as prefix tokens for text output generation (Zhang et al., 5 May 2026). GRAFT internalizes the tool graph more aggressively: instead of retrieving tools from an external graph or serializing the graph into a prompt, it parameterizes the tool graph as dependency-aware tool tokens and trains on-policy tool context distillation so that graph legality influences next-token prediction directly (Gao et al., 12 May 2026).

Knowledge-graph completion illustrates a second pattern: full-space prediction over a graph-native vocabulary. KGT uses specialized entity and relation tokens, relation-guided gating to fuse textual and structural priors, and dual-view decoupled prediction so that semantic and structural reasoning are separated and combined at the output layer (Su et al., 26 Feb 2026). In graph-language modeling for molecules, HIGHT uses separate adapters for atom-level, motif-level, and graph-level tokens before instruction tuning with a Vicuna backbone, while its HiPubChem dataset augments caption supervision with positive and negative motif statements (Chen et al., 2024).

Graph tokenization also enables standard sequence models to be used without graph-specific architectural modifications. GQT explicitly decouples tokenizer learning from Transformer learning and then trains a standard Transformer encoder on precomputed discrete graph tokens (Wang et al., 2024). The reversible-serialization-plus-BPE framework applies ordinary BERT and GTE backbones directly to graph benchmarks, and reports that the graph-to-sequence interface can bridge graphs and the broader ecosystem of sequence models (Guo et al., 11 Mar 2026). In dynamic graphs, Todyformer alternates between local tokenization by an MPNN and global contextualization by a Transformer with temporal positional encoding, thereby using tokenization as the handshake between local graph computation and long-range sequence modeling (Biparva et al., 2024).

5. Theoretical trade-offs and empirical behavior

Theoretical work has made explicit that tokenization is part of the model’s expressivity. “Lost in Tokenization” shows that different tokenizations induce distinct depth regimes: the same graph computation may be realizable by a shallow transformer under one tokenization and require substantially larger depth under another. In that analysis, adjacency tokenization is local and exact but order-dependent; full spectral tokenization is lossless but ill-conditioned for local tasks; and random-walk tokenization is permutation-equivariant but lossy for any walk length (Bechler-Speicher et al., 21 May 2026). The same work further argues that limited-depth transformers cannot in general convert between tokenization families efficiently, so an unfavorable tokenization may preclude efficient recovery of a more suitable structural representation.

This theoretical picture is consistent with empirical findings that different tasks favor different structural views. The same study reports that adjacency performs best on tasks with explicit local structure, Laplacian tokenization performs best on some globally organized tasks such as Tox21, and combined tokenization often performs best or near best overall (Bechler-Speicher et al., 21 May 2026). The consequence is not that one universal tokenizer has been identified, but that tokenization choice is task-dependent and tightly coupled to the downstream notion of locality, globality, or diffusion.

Empirical work on LLMs adds a second caution: token availability does not imply token understanding. GTEval finds that existing Graph-Tokenizing LLMs do not fully understand graph tokens, exhibit over-sensitivity or over-insensitivity to instruction changes, and rely heavily on text for reasoning. The same evaluation shows that graph tokens preserve task-relevant graph information and receive attention across LLM layers, but their utilization varies across models and instruction variants; additional instruction tuning improves performance on original and seen instructions without fully resolving graph-token understanding (Zhang et al., 5 May 2026).

At the same time, specialized tokenization schemes have been associated with concrete gains in specific regimes. GRAFT reports a hallucination ratio of 0.00% and the best EM and ACPL across all backbones and datasets in dependency-aware tool planning (Gao et al., 12 May 2026). HIGHT reports reducing hallucination by 40% in molecule-language alignment (Chen et al., 2024). GQT reports a 270-fold memory reduction in its cited large-graph example through residual quantization (Wang et al., 2024). The reversible graph tokenization framework with frequency-guided serialization and BPE reports state-of-the-art results on 14 benchmark datasets while using standard Transformers without architectural changes (Guo et al., 11 Mar 2026). These results indicate that graph tokenization can improve reliability, compactness, and transfer, but only when the chosen tokens expose the structural signals that the task actually requires.

6. Applications, limitations, and open directions

Graph tokenization has already been applied across a wide range of settings: node classification, link prediction, graph property prediction, knowledge graph completion, dynamic graph learning, graph retrieval, cold-start recommendation, next POI recommendation, molecule-language alignment, and tool planning (Yu et al., 2 Jan 2025). The application-specific token units vary widely—entity tokens, structural IDs, semantic codes, patch tokens, dedicated tool tokens, walk tokens, motif tokens—but the underlying goal is consistent: bridge the modality gap by converting structural priors into a token interface that sequence models already understand.

Several open problems recur across the literature. The LLM4graph survey highlights domain-specific instructions and a general learning framework, explicitly suggesting that Graph2text and Graph2token should be combined so that Graph2text supplies domain-specific modules and Graph2token supplies general structural learning. It also identifies theoretical questions around permutation invariance and geometric equivariance, fair graph learning, efficiency and scalability through hierarchical attention, neighborhood sampling, model distillation, and pruning, and tokenization for dynamic graphs whose structure evolves over time (Yu et al., 2 Jan 2025).

Other frameworks expose more specific limitations. The reversible serialization approach notes difficulties with continuous node and edge features, node- and edge-level prediction under BPE merging, and context-window limits for very large graphs (Guo et al., 11 Mar 2026). NT-LLM leaves weighted graphs for future work and makes the cost of anchor selection explicit through its dependence on coverage radius, coverage ratio, and anchor count (Ji et al., 2024). GRAFT is still a pre-execution planning method because it does not use tool execution feedback (Gao et al., 12 May 2026). QUIET does not deeply explore pre-training at truly foundation-model scale, multimodal graph settings, or domain transfer beyond its evaluated benchmarks (Xiang et al., 14 Oct 2025).

A final unresolved issue is semantic robustness. Current evidence suggests that graph tokens can preserve graph information and still fail to be interpreted robustly by LLMs under instruction variation (Zhang et al., 5 May 2026). This suggests that future graph tokenization research will not be limited to designing better graph-to-token maps. It will also require stronger alignment objectives, explicit graph supervision, and evaluation protocols that test whether graph tokens are merely present in the prompt or are genuinely usable as semantic units of reasoning.