Tokenphormer: Multi-Token Graph Transformer

• The paper introduces Tokenphormer, a multi-token graph Transformer that addresses over-smoothing and over-squashing in traditional GNNs.
• It fuses walk-tokens, SGPM-tokens, and hop-tokens through a Transformer encoder to capture fine-grained structural, global, and local neighborhood information.
• Empirical results on datasets like Cora and Citeseer demonstrate accuracy improvements over models like NAGphormer, validating its tokenization strategy.

Tokenphormer is a structure-aware multi-token Graph Transformer for node classification that represents each target node not by a single token or a single hop-wise summary, but by multiple complementary tokens capturing path structure, local neighborhoods, and broader graph context. Its defining components are the walk-token, the SGPM-token derived from a self-supervised graph pre-train model, and the hop-token derived from decoupled message passing. These tokens are processed jointly by a Transformer and fused by attention for downstream prediction. The method is motivated by two limitations in prior graph learning: message-passing GNNs suffer from over-smoothing and over-squashing, while graph Transformers can incur irrelevant-node noise and weak structural preservation when nodes are treated as isolated tokens in globally connected attention (Zhou et al., 2024).

1. Conceptual position and motivation

Tokenphormer addresses semi-supervised node classification on graphs $G=(V,E)$ with node features $X$. Its central thesis is that graph learning benefits from a richer tokenization scheme analogous to fine-grained token-based representation learning in NLP. Instead of assigning one coarse representation to each node, the method assigns several tokens to each node, with each token encoding a different structural view or receptive-field scale (Zhou et al., 2024).

The motivation is organized around three contrasts. First, traditional GNNs follow message passing and are effective at exploiting local and structural information, but stacking many layers introduces over-smoothing and over-squashing, limiting effective receptive field. Second, graph Transformers provide global receptive field, but full attention over graph nodes can introduce noise from irrelevant nodes and lose structural information. Third, prior tokenized graph Transformers, especially hop-based ones, improve scalability but use tokenizations that the paper characterizes as too monotonous or coarse, because hop-only summaries can miss relations among nodes within the same hop and beyond a fixed $k$-hop range (Zhou et al., 2024).

This design places Tokenphormer within the line of node-sequence graph Transformers, but with a broader token vocabulary. The paper explicitly positions walk-token as fine-grained and structurally expressive, SGPM-token as broader-contextual, and hop-token as locally dense. A plausible implication is that Tokenphormer treats token construction itself as the main inductive-bias mechanism, rather than relying primarily on architectural changes inside attention.

2. Graph serialization and overall architecture

A foundational idea in Tokenphormer is graph serialization: graph structure is converted into sequence-like objects. At graph level, long random walks form a graph document composed of graph sentences. At node level, multiple walks starting from a target node serialize its local-to-mid-range neighborhood. The paper motivates this with random-walk convergence: $\lim_{n \to \infty} (S_n = s_i) = \pi(s_i),$ and for a connected, non-bipartite graph,

$\lim_{n \to \infty} (\pi_0 P^n)(v) = \pi(v).$

These statements justify using walk-based sequences as carriers of graph structure and context (Zhou et al., 2024).

For each node $v$, Tokenphormer constructs a token sequence consisting of $m$ walk-tokens, $n$ hop-tokens, and $1$ SGPM-token. These tokens are concatenated and fed into a Transformer encoder. For Transformer layer $l$, the paper uses a standard pre-norm residual form: $X$0

$X$1

The final token representations are fused by attention: $X$2 after which an MLP-based module performs node classification (Zhou et al., 2024).

The architecture is therefore not a full-graph Transformer. It is a per-node token Transformer whose expressivity depends on how graph structure is serialized into tokens before attention. This distinguishes Tokenphormer from methods that attempt to solve graph learning primarily through global attention patterns.

3. Token taxonomy: walk-token, SGPM-token, and hop-token

The model’s token inventory is organized around three distinct mechanisms.

Token type	Construction	Intended role
Walk-token	Mixed walks from the target node	Fine-grained structural and contextual information
SGPM-token	Self-supervised graph pre-train model	Global/contextual reach beyond limited walk length
Hop-token	Decoupled propagation $X$3	Local dense neighborhood coverage

Walk-token is the paper’s main fine-grained token type. It is generated from mixed walks rather than a single walk process. The four walk types are uniform random walk (URW), non-backtracking random walk (NBRW), neighborhood jump walk (NJW), and non-backtracking neighborhood jump walk (NBNJW). The initial walk-type ratio is $X$4. NBRW is defined by the immediate non-return constraint

$X$5

with transition matrix over directed edges

$X$6

NJW generalizes the step relation by allowing transitions within the $X$7-hop neighborhood of the current node, using $X$8-step propagated probabilities. The paper argues that walk-token preserves path/order information, local and distant node co-occurrence, structural motifs encountered along paths, and contextual relationships induced by surrounding nodes (Zhou et al., 2024).

SGPM-token is derived from SGPM, the Self-supervised Graph Pre-train Model. SGPM is trained on a graph document built from non-backtracking random walks. Walk lengths follow $X$9, with $k$0, the graph radius, and $k$1; in implementation the mean is the graph radius or $k$2. Each node is treated as a token in an NLP-style vocabulary augmented with five special tokens: PAD, UNK, CLS, SEP, and MASK. SGPM input representation is

$k$3

where $k$4, $k$5, $k$6, and $k$7 are token embedding, position embedding, node feature embedding, and centrality embedding. Pretraining uses a BERT-style MLM loss: $k$8 Each node then receives one SGPM-token for downstream Tokenphormer, intended to extend the length limit of walk-token (Zhou et al., 2024).

Hop-token is the explicitly local component. It is defined from decoupled message passing: $k$9 For node $\lim_{n \to \infty} (S_n = s_i) = \pi(s_i),$0, $\lim_{n \to \infty} (S_n = s_i) = \pi(s_i),$1 is the $\lim_{n \to \infty} (S_n = s_i) = \pi(s_i),$2-hop aggregated information. The paper fixes hop number to $\lim_{n \to \infty} (S_n = s_i) = \pi(s_i),$3 in experiments. Hop-token is introduced because walk-token is sparse and path-based, and therefore limited by sampled path length and path density; hop-token systematically aggregates all nodes within a given hop radius and addresses what the paper calls the density limit of walk-token (Zhou et al., 2024).

4. Mathematical formulation and training pipeline

The training pipeline has two stages: SGPM pretraining and supervised Tokenphormer training. In SGPM pretraining, the model generates $\lim_{n \to \infty} (S_n = s_i) = \pi(s_i),$4 non-backtracking random walks per node for training and $\lim_{n \to \infty} (S_n = s_i) = \pi(s_i),$5 walks per node for validation, then optimizes the MLM objective over graph-document sequences. In downstream training, Tokenphormer uses optimizer AdamW, learning rate $\lim_{n \to \infty} (S_n = s_i) = \pi(s_i),$6 for Flickr and Photo and $\lim_{n \to \infty} (S_n = s_i) = \pi(s_i),$7 for other datasets, weight decay $\lim_{n \to \infty} (S_n = s_i) = \pi(s_i),$8, dropout $\lim_{n \to \infty} (S_n = s_i) = \pi(s_i),$9, Transformer heads $\lim_{n \to \infty} (\pi_0 P^n)(v) = \pi(v).$0, and batch size $\lim_{n \to \infty} (\pi_0 P^n)(v) = \pi(v).$1. SGPM pretraining uses DGX-H800 with $\lim_{n \to \infty} (\pi_0 P^n)(v) = \pi(v).$2 NVIDIA H800 80GB, while Tokenphormer training uses RTX 3090Ti 24GB (Zhou et al., 2024).

The model’s online complexity excludes token generation and precomputation. Time complexity is

$\lim_{n \to \infty} (\pi_0 P^n)(v) = \pi(v).$3

where $\lim_{n \to \infty} (\pi_0 P^n)(v) = \pi(v).$4 is the number of nodes, $\lim_{n \to \infty} (\pi_0 P^n)(v) = \pi(v).$5 the number of tokens per node, and $\lim_{n \to \infty} (\pi_0 P^n)(v) = \pi(v).$6 the feature dimension. Space complexity is

$\lim_{n \to \infty} (\pi_0 P^n)(v) = \pi(v).$7

where $\lim_{n \to \infty} (\pi_0 P^n)(v) = \pi(v).$8 is the number of Transformer layers and $\lim_{n \to \infty} (\pi_0 P^n)(v) = \pi(v).$9 the batch size. The paper states that SGPM-token is available in $v$0 at downstream training time once pretrained, and that walk-tokens and hop-tokens can also be precomputed (Zhou et al., 2024).

The method also includes a coverage analysis for walk sampling. Using a label-limited transition matrix

$v$1

the probability of a walk information type $v$2 is

$v$3

and for source node $v$4,

$v$5

If $v$6 is the number of times this information type is sampled in $v$7 trials, then

$v$8

and Hoeffding’s inequality gives

$v$9

Within the paper, this is used to argue that coverage improves rapidly as the number of sampled walks increases (Zhou et al., 2024).

5. Empirical performance and ablation findings

On homogeneous node classification datasets, Tokenphormer reports the following accuracies: Cora $m$0, Citeseer $m$1, Flickr $m$2, Photo $m$3, DBLP $m$4, and Pubmed $m$5. These are the best reported values in the table on Cora, Citeseer, Flickr, Photo, and Pubmed; on DBLP, Gapformer reports $m$6 versus Tokenphormer’s $m$7 (Zhou et al., 2024).

Relative to NAGphormer, Tokenphormer is stronger on all six homogeneous datasets: Cora $m$8 vs $m$9, Citeseer $n$0 vs $n$1, Flickr $n$2 vs $n$3, Photo $n$4 vs $n$5, DBLP $n$6 vs $n$7, and Pubmed $n$8 vs $n$9. On heterogeneous datasets, reported without SGPM-token, Tokenphormer obtains Cornell $1$0, Wisconsin $1$1, and Actor $1$2. Compared with Gapformer, it is lower on Cornell but higher on Wisconsin and Actor (Zhou et al., 2024).

Ablation results show that removing any token type reduces accuracy. Removing SGPM-token lowers performance on all six homogeneous datasets, for example Cora $1$3, Citeseer $1$4, and DBLP $1$5. Removing walk-token produces the largest degradation overall, including Cora $1$6, Citeseer $1$7, Photo $1$8, and DBLP $1$9. Removing hop-token causes smaller but consistent drops, including Cora $l$0, Citeseer $l$1, and Pubmed $l$2 (Zhou et al., 2024).

Walk-type ablations further show that single walk variants—only URW, only NBRW, only NJW, or only NBNJW—perform worse than mixed walks. The paper interprets this as evidence that different graph types benefit from different exploration biases. Expressiveness analysis comparing Tokenphormer with NAGphormer as token number increases shows that NAGphormer improves initially, peaks before token number reaches graph diameter, then declines, whereas Tokenphormer continues improving and becomes more stable as more tokens are added. The paper interprets this as evidence that hop-wise tokenization still suffers some over-smoothing as token depth increases, while the multi-token design is more stable (Zhou et al., 2024).

6. Relation to adjacent models, misconceptions, and limitations

The closest antecedent in the provided literature is NAGphormer, which tokenizes each node as a sequence of hop-wise neighborhood summaries produced by Hop2Token. NAGphormer’s core sequence for node $l$3 is $l$4, with hop-wise propagation $l$5, and it is explicitly motivated as a tokenized graph Transformer for node classification on large graphs (Chen et al., 2022). Tokenphormer inherits the per-node token-sequence viewpoint, but the paper argues that hop-only tokenization is too coarse and introduces walk-token and SGPM-token to complement hop-token. This suggests that Tokenphormer can be read as a direct generalization of hop-token graph Transformers rather than a departure from that line.

A common naming confusion arises with TokenFormer, a distinct architecture that tokenizes model parameters rather than graph structure. In TokenFormer, model parameters are represented as learnable key-value tokens and input tokens attend to them through token-Parameter attention, replacing the linear projections used for $l$6, $l$7, $l$8, output projection, and FFN (Wang et al., 2024). Despite the similarity in names, the two methods operate in different design spaces: Tokenphormer is a graph node-representation method based on multi-token graph serialization, whereas TokenFormer is a parameterization and scaling method for Transformers.

Several limitations are explicit or near-explicit in the Tokenphormer paper. First, SGPM pretraining and token generation increase pipeline complexity through precomputation and pretraining overhead. Second, performance depends on token-hyperparameter choices such as number of walks, walk length, walk-type ratio, and hop number. Third, SGPM-token is omitted in heterogeneous experiments, indicating that it may not always be straightforward or necessary. Fourth, the method is modular but hand-designed: token types and walk variants are engineered rather than learned fully automatically. Fifth, although per-node attention is far cheaper than full-graph attention, complexity remains quadratic in token count $l$9 (Zhou et al., 2024).

Within the broader graph Transformer literature, Tokenphormer’s substantive claim is that graph tokenization should not be limited to one structural view. Its walk-token encodes fine-grained path structure, its SGPM-token injects graph-document-level context, and its hop-token guarantees local neighborhood coverage. The Transformer then serves as the mechanism for collaborative fusion among these heterogeneous token types rather than as the primary source of graph inductive bias.