Random-Walk Tokenization for Graph Transformers

• Random-walk tokenization is a method for encoding graph structure by representing each node as a sequence of return probabilities from short random walks.
• It enables efficient, constant-depth inference tasks such as k-closed-walk detection, while inherently sacrificing fine-grained edge-level details.
• Compared to adjacency and spectral tokenizations, it offers a unique diffusion-oriented summary that balances computational efficiency with information loss.

Random-walk tokenization is a principled method for encoding graph structure as input for transformer models, in which each node is represented by a feature vector consisting of return probabilities from short random walks on the graph. This approach provides a diffusion-oriented summary of the local and global connectivity patterns surrounding each node, making it a fundamental tokenization paradigm in graph-transformer architectures. Its expressivity and limitations have been rigorously analyzed, especially in contrast to adjacency and spectral tokenizations, revealing fundamental trade-offs regarding information retention, inference depth, and task suitability (Bechler-Speicher et al., 21 May 2026).

1. Formal Construction

Let $G = (V, E)$ be an undirected graph with $n$ vertices, adjacency matrix $A$, degree matrix $D$, and random-walk transition matrix $P = D^{-1}A$. Select a walk length function $t(n)$. The random-walk tokenization $$\mathcal{P}_{\mathrm{RW},\,t} \colon \mathcal{G}_n \to \mathbb{R}^{n \times t(n)}$$ constructs a token $x_v \in \mathbb{R}^{t(n)}$ for each node $v$ by recording the sequence of return probabilities:

$$x_v = \left( (P^1)_{vv},\, (P^2)_{vv},\, \dots,\, (P^{t(n)})_{vv} \right),$$

where $n$0 is the probability that a random walk of length $n$1 starting at $n$2 returns to $n$3. The full token matrix is

$n$4

These token vectors are linearly projected to the transformer’s hidden dimension and used as the initial token inputs. The transformer encoder architecture then processes these tokens, with each node able to attend globally, but its positional information is constrained to the random-walk statistics for its location in the graph.

2. Theoretical Expressivity and Limitations

Random-walk tokenization exhibits categorical expressivity limitations. The core limitation is its fundamental lossiness: key structural information, including global topological properties, is irrecoverable from random-walk tokens for any finite walk length.

Planarity Undecidability Theorem:

For any walk-length function $n$5, there exist planar and non-planar graphs $n$6 with

$n$7

yet one can arrange for $n$8 to be non-planar (contain a Kuratowski minor such as $n$9) while $A$0 is planar. The construction leverages Godsil–McKay switching: for suitably chosen switching sets, the modified graph $A$1 satisfies $A$2, with $A$3 an orthogonal involution, yielding invariant return probabilities for all $A$4. As a result, no Graph Transformer, regardless of depth or width, can decide global planarity or reconstruct the full adjacency matrix from these tokens (Bechler-Speicher et al., 21 May 2026).

3. Inference Depth and Computational Regimes

Random-walk tokenization enables certain computations inaccessible to shallow transformers when using other tokenizations. Specifically, it directly exposes $A$5 statistics, allowing tasks such as $A$6-closed-walk detection to be solved in constant ($A$7) transformer layers.

Depth Separation:

• With random-walk tokens, a transformer requires only $A$8 layers to decide if $A$9 exceeds a threshold.
• By contrast, when input tokens are only adjacency rows, extracting $D$0-closed-walks incurs a lower bound of $D$1 layers under the conjecture $D$2 (Bechler-Speicher et al., 21 May 2026).

Tokenization	Information Retention	Inference Depth
Adjacency	Lossless (full reconstruction)	$D$3 for edge queries; $D$4 for $D$5-walks
Spectral (full)	Lossless up to isomorphism	$D$6 for global tasks
Random-walk	Lossy (fixed $D$7)	$D$8 for $D$9-walks, ill-suited for edges

This table summarizes several depth and information trade-offs under each tokenization regime.

4. Comparison with Adjacency and Spectral Tokenizations

Relative to other tokenizations, random-walk tokenization offers a unique set of capabilities and shortcomings:

• Adjacency ($P = D^{-1}A$0): Enables exact graph reconstruction; local queries (edge existence) are trivial with $P = D^{-1}A$1 layers, but recovering diffusion statistics like $P = D^{-1}A$2-walks requires $P = D^{-1}A$3 layers.
• Spectral ($P = D^{-1}A$4): Full spectral tokenization (all Laplacian eigenpairs) is also lossless up to graph isomorphism. However, extracting local edge information is ill-conditioned—gradient magnitudes required for local queries scale at least linearly in the maximum degree.
• Random-walk ($P = D^{-1}A$5): Intrinsically lossy—cannot reconstruct the adjacency matrix or decide planarity. However, provides well-conditioned, shallow access to diffusion-oriented statistics and $P = D^{-1}A$6-walk counts, and does not experience the conditioning issues of the spectral encoding.

A crucial result is that, despite both random-walk and spectral tokenizations deriving from adjacency information, there is a provable depth separation: a limited-depth transformer cannot in general convert between these representations.

5. Empirical Performance Across Tasks

Empirical experiments confirm the theoretical separations:

• Synthetic $P = D^{-1}A$7-walk Recovery:

Training a transformer to predict $P = D^{-1}A$8 from adjacency tokens produces a relative error increasing with $P = D^{-1}A$9, manifesting the depth barrier. Conversely, using random-walk tokens as input, the error remains stable at around 6–8%, regardless of $t(n)$0.

• Downstream Benchmarks:

On applied datasets (e.g., molecular regression, node classification), random-walk tokenization excels where global connectivity or diffusion-like properties are target-relevant (such as ZINC property prediction). It is suboptimal for tasks requiring precise edge-level information, such as MaxClique and Topological Ordering, where adjacency tokens excel. Combining all three tokenizations—random-walk, adjacency, and spectral—by concatenating embeddings per node, often achieves the best aggregate results, as complementary information is exploited by the transformer (Bechler-Speicher et al., 21 May 2026).

6. Practical Guidance and Application Suitability

Random-walk tokenization is well-suited for tasks that depend primarily on diffusion processes or random-walk return probabilities, such as cycle detection or regression problems sensitive to spectral graph properties (e.g., spectral gap). It is particularly advantageous when shallow, constant-depth inference for such statistics is needed.

However, as it is provably blind to finer structural details—specifically, exact adjacency and global planarity—it should not be used in isolation for tasks requiring these properties. For such scenarios, it should be paired with:

• Adjacency tokens where fine-grained edge information is essential (e.g., clique detection, shortest paths, planarity),
• Spectral tokens when a globally well-conditioned representation is desired, especially for connectivity or community structure detection, while being aware of spectral tokenization’s ill-conditioning for local queries.

A highly effective practice is to combine the three paradigms—random-walk, adjacency, and spectral—by concatenating their respective node embeddings, enabling the model to exploit the distinct local and global signals available at shallow depth (Bechler-Speicher et al., 21 May 2026).