Adjacency Tokenization: Methods & Trade-offs

• Adjacency tokenization is a method that encodes local structures by mapping discrete elements to tokens that represent explicit pairwise relationships in texts and graphs.
• It applies to string and graph domains to optimize representation by covering adjacent symbol pairs or node connections, highlighting coverage and computational complexity trade-offs.
• Empirical evaluations show improved token compression and neighborhood accuracy for local tasks, though scalability and global reasoning remain significant challenges.

Adjacency tokenization is a family of encoding methods that directly expose local adjacency information to models, typically by mapping discrete objects (such as characters in strings or nodes in graphs) to tokens representing explicit pairwise or neighborhood relationships. In Natural Language Processing, adjacency-driven approaches such as the partition–cover tokenization framework formalize the vocabulary learning problem in terms of covering adjacent symbol pairs within corpora. In the context of graph learning, adjacency tokenization encodes each node’s direct connections as tokens (often as rows of the adjacency matrix), making local structure directly accessible to models such as transformers. This class of tokenizations offers strong locality guarantees, is lossless up to node ordering, and provides a rigorous connection to combinatorial optimization, but is limited by scalability and depth requirements for global tasks. Adjacency tokenization stands in stark methodological contrast to spectral and random-walk tokenizations and exposes fundamental trade-offs in expressivity and efficiency for both language and graph models (Lim et al., 8 Jan 2025, Bechler-Speicher et al., 21 May 2026).

1. Formal Definitions

Adjacency tokenization manifests in two premier domains: string tokenization (partition–cover) and graph tokenization.

Partition–Cover for Strings:

Given a corpus $C=(W, \mathrm{count})$ over finite alphabet $\Sigma$, with each word $W\in\Sigma^+$, adjacency tokenization seeks a vocabulary $S$ of size $k$ to minimize the total number of tokens needed to represent the corpus, subject to a constraint that tokens explain adjacent character pairs:

• $\mathrm{partition}(W, S\cup B)$: Minimum number of tokens from $S$ (and singleton base tokens $B$) whose concatenation yields $W$.
• $\mathrm{cover}(W, S)$: Maximum number of adjacent singleton pairs in $\Sigma$0 explained by $\Sigma$1, with non-overlapping assignments. The optimization objective (Tok–min) is to minimize $\Sigma$2, or equivalently (Tok–max), maximize $\Sigma$3, with $\Sigma$4 (Lim et al., 8 Jan 2025).

Adjacency Tokenization for Graphs:

For a graph $\Sigma$5 with adjacency matrix $\Sigma$6, node-level adjacency tokenization maps $\Sigma$7 to $\Sigma$8, where $\Sigma$9 is the indicator vector of $W\in\Sigma^+$0’s neighbors. This map is lossless up to node ordering, and admits efficient $W\in\Sigma^+$1 preprocessing (Bechler-Speicher et al., 21 May 2026).

2. Optimization Landscape and Computational Barriers

In the partition–cover paradigm, the core token selection problem is provably intractable:

• NP-Hardness: The decision version of Tok–min is NP-hard via a linear-time reduction from the classic Vertex Cover problem: covering all requisite adjacent pairs with a limited vocabulary is equivalent to selecting covering vertices in a hypergraph representation (Lim et al., 8 Jan 2025).
• Greedy Algorithms: Despite the NP-hardness, greedy algorithms achieve strong practical performance. GreedTok iteratively selects tokens with maximal marginal increase in covered adjacent pairs, updating coverage states at each step. The time complexity is $W\in\Sigma^+$2. No formal approximation ratio is proven for GreedTok, but empirical results indicate it approaches the (1–$W\in\Sigma^+$3) bound achieved by solving the relaxed weighted maximum coverage (GreedWMC), where overlap constraints are dropped (Lim et al., 8 Jan 2025).

Algorithm	Type	Coverage Guarantee
GreedTok	Greedy	Empirically near-optimal
GreedWMC	Greedy	$W\in\Sigma^+$4-approximation

In graph tokenization, the conversion between tokenization schemes (adjacency, spectral, and random-walk) poses major complexity-theoretic barriers for shallow neural architectures (Bechler-Speicher et al., 21 May 2026).

3. Expressivity, Depth Bounds, and Comparisons

Adjacency tokenization establishes a distinct expressivity profile relative to spectral and random-walk tokenizations:

• Losslessness: Node-level adjacency tokens fully specify the graph topology, up to node order.
• Depth Requirements: Several complexity-theoretic lower bounds hold for transformers:
  • Any transformer using adjacency tokenization requires depth $W\in\Sigma^+$5 to decide if a node is on a closed walk of length $W\in\Sigma^+$6 (Theorem 1).
  • Deciding graph connectivity from adjacency tokens requires $W\in\Sigma^+$7 layers (Theorem 2).
  • Truncating adjacency rows (to reduce per-token dimension) imposes a bottleneck for triangle counting: any transformer must satisfy $W\in\Sigma^+$8 (with residuals), or $W\in\Sigma^+$9 (without residuals) for hidden dim $S$0, $S$1 heads, precision $S$2 (Theorem 3) (Bechler-Speicher et al., 21 May 2026).

Transformational Incompatibility:

Depth-limited transformers cannot generally map between adjacency, spectral, and random-walk tokens:

• Adjacency $S$3 random-walk: requires $S$4 layers.
• Random-walk $S$5 adjacency: impossible for any finite walk length (due to Godsil–McKay switching).
• Adjacency $S$6 spectral: for global properties, requires $S$7 depth; spectral $S$8 adjacency (local properties) is extremely ill-conditioned (Bechler-Speicher et al., 21 May 2026).

Tokenization	Local Properties	Global Properties	Lossiness
Adjacency	Strong	Weak (deep)	Lossless
Spectral	Weak	Strong	Lossless
Random-Walk	Moderate	Moderate	Lossy

4. Illustrative Examples and Intuitive Coverage

Adjacency tokens enable direct access to 1-hop neighborhood structure but reveal limitations for higher-order tasks and inherent symmetries:

• Triangle ($S$9): Each node’s token codes neighbors, enabling a depth-1 transformer to confirm the clique.
• Cycle ($k$0): Adjacency tokens for diametrically opposed nodes coincide, reflecting symmetry and highlighting order dependence.
• Star: All leaf nodes receive identical tokens, so distinguishing between leaves and center demands further context or increased model depth.
• Truncated Adjacency: Projecting tokens via random projection reduces dimensionality but induces collisions, potentially rendering distinct subgraphs indistinguishable (Bechler-Speicher et al., 21 May 2026).

5. Empirical Evaluation and Comparative Performance

Comprehensive empirical studies demonstrate the comparative compression, expressivity, and predictive power of adjacency tokenization:

• Partition–cover (Text):
  • GreedTok achieves average token-per-word improvements of 13% vocabulary reduction relative to BPE for fixed compression targets, and 3–5% fewer tokens per word when using identical vocabulary sizes.
  • The overlap between top $k$1 GreedTok and BPE vocabularies is low, indicating the former’s tendency to favor different substrings.
  • In CJK Wikipedia, language-specific GreedTok vocabularies fused together nearly match per-language optima and exceed cross-language BPE (Lim et al., 8 Jan 2025).
• Graph Transformers:
  • Synthetic tasks confirm that connectivity and long-path prediction with adjacency tokens break down at small network depth, matching theoretical lower bounds.
  • On benchmarks:
  • Local tasks (Max Clique, Topological Ordering): adjacency tokens achieve highest F1/MAE.
  • Global tasks (molecular property prediction): spectral/random-walk dominate.
  • Mixed tasks: Combining tokenizations yields the best results, reflecting complementary strengths (Bechler-Speicher et al., 21 May 2026).

6. Limitations, Practical Guidelines, and Open Directions

Adjacency tokenization exhibits both unique advantages and critical pitfalls:

• Strengths: Most effective when immediate local structure or edge-level details are paramount, and for small-to-mid-size graphs or compact strings.
• Scalability Barrier: Input size and dimensionality scale linearly with $k$2, limiting practical use on large graphs or corpus slices.
• Symmetry/Order Sensitivity: Not permutation-equivariant; node (or character) ordering must be externally fixed.
• Global Reasoning Costs: Ω(log n)–depth models are necessary for connectivity, planarity, or high-order structural inference.

Practical Recommendations:

• Use adjacency tokenization for edge-sensitive, local, and moderately sized domains.
• For large graphs, truncated or mixed-token approaches can offer trade-offs, albeit with lossiness.
• Combining adjacency, spectral, and random-walk tokenizations can fuse local and global signals but increases model complexity.
• Open questions include the efficacy of edge-level (rather than node-level) adjacency tokens, training dynamics given the raw binary input structure, and mitigating depth/precision bottlenecks with architectural innovations (Bechler-Speicher et al., 21 May 2026).

7. Theoretical Significance and Future Prospects

Adjacency tokenization operationalizes the exposure of explicit pairwise structure, bridging combinatorial optimization with practical system design. In the partition–cover regime, it unifies compression, coverage, and approximate algorithms under a tractable framework with proven empirical superiority over BPE. In graph learning, it formalizes the trade-off between locality, expressivity, and complexity, revealing inherent limits for transformer architectures and illuminating the necessity of tailored, task-specific tokenization schemes. Future developments are likely to focus on scalable, hybrid tokenization methodologies and on deeper analytical understanding of training and generalization behaviors as induced by local adjacency cues (Lim et al., 8 Jan 2025, Bechler-Speicher et al., 21 May 2026).