---
title: 'Adjacency Tokenization: Methods & Trade-offs'
url: https://www.emergentmind.com/topics/adjacency-tokenization
type: topic
---

# Adjacency Tokenization: Methods & Trade-offs

Adjacency tokenization is a family of encoding methods that directly expose local adjacency information to models, typically by mapping discrete objects (such as characters in strings or nodes in graphs) to tokens representing explicit pairwise or neighborhood relationships. In Natural Language Processing, adjacency-driven approaches such as the partition–cover tokenization framework formalize the vocabulary learning problem in terms of covering adjacent symbol pairs within corpora. In the context of graph learning, adjacency tokenization encodes each node’s direct connections as tokens (often as rows of the adjacency matrix), making local structure directly accessible to models such as transformers. This class of tokenizations offers strong locality guarantees, is lossless up to node ordering, and provides a rigorous connection to combinatorial optimization, but is limited by scalability and depth requirements for global tasks. Adjacency tokenization stands in stark methodological contrast to spectral and random-walk tokenizations and exposes fundamental trade-offs in expressivity and efficiency for both language and graph models [2501.06246] [2605.22471].

## 1. Formal Definitions

Adjacency tokenization manifests in two premier domains: string tokenization (partition–cover) and graph tokenization.

**Partition–Cover for Strings:**  
Given a corpus $C=(W, \mathrm{count})$ over finite alphabet $\Sigma$, with each word $W\in\Sigma^+$, adjacency tokenization seeks a vocabulary $S$ of size $k$ to minimize the total number of tokens needed to represent the corpus, subject to a constraint that tokens explain adjacent character pairs:
- $\mathrm{partition}(W, S\cup B)$: Minimum number of tokens from $S$ (and singleton base tokens $B$) whose concatenation yields $W$.
- $\mathrm{cover}(W, S)$: Maximum number of adjacent singleton pairs in $W$ explained by $S$, with non-overlapping assignments.
The optimization objective (Tok–min) is to minimize $\sum_W \mathrm{count}(W) \cdot \mathrm{partition}(W, S\cup B)$, or equivalently (Tok–max), maximize $\sum_W \mathrm{count}(W) \cdot \mathrm{cover}(W, S)$, with $|S|\leq k$ [2501.06246].

**Adjacency Tokenization for Graphs:**  
For a graph $G=(V,E)$ with adjacency matrix $A\in\{0,1\}^{n\times n}$, node-level adjacency tokenization maps $G$ to $P_{\mathrm{Adj}}(G) = \bigl[ x_1; x_2; \ldots; x_n\bigr]$, where $x_v=A_{v,:}\in\{0,1\}^n$ is the indicator vector of $v$’s neighbors. This map is lossless up to node ordering, and admits efficient $O(|E|)$ preprocessing [2605.22471].

## 2. Optimization Landscape and Computational Barriers

In the partition–cover paradigm, the core token selection problem is provably intractable:

- **NP-Hardness:** The decision version of Tok–min is NP-hard via a linear-time reduction from the classic Vertex Cover problem: covering all requisite adjacent pairs with a limited vocabulary is equivalent to selecting covering vertices in a hypergraph representation [2501.06246].
- **Greedy Algorithms:** Despite the NP-hardness, greedy algorithms achieve strong practical performance. GreedTok iteratively selects tokens with maximal marginal increase in covered adjacent pairs, updating coverage states at each step. The time complexity is $O(|T|\cdot k \cdot \sum_W |W|)$. No formal approximation ratio is proven for GreedTok, but empirical results indicate it approaches the (1–$1/e$) bound achieved by solving the relaxed weighted maximum coverage (GreedWMC), where overlap constraints are dropped [2501.06246].

| Algorithm   | Type     | Coverage Guarantee           |
|-------------|----------|-----------------------------|
| GreedTok    | Greedy   | Empirically near-optimal    |
| GreedWMC    | Greedy   | $(1-1/e)$-approximation     |

In graph tokenization, the conversion between tokenization schemes (adjacency, spectral, and random-walk) poses major complexity-theoretic barriers for shallow neural architectures [2605.22471].

## 3. Expressivity, Depth Bounds, and Comparisons

Adjacency tokenization establishes a distinct expressivity profile relative to spectral and random-walk tokenizations:

- **Losslessness:** Node-level adjacency tokens fully specify the graph topology, up to node order.
- **Depth Requirements:** Several complexity-theoretic lower bounds hold for transformers:
    - Any transformer using adjacency tokenization requires depth $L=\Omega(\log k)$ to decide if a node is on a closed walk of length $k$ (Theorem 1).
    - Deciding graph connectivity from adjacency tokens requires $L=\Omega(\log n)$ layers (Theorem 2).
    - Truncating adjacency rows (to reduce per-token dimension) imposes a bottleneck for triangle counting: any transformer must satisfy $mpHL = \Omega(n)$ (with residuals), or $mpH=\Omega(n)$ (without residuals) for hidden dim $m$, $H$ heads, precision $p$ (Theorem 3) [2605.22471].

**Transformational Incompatibility:**  
Depth-limited transformers cannot generally map between adjacency, spectral, and random-walk tokens:
- Adjacency $\to$ random-walk: requires $\Omega(\log k)$ layers.
- Random-walk $\to$ adjacency: impossible for any finite walk length (due to Godsil–McKay switching).
- Adjacency $\to$ spectral: for global properties, requires $\Omega(\log n)$ depth; spectral $\to$ adjacency (local properties) is extremely ill-conditioned [2605.22471].

| Tokenization | Local Properties | Global Properties | Lossiness    |
|--------------|-----------------|------------------|--------------|
| Adjacency    | Strong          | Weak (deep)      | Lossless     |
| Spectral     | Weak            | Strong           | Lossless     |
| Random-Walk  | Moderate        | Moderate         | Lossy        |

## 4. Illustrative Examples and Intuitive Coverage

Adjacency tokens enable direct access to 1-hop neighborhood structure but reveal limitations for higher-order tasks and inherent symmetries:
- **Triangle ($K_3$):** Each node’s token codes neighbors, enabling a depth-1 transformer to confirm the clique.
- **Cycle ($C_4$):** Adjacency tokens for diametrically opposed nodes coincide, reflecting symmetry and highlighting order dependence.
- **Star:** All leaf nodes receive identical tokens, so distinguishing between leaves and center demands further context or increased model depth.
- **Truncated Adjacency:** Projecting tokens via random projection reduces dimensionality but induces collisions, potentially rendering distinct subgraphs indistinguishable [2605.22471].

## 5. Empirical Evaluation and Comparative Performance

Comprehensive empirical studies demonstrate the comparative compression, expressivity, and predictive power of adjacency tokenization:

- **Partition–cover (Text):**
    - GreedTok achieves average token-per-word improvements of 13% vocabulary reduction relative to BPE for fixed compression targets, and 3–5% fewer tokens per word when using identical vocabulary sizes.
    - The overlap between top $k$ GreedTok and BPE vocabularies is low, indicating the former’s tendency to favor different substrings.
    - In CJK Wikipedia, language-specific GreedTok vocabularies fused together nearly match per-language optima and exceed cross-language BPE [2501.06246].

- **Graph Transformers:**
    - Synthetic tasks confirm that connectivity and long-path prediction with adjacency tokens break down at small network depth, matching theoretical lower bounds.
    - On benchmarks:
        - **Local tasks** (Max Clique, Topological Ordering): adjacency tokens achieve highest F1/MAE.
        - **Global tasks** (molecular property prediction): spectral/random-walk dominate.
        - **Mixed tasks:** Combining tokenizations yields the best results, reflecting complementary strengths [2605.22471].

## 6. Limitations, Practical Guidelines, and Open Directions

Adjacency tokenization exhibits both unique advantages and critical pitfalls:
- **Strengths:** Most effective when immediate local structure or edge-level details are paramount, and for small-to-mid-size graphs or compact strings.
- **Scalability Barrier:** Input size and dimensionality scale linearly with $n$, limiting practical use on large graphs or corpus slices.
- **Symmetry/Order Sensitivity:** Not permutation-equivariant; node (or character) ordering must be externally fixed.
- **Global Reasoning Costs:** Ω(log n)–depth models are necessary for connectivity, planarity, or high-order structural inference.

**Practical Recommendations:**
- Use adjacency tokenization for edge-sensitive, local, and moderately sized domains.
- For large graphs, truncated or mixed-token approaches can offer trade-offs, albeit with lossiness.
- Combining adjacency, spectral, and random-walk tokenizations can fuse local and global signals but increases model complexity.
- Open questions include the efficacy of edge-level (rather than node-level) adjacency tokens, training dynamics given the raw binary input structure, and mitigating depth/precision bottlenecks with architectural innovations [2605.22471].

## 7. Theoretical Significance and Future Prospects

Adjacency tokenization operationalizes the exposure of explicit pairwise structure, bridging combinatorial optimization with practical system design. In the partition–cover regime, it unifies compression, coverage, and approximate algorithms under a tractable framework with proven empirical superiority over BPE. In graph learning, it formalizes the trade-off between locality, expressivity, and complexity, revealing inherent limits for transformer architectures and illuminating the necessity of tailored, task-specific tokenization schemes. Future developments are likely to focus on scalable, hybrid tokenization methodologies and on deeper analytical understanding of training and generalization behaviors as induced by local adjacency cues [2501.06246] [2605.22471].

Source: https://www.emergentmind.com/topics/adjacency-tokenization