NT-LLM: Graph-Aware Tokenization for LLMs

• NT-LLM is a framework that equips large language models with learnable node position embeddings to capture both local and global graph topology.
• The method employs a greedy anchor node selection and relative distance encoding to efficiently summarize graph structure for integration with LLMs.
• Empirical results show significant gains in tasks like node classification and link prediction using minimal parameter updates during downstream adaptation.

NT-LLM refers to the "Node Tokenizer for LLMs," a framework designed to enable LLMs, which are natively sequential, to efficiently encode and reason over graph-structured data by introducing structured node position embeddings anchored in graph topology (Ji et al., 2024).

1. Motivation and Background

LLMs excel on sequential text, but are fundamentally ill-suited for graph input due to their lack of native structure-aware mechanisms. Two approaches have previously dominated attempts to adapt LLMs for graph learning tasks:

• Chain-of-Tasks (GNN + LLM): A specialized Graph Neural Network (GNN) generates node/edge embeddings reflecting structural information, which are then provided as prefix tokens to an LLM for downstream processing. This leverages the reasoning ability of LLMs and GNNs' structural sensitivity, but introduces a performance bottleneck at the GNN and incurs nontrivial engineering overhead when scaling GNNs to LLM-sized parameters.
• Graph-to-Text Conversion: The graph structure is linearized into a (possibly very long) natural-language text description (e.g., sequences of neighbors, paths), which the LLM then processes. This allows LLMs to be used as-is, but typically leads to loss of global topological relationships, and the resulting lengthy prompts can be brittle.

NT-LLM addresses these deficiencies by equipping LLMs with an explicit "topological view" of the graph via compact, learnable node position embeddings, directly amenable to attention mechanisms.

2. Node Tokenizer Construction

NT-LLM's key innovation is a modular node tokenizer, constructed in three stages:

2.1 Anchor Node Selection

A greedy algorithm identifies a covering set of anchor nodes $A$ under constraints of a coverage radius $c$ and a coverage ratio $CR$. Each anchor is selected to maximize the number of currently uncovered nodes within $c$ hops, until at least $CR \cdot |V|$ nodes are covered, where $V$ is the set of all nodes:

$$\text{Algorithm:} \quad \begin{align*} &\text{Input:}\ G=(V,E),\ c,\ CR \ &\text{Initialize:}\ A\gets\emptyset,\ \text{Covered}\gets\emptyset \ &\text{Precompute}\ N_c(v)\ \forall v\in V \ &\text{While}\ |Covered| < CR \cdot |V|: \ &\qquad \text{For}\ v\notin A,\ \text{compute}\ gain(v) = |N_c(v) \setminus Covered| \ &\qquad \text{anchor} \gets \arg\max_v gain(v) \ &\qquad \text{If } gain(anchor)=0,\ \text{break} \ &\qquad A \gets A \cup \{anchor\},\ Covered \gets Covered \cup N_c(anchor) \ \end{align*}$$

This ensures $|A| \ll |V|$ in practice, offering a scalable summary of structure.

2.2 Relative Distance Encoding

Each node $v$ is encoded as a $K$-dimensional vector (where $c$0):

$c$1

where $c$2 denotes the shortest-path (hop) distance from $c$3 to anchor $c$4. This vector jointly expresses the node's local and global position within the graph. For downstream use, the topology-aware pairwise node distance can be upper-bounded via

$c$5

with a bounded error (see Lemma 1 in (Ji et al., 2024)) when coverage is sufficient.

2.3 Positional Embedding Pretraining

Because discrete distance vectors are poorly matched to the continuous, Euclidean geometry of transformer embeddings, NT-LLM learns a mapping $c$6 such that for node pairs $c$7, Euclidean distances between $c$8 and $c$9 preserve the rank-ordering of graph-space distances. The objective is a binary cross-entropy loss over quadruples:

$CR$0

where $CR$1 and $CR$2. The mapping enforces that $CR$3 reflects ordering of graph distances.

3. LLM Integration and Architectural Adaptation

NT-LLM interfaces with the LLM via structured embedding streams:

• Graph Embeddings $CR$4: Rank-preserving positional embeddings for relevant nodes.
• Textual Embeddings $CR$5: Node and edge attribute embeddings, typically using SentenceBERT or equivalent.
• Prompt Embedding $CR$6: Standard LLM text prompt embedding.

A soft prompt adapter $CR$7 (shallow MLP) transforms $CR$8 into a "fake token" sequence $CR$9 compatible with the LLM's embedding space. Inputs to the frozen LLM are concatenated as $c$0.

During adaptation, only the adapter $c$1 and low-rank (LoRA) updates to select LLM matrices are trained, keeping the underlying large model weights unchanged.

4. Task-Specific Tuning and Training Procedure

NT-LLM uses a two-stage training protocol:

• (1) Pretraining $c$2: Minimizes the rank-preserving BCE loss over pairs sampled from the graph, typically employing a 3-layer MLP, with hyperparameters $c$3.
• (2) Downstream Adaptation: For tasks such as node classification, link prediction, and graph property prediction—standard losses (cross-entropy, margin ranking) are used, with prompt tuning and LoRA applied (e.g., rank $c$4, scaling $c$5). Optimizer is AdamW.

The modularity allows all graph, textual, and prompt streams to be accommodated efficiently. Downstream tuning updates $c$61–2 million parameters, several orders of magnitude fewer than full GNN–LLM hybridization.

5. Empirical Results and Quantitative Analysis

NT-LLM demonstrates substantial gains on diverse graph learning benchmarks:

Task	Dataset	Best Baseline	NT-LLM Improvement
Node Classification	Cora	LLM (Prompt)	+19.93%
Node Classification	OGBN-arxiv	LLM (Prompt)	see main text
Link Prediction	OGBL-ddi	LLM (Prompt)	+74.47%
Graph Property Prediction	OGBG-molhiv	LLM (Prompt)	ROC-AUC 0.8045 (vs 0.7529)
Structured QA/Explanation	ExplaGraphs	LLM (Prompt)	see main text

Main LLM baselines include zero-shot, prompt tuning, and LoRA-only adaptation. Further qualitative analysis shows that after $c$7-pretraining, same-class nodes in Cora cluster spatially tighter, and the greedy anchor selection provides superior graph coverage compared to degree/PageRank heuristics.

6. Limitations, Scalability, and Applicability

NT-LLM's efficacy is contingent on adequate anchor coverage: if coverage radius $c$8 or ratio $c$9 are set too low, some nodes remain distant from all anchors, increasing the approximation error (guaranteed to be $CR \cdot |V|$0 with probability at least $CR \cdot |V|$1). Without pretraining the mapping $CR \cdot |V|$2, the embeddings can distort, leading to performance drops.

Computational costs are dominated by anchor selection ($CR \cdot |V|$3) and positional embedding pretraining (sampling $CR \cdot |V|$4 node pairs for loss computation), but both are tractable with subsampling and the limited number of anchors. Downstream adaptation is highly parameter and memory efficient, since only adapters and prompt layers are tuned.

A plausible implication is that NT-LLM enables LLM-based architectures to process graph data at scale without the duplicative resource consumption of full-graph neural architectures, providing a flexible, structure-sensitive interface for both graph and hybrid graph-text problems.

7. Significance and Future Directions

NT-LLM provides a principled and scalable mechanism to unify graph structure with LLM reasoning, outperforming both standalone LLMs and GNN baselines across node, edge, and graph-level tasks (Ji et al., 2024). It achieves this with dramatically less engineering overhead than GNN–LLM systems or fully customized architectures.

Open directions include optimizing anchor selection heuristics for even larger graphs, joint graph-text co-training, and integration into prompt-based multi-modal LLMs. Extending node tokenizers to encode temporal or weighted relationships may broaden applicability, particularly in domains (social networks, biological interactomes) where topology is nontrivial.

NT-LLM thus stands as a representative advance in equipping LLMs with the ability to ingest and reason over nonsequential, relational data structures endemic to many scientific and real-world applications.