Graph Language Models (GLMs)

• Graph Language Models (GLMs) are methods that integrate graph topology with textual representations to enable tasks such as node classification, link prediction, and graph reasoning.
• GLM architectures range from graph-biased transformers to GNN-LLM hybrids and LLM-only systems that serialize graphs into prompts, each offering distinct processing advantages.
• GLMs leverage training strategies like text-to-text supervision, graph-text alignment, and reasoning-path guidance to optimize performance across diverse graph-based benchmarks.

Graph LLMs (GLMs) are a family of methods that adapt LLMs to graph-structured data by combining graph topology with textual or tokenized representations. In current usage, the term spans several related designs: graph-biased transformers that process graphs, text, and interleaved graph-text inputs within a single architecture; hybrid systems that align Graph Neural Network (GNN) representations with LLM token spaces; and LLM-only systems that serialize graph structure into prompts or graph tokens. Across these variants, GLMs are used for node classification, link prediction, relation classification, graph question answering, summary generation, and graph-structured reasoning (Plenz et al., 2024, Shang et al., 2024, Yu et al., 2 Jan 2025).

1. Definitions and conceptual scope

A common formalization treats a GLM as a function

$f : (G,T) \mapsto A,$

where $G=(V,E,F^v,F^e)$ is a text-attributed graph, $T$ is a natural-language query, and $A$ is the generated answer token sequence. In one representative decomposition, a GLM consists of a graph encoder $M_g$, a projector $M_P$ that maps graph representations into the LLM embedding space, and an LLM $M_l$ that consumes graph tokens together with textual input (Petkar et al., 28 Aug 2025). In benchmark work on text-attributed graphs, related notation uses $\mathcal G=(\mathcal V,A,\{s_n\}_{n\in\mathcal V})$, where $A$ is the adjacency matrix and $s_n$ is the raw text attached to node $G=(V,E,F^v,F^e)$0 (Li et al., 2024).

The literature organizes GLMs along more than one axis. In the transformation-oriented taxonomy, methods fall into Graph2text and Graph2token paradigms: the former converts graphs into natural or domain-specific text, while the latter converts graphs into discrete or soft tokens for LLM consumption (Yu et al., 2 Jan 2025). In the functional-role taxonomy, GraphLLM methods are divided into LLM-as-Enhancer, LLM-as-Predictor, and LLM-as-Aligner. In enhancer designs, the LLM enriches node or text features before graph propagation; in predictor designs, the LLM directly outputs the task answer from a serialized graph prompt; in aligner designs, a GNN and an LLM are co-adapted in a shared or aligned space (Li et al., 2024).

A narrower architectural definition appears in work that introduces a transformer-based GLM as a pretrained LLM whose self-attention is modified to respect graph structure through relative positional biases and sparsity masks. Under that view, GLMs are neither simple graph linearizations nor late-fusion LM→GNN pipelines, but a single model class in which linguistic contextualization and graph-structured message passing occur inside the same transformer stack (Plenz et al., 2024). The coexistence of these definitions is characteristic of the field rather than anomalous: some works reserve “GLM” for graph-native transformer architectures, while others use it as an umbrella term for LLM–graph integration more broadly.

2. Architectural paradigms

One major line of work implements graph structure directly inside transformer attention. In the graph-biased transformer formulation, inputs are converted into an extended Levi graph, tokenized, and passed through a pretrained transformer whose self-attention is computed as

$G=(V,E,F^v,F^e)$1

where $G=(V,E,F^v,F^e)$2 is a learned relative-position bias derived from signed graph distance and $G=(V,E,F^v,F^e)$3 is a structural attention mask. This design allows the same model to process graphs only, text only, or interleaved graph-text inputs (Plenz et al., 2024).

A second paradigm uses a dedicated graph encoder and then injects graph information into an LLM by prefixes, projectors, or learned node tokens. GraphLLM is a representative end-to-end example: it combines a node-understanding module, a graph transformer for structure understanding, and graph-enhanced prefix tuning, while keeping the base LLM frozen and training only graph-related modules through sequence cross-entropy (Chai et al., 2023). GraphLAMA adopts a different hybrid design: a 3-layer Graph Transformer backbone maps $G=(V,E,F^v,F^e)$4-hop subgraphs into the LLM token space, hop encodings are added, Task-Invariant Gate and Task-Related Gate modules separate invariant and task-dependent channels, and node tokens are concatenated with raw text tokens and natural-language instructions before decoding (Chen et al., 11 Jun 2025).

A third paradigm leaves the transformer architecture unchanged and instead turns graph structure into a promptable language. InstructGLM expands the token vocabulary with one new token per graph node and describes multi-scale graph structure in natural language, so that node classification and link prediction become ordinary sequence-generation tasks under standard negative log-likelihood training (Ye et al., 2023). SDM-InstructGLM is even more explicit about the LLM-only design: it adds no graph-attention masks or adjacency-based convolutions, serializes sampled nodes and edges into natural-language-like fragments, and orders them using a similarity-degree-based biased random walk whose transition score is

$G=(V,E,F^v,F^e)$5

The resulting transition probability is normalized over the neighborhood of the current node, yielding a token-efficient subgraph prompt for large graphs (Lee et al., 2 May 2025).

A fourth paradigm emphasizes structural priors and graph construction. Scale-Free Graph-LLMs integrate graph generation and text embedding within a unified framework by using the scale-free property as a structural prior, approximating that prior with a cosine $G=(V,E,F^v,F^e)$6-nearest-neighbor graph, and then using a graph-based pseudo-labeler to improve LM fine-tuning under label scarcity (Lu et al., 21 Feb 2025). Related graph-aware language-model pretraining on large heterogeneous graph corpora uses a serial LLM→GNN fusion pipeline, link-prediction pretraining, and graph stitching across downstream applications (Xie et al., 2023).

3. Representation learning and optimization objectives

Despite their architectural diversity, GLMs are unified by a small set of recurring training strategies. The most common is autoregressive or text-to-text supervision. InstructGLM formulates graph tasks as sequence generation and optimizes the standard token-level negative log-likelihood over the answer sequence (Ye et al., 2023). SDM-InstructGLM uses the same generate-and-score principle on prompts consisting of an instruction prefix, serialized subgraph, question tokens, and answer tokens, with teacher forcing and no auxiliary losses (Lee et al., 2 May 2025).

Hybrid GLMs often add graph-text alignment and multi-task pretraining. GraphLAMA pretrains all components except the LLM on node matching, paper classification, and link prediction, then performs few-shot adaptation by updating only a small subset of parameters—approximately $G=(V,E,F^v,F^e)$7M parameters, or about $G=(V,E,F^v,F^e)$8 of the LLM—while freezing the 7B-parameter LLM and the full GNN encoder (Chen et al., 11 Jun 2025). UniGraphLM extends this alignment view to multi-domain and multi-task settings: a multi-scale GNN encoder is first pretrained on graph–text pairs using domain-aware contrastive learning (“DR-CLIP”), and only a projector is trained during curriculum alignment tuning, with per-domain weights adjusted online according to estimated difficulty (Chen et al., 12 May 2026).

Another recurring strategy is pseudo-labeling and iterative supervision transfer. Scale-Free Graph-LLMs train a GNN classifier on labeled nodes of a scale-free KNN graph, generate pseudo-labels for unlabeled nodes, fine-tune a pretrained LM on both labeled and pseudo-labeled texts, and then retrain a final GNN on refined LM embeddings (Lu et al., 21 Feb 2025). In benchmark taxonomies, this pattern appears more generally as LLM-as-Aligner, often in Expectation–Maximization-style or co-training loops between graph and language components (Li et al., 2024).

A distinct branch of GLM training uses reasoning-path supervision rather than only final-answer supervision. GUNDAM constructs “hard” reasoning paths from classical graph algorithms and trains a fine-tuned LLM with a dual objective

$G=(V,E,F^v,F^e)$9

where $T$0 is answer loss and $T$1 is reasoning-path loss. The paper also states an information-theoretic theorem: if the reasoning path $T$2 carries additional information about the correct answer $T$3 beyond the model’s direct response $T$4, then conditioning on $T$5 strictly reduces uncertainty about $T$6 (Ouyang et al., 2024).

4. Tasks, benchmarks, and empirical record

GLMs have been evaluated on both conventional graph-learning tasks and explicit graph-reasoning problems. On node classification benchmarks, InstructGLM reports best single-model accuracies of $T$7 on ogbn-arxiv, $T$8 on Cora, and $T$9 on PubMed, and under a $A$0 label ratio on PubMed it reports $A$1, compared with approximately $A$2 for the best GNN and approximately $A$3 for the best transformer (Ye et al., 2023). AugGLM, which preserves the original Flan-T5 architecture and augments inputs with topological retrieval, semantic retrieval, and GNN-based candidate pruning, reports $A$4 on Cora, $A$5 on PubMed, $A$6 on ogbn-arxiv, and $A$7 on an ogbn-products subset for Flan-T5-large (Xu et al., 2024).

Few-shot and cross-domain adaptation have become a separate evaluation regime. GraphLAMA reports $A$8 on 5-way 5-shot node classification averaged over Cora, Wiki-CS, and Products, with an absolute gain of about $A$9 over the best baseline, and reports inference latency of approximately $M_g$0 s per target versus $M_g$1–$M_g$2 s for in-context learning, giving $M_g$3 faster inference at 5-shot (Chen et al., 11 Jun 2025). UniGraphLM, evaluated on node-, edge-, and graph-level tasks, reports an average of $M_g$4 in multi-domain multi-task joint alignment versus $M_g$5 for the strongest baseline, and also reports strong single-dataset transfer across unseen datasets and task types (Chen et al., 12 May 2026).

On graph-reasoning benchmarks, GraphLLM reports a substantial average accuracy enhancement of $M_g$6 together with a context reduction of $M_g$7 across four fundamental reasoning tasks: substructure counting, maximum triplet sum, shortest path, and bipartite graph matching (Chai et al., 2023). GUNDAM, evaluated on NLGraph tasks such as connectivity, cycle detection, topological sort, shortest path, maximum flow, bipartite matching, Hamilton path, and two-layer GNN message passing, reports overall averages of $M_g$8 for GUNDAM-V and $M_g$9 for GUNDAM-L, compared with $M_P$0 for GPT-4 on the same suite (Ouyang et al., 2024).

The scope of GLM evaluation has also expanded beyond classification and reasoning. TinyGraphEstimator treats graph structure inference itself as the task: small instruct models under 5B parameters receive graphs serialized as edge lists or adjacency lists and output structural metadata in JSON. With LoRA fine-tuning, Llama-3B reduces normalized RMSE on nodes from $M_P$1 to $M_P$2, on edges from $M_P$3 to $M_P$4, on density from $M_P$5 to $M_P$6, on average clustering from $M_P$7 to $M_P$8, and on chromatic number from $M_P$9 to $M_l$0 (Podstawski, 9 Oct 2025). This suggests that the empirical domain of GLMs now includes not only text-attributed graphs and graph QA, but also direct estimation of graph-theoretic parameters from textual graph encodings.

5. Evaluation debates, bottlenecks, and mechanistic limitations

A central controversy concerns whether widely used benchmarks actually require graph–language fusion. GLBench finds that GraphLLM methods outperform traditional baselines in supervised settings, with LLM-as-enhancers showing the most robust performance, but it also reports that LLM-as-predictor methods are less effective and often suffer from uncontrollable output issues. The benchmark further states that no clear scaling laws exist for current GraphLLM methods (Li et al., 2024). These findings complicate simple narratives in which larger LLMs or more graph tokens automatically yield better graph learning.

A stronger critique is formulated through the CLEGR benchmark. That work argues that current GLM evaluations, largely based on repurposed node-level classification datasets, are insufficient for assessing multimodal reasoning because strong performance is often achievable with unimodal information alone. On the synthetic CLEGR benchmark, GLMs and soft-prompted LLM baselines both saturate at approximately $M_l$1+ accuracy on the Facts subset, but on the Reasoning subset all models collapse to approximately $M_l$2–$M_l$3 accuracy, with no meaningful gain from full GNN backbones. The same paper reports that soft-prompted LLM baselines perform on par with GLMs that incorporate a full GNN backbone, directly questioning the architectural necessity of graph encoders under current evaluations (Petkar et al., 28 Aug 2025).

Mechanistic analysis has exposed an additional limitation inside graph-token pipelines themselves. In representative GLMs such as LLaGA-style node-aligned tokens and TEA-GLM-style encoder-generated tokens, “graph sink tokens” emerge as activation-level outliers concentrated in a few hidden-state dimensions and biased toward early graph-token positions. However, pruning, repositioning, and swapping experiments show that these sink tokens are not the most important semantic or structural tokens for downstream prediction; in LLaGA, many sink slots are even $M_l$4 tokens. The reported conclusion is a decoupling between activation-level saliency and graph-semantic utility (Zhang et al., 2 Jun 2026).

Practical limitations recur across architectural families. LLM-only approaches face strict prompt-length budgets and must use neighborhood sampling, biased random walks, or retrieval to prevent context explosion (Ye et al., 2023, Lee et al., 2 May 2025). Few-shot in-context learning on graphs is criticized for fixed parameters and long contexts, motivating extra adaptation stages such as GraphLAMA (Chen et al., 11 Jun 2025). Retrieval-augmented node classifiers such as AugGLM inherit manual prompt-template design, the need for a pretrained auxiliary GNN, and added retrieval cost at inference time (Xu et al., 2024). Taken together, these results suggest that current GLMs are constrained as much by representation and evaluation design as by raw model capacity.

6. Research directions

Several trajectories are now visible. One is stronger and more general alignment. UniGraphLM argues that existing GNN-encoder-based GLMs largely overlook the problem of aligning graph representations across domains and tasks with the LLM token space, and it responds with multi-domain graph–text pretraining plus adaptive curriculum alignment (Chen et al., 12 May 2026). A plausible implication is that future GLMs will treat alignment not as a final projector layer, but as a first-class multi-domain learning problem.

A second direction is more explicit topology-aware internalization inside transformers. Mechanistic analysis recommends structure-aware positional encodings, attention biases that favor linked or $M_l$5-hop-related nodes, stronger semantic–structural alignment objectives, and specialized “gateway” tokens that aggregate and redistribute graph structure, rather than relying on vanilla graph-token placement (Zhang et al., 2 Jun 2026). This suggests movement away from purely content-based graph tokens toward architectures that preserve topology throughout the transformer computation.

A third direction is better evaluation. The CLEGR results motivate benchmarks that require joint reasoning over language and structure at controlled levels of compositional complexity, including temporal graphs, higher-order queries, and dynamic structure (Petkar et al., 28 Aug 2025). GLBench reaches a related conclusion from a different angle: unified experimental protocols, broader task coverage, and efficiency-aware comparison are prerequisites for credible progress claims (Li et al., 2024).

A fourth direction is scalability under constrained context and annotation budgets. GraphLAMA focuses on efficient adaptation with only a few labeled examples and future extensions such as dynamic $M_l$6-hop selection and tighter GNN–LLM fusion (Chen et al., 11 Jun 2025). TinyGraphEstimator points toward compact models, parameter-efficient LoRA adaptation, and alternative encodings such as mini-DSLs, hierarchical chunking, and graph grammars (Podstawski, 9 Oct 2025). Survey work broadens these concerns to fairness, dynamic graphs, prompt compression, and graph-native foundation models (Yu et al., 2 Jan 2025, Shang et al., 2024).

The resulting picture is not that of a single settled model class, but of a rapidly diversifying research program. GLMs now include graph-biased transformers, graph-token aligners, GNN–LLM hybrids, prompt-based graph reasoners, and lightweight structure estimators. Their common ambition is stable: to make graph structure as tractable to language-model machinery as text already is. Whether that ambition is best realized through Graph2text, Graph2token, tighter graph-aware attention, or new evaluation regimes remains an open question, and current evidence indicates that future progress will depend at least as much on representation fidelity and benchmark validity as on model scale alone.