Graph Vocabulary Learning

• Graph vocabulary learning is the extraction and utilization of recurring subgraph patterns to create a finite set of tokens representing complex graph structures.
• It enables efficient graph compression, summarization, and reasoning by mapping structural components and semantic attributes into transferable units.
• The approach underpins diverse applications including foundation models, scene graph generation, knowledge graphs, and code analysis.

Graph vocabulary learning refers to the process of defining, constructing, and utilizing a discrete set of recurring substructures, patterns, or symbolic units—termed a "vocabulary"—within graphs, for the purpose of compressing, summarizing, reasoning about, or transferring knowledge across diverse graph-structured data. This approach leverages the analogy to discrete vocabularies in language and vision, serving as the foundation for a range of advances in foundation models, open-vocabulary prediction, structural summarization, and compositional transfer across domains and tasks.

1. Formal Definitions and Paradigms

Graph vocabulary learning seeks explicit or implicit atomic representations of patterns in graphs, mapping complex structures into a finite (but potentially open or extensible) set of tokens, codes, or building blocks.

Key paradigms include:

• Subgraph vocabulary: A small set of canonical subgraph types (e.g., stars, cliques, chains, treelets, cycles, graphlets) that serve as basis elements for reconstructing or summarizing a graph (Koutra et al., 2014, Amouzouvi et al., 7 May 2026, Sun et al., 5 Feb 2025).
• Computation tree vocabulary: Encoding all local L-hop computation trees (as induced via message passing in GNNs) and discretizing them as transferable tokens (Wang et al., 2024).
• Structural vocabulary derived from geometry: Decomposing graphs into trees and cycles, then embedding these in Riemannian manifolds suited to their geometry (Sun et al., 5 Feb 2025).
• Language-based vocabulary: Assigning readable, language-grounded IDs to nodes, subgraphs, or attributes, enabling direct alignment with LLMs (Zhu et al., 5 Mar 2025).
• Relation lattice vocabulary: Enumerating the open universe of predicates/relations in scene graphs and explicitly modeling semantic dependencies (entailment, synonymy, contradiction) among them (Hosseini et al., 24 Apr 2026).

This field systematically addresses the lack of a universal "token" set for graphs, which lack the fixed alphabet of language or the gridded pixel basis of images.

2. Structural and Semantic Vocabularies: Construction and Characteristics

Graph vocabularies can be grouped into structural and semantic types.

Structural vocabularies enumerate subgraph patterns:

Method	Vocabulary Atoms (Examples)	Learning/Extraction Procedure
VoG (Koutra et al., 2014)	full cliques, near-cliques, bipartite cores, near-bipartite cores, stars, chains	MDL-based selection over candidate subgraphs
Graphlet-based KGFMs (Amouzouvi et al., 7 May 2026)	open/closed 2- and 3-paths, star graphlets	SPARQL pattern mining, adjacency pattern matching
RiemannGFM (Sun et al., 5 Feb 2025)	all small trees $T_k$, cycles $C_m$	Sampling and geometry-driven embedding
GFT (Wang et al., 2024)	computation trees (unfolded message-passing trees)	Neural encoding + vector quantization

Semantic/language-based vocabularies focus on tokens, predicates, or node labels aligned with language or task-specific semantics:

Method	Vocabulary Atoms	Construction/Alignment
PromptGFM (Zhu et al., 5 Mar 2025)	node-IDs tokenized as readable language sequences	Prompt-based GNN, unsupervised + instruction tuning
RAHP (Liu et al., 2024), OvSGTR (Chen et al., 26 May 2025)	subject/object/predicate triplets, region-aware natural language prompts	LLM-aided clustering, dynamic prompt selection
ReLIC-SGG (Hosseini et al., 24 Apr 2026)	semantic relation lattice (phrases)	Lattice built over predicate phrases, relation dependencies from CLIP embeddings and ontology

Some architectures such as RAHP combine entity clustering and LLM-based prompt generation to manage the combinatorial explosion of possible triplets, while others (e.g., computation-tree vocabularies) leverage the inherent recursive structure of message passing in GNNs.

3. Learning Algorithms and Objectives

Learning graph vocabularies typically involves a two-stage process: mining or constructing the vocabulary, followed by end-to-end optimization to represent, reconstruct, or classify graph data.

MDL-based compression (VoG (Koutra et al., 2014)):

• Subgraphs are chosen if they lower the total description length $L(G, M) = L(M) + L(E)$ via the Minimum Description Length principle.
• $\Delta L(C)$ is used to select structures for the summary.

Vector-quantized tree encoding (GFT (Wang et al., 2024)):

• Embeddings of computation trees are quantized into a discrete codebook via

$$L_\mathrm{pretrain} = L_\mathrm{tree} + \frac{1}{m} \sum_i \left\| \mathrm{sg}[e_i] - z_i \right\|_2^2 + \beta_1 \frac{1}{m} \sum_i \left\| e_i - \mathrm{sg}[z_i] \right\|_2^2$$

• Multi-aspect reconstruction losses $L_\mathrm{feat}, L_\mathrm{sem}, L_\mathrm{topo}$ enforce feature, semantic, and topological consistency.

Geometry-aware structural vocabulary (RiemannGFM (Sun et al., 5 Feb 2025)):

• Self-supervised contrastive loss between geometric views:

$$\mathcal J_0 = \mathcal J_{H \to S} + \mathcal J_{S \to H}$$

aligns hyperbolic and spherical tangent-space encodings of sampled tree/cycle instances.

Open-vocabulary semantic lattices (ReLIC-SGG (Hosseini et al., 24 Apr 2026)):

• The positive-unlabeled (PU) objective combines

$$\mathcal{L}_{\rm PU} = \mathcal{R}_p^+ + \lambda_u \mathcal{R}_u$$

with consistency terms over semantic similarity, entailment, and contradiction edges among relations.

Language-based graph IDs (PromptGFM (Zhu et al., 5 Mar 2025)):

• Vocabulary is acquired via prompting LLMs to perform iterative aggregation and summarization, then fine-tuned by cross-entropy on multi-task instruction-answer pairs.

4. Transferability, Generalization, and Cross-Domain Impact

A core motivation behind graph vocabulary learning is transferability: the ability to reuse learned atomic units across disparate domains, tasks, or label spaces.

• Computation-tree vocabularies (GFT): The theoretical generalization bounds (Theorem 2.3 in (Wang et al., 2024)) and empirical results show that encoding instances as tokens in the same tree-vocabulary provides robust cross-task and cross-domain transfer for node classification, link prediction, and graph-level prediction, with minimal negative transfer.
• Structural vocabularies (RiemannGFM): The embedding of universal atoms (trees/cycles) in principled manifolds enables foundation models trained solely on structure to generalize across graphs without attribute information or domain-specific vocabulary (Sun et al., 5 Feb 2025).
• Language-driven vocabularies: Instruction tuning using consistent node IDs (PromptGFM) supports rapid adaptation to unseen graphs and tasks, bridging structured data and language (Zhu et al., 5 Mar 2025).
• Semantic lattices: Handling partial labels in open-vocabulary SGG is improved by leveraging predicate dependencies, increasing rare/unseen predicate accuracy while maintaining semantic compactness (Hosseini et al., 24 Apr 2026).
• Graphlets in KGFMs: Adding closed and open graphlet patterns to the structural vocabulary strictly improves inductive and transductive link prediction across 51 diverse knowledge graphs, with especially pronounced benefits in sparse regimes (Amouzouvi et al., 7 May 2026).

5. Applications: Foundation Models, Scene Graph Generation, Knowledge Graphs, and Code

Graph Foundation Models (GFMs)

• GFT: Utilizes computation-tree vocabularies to enable few-shot and zero-shot task transfer (Wang et al., 2024).
• RiemannGFM: Uses the universal vocabulary of trees/cycles, with transferability tested on node/edge classification in citation, code, and infrastructure graphs (Sun et al., 5 Feb 2025).
• PromptGFM: Provides unified node/edge vocabulary for text-attributed graphs, enabling instruction-based multi-task transfer (Zhu et al., 5 Mar 2025).

Scene Graph Generation (SGG)

• OvSGTR, RAHP, PGSG: Open-vocabulary SGG relies on extracting, augmenting, or dynamically selecting large predicate/entity vocabularies grounded in natural language and region-aware descriptors. Weak supervision pipelines and LLMs are used to synthesize and cluster vocabulary entries (Chen et al., 26 May 2025, Liu et al., 2024, Li et al., 2024).
• ReLIC-SGG: Imposes a structured semantic lattice over open-predicate vocabulary, improving rare predicate discovery and mitigating label incompleteness (Hosseini et al., 24 Apr 2026).

Knowledge Graphs

• Graphlet-based vocabulary (KGFMs): Mining fixed-size graphlets—open/closed k-paths, stars, etc.—yields transferable tokens suitable for foundational reasoning about unseen KGs (Amouzouvi et al., 7 May 2026).
• Application to link prediction, zero-shot entity/relation transfer, and robustness in sparse KGs.

Source Code Modeling

• Graph-Structured Cache (GSC): Treats each unique word in code as a graph node, enabling scalable open-vocabulary prediction in variable naming and code completion (Cvitkovic et al., 2018).

Graph Summarization and Compression

• VoG: The MDL-driven selection of a small interpretable set of subgraph types yields highly compressed and human-readable graph summaries (Koutra et al., 2014).

6. Limitations, Challenges, and Extensions

While graph vocabulary learning provides significant advances, several open issues persist:

• Curse of size: The vocabulary size may grow combinatorially with k in k-node subgraph vocabularies, necessitating clustering, abstraction (super-entities), or dynamic selection (Liu et al., 2024).
• Model-agnostic integration: Certain vocabularies (e.g., graphlets) are designed for easy adaptation into various architectures but may not capture domain-specific semantics (Amouzouvi et al., 7 May 2026).
• Balancing expressiveness and compactness: Overly fine-grained vocabularies may overfit or fail to generalize; underspecified ones miss salient patterns (Hosseini et al., 24 Apr 2026).
• Open-vocabulary alignment: In language-grounded SGG, aligning visual and text vocabularies requires sophisticated prompt, distillation, or lattice strategies; catastrophic forgetting remains a risk (Chen et al., 26 May 2025).
• Partial annotation: Large vocabularies facing incomplete labels benefit from positive-unlabeled approaches and semantic constraint (entailment, contradiction) regularization (Hosseini et al., 24 Apr 2026).

Potential extensions include scaling vocabularies to millions of classes (scene graphs), exploiting richer ontologies in knowledge graphs, and integrating multi-modal foundation models (text/graph/vision) via unified vocabularies (Chen et al., 26 May 2025).

7. Summary Table: Representative Methods and Their Graph Vocabularies

Reference	Domain	Vocabulary Type	Selection/Training Principle
VoG (Koutra et al., 2014)	General graphs	cliques, bipartite cores, stars, chains	Minimum Description Length (MDL)
GFT (Wang et al., 2024)	GNNs/foundation models	computation trees	Vector quantization, multi-objective reconstruction
RiemannGFM (Sun et al., 5 Feb 2025)	Foundation models	trees, cycles (structural atoms)	Riemannian geometry, contrastive loss
KGFMs (Amouzouvi et al., 7 May 2026)	Knowledge graphs	open/closed k-paths, stars	SPARQL pattern mining
OvSGTR/PGSG/RAHP (Chen et al., 26 May 2025, Li et al., 2024, Liu et al., 2024)	Scene graph generation	semantic triplets, LLM-driven region prompts	Weak supervision, entity clustering, dynamic selection
ReLIC-SGG (Hosseini et al., 24 Apr 2026)	SGG/open predicate prediction	semantic relation lattice	PU training, entailment/synonym/contradict mapping
PromptGFM (Zhu et al., 5 Mar 2025)	Text-attributed graphs	language-based node IDs	Prompted graph aggregation, instruction tuning
GSC (Cvitkovic et al., 2018)	Source code (AST)	unique identifier word nodes	Graph expansion, message passing