GraphMERT: Efficient and Scalable Distillation of Reliable Knowledge Graphs from Unstructured Data (2510.09580v1)

Abstract: Researchers have pursued neurosymbolic AI applications for nearly three decades because symbolic components provide abstraction while neural components provide generalization. Thus, a marriage of the two components can lead to rapid advancements in AI. Yet, the field has not realized this promise since most neurosymbolic AI frameworks fail to scale. In addition, the implicit representations and approximate reasoning of neural approaches limit interpretability and trust. Knowledge graphs (KGs), a gold-standard representation of explicit semantic knowledge, can address the symbolic side. However, automatically deriving reliable KGs from text corpora has remained an open problem. We address these challenges by introducing GraphMERT, a tiny graphical encoder-only model that distills high-quality KGs from unstructured text corpora and its own internal representations. GraphMERT and its equivalent KG form a modular neurosymbolic stack: neural learning of abstractions; symbolic KGs for verifiable reasoning. GraphMERT + KG is the first efficient and scalable neurosymbolic model to achieve state-of-the-art benchmark accuracy along with superior symbolic representations relative to baselines. Concretely, we target reliable domain-specific KGs that are both (1) factual (with provenance) and (2) valid (ontology-consistent relations with domain-appropriate semantics). When a LLM, e.g., Qwen3-32B, generates domain-specific KGs, it falls short on reliability due to prompt sensitivity, shallow domain expertise, and hallucinated relations. On text obtained from PubMed papers on diabetes, our 80M-parameter GraphMERT yields a KG with a 69.8% FActScore; a 32B-parameter baseline LLM yields a KG that achieves only 40.2% FActScore. The GraphMERT KG also attains a higher ValidityScore of 68.8%, versus 43.0% for the LLM baseline.

• The paper introduces GraphMERT, an encoder-based framework that efficiently extracts reliable and ontology-consistent knowledge graphs from unstructured text.
• The methodology integrates Hierarchical Graph Attention Networks with a transformer architecture to fuse syntactic and semantic information for robust dual-space learning.
• The paper demonstrates significant performance gains on medical datasets, achieving 69.8% FActScore and 68.8% ValidityScore, outperforming traditional LLM approaches.

GraphMERT: Efficient and Scalable Distillation of Reliable Knowledge Graphs from Unstructured Data

Introduction

The paper presents GraphMERT, a novel framework designed to efficiently and scalably distill reliable Knowledge Graphs (KGs) from unstructured data. The framework aims to overcome the challenges faced by conventional neurosymbolic AI systems, which struggle to scale and maintain interpretability, by effectively integrating neural and symbolic approaches to create a verifiable, reliable domain-specific KG.

Overview of GraphMERT Framework

GraphMERT leverages a small graphical encoder-only model (80M parameters) to translate its internal representations and high-quality text corpora into a structured KG. The system targets domain-specific KGs that are factual, ontology-consistent, and suitable for high-stakes environments, such as medicine. Traditional LLMs generate domain KGs that are unreliable due to prompt sensitivity and limited domain specificity. In contrast, GraphMERT demonstrates high factuality and validity with significantly superior scores compared to its LLM counterparts.

GraphMERT Architecture

GraphMERT is built on an encoder-only transformer architecture, which is enhanced by Hierarchical Graph Attention Networks (H-GAT). This architecture allows GraphMERT to incorporate both semantic and syntactic information effectively. Key components of the architecture include:

• Leafy Chain Graphs: These graphs unify syntactic contexts from text corpora with semantic examples from a seed KG, enabling dual-space learning.
• H-GAT: This mechanism fuses relation embeddings into semantic graph nodes, enhancing the transformer with graph-based contextual information.
• Graph Attention: Modifies attention weights with an exponential decay function, encoding spatial distances between graph nodes to ensure that close nodes exert more influence during learning.

Training and Dual Loss Objectives

GraphMERT's training involves two primary loss objectives:

1. Masked Language Modeling (MLM): Trains GraphMERT on syntactic text corpora by predicting masked text portions.
2. Masked Node Modeling (MNM): Focuses on semantic leaves, training relation embeddings through masked tail prediction within the chain graphs.

These objectives ensure that GraphMERT learns robust cross-modal representations, capturing knowledge that simultaneously reflects syntactic context and semantic meaning.

Seed KG Creation and Injection

A critical component of the GraphMERT pipeline is the creation and injection of a seed KG, built from high-quality domain-specific sources. Key steps include:

• Entity Linking: Processes to match text entities with a domain ontology, ensuring genuine relevance to the target domain.
• Similarity Matching: Chooses triples from a seed KG based on their semantic similarity with the text corpus, aided by efficient embedding-based ranking.
• Injection Algorithm: Ensures that each head entity in the text corpus is associated with a semantically meaningful triple, maximizing predictive relevance and diversity.

KG Extraction and LLM Assistance

For KG extraction, GraphMERT predicts masked tail tokens from text sequences conditioned on their context. This prediction benefits from a helper LLM in combining token predictions into coherent prompts, ensuring contextual and semantic congruence in assembled triples.

Figure 1: Overview of the GraphMERT framework. It illustrates the training and semantic augmentation processes involving both chain graphs and LLM collaboration.

Evaluation and Results

GraphMERT's efficacy is evaluated against a baseline LLM KG on medical datasets, showing notable improvements in reliability and accuracy:

• FActScore and ValidityScore: GraphMERT achieves a 69.8% FActScore and a 68.8% ValidityScore, outperforming LLM baseline scores of 40.2% and 43.0% respectively.
• GraphRAG Evaluation: Demonstrates superior performance in tasks requiring multi-hop reasoning and sustained context understanding, successfully answering domain-specific questions.
• Domain Flexibility: Despite potential constraints, GraphMERT adapts well to sparse high-quality training data, emphasizing the importance of data quality over sheer volume.

Conclusion

GraphMERT offers a robust and scalable approach to distilling KGs from unstructured data. With its advanced integration of syntactic and semantic learning, the framework not only overcomes limitations of traditional neurosymbolic systems but also enhances interpretability and reliability. Future work will focus on extending GraphMERT's applicability to other domains and improving multi-token span prediction capabilities. These advancements promise to bridge the gap between neural and symbolic reasoning, laying a path toward domain-specific superintelligence.