GraphMERT: Compact Neurosymbolic KG Extraction

• GraphMERT is a compact encoder-only transformer that integrates hierarchical graph attention with dual objectives to distill factual, ontology-consistent knowledge graphs from unstructured text.
• It employs exponential decay-modulated attention weights and treats syntactic tokens and semantic triple components through a dual loss framework combining masked language and node modeling.
• Benchmark results show GraphMERT’s superior factuality (FActScore of 69.8%) over larger LLM baselines, making it ideal for high-stakes domains like medical and legal applications.

GraphMERT is a compact encoder-only transformer model designed to distill high-quality knowledge graphs (KGs) from unstructured text and internal neural representations. By integrating hierarchical graph attention mechanisms with dedicated symbolic losses, GraphMERT enables efficient and scalable neurosymbolic AI that delivers reliable, ontology-consistent KGs without reliance on prompt engineering or excessively large models. It targets factual and valid KGs suited for high-stakes domains, achieving strong benchmark results compared to LLM baselines.

1. Architectural Design: Encoder-Only Graph-Infused Transformer

GraphMERT employs a modular, encoder-only transformer backbone tailored for text-to-KG distillation. It operates on specially constructed “leafy chain graphs,” where root nodes correspond to syntactic tokens from text and leaf nodes correspond to injected semantic triple elements drawn from a curated seed KG. The embedding layer incorporates a hierarchical graph attention network (H-GAT) to fuse semantic relations into leaf token embeddings. Attention weights are modulated by an exponentially decaying mask, which prioritizes token pairs connected by short graph paths, operationalized as:

$$f(\text{sp}(i,j)) \equiv \lambda^{\text{GELU}(\sqrt{\text{sp}(i,j)} - p)}$$

Here $\text{sp}(i,j)$ is the shortest path distance between nodes, $\lambda$ $(0<\lambda<1)$ is a decay hyperparameter, and $p$ is learnable.

For each semantic triple $\langle h, r, t \rangle$, the tail token embedding is conditioned on the head token via relation-specific transformations:

$$e_{ij}^{(r)} = \text{LeakyReLU}(a_r^T [W_r t_i \,\|\, W_r h_j])$$

with softmax yielding attention weights $\alpha_{ij}^{(r)}$, and the final tail embedding computed as:

$t'_i = \sum_j \alpha_{ij}^{(r)} \cdot W_r h_j$

This enables the transformer to encode both syntactic and KG-derived contextual dependencies.

2. Symbolic-Neural Dual Objective

GraphMERT’s loss function comprises two objectives:

• A standard masked language modeling (MLM) over root tokens, capturing syntactic and semantic abstraction from text.
• A masked node modeling (MNM) loss focused on semantic leaf nodes associated with triples, enforcing symbolic constraints and favoring ontology-compliant relations.

Joint training on both losses is shown to align the model’s neural representations with the external, curated KG, allowing it to distill factual and structurally valid triples during extraction. The model requires a seed KG with ideally 100–1,000 examples per relation to initialize robust relation embeddings and constrain the semantic space.

3. Benchmark Performance: Factuality and Validity

To assess KG reliability, GraphMERT introduces two metrics:

Model	FActScore (%)	ValidityScore (%)
GraphMERT (80M params)	69.8	68.8
LLM baseline (32B params)	40.2	43.0

• FActScore quantifies factual correctness against ground-truth information.
• ValidityScore measures adherence to the underlying ontology (e.g., proper usage of domain-restricted relations).

On domain-specific corpora (e.g., PubMed diabetes papers), GraphMERT outperforms a Qwen3-32B LLM, demonstrating greater reliability and conformance to semantic constraints.

4. Technical Innovations and Functionality

GraphMERT’s compact architecture emphasizes efficiency and scalability:

• Encoder-only design with approximately 80 million parameters, yielding favorable resource requirements compared to LLM baselines.
• H-GAT modifications and exponential attention decay mask ensure the transformer capture relevant symbolic context with minimal token-level computation.
• During extraction, candidate triple components predicted by GraphMERT are post-processed using an external LLM to combine head-tail pairs and finalize KG triples—a hybrid neurosymbolic inference step.

While effective, this process may sometimes produce incomplete or vague tails, and tends to favor frequent entities prevalent in the training data.

5. Applications, Impact, and Limitations

GraphMERT’s reliable KG extraction is particularly suitable for domains where factuality and semantic validity are critical, such as:

• Medical decision support (e.g., PubMed literature distillation)
• Legal and regulatory compliance
• Scientific knowledge management
• Retrieval-augmented generation systems requiring explicit provenance and reasoning over structured KGs

The modular neurosymbolic stack facilitated by GraphMERT and its KG output enhances downstream interpretability and verifiability. Organizations benefit from maintaining auditable, specialist KGs without resorting to opaque, general-purpose LLMs.

Limitations include:

• Dependence on a high-quality, domain-curated seed KG, constraining relation and entity vocabulary.
• Necessity to retrain for new relations or substantial ontology changes.
• Over-representation of common entities and possible incompleteness in tail prediction.
• Reliance on helper LLMs for triple completion during extraction.

6. Future Directions and Research Opportunities

Potential avenues for improving GraphMERT encompass:

• Extension to direct multi-token semantic span prediction, mitigating dependence on post-processing LLMs.
• Refinement of graph-level evaluation metrics to isolate KG quality from neural representation learning artifacts.
• Domain adaptation strategies to enable broader applicability across heterogeneous data sets and ontologies.
• Investigations into regularization techniques for relation and entity embeddings to enhance generalization.
• Enhancements for operating with sparser or less curated seed knowledge graphs.

This suggests ongoing evolution toward more autonomous, robust neurosymbolic models capable of reliable KG distillation in diverse domains.

7. Comparative Analysis and Contextual Significance

GraphMERT marks a notable advancement in neurosymbolic AI, setting new benchmarks for reliable KG extraction from unstructured data. In contrast to prompt-sensitive LLM-based extractors, it delivers explicit symbolic reasoning and interpretable outputs with tangible performance and validity guarantees. Its integration strategy, hybrid neural-symbolic objectives, and compact design differentiate it from contemporaries and position it as a foundational framework for practical, high-assurance knowledge graph construction.