KGGen: LLM-Powered Knowledge Graphs

• KGGen is an open-source framework that constructs dense, semantically coherent knowledge graphs from text using LLM-based entity and relation extraction.
• It employs a novel iterative clustering process to consolidate synonyms and reduce redundancy, enhancing graph quality for downstream tasks.
• The system is validated with the MINE benchmark, demonstrating significant performance improvements over traditional extraction methods like OpenIE and GraphRAG.

KGGen is an open-source framework for constructing dense, semantically coherent knowledge graphs (KGs) from plain text using LLMs. Addressing persistent challenges of data scarcity and noisy automatic extraction in knowledge graph construction, KGGen combines modern LLM-based entity/relation extraction with an iterative clustering procedure to reduce sparsity and redundancy, enabling the creation of high-quality KGs suitable for downstream embedding and retrieval tasks. The KGGen package is distributed via Python (pip install kg-gen) and is accompanied by the MINE benchmark, the first systematic test for evaluating the informativeness of automatically generated KGs from text (Mo et al., 14 Feb 2025).

1. Motivation and Problem Definition

High-quality KGs, structured collections of ⟨subject, predicate, object⟩ triples, are essential for graph embeddings, retrieval-augmented generation, multi-hop reasoning, and systematic knowledge synthesis. However, the field faces two central bottlenecks:

• Data scarcity: Existing high-quality KGs (e.g., Wikidata, DBpedia, YAGO) are mostly human-curated and do not comprehensively cover domain-specific or emergent facts.
• Extraction quality: Automatic pipelines—both traditional (OpenIE) and early LM-based methods (e.g., GraphRAG's KG extraction component)—produce graphs plagued by isolated nodes, redundant entities (e.g., “labors” vs. “labor”), and semantically noisy relations.

Such sparsity and redundancy undermine the effectiveness of graph-based embeddings and limit the utility of KGs for in-context retrieval or reasoning. KGGen was developed to directly address these limitations by leveraging LLMs for higher-precision extraction and incorporating a novel clustering stage to collapse synonyms and related entities (Mo et al., 14 Feb 2025).

2. System Architecture and Methodology

KGGen implements a modular three-stage pipeline:

A. Entity & Relation Extraction (generate)

1. Entity Extraction: The framework prompts an LLM (e.g., GPT-4o) to enumerate all key entities (nouns, adjectives, verbs) within a document in JSON format.
2. Relation Extraction: The LLM, given both the list of extracted entities and the source text, is prompted to output a set of ⟨subject, predicate, object⟩ triples. This two-pass design ensures maximal consistency between entities and relations.

B. Aggregation (aggregate)

• All per-document KGs are merged into a master graph.
• Label normalization (e.g., lowercase) and duplicate removal are performed. This stage is deterministic and does not involve further LLM calls.

C. Iterative Clustering (cluster)

KGGen's core innovation is its LM-based clustering approach. The algorithm operates as follows:

1. The entity set $E$ is iteratively scanned. For each iteration:
   • The LLM is prompted to propose a cluster $C_t \subseteq E$ (entities with the same meaning, differing only in tense, plurality, etc.).
   • A second LLM call validates $C_t$; if valid, the cluster receives a canonical label $\ell_t$ and is removed from $E$.
2. This procedure repeats until $N$ consecutive failed clustering attempts occur.
3. Remaining singleton entities are batched and (via further prompts) added to appropriate clusters if semantically justified.
4. An identical procedure is applied to edge labels.

This process can be informally viewed as attempting to maximize average intra-cluster semantic similarity as judged by the LM, subject to cluster disjointness: $\max_{\{C_k\}}\sum_k\sum_{e_i,e_j \in C_k} \mathrm{sim}(e_i, e_j) \quad \text{s.t. %%%%0%%%% disjoint}$ where “sim” is approximated by LLM semantic judgments (Mo et al., 14 Feb 2025).

3. The MINE Benchmark

To systematically evaluate KG extraction quality, KGGen introduces the MINE (Measure of Information in Nodes and Edges) benchmark:

• For a corpus of length-standardized articles ($\mathcal{A} = \{A_1, \ldots, A_{100}\}$), each is annotated with 15 ground-truth facts $\mathcal{F}_i$.
• For each method, the autogenrated KG $G_i$ is compared to the facts by:
  1. Vectorizing all KG node labels and each fact using an encoder (e.g., SentenceTransformers all-MiniLM-L6-v2).
  2. For each fact, the top-$k$ nearest nodes are found; nodes within two hops are collected into a subgraph.
  3. An LLM judge is prompted to output 1 (fact inferable) or 0 (not inferable) given the subgraph.

The MINE score is: $$\mathrm{MINE}(G_i) = \frac{1}{15}\sum_{j=1}^{15} \mathbb{1}[\mathrm{Inf}(f_{i,j},S_{i,j}) = 1]$$ The method’s aggregate MINE is averaged over all articles.

4. Experimental Evaluation

Datasets and Baselines

• 100 LLM-generated Wikipedia-style articles, each ~1000 tokens.

• Comparison methods: OpenIE (Stanford CoreNLP); Microsoft’s GraphRAG (including community summarization).
• Metric: MINE (see previous section).

Results

Method	MINE (%)
KGGen	66.07
GraphRAG	47.80
OpenIE	29.84

KGGen achieves an 18% absolute improvement over GraphRAG and a 36% lift over OpenIE. Qualitatively, KGGen graphs have greater density (fewer singleton nodes), reduced semantic redundancy, and support more robust fact inference and retrieval. In contrast, OpenIE is prone to generic or spurious nodes (e.g., “it,” “are”) and GraphRAG typically produces sparse, under-connected graphs (Mo et al., 14 Feb 2025).

5. Package Usage and Configuration

KGGen is available via PyPI and is compatible with popular LLM back-ends (e.g., GPT-4o via DSPy). A typical workflow:

from kg_gen import KGGenerator kggen = KGGenerator(model="gpt-4o") raw_kg = kggen.generate(text=document_text) merged_kg = kggen.aggregate([raw_kg]) final_kg = kggen.cluster(merged_kg, max_loops=10, batch_size=20) for (s, p, o) in final_kg.triples: print(f"{s} —[{p}]→ {o}")

Key parameters include the LLM model, clustering loop threshold (max_loops), batch size for singleton assignment, and customizable prompt templates.

The output is a KnowledgeGraph object:

class KnowledgeGraph: triples: List[Tuple[str,str,str]] clusters: { "entities": List[Cluster], # {label: str, members: List[str]} "edges": List[Cluster] } }

This structure facilitates downstream conversion to standard graph libraries such as NetworkX or RDFLib.

6. Comparative Context and Limitations

KGGen advances beyond previous fully automatic KGC methods by combining LLM triple extraction with clustering to minimize node redundancy and sparsity, overcoming key weaknesses of both OpenIE and early LM-based systems (e.g., GraphRAG). The reliance on LLMs for both extraction and semantic consolidation allows the system to adapt flexibly across domains.

However, the pipeline builds exclusively from textual input, with no integration of multimodal or tabular sources. All semantic consolidation relies on prompt engineering and LM judgment, with no structured loss function or explicit optimization for coverage or factual consistency beyond the clustering heuristic. A plausible implication is that domain shifts (e.g., biomedical terminology) or non-English input may challenge the robustness of synonym identification and relation extraction due to prompt or model limitations.

7. Outlook and Significance

KGGen provides a reproducible, open-source baseline for KGC from plain text, establishing a high bar for graph density, coverage, and semantic quality. The release of the MINE benchmark catalyzes rigorous comparison of new extractors. Future extensions—such as end-to-end retrieval-augmented KG construction, dynamic multi-hop extraction, or multimodal KGGen variants—are active directions in both academia and industry (Mo et al., 14 Feb 2025, Zhang et al., 14 Apr 2025, Zhang et al., 14 Mar 2025).