---
title: GPTKB Methodology
url: https://www.emergentmind.com/topics/gptkb-methodology
type: topic
---

# GPTKB Methodology

Searching arXiv for the GPTKB methodology papers and closely related follow-up work.
GPTKB methodology is a framework for materializing a language model’s factual knowledge into a persistent knowledge base by recursively prompting the model, extracting triples, and consolidating the results into a usable graph [2507.05740]. In the LLM-knowledge literature, it is proposed as a response to **availability bias** in benchmark-based evaluation: instead of testing only preselected facts, GPTKB recursively discovers the model’s own factual content, beginning from a seed entity and expanding breadth-first through newly mentioned named entities [2411.04920].

## 1. Concept and motivation

GPTKB is designed for **recursive knowledge elicitation** followed by **post-hoc knowledge consolidation**. The motivating claim is methodological rather than merely infrastructural: fixed benchmarks measure only facts that researchers already anticipated, whereas GPTKB attempts to discover the scope, structure, bias, cutoff behavior, and consistency of an LLM’s factual knowledge by letting one elicited fact lead to the next [2411.04920].

The methodology begins from a single seed entity. In the prototype and subsequent large-scale realization, that seed is **Vannevar Bush**, chosen symbolically because of his “As We May Think” vision of linked knowledge; the choice is described as arbitrary in the sense that any seed connected to common entities would work because large knowledge graphs are densely connected [2411.04920]. From that starting point, GPTKB treats the LLM as both the source of triples and the source of candidate frontier entities for further expansion.

A central conceptual distinction is between **benchmark probing** and **materialization**. Traditional evaluation asks fixed questions about a handpicked set of entities, relations, or QA items. GPTKB instead builds a persistent resource that can later be queried, browsed, and statistically analyzed. This suggests a shift from isolated factual testing to open-ended structural analysis of LLM knowledge, although the resulting graph still reflects the probabilistic and prompt-sensitive nature of the underlying model [2411.04920].

## 2. Recursive elicitation pipeline

The elicitation stage is a **BFS-based graph exploration** over the model’s latent factual graph. For each subject entity, the LLM is prompted to return factual triples in subject–predicate–object form. The prompt constrains output behavior: return **all facts** it knows for the subject, vary output counts with popularity, return an empty list if the subject is unknown or not a named entity, include at least one `instanceOf` triple for named entities, keep outputs concise, and split multi-object statements into separate triples [2411.04920].

Two closely related implementation strategies are reported. The prototype uses OpenAI’s **structured output** feature to force a JSON schema, thereby reducing verbose free-form responses [2411.04920]. The later v1.5 system describes the same requirement as **constrained decoding** to keep outputs within triple form [2507.05740]. In both cases, the methodological aim is to make recursive extraction operationally stable enough for large-scale crawling.

New candidate entities are obtained from triple objects. GPTKB does not rely on conventional NER for this step, because standard NER models were reported to perform poorly on short, context-free labels. Instead, it uses **LLM-based NER** to classify candidate phrases as named entities or not; only accepted named entities are enqueued for further crawling [2411.04920]. This filtering is crucial because naive recursive expansion was observed to cause “topical runaway” into translations, literals, and other non-entity artifacts.

The prototype realizes this procedure at substantial scale: **2,200 batches**, up to **BFS depth 10**, **5.8 million prompted entities**, and **2.9 million entities** with non-empty answers, using up to **100 batches of 10,000 entities in parallel** after startup [2411.04920]. The v1.5 realization preserves the same recursive logic but reports that parallel crawling distorts strict BFS order; it therefore recomputes each node’s shortest path to the root afterward and stores it as the meta-relations `bfsLayer` and `bfsParent` [2507.05740].

## 3. Consolidation, normalization, and taxonomy construction

Recursive elicitation produces a large but noisy graph. GPTKB therefore performs **post-hoc consolidation** rather than attempting to keep the entire evolving KB in-context during generation [2411.04920]. Consolidation targets redundant relation labels, redundant class labels, structural metadata, and duplicate entities.

For relations, the prototype reports **788K distinct relation names** before clustering. It applies an embedding-based greedy clustering algorithm that processes relations from most frequent to least frequent and merges a relation into the most similar more-frequent relation if similarity exceeds an adaptive threshold [2411.04920]:

$$
threshold \gets \alpha \times \frac{\log(\text{frequency}(r))}{\log(\text{frequency}(\text{first}(\mathcal{R})))}
$$

with embeddings from **SentenceTransformers**, cosine similarity, and $\alpha = 1.2$ chosen by manual inspection of held-out examples [2411.04920]. The same procedure is applied to class names, i.e., the objects of `instanceOf`. In the prototype, this reduces **788K** relations to **1,804** and **103K** classes to **473** [2411.04920]. In v1.5, the corresponding consolidation is described more generally as a greedy clustering algorithm that merges relations and classes into more frequent canonical ones when label embeddings are sufficiently similar; the reported counts are **936k raw relations → 381k canonicalized** and **220k raw classes → 32k canonicalized** [2507.05740].

GPTKB also constructs a taxonomy for classes. The prototype describes **Algorithm 2: Taxonomy construction**, which first asks the LLM to generate a seed taxonomy, then asks it to score each class by generality on a 1–10 integer scale, and finally inserts classes into the taxonomy by recursive depth-first decisions over existing branches [2411.04920]. The LLM may introduce intermediate nodes during insertion, so taxonomy formation is not merely a direct projection of observed `instanceOf` triples.

Entity deduplication is deliberately lightweight. For humans, GPTKB uses a **blocking-based deduplication** procedure keyed by **birth date**; within a block, two entities are treated as duplicates if their labels match and **30% of their triples are exactly the same** [2411.04920]. This suggests a scalability-oriented compromise: the system avoids costly global entity linking while accepting that some duplicates or under-canonicalized entities remain.

## 4. Large-scale realizations and system infrastructure

Two closely related large-scale realizations anchor the methodology: the GPT-4o-mini prototype and GPTKB v1.5 built from GPT-4.1.

| Realization | Core construction facts | Storage and access |
|---|---|---|
| **GPTKB prototype** [2411.04920] | **101 million triples**, **2.9 million entities**, **1,804 relations**, **473 classes**, runtime **27 hours**, cost **$3,500** | **3.8 GB Turtle (TTL)** download, browsing interface, **SPARQL** endpoint |
| **GPTKB v1.5** [2507.05740] | **100 million triples**, **6.1 million entities**, **381k** canonicalized relations, **32k** canonicalized classes, runtime **18 days**, cost **$14,136** | converted into **RDF**, serialized in **Turtle**, hosted in **Virtuoso**, exposed through a **SPARQL endpoint** |

The move from GPT-4o-mini to GPT-4.1 is explicitly motivated by quality problems in GPTKB v1.1: over **60% hallucinated triples** estimated, severe output skew, and some entities with **over 100k mostly hallucinated triples** [2507.05740]. The v1.5 paper reports improvements of **+110% entities** and **+44 percentage points accuracy** at a total cost of about **$14,136** [2507.05740].

The demonstrator built on v1.5 emphasizes three interaction modes: **link-traversal-based knowledge exploration**, **SPARQL-based structured querying**, and **comparative exploration of the strengths and weaknesses of LLM knowledge** [2507.05740]. The KB is therefore not only an extraction artifact but also an analysis platform. Users can search entities, inspect statements, traverse linked objects, and examine `bfsLayer` and `bfsParent` to reconstruct the recursive expansion structure [2507.05740].

## 5. Analytical uses and empirical findings

GPTKB is intended to support analyses of **accuracy, bias, cutoff, consistency, coverage, taxonomy structure, and overlap with external KBs** [2411.04920]. The prototype and v1.5 papers report such analyses at scale.

For the prototype, entity-level verifiability is assessed via web search snippets and LLM judgment, yielding **74% verifiable**, **9% plausible**, and **17% unverifiable** entities, with a gradual decline across BFS layers from 100% at the root to about 70% by layer 10 [2411.04920]. Triple-level evaluation reports **31% true/entailed**, **61% plausible**, **1% implausible**, and **7% false** [2411.04920]. Taxonomy evaluation gives **64%** correct subclass–superclass edges and **70%** best-alternative superclass choices in a sibling-choice setting [2411.04920].

For v1.5, evaluation is stronger under its reported protocols: on **1,000 random triples**, the automated web-search-based assessment yields **75.5% true**, **5.0% plausible**, and **19.5% false**; a manual assessment of **100 triples** yields **75% correct**, **14% false**, with remaining cases undecidable or semantically unclear [2507.05740]. Subject-level precision is reported as **85.3% verifiable**, **3.4% plausible**, and **11.3% unverifiable** [2507.05740]. Because the two papers use different models and somewhat different evaluation setups, this suggests caution in treating their headline precision numbers as directly interchangeable.

The materialized KB enables analyses that ordinary benchmark probes do not. The prototype observes a sharp drop from **2023** to **2024** in year frequencies, matching GPT-4o-mini’s stated cutoff [2411.04920]. The v1.5 demonstrator uses SPARQL to study nationality distributions, gender distributions, EU-citizen counts, and relation symmetry. For example, the spouse relation is highly asymmetric: `numMutual = 65,339`, `total = 402,333`, and `fraction = 0.162` [2507.05740]. The demonstrator also supports direct cross-model comparison across **Llama 3.3-70B-Instruct**, **Llama 4-Scout-17B-16E-Instruct**, **GPT-4o-mini**, **GPT-4.1**, and **DeepSeek-R1** for a selected set of 100 entities [2507.05740].

## 6. Reliability, limitations, and methodological status

A subsequent study on **miniGPTKBs** focuses on the foundational questions of **termination, reproducibility, and robustness** for GPTKB-style elicitation [2510.06780]. It studies three domains—ancient Babylon, *The Big Bang Theory*, and DAX 40—and defines three metric families: **yield**, **lexical similarity**, and **semantic similarity**. Yield variation is summarized by the coefficient of variation $Y_{CV} = \sigma / \mu$; lexical similarity uses average Jaccard similarity; semantic similarity is measured both by average cosine-based Hausdorff similarity and by a thresholded semantic-match percentage with $\tau = 0.95$ [2510.06780].

The main result is that GPTKB-style knowledge materialization shows **high termination rates overall**, but with clear dependence on model and language [2510.06780]. All base runs, all seed-variation runs, and all temperature-variation runs terminated; by contrast, **3 of the 10 language runs** did not terminate organically and had to be stopped after **96 hours**, and among local or open models, **Llama 4 Scout**, **DeepSeek-R1**, and **Teuken 7b Instruct** did not terminate [2510.06780]. The reported non-termination modes include repetitive loops, invented Wikidata-like identifiers, and questionable NER classifications.

Reproducibility is described as **mixed**. Yield is stable across repeated runs, with named-entity coefficients of variation of **0.12** for babylonGPTKB, **0.18** for tbbtGPTKB, and **0.13** for dax40GPTKB [2510.06780]. Exact lexical overlap is modest—**0.33**, **0.41**, and **0.51** Jaccard similarity for named entities in the three domains—but semantic similarity is substantially higher, with cosine-based Hausdorff similarity around **0.89** and semantic-match percentages of **58.3%**, **61.3%**, and **69.0%** [2510.06780]. Robustness is high for **seed** and **temperature**, lower for **language**, and intermediate for **model** variation [2510.06780].

The broader methodological caveats remain substantial. The core GPTKB papers note that some triples remain undecidable because their semantics are unclear, hallucinations persist even in GPT-4.1, prompt wording affects the precision–recall tradeoff, and the materialized KB should not be used blindly in production [2411.04920]. The miniGPTKB study reinforces that exact reproducibility is limited, multilingual robustness is weak, and open-domain termination is not proved merely by the good behavior of subcrawls [2510.06780]. At the same time, the combined evidence indicates that GPTKB can recover a stable semantic core of an LLM’s factual knowledge and convert that core into a queryable graph suitable for browsing, SPARQL analysis, and comparative model study [2507.05740].

Source: https://www.emergentmind.com/topics/gptkb-methodology