KG20C: Scholarly KG Benchmark

• KG20C is a curated scholarly knowledge graph benchmark derived from the Microsoft Academic Graph that standardizes evaluation for link prediction and question answering.
• It applies strict filtering and a controlled schema to ensure high data quality, reducing leakage and redundant relations.
• Its extension, KG20C-QA, converts graph triples into natural-language QA pairs, offering a synchronized testbed for heterogeneous scholarly queries.

Searching arXiv for KG20C and related benchmark papers. KG20C is a curated scholarly knowledge graph benchmark derived from the Microsoft Academic Graph (MAG) for evaluating link prediction, knowledge graph representation learning, reasoning, recommendation, and, through its extension KG20C-QA, question answering over scholarly metadata. It was designed to provide for scholarly metadata the kind of controlled, reproducible benchmark that WN18RR and FB15k-237 provide for general knowledge-graph completion, with standardized train/validation/test splits, explicit efforts to avoid leakage, and a schema centered on papers, authors, affiliations, venues, domains, and citations rather than lexical or encyclopedic facts (Tran et al., 25 Dec 2025).

1. Definition and benchmark rationale

KG20C addresses a specific gap in knowledge-graph benchmarking. Standard benchmarks such as WN18RR and FB15k-237 are lexical or encyclopedic, whereas scholarly applications require structured data about the research ecosystem itself: papers, authors, affiliations, venues, domains, and citations. MAG provides such coverage at large scale, but it is not benchmark-ready; KG20C therefore adopts a narrower but higher-quality design intended to improve reliability, comparability, and ease of reuse (Tran et al., 25 Dec 2025).

The benchmark’s scope is intentionally constrained. It is limited to top-tier computer science conferences and a bounded publication period, and it applies quality-based filtering to remove sparse or unreliable records. This makes KG20C substantially smaller than raw scholarly graphs such as MAG or AMiner, but also more coherent as a research substrate. The underlying design principle is not maximum coverage; it is benchmark discipline. That includes a controlled schema, non-redundant relations, and split construction informed by lessons from earlier KG benchmarks in which inverse-relation leakage inflated apparent performance (Tran et al., 25 Dec 2025).

A useful way to interpret KG20C is as a domain-specific benchmark rather than a comprehensive scholarly database. This suggests that its principal scientific role is not archival completeness, but controlled experimentation on scholarly metadata under standardized conditions.

2. Curation pipeline and corpus boundaries

The dataset is derived from MAG through a focused curation pipeline. The authors first extracted a high-quality subset they call MAG20C. Papers were restricted to the years 1990–2010 and to twenty top-tier computer science conferences selected using the CORE 2020 A* conference ranking: AAAI, AAMAS, ACL, CHI, COLT, DCC, EC, FOCS, ICCV, ICDE, ICDM, ICML, ICSE, IJCAI, NIPS, SIGGRAPH, SIGIR, SIGMOD, UAI, and WWW (Tran et al., 25 Dec 2025).

Additional quality controls were then applied. Venues with fewer than 300 publications were excluded, as were papers with fewer than 20 recorded citations. From MAG’s 123,056,983 papers, 114,698,044 authors, 19,843 affiliations, 1,283 conferences, 23,404 journals, and 53,834 domains over 1800–2017, the curated MAG20C contains 5,047 papers, 8,680 authors, 692 affiliations, 20 conferences, 0 journals, and 1,923 domains over 1990–2010 (Tran et al., 25 Dec 2025).

These filters define the benchmark’s strengths and its biases. The resulting graph is high-quality and comparatively clean, but it is also intentionally narrow in subject area and prestige profile. Lower-tier venues, journals, newer publications, and less-cited work are excluded by construction. The benchmark is therefore best understood as a curated slice of the scholarly ecosystem rather than a representative sample of scholarship as such.

3. Schema, release format, and dataset statistics

From MAG20C, the authors built a labeled multi-digraph with five entity types: Paper, Author, Affiliation, Venue, and Domain. They define five intrinsic relation types: author_in_affiliation, author_write_paper, paper_in_domain, paper_cite_paper, and paper_in_venue. A central design choice is that these relations are intended to be non-redundant and not directly implied by one another, reducing trivial inference paths and making the benchmark more suitable for meaningful representation learning (Tran et al., 25 Dec 2025).

The release format follows the benchmark conventions common in KG completion. all_entity_info.txt contains entity ID, name, and type; all_relation_info.txt lists relation IDs; and train.txt, valid.txt, and test.txt contain triples of the form entity_id relation_id entity_id. An example triple is 28674CFA author_in_affiliation 075CFC38, meaning that the author with ID 28674CFA is affiliated with institution 075CFC38 (Tran et al., 25 Dec 2025).

KG20C contains 16,362 entities and 5 relation types, with 48,213 training triples, 3,670 validation triples, and 3,724 test triples, for a total of 55,607 triples. The paper positions this as comparable in spirit and rough scale to classical benchmarks, especially WN18RR. For reference, it reports WN18RR as having 40,943 entities, 11 relations, and splits of 86,835/3,034/3,134, while FB15K-237 has 14,541 entities, 237 relations, and 272,115/17,535/20,466 (Tran et al., 25 Dec 2025).

The split construction enforces that every entity and relation appearing in validation or test already appears in training, so models can learn embeddings for all symbols they must score. Duplicate triples and inverse-relation leakage are explicitly eliminated. This is a comparatively strict benchmark protocol, and it is central to the claim that KG20C is “high-quality” in the benchmarking sense, rather than merely curated.

4. Link prediction formulation and empirical difficulty

KG20C is evaluated primarily as a link prediction dataset. The task is standard KG completion: given either $(h, r, ?)$ or $(?, r, t)$, the model ranks candidate entities to recover the missing tail or head. Evaluation uses the standard filtered protocol from Bordes et al., so other known true triples are filtered out from the candidate set and are not treated as false negatives. The reported metrics are mean reciprocal rank (MRR) and Hits@$\{1,3,10\}$; the paper also reports type-filtered metrics, in which candidate entities are restricted to the correct semantic type, a particularly meaningful variant in a heterogeneous scholarly graph (Tran et al., 25 Dec 2025).

Training is described in reproducible benchmark terms rather than through a new task-specific formalism. Models are trained with mini-batch stochastic gradient descent using Adam, with full softmax cross-entropy and both 1-vs-all and k-vs-all negative sampling mentioned as training settings; hyperparameters such as dropout and weight decay are tuned by random search; embedding dimensionality is held to about 100 effective dimensions for comparability; early stopping is based on validation MRR; and reported scores are medians over three random seeds (Tran et al., 25 Dec 2025).

The baseline results indicate that the benchmark is challenging and unsaturated. On the link prediction task, the reported scores are as follows.

Model	MRR	Hits@10
Random	0.001	$< 5e{-4}$
Word2vec skipgram	0.068	0.177
CP$_h$	0.215	0.348
MEI	0.230	0.368

The pattern is consistent with the dataset’s design. A single-relational co-occurrence baseline such as word2vec is much weaker than genuinely multi-relational embedding models; CP$_h$ is a strong baseline; and MEI performs best overall. Yet the best reported MRR of 0.230 remains modest, which supports the claim that KG20C is far from solved. Relation semantics also matter: because the graph is constructed to reduce redundancy and trivial implication, success depends less on exploiting benchmark artifacts and more on learning the structure of scholarly metadata.

5. KG20C-QA and one-hop scholarly question answering

KG20C-QA extends the graph benchmark into a one-hop question answering benchmark by converting every KG20C triple into question–answer pairs in two synchronized representations: a graph form suitable for embedding models and a natural-language form suitable for text-based systems, including LLMs. Each triple $(h, r, t)$ generates two QA instances, one in each direction: a forward query asks for the tail given the head and relation, and a reverse query asks for the head given the tail and relation (Tran et al., 25 Dec 2025).

The graph-form instances are incomplete triples such as $(h, r, ?)$. The text-form instances are generated through hand-designed relation-specific templates that map the KG schema into readable questions using entity names.

Relation	Forward template	Reverse template
author_in_affiliation	“Where does this author work?”	“Who works at this organization?”
author_write_paper	“What papers did this author write?”	“Who wrote this paper?”
paper_in_domain	“What domains does this paper belong to?”	“What papers belong to this domain?”
paper_cite_paper	“Which papers does this paper cite?”	“Which papers cite this paper?”
paper_in_venue	“Where was this paper published?”	“What papers were published in this conference?”

Because each KG triple yields two QA pairs, KG20C-QA doubles the instance count of KG20C. Its reported size is 96,426 training questions, 7,340 validation questions, and 7,448 test questions, for 111,214 question instances across graph and natural-language forms. The splits are aligned with the base graph, so the QA benchmark remains synchronized with the link prediction benchmark (Tran et al., 25 Dec 2025).

The current release is explicitly limited to one-hop QA based on single triples. That limitation is methodologically important. It makes the benchmark simple relative to multi-hop scholarly QA, but it also provides a controlled substrate for comparing graph-based models and text-based models on exactly aligned supervision. In the reported experiments, however, only graph-based answering in entity–relation form is evaluated; natural-language QA with LLMs is left for future work.

The type-filtered MEI results on KG20C-QA show marked variation by relation and direction. Queries with strongly constrained answer spaces are comparatively easy: “Which conferences may this paper publish in?” reaches MRR 0.693, Hit@1 0.542, Hit@3 0.810, Hit@10 0.976, and “Where may this author work at?” reaches MRR 0.626, Hit@1 0.562, Hit@3 0.669, Hit@10 0.731. By contrast, citation and domain questions are much harder: “Which papers may cite this paper?” gives MRR 0.116, “Which papers may this paper cite?” gives 0.193, “Which papers may belong to this domain?” gives 0.052, and “Which may be the domains of this paper?” gives 0.189 (Tran et al., 25 Dec 2025).

These results indicate that KG20C-QA is not merely a mechanical rendering of triples into questions. It exposes substantial heterogeneity in relation difficulty, with venue and affiliation links comparatively predictable, and domains and citations comparatively diffuse.

6. Position in the literature, downstream use, and limitations

KG20C is situated as a benchmark pair alongside KG20C-QA. Relative to WN18RR and FB15k-237, it offers a benchmark of similar intent and cleaner protocol in the scholarly domain; relative to raw scholarly graphs such as MAG and AMiner, it is smaller but benchmark-ready and reproducible; and relative to scholarly QA datasets such as SciQA and DBLP-QuAD, KG20C-QA emphasizes one-hop template grounding, alignment with a curated KG completion benchmark, and dual release in graph and text forms (Tran et al., 25 Dec 2025).

Subsequent work has used KG20C to study unintended information leakage in embeddings. The EXTRACT paper describes KG20C as a citation-network knowledge graph constructed from 20 top AI conferences and uses it to show that conference identity can be inferred from paper embeddings even when conference labels are not used for training. In that study, a logistic classifier predicts conference from paper embeddings with 55% accuracy against a baseline reported as 13% in the prose and 15% in the table, and transparent multiclass debiasing methods reduce conference prediction to around 0.10–0.14 while lowering link-prediction utility (Guo et al., 2023). This use positions KG20C not only as a completion benchmark, but also as a testbed for examining what scholarly graph embeddings implicitly encode.

The limitations of KG20C are explicit rather than incidental. Coverage is intentionally narrow: only twenty top-tier computer science conferences, only 1990–2010, and only records passing publication-count and citation thresholds. This produces domain and prestige bias. KG20C-QA covers only one-hop questions and uses template-based natural-language generation, so it does not yet test compositional, multi-hop, or highly paraphrastic QA. The evaluated QA baselines are only graph-based. These constraints do not invalidate the benchmark, but they define its valid use cases.

A common source of confusion is the identifier itself. In unrelated arXiv contexts, “KG20C” does not denote the DARTWARS KITWPA prototype line, and the DARTWARS progress paper contains no explicit reference to KG20C; likewise, “DarkSide-20k” is an unrelated direct dark matter detection experiment and not a knowledge-graph benchmark (Borghesi et al., 2022, Aalseth et al., 2017). Within scholarly KG research, however, KG20C refers specifically to the curated MAG-derived benchmark and its aligned QA extension.

Taken together, KG20C and KG20C-QA define a small-to-medium sized but carefully curated benchmark pair for scholarly metadata. Their main contribution lies less in raw scale than in protocol design: targeted venue selection, quality filtering, a clean schema, leakage-aware splitting, and synchronized graph and question-answer views. That combination makes them reusable resources for link prediction, representation learning, one-hop scholarly QA, and analysis of the structure and biases encoded in scholarly knowledge-graph embeddings (Tran et al., 25 Dec 2025).