KG20C-QA: Scholarly QA Benchmark

• KG20C-QA is a scholarly QA benchmark built on a curated knowledge graph that captures authorship, affiliation, venue, domain, and citation relations.
• It converts graph triples into paired graph and natural-language QA instances, supporting both link prediction and text-based entity prediction.
• The benchmark features standardized splits, rigorous filtering, and controlled evaluation protocols to ensure reproducible evaluation of scholarly metadata.

Searching arXiv for papers on KG20C-QA and related scholarly QA benchmarks. KG20C-QA is a curated, one-hop question answering benchmark built on top of KG20C, a scholarly knowledge graph derived from the Microsoft Academic Graph. It is intended to support evaluation of both graph-based reasoning and text-based question answering on scholarly metadata rather than on encyclopedic world knowledge. In this setting, the benchmark operationalizes questions over entities and relations such as authorship, affiliation, venue, field of study, and citation, and it does so by converting graph triples into paired graph-form and natural-language QA instances. The benchmark is formally described alongside KG20C in "KG20C & KG20C-QA: Scholarly Knowledge Graph Benchmarks for Link Prediction and Question Answering" (Tran et al., 25 Dec 2025).

1. Position within scholarly knowledge graph research

KG20C-QA emerges from the observation that scholarly data encodes the "metadata of science": who authored which paper, where it was published, under which affiliation, and who cited whom. Large scholarly graphs such as the Microsoft Academic Graph are extensive but noisy, and they do not inherently provide benchmark-ready splits or reproducible QA formulations. KG20C addresses this by offering a curated scholarly KG with standardized train, validation, and test partitions, while KG20C-QA extends that resource into a QA benchmark through question templates defined over graph triples (Tran et al., 25 Dec 2025).

The benchmark is explicitly designed to bridge two evaluation regimes. On one side are graph-based models, especially knowledge graph embedding methods, which can treat QA as triple completion over incomplete triples such as $(h, r, ?)$ or $(?, r, t)$. On the other side are text-based models, including LLMs, which can be prompted with natural language questions whose answers are entities in the graph. This dual representation is a defining feature of KG20C-QA and a plausible implication is that it enables controlled comparison between symbolic or embedding-based retrieval and text-conditioned entity prediction under a shared data substrate (Tran et al., 25 Dec 2025).

A common source of confusion is nomenclature. KG20C-QA is a scholarly benchmark built from KG20C (Tran et al., 25 Dec 2025), whereas KG2QA is a communication-standards question answering system that combines a domain knowledge graph with a fine-tuned LLM and RAG (Luo et al., 8 Jun 2025). The two are distinct in domain, construction, and intended evaluation.

2. Source data, curation, and split design

KG20C-QA is derived from KG20C, whose source is the Microsoft Academic Graph, a large bibliographic graph covering the period 1800–2017. The curation procedure narrows this source to computer science by using the CORE 2020 A* conference ranking to identify top-tier venues, and then restricts the time range to 1990–2010. This range is described as balancing coverage and quality while constraining the scope to a stable period of CS research output (Tran et al., 25 Dec 2025).

The quality-based filtering is stringent. Venues with fewer than 300 publications are removed, and papers with fewer than 20 recorded citations are also removed. The final venue list contains 20 leading CS conferences: AAAI, AAMAS, ACL, CHI, COLT, DCC, EC, FOCS, ICCV, ICDE, ICDM, ICML, ICSE, IJCAI, NIPS, SIGGRAPH, SIGIR, SIGMOD, UAI, and WWW. Before final KG construction, the intermediate curated subset MAG20C contains 5,047 papers, 8,680 authors, 692 affiliations, 20 conferences, 0 journals, and 1,923 domains, with years spanning 1990–2010 (Tran et al., 25 Dec 2025).

The splitting protocol is central to the benchmark’s reproducibility claims. KG20C uses a random uniform split into train, validation, and test over triples. All entities and relations appearing in validation and test also appear in training, so there are no out-of-vocabulary entities at evaluation time. The benchmark further avoids duplicate triples and explicitly takes care to avoid inverse-relation leakage, following best practices motivated by issues identified in earlier benchmarks such as WN18 versus WN18RR (Tran et al., 25 Dec 2025). This suggests that KG20C-QA is intended not merely as a dataset conversion layer, but as a benchmark whose design choices are tightly coupled to methodological validity.

3. Schema, semantics, and QA derivation

KG20C contains five entity types: Paper, Author, Affiliation, Venue, and Domain. It also contains five intrinsic relation types selected to be non-redundant and not trivially implied by one another: author_in_affiliation, author_write_paper, paper_in_domain, paper_cite_paper, and paper_in_venue (Tran et al., 25 Dec 2025).

These relations encode core scholarly metadata. author_in_affiliation(author → affiliation) records organizational association; author_write_paper(author → paper) records authorship; paper_in_domain(paper → domain) records field-of-study assignment; paper_cite_paper(paper → paper) records directed citation; and paper_in_venue(paper → venue) records the conference where a paper was published. Representative examples include 28674CFA author_in_affiliation 075CFC38 and generic triples such as (AuthorID_A, author_write_paper, PaperID_P) and (PaperID_P1, paper_cite_paper, PaperID_P2) (Tran et al., 25 Dec 2025).

KG20C-QA is generated by converting each triple $(h, r, t)$ into two QA items. The first is a forward question that predicts the tail given the head and relation; the second is a reverse question that predicts the head given the tail and relation. This produces two synchronized forms: a graph form based on incomplete triples and a text form based on natural language questions whose answers are entities. Because each triple yields two QA items, the QA benchmark directly mirrors the relational content of the underlying KG while making directionality explicit (Tran et al., 25 Dec 2025).

The current release focuses on one-hop QA, meaning that each question corresponds to a single triple rather than to a compositional or multi-hop reasoning chain. That design keeps the benchmark tightly specified and reproducible. A plausible implication is that one-hop QA serves as a controlled baseline for future extensions to more complex scholarly reasoning tasks.

4. Templates and benchmark formulation

The QA templates are defined per relation and for both query directions. In the graph form, the question is an incomplete triple. In the text form, variables are replaced with entity names such as authors, paper titles, venue names, organization names, or domain labels (Tran et al., 25 Dec 2025).

The template inventory is as follows:

Relation	Forward template	Reverse template
author_in_affiliation	"Where does this author work?"	"Who works at this organization?"
author_write_paper	"What papers did this author write?"	"Who wrote this paper?"
paper_in_domain	"What domains does this paper belong to?"	"What papers belong to this domain?"
paper_cite_paper	"Which papers does this paper cite?"	"Which papers cite this paper?"
paper_in_venue	"Where was this paper published?"	"What papers were published in this conference?"

The templates are deliberately simple and relation-aligned. The paper states that they avoid ambiguity and are readable, mapping directly to single triples for one-hop QA (Tran et al., 25 Dec 2025). This matters because the benchmark is intended to evaluate both ranking-based graph models and natural-language QA models under semantically equivalent task definitions.

An example instantiation for affiliation is "Where does author 28674CFA work?" with answer 075CFC38. For authorship, "Who wrote PaperID_P?" yields AuthorID_A (Tran et al., 25 Dec 2025). These examples underscore that the benchmark’s canonical answers are graph entities, even when the question is phrased in natural language. For LLM evaluation, this creates a practical requirement for mapping surface strings back to entity identifiers or canonical labels.

5. Statistics, task settings, and evaluation protocols

The underlying KG20C contains 16,362 entities, 5 relation types, 48,213 train triples, 3,670 validation triples, and 3,724 test triples. Because KG20C-QA generates two QA items per triple, it contains 96,426 training QA items, 7,340 validation QA items, and 7,448 test QA items (Tran et al., 25 Dec 2025).

For graph-based QA, the input is an incomplete triple such as $(h, r, ?)$ or $(?, r, t)$, and the output is a ranking over candidate entities of the correct semantic type. Evaluation uses standard link-prediction metrics. Mean Reciprocal Rank is defined as

$$\mathrm{MRR} = \frac{1}{N}\sum_{i=1}^{N}\frac{1}{\mathrm{rank}_i}.$$

Hits@$k$ is the proportion of queries whose correct answer is ranked in the top $k$, and the benchmark reports Hits@1, Hits@3, and Hits@10 (Tran et al., 25 Dec 2025).

The protocol uses the standard filtered setting. When ranking candidates for $(h, r, ?)$, other true tails for the same $(h, r)$ found in train, validation, or test are filtered out so they are not treated as negatives. Type-filtered evaluation further restricts candidates to the correct semantic type, for example venues when $(?, r, t)$0 (Tran et al., 25 Dec 2025). This is an important design feature because scholarly relations are type-constrained; ignoring type can make evaluation less realistic and noisier.

Training uses mini-batch Adam on a softmax cross-entropy objective, with both 1-vs-all and k-vs-all negative sampling regimes. Early stopping is based on validation MRR, and results are reported as the median across three random seeds (Tran et al., 25 Dec 2025). For text-based QA, the benchmark permits evaluation by exact match or normalized string match to the canonical entity label, with optional mapping of model outputs to entity IDs before computing accuracy. The paper emphasizes graph-form experiments and leaves natural-language QA experiments for future work (Tran et al., 25 Dec 2025).

6. Baselines, empirical patterns, and methodological significance

The reported baselines include Random, Word2vec skip-gram, CP_h, and MEI. The effective embedding dimensionality is 100, with MEI using 10×10 partitioned embeddings and CP_h and word2vec using 50×2. Optimization uses Adam with early stopping on validation MRR, and results are aggregated as the median over three seeds (Tran et al., 25 Dec 2025).

On link prediction over KG20C, the reported results are:

Model	MRR	Hits@1 / Hits@3 / Hits@10
Random	$(?, r, t)$1	not reported
Word2vec	0.068	0.011 / 0.070 / 0.177
CP_h	0.215	0.148 / 0.234 / 0.348
MEI	0.230	0.157 / 0.258 / 0.368

These results indicate that multi-relational KGE methods outperform a single-relational co-occurrence baseline, with MEI performing best among the reported methods (Tran et al., 25 Dec 2025). The absolute values also show that the benchmark is not saturated.

The per-relation QA-style patterns for MEI are particularly informative. Affiliation queries in the forward direction, corresponding to "Where does this author work?", are comparatively easy with MRR $(?, r, t)$2, while the reverse direction "Who works at this organization?" is lower at $(?, r, t)$3. Venue queries in the forward direction, "Where was this paper published?", are the easiest with MRR $(?, r, t)$4. Authorship queries are of moderate difficulty at MRR $(?, r, t)$5–$(?, r, t)$6. Citation queries are challenging at $(?, r, t)$7–$(?, r, t)$8, and domain queries are the most challenging at $(?, r, t)$9–$(h, r, t)$0 (Tran et al., 25 Dec 2025).

The reported interpretation is that current embedding models excel on venue and some affiliation predictions while struggling on domain and citation queries. A plausible implication is that these harder relations require richer structural inductive bias or supplementary signals such as textual semantics. The paper explicitly notes that domain assignment and citation prediction are difficult and suggests room for improvement (Tran et al., 25 Dec 2025).

The benchmark documentation also lists common KGE scoring functions as natural future baselines, including TransE, DistMult, ComplEx, and RotatE:

$(h, r, t)$1

for TransE,

$(h, r, t)$2

for DistMult,

$(h, r, t)$3

for ComplEx, and

$(h, r, t)$4

for RotatE, where $(h, r, t)$5 encodes a complex-phase rotation (Tran et al., 25 Dec 2025). The paper does not report these models in the baseline table, and that distinction is important for accurate interpretation.

7. Data packaging, usage, and limitations

KG20C and KG20C-QA are packaged in plain-text TSV format consistent with WN18RR and FB15k-237 conventions. The distribution includes all_entity_info.txt, containing entity_id, name(label), and type; all_relation_info.txt, containing the relation inventory; and train.txt, valid.txt, and test.txt, containing triples as (head_id, relation_id, tail_id) (Tran et al., 25 Dec 2025). KG20C-QA is released in synchronized graph and text forms, enabling both KGE-style and LLM-style experimentation.

The benchmark is intended to be practical. For KGE training, one trains on KG20C train.txt, evaluates on valid and test under filtered and type-filtered protocols, and then evaluates graph-form QA by ranking candidates for incomplete triples derived from KG20C-QA. For LLM-based QA, the benchmark supports prompting with the text-form questions and evaluating against canonical entity labels, with optional output restriction to the correct type and post-processing by entity-ID mapping (Tran et al., 25 Dec 2025). The documentation also advises normalization, alias handling if available, and caution with homonyms.

Several limitations are explicit. First, KG20C-QA currently covers only one-hop questions; multi-hop reasoning is identified as future work. Second, the benchmark’s coverage is selective: it is restricted to CORE A* conferences, the 1990–2010 period, and papers with at least 20 citations. This yields a high-quality but filtered snapshot of CS scholarship, and results may not generalize to other venues or periods (Tran et al., 25 Dec 2025). Third, natural-language QA baselines are not yet reported, so the text form presently functions more as an enabling resource than as a completed evaluation track.

These limitations define the benchmark’s role with unusual clarity. KG20C-QA is not a general scholarly QA corpus and not a multi-hop reasoning benchmark. Rather, it is a principled one-hop benchmark over curated scholarly metadata with aligned graph and text formulations. Within that scope, it offers a reproducible testbed for comparing embedding-based triple completion, ranking-based QA, and future LLM-based entity prediction over the structure of scientific publication data (Tran et al., 25 Dec 2025).