---
title: 'ScholarQA: Scholarly Question Answering'
url: https://www.emergentmind.com/topics/scholarqa-sqa
type: topic
---

# ScholarQA: Scholarly Question Answering

ScholarQA (SQA) addresses question answering in the scholarly domain, encompassing techniques, benchmarks, and systems designed to reason over heterogeneous data sources such as bibliographic knowledge graphs (KGs), full-text corpora, and metadata. SQA encompasses both open-domain scientific QA over literature and highly structured queries over scholarly KGs, increasingly utilizing hybrid retrieval, generation, and attribution mechanisms with large language models (LLMs) and retrieval-augmented generation (RAG) pipelines.

## 1. Problem Scope and Challenges

Scholarly QA diverges from generic open-domain QA due to the fragmentation of relevant information across structured KGs (examples: DBLP, SemOpenAlex) and unstructured texts (such as millions of full-text research papers and Wikipedia articles). Traditional QA systems targeting only KG triples cannot extract biographical or discursive information, while purely text-based models fail to capture precise bibliometric facts or structured relationships. Realistic scholarly queries (“main research focus of author X,” “citation metrics of institution Y”) frequently necessitate cross-source traversal, entity disambiguation, and multi-hop reasoning, requiring QA models to integrate KG precision and broad text coverage [2412.02788] [2409.18969].

## 2. Benchmark Datasets and Task Formulations

The evaluation of SQA systems builds upon benchmarks that enforce reasoning across both KGs and unstructured text:

- **Hybrid-SQuAD** (“Hybrid Scholarly Question Answering Dataset”) contains 10,581 question-answer pairs, generated by LLMs with structured sources from DBLP, SemOpenAlex, and Wikipedia. Its design ensures that answers often require bridging multiple data types (e.g., KG→Text, KG→KG→Text). The dataset captures a balanced distribution of bibliometric, biographical, organizational, and research-focused questions, with a mean question length of ~14 tokens and short span answers. LLM-driven generation was used for scalability, with human spot checks for quality. Traversal statistics highlight the dominance of KG→Text paths (55.6% of test cases), underscoring the necessity of integrating heterogeneous data [2412.02788].

- **QALD-2024 Scholarly Hybrid QA** (as in [2409.18969]) and other scenario-based datasets define rigorous test regimes with precise entity linking, structured intent labeling, and explicit evaluation of multi-type questions (counts, affiliations, biographies).

- **Scenario-Based Multiple-Choice SQA** (JEEVES architecture) expands on SQA for multi-choice scenarios, combining scenario contextualization with evidence retrieval from large external corpora, modeling enriched option structures ([S; Q; O_i]) [2108.13875].

## 3. System Architectures

### 3.1 Hybrid RAG Pipelines

Modern SQA systems employ retrieval-augmented generation (RAG) pipelining, jointly leveraging dense and sparse retrieval, KG querying, cross-encoder and LLM-based re-ranking, and structured answer generation:

- **Ai2 Scholar QA**: Integrates over 100M abstracts (Semantic Scholar) and 11.7M full-text papers (Vespa cluster, 285.6M passages, max 480 tokens). The retrieval stack combines sparse (BM25) and dense (embedding) indexing, with ensemble ranking $Score(p) = \alpha \cdot cosine(e_{query}, e_p) + (1–\alpha) \cdot BM25\_{norm}(p)$ ($\alpha=0.6$). Up to 276 top passages are selected, re-ranked with mxbai-rerank-large-v1, and funneled into a multi-step generative LLM workflow. The answer module executes quote extraction, thematic outlining, section-wise synthesis, and tabularization, inserting fine-grained inline citations ([PaperID, ¶n]) for every factual claim. Attribution is systematically enforced and validated with an entailment model (GPT-4o judge), with citation precision/recall audited using protocols such as ALCE [2504.10861].

- **Hybrid-SQuAD Baseline**: Applies a three-stage pipeline: (a) sub-question extraction and SPARQL linking; (b) hybrid KG + dense text retrieval (FAISS, top-k cosine similarity); (c) answer synthesis that concatenates KG triples (as natural language spans) and text passages in a prompt for LLM generation [2412.02788].

- **SPARQL–LLM Hybrid Systems**: Use divide-and-conquer routines to classify entity/intent, route queries to the appropriate KG (DBLP, SemOpenAlex) or text endpoint, execute parameterized SPARQL, and, for context-heavy cases, apply extractive QA (e.g., BERT-base-cased-SQuAD2 with top-N textual and KG-derived context). Aggregation and answer selection follow, using simple confidence ensembles between structured and unstructured sources [2409.18969].

### 3.2 Scenario-Based Models

- **JEEVES**: Joint retriever-reader architecture for scenario-based QA. The retriever is trained with implicit supervision on QA labels via a novel word-weighting network; retrieved paragraphs are scored with both sparse BM25 and word salience; reader fuses k top paragraphs via intra/inter-paragraph dual attention and self-attention mechanisms. Both retriever and reader participate in a joint end-to-end loss function, enhancing retrieval relevance and answer grounding [2108.13875].

## 4. Attribution, Validation, and Evaluation

### 4.1 Attribution Strategies

Attribution is central to SQA, particularly in generative models:

- LLMs are instructed to append inline citations “[PaperID, ¶n]” to every factual claim, and to cite “LLM Memory” if introducing unsupported information [2504.10861].
- Post-generation validation uses entailment models to classify each (claim, citation) pair as “Attributable” or not; citation precision and recall are calculated as per the ALCE protocol [2504.10861].

### 4.2 Evaluation Metrics

SQA systems are evaluated across retrieval, reranking, and generation:

| Metric     | Definition/Scope                                                    |
|------------|---------------------------------------------------------------------|
| Exact Match (EM) | Fraction of answers exactly matching the gold span            |
| F1 Score   | Span-level token overlap                                            |
| nDCG@10    | $\sum_{i=1}^{10} \frac{2^{rel_i}–1}{\log_2(i+1)}$; rel_i: graded relevance|
| mRR        | Mean Reciprocal Rank, averaged over queries                         |
| RubricsScore, TotalScore | Human-composed rubrics (60% question-specific, 40% global), plus bonuses for answer length, expertise, citations [2504.10861]|
| Citation Precision/Recall | As defined above, using entailment-based audit       |

- **Hybrid-SQuAD Baseline**: RAG+ChatGPT-3.5 achieves EM=69.65%, F1=74.91%; best KG-only and text-only systems are 15–37 points lower [2412.02788].
- **Ai2 Scholar QA**: Outperforms all baselines on ScholarQA-CS (Rubrics: 58.0, Total: 61.9, Citation P/R: 48.1), with cross-encoder reranking nDCG@10=0.927, mRR=0.975 [2504.10861].
- **Divide-and-Conquer Hybrid**: Combined SPARQL–LLM pipeline surpasses both SPARQL-only and LLM-only on QALD-2024 (EM: 61.8%, F1: 72.4%) [2409.18969].

## 5. Implementation, Accessibility, and Reproducibility

- **Open-Source Frameworks**: ai2-scholar-qa publicly releases all pipeline modules (retrieval, reranking, generation), evaluation scripts, and index-building utilities. Web applications (Typescript + React) provide interactive QA, with full documentation on data ingestion, API usage, and reproduction of benchmark results [2504.10861].
- **Dataset Creation**: Hybrid-SQuAD generation employs automated LLM prompting from LLM-parsed KGs and Wikipedia, with strict post-processing and attribution tracing for QA pairs [2412.02788].

## 6. Limitations, Design Insights, and Future Directions

SQA systems confront multiple technical and computational barriers:

- **Retrieval-Bottleneck**: Real-time SPARQL performance and index scalability remain limiting for multi-source retrieval [2409.18969].
- **Entity Linking and Multi-Hop Reasoning**: Current approaches are sensitive to ambiguous author names and multi-hop KG→KG→Text traversals; these remain dominant error sources [2412.02788].
- **Attribution Robustness**: Quote extraction, enforced attribution, and section-wise synthesis measurably reduce hallucinations but do not eliminate them entirely; abstention (“LLM Memory”) is used to signal unverifiable claims [2504.10861].
- **Generalizability**: Most SQA systems are concentrated on English, with limited evaluation for multilingual, non-traditional, or student-centered tutoring queries [2412.02788].

Key design insights include: hybrid sparse+dense retrieval with tuned weighting optimizes recall/precision tradeoff (α=0.6); cross-encoder reranking outperforms bi-encoder baselines; multi-step prompting with forced thematic sections enhances answer structure and user experience [2504.10861]. Potential advances involve end-to-end fine-tuning, dynamic KG indexing, robust entity linking via neural methods, and extension to non-English corpora [2412.02788] [2409.18969].

## 7. Representative Systems and Comparative Results

| System              | Architecture             | Main Data Sources             | Key Metrics                    |
|---------------------|--------------------------|-------------------------------|--------------------------------|
| Ai2 Scholar QA      | Full RAG, hybrid retrieval, LLM gen., explicit attribution | S2ORC full-text, Semantic Scholar | Rubrics 58.0, nDCG@10 0.927   |
| Hybrid-SQuAD Baseline | Hybrid RAG (SPARQL + dense text) | DBLP, SemOpenAlex, Wikipedia    | EM 69.65%, F1 74.91%           |
| SPARQL–LLM Hybrid   | Divide-and-conquer, extractive QA | DBLP, SemOpenAlex, Wikipedia    | EM 61.8%, F1 72.4% (QALD-2024) |
| JEEVES              | Joint retriever-reader   | Scenario, large corpus         | 2–6% absolute accuracy gain    |

These systems collectively demonstrate the superiority of hybrid, multi-source RAG models over strict KG- or text-only approaches, and illustrate the necessity of structured attribution and fine-grained evaluation for advancing SQA.

---

*References: [2504.10861], [2412.02788], [2409.18969], [2108.13875]*

Source: https://www.emergentmind.com/topics/scholarqa-sqa