ExpertQA: Domain-Specific QA Benchmark

• ExpertQA is an ecosystem of methods and benchmark datasets designed for generating long-form, domain-specific answers with expert validation and accurate evidence citation.
• It integrates advanced techniques such as few-shot prompting, style-imitative generation, and efficient deduplication to ensure diverse and high-quality question-answer pairs.
• Robust evaluation frameworks in ExpertQA combine multi-axis expert ratings, detailed evidence grounding, and user expertise analysis to optimize accuracy and reliability.

ExpertQA refers to methodologies, systems, and datasets designed to enable, evaluate, and enhance long-form question answering for expert-level, domain-specific queries. The term encompasses both benchmark resources for rigorous empirical assessment and algorithmic frameworks for expert identification, expert-imitative question generation, evidence grounding, factuality attribution, expert ranking, and system diagnosis in high-stakes or specialized domains. ExpertQA resources are widely used to benchmark LLMs, retrieval-augmented QA systems, and recommendation frameworks in contexts where accuracy, transparency, and explainability are critical.

1. Benchmark Construction and Dataset Properties

The canonical ExpertQA dataset was constructed via expert-sourced queries and attributed long-form answers, with a focus on multi-domain coverage and rigorous annotation (Malaviya et al., 2023). Expert recruitment involved 484 domain-qualified professionals across 32 fields who contributed over 2,100 technically challenging, open-ended questions, mandating the inclusion of scenario-based (complex, context-rich) prompts. Each question was annotated for information-need type and linked to both machine- and human-generated answers. Subsequently, the original experts evaluated model-produced responses in a multi-axis schema covering usefulness, informativeness, factual accuracy, cite-worthiness, attribution completeness, and source reliability. This dual-loop process—evaluation and expert revision—resulted in high-quality, fact-checked target answers and granular claim-level attributions, forming a gold standard for subsequent QA evaluation and training.

The dataset spans critical domains (Medicine, Law, Engineering, etc.), with high representation in areas where factuality stakes are highest. Manual filtering and Cohen's κ analysis indicate high-classification agreement (κ > 0.85), signifying strong inter-annotator reliability.

2. Algorithmic Approaches to ExpertQA Generation

ExpertQA system design integrates formatting, systematic topic coverage, style imitation, and downstream performance metrics (Shahgir et al., 4 Mar 2025). The ExpertGenQA protocol exemplifies modern synthetic expert QA generation for specialized domains, particularly technical regulations:

• Few-Shot Prompting: Utilizes GPT-4o with temperature T=1, top-k=5, and n=10 few-shot exemplars per generation, achieving a balance of QA diversity and modeling efficiency.
• Style-Topic Dual Categorization: Manual taxonomy segments expert questions into policy application, scenario-based, and terminology clarification styles. Each document is partitioned into topics using LLM-driven extraction, with a separate QA generation loop enforcing coverage over all topics/styles.
• Deduplication and Efficiency: Unique QAs filtered by bigram-overlap threshold (0.3). Efficiency, defined as (#unique questions) / (#LLM calls), reaches 40.5% for ExpertGenQA—double standard few-shot approaches.
• Coverage and Retrieval: Topic coverage (TC) achieves 94.4%. Downstream, retriever fine-tuning (InfoNCE loss, in-batch negatives, τ=0.1) yields a +13.02 percentage point top-1 accuracy gain over baseline, outperforming larger embedding models and template-based approaches.

An analysis using Bloom’s Taxonomy confirms that ExpertGenQA maintains the cognitive complexity distribution of expert-written QAs, unlike template methods that overemphasize rote recall.

3. Evidence Grounding and Citation Faithfulness

Robust ExpertQA entails not only answer accuracy but also rigorous source attribution. Research on explicit evidence grounding interrogates inline citation generation, span alignment, and claim-citation faithfulness (Yeginbergen et al., 5 Jun 2026, Huang et al., 2024):

• Structured Inline Citations: Frameworks such as FullCite enforce strict association of claims to source document IDs and verbatim evidence spans using prompt-based generation, constrained decoding with finite-state automata enforcing BNF citation grammars, and posthoc span alignment (max-Jaccard search, τ=0.7).
• Evaluation Metrics: Document-level F1 (Doc-F1), snippet-level F1 (Snippet-F1), and claim-citation semantic similarity are computed. On ExpertQA, document identification is relatively robust (Doc-F1 up to 82.5%), but snippet-level F1 for precise span grounding rarely exceeds 33.5%, revealing that pinpointing atomic evidence is a limiting factor.
• Systematic Failure Modes: Primacy bias (64% of citations restricted to first two documents), citation omissions on yes/no questions, and refusals (17.9% of posthoc runs cannot cite legally) are documented.

Fine-grained reward functions for citation generation (correctness recall, citation recall, citation precision) and multi-stage training (distillation, rejection sampling, PPO-RL) lead LLMs (e.g., LLaMA-2-7B) to outperform even GPT-3.5-turbo on support metrics such as AutoAIS (achieving 66.12% vs. 56.98%) (Huang et al., 2024).

4. Attribution Quality, Reference Verification, and Hallucination

The reliability of claim attributions is systematically scrutinized using URL liveness auditing, hallucination detection, and agentic self-correction (Rao et al., 3 Apr 2026). Key findings include:

• Reference Hallucination: Among 168,000+ ExpertQA citation URLs across 32 fields and multiple LLMs, non-resolving rates span 4.2–9.4% (up to 18.2% for Reddit URLs in some models), with hallucinated URLs (never existed) at 3–13%.
• Domain/Model Effects: Healthcare/Medicine and Theology exhibit highest non-resolving rates (∼11%), while Business and Engineering maintain lowest (<6%). Some models fabricate all non-resolving URLs; others present substantial link rot.
• Automated Verification: The open-source urlhealth tool programmatically checks URL liveness and classifies dead links as hallucinated or stale (Wayback snapshot check). Integration as an agentic skill enables LLMs to self-correct; non-resolving rates decrease by up to 79× (e.g., GPT-5.1 from 16% to 0.6%).
• Practical Ceiling: 10–20% of URLs fall in an "unknown" class due to paywalls, bot blocks, or ambiguous status codes—headless browser audits reveal most are actually alive or access-blocked, setting an empirical limit for automated checking.

5. User Expertise and Difficulty Estimation

ExpertQA systems in collaborative networks rely on robust modeling of user expertise and question difficulty to optimize expert routing (Huang et al., 2018, Sun et al., 2018):

• Semantic Embedding and Clustering: Posts are embedded using tf-weighted Word2Vec vectors; k-means clustering in embedding space identifies latent knowledge domains. Incoming queries are assigned to domains to facilitate expert retrieval.
• Expertise Quantification: Hybrid scores combine context similarity (cosine to query vector) and latent reputation (NMF voting scores). Empirical results on Stack Overflow show stable gains: precision@5 reaches 0.1367, outperforming tag-based and PMF/BPMF baselines.
• Ranking via Graph Hierarchies: QDEE frames question-user interactions as a competition graph. Social agony minimization [Gupte et al. 2011] yields a total order on nodes, providing expertise scores for users and difficulty ranks for questions. For cold-start questions, text-based regression models or k-NN impute difficulty.
• Routing: Candidates are ranked using (e_u – d_q)·sim_topic(u, q), prioritizing users whose expertise slightly exceeds question difficulty and incorporating topic proximity.

6. Evaluation Frameworks and Medical ExpertQA

MEDIC exemplifies multi-dimensional evaluation of LLM-based ExpertQA systems, particularly in healthcare contexts (Kanithi et al., 2024):

• Dimensions: Medical reasoning (knowledge recall, logical inference, evidence justification), ethics, language understanding, in-context learning, and clinical safety.
• Cross-Examination (4C): Automatic closed-ended question generation from both model outputs and gold standards with bidirectional checking yields four scores: Coverage, Conformity, Consistency, Conciseness (see Section 2 in (Kanithi et al., 2024)).
• Metrics: Closed-ended accuracy, F1, open-ended LLM-as-judge scoring (across 12 axes), pairwise win-rates/Elo, and safety violation rates. Preference-aligned, medically fine-tuned 70B models (e.g., Med42-Llama3-70B) deliver top performance (Consistency ≥ 98%, Conformity ≥ 96%).
• Cost and Safety Tradeoffs: Mid-sized models offer good cost-utility tradeoffs; safety-focused models score ≤2 on harmfulness metrics, crucial for clinician-facing deployments.

By integrating scenario-rich expert-authored queries, claim-level attribution, explicit evidence verification, and sophisticated expertise modeling, the ExpertQA paradigm has enabled rigorous empirical progress in high-accuracy, trustworthy domain-specific QA—setting practical standards for LLM benchmarking, pipeline design, and system safety in critical applications.