Real-POCQi: Real-World Clinical Queries

• Real-POCQi are genuine clinical queries characterized by unstructured language, ambiguity, and diverse inquiry types at the point of care.
• Datasets like QuarkMedBench and Real-POCQi collect thousands of authentic, multi-turn queries to rigorously benchmark medical AI systems.
• Advanced evaluation methods, including automated rubric scoring and dynamic knowledge calibration, enhance model safety and real-world performance.

Real-world Point-Of-Care Queries (Real-POCQi) represent the genuine, context-rich, and frequently unstructured inquiries posed by clinicians, healthcare professionals, and sometimes patients during clinical workflows. Unlike exam-style or template-based questions, Real-POCQi encompass the complexity, ambiguity, and diversity of authentic clinical information needs at the bedside, in outpatient settings, and across patient–healthcare-provider interactions. The study of Real-POCQi is central to the development, benchmarking, and safe deployment of medical AI systems, as these queries drive the information-seeking and decision support tasks most critical to health outcomes.

1. Definition and Characteristics of Real-POCQi

Real-POCQi are ecologically valid clinical questions, capturing the true spectrum of information sought at the point of care. Their defining features include:

• Colloquial, Unstructured Language: Real-POCQi are short (mean ≈ 11.7 characters in Chinese; varied in English) and often lack standardized phrasing, reflecting the brevity or urgency of clinical interactions.
• Ambiguity and Imprecise Intent: These questions may embed ambiguous intent, lack key details, or assume shared context between questioner and answerer.
• Long-Tail Distribution and Heterogeneity: The vast majority represent low-frequency, “long-tail” needs (i.e., questions not well represented in training data or exam sets), spanning rare conditions, nuanced patient scenarios, or combinations thereof.
• Diverse Clinical Domains: Governed by the demands of active patient management, Real-POCQi span clinical care (acute/chronic disease management), wellness, and professional/procedural inquiry. For example, in the QuarkMedBench benchmark: 66.2% are clinical care, 27.6% wellness, 6.1% professional inquiry (Wu et al., 14 Mar 2026).
• Multi-Turn, Contextual Sequences: Many queries occur as part of dialogic, multi-turn episodes, requiring the system to track context or resolve temporally unfolding clinical problems (Wu et al., 14 Mar 2026).

2. Datasets and Benchmarks for Real-POCQi

Multiple initiatives have sought to capture, curate, and benchmark Real-POCQi:

QuarkMedBench

• Comprises 20,821 single-turn queries and 3,853 multi-turn sessions (6,215 turns), reflecting authentic, unfiltered point-of-care medical questions (Wu et al., 14 Mar 2026).
• Queries are annotated and decomposed using an automated rubric pipeline that yields an average of 9.8 rubrics/query (single-turn) for granular evaluation of LLM-generated clinical responses.
• The dataset intentionally mimics the heterogeneity, frequency distribution, and real-world phrasing of practitioner and patient queries.

Real-POCQi Evaluation Set (OpenEvidence)

• Drawn from >1 million physician-submitted queries per week on a live clinical decision support platform, with 620 fully annotated, specialty-matched questions for head-to-head AI evaluation (Feng et al., 27 Jun 2026).
• Covers 30 specialties, stratified by explicit patient context and general clinical questions, systematically de-identified and pre-processed to remove protected health information while preserving clinical content.

RealMedQA, DiSCQ, FHIRPath-QA

• RealMedQA: 230 “ideal” QA pairs focusing on plausibly realistic, guideline-anchored questions challenging for retrieval models due to lower lexical overlap between questions and answers (Kell et al., 2024).
• DiSCQ: 2,029 truly physician-authored questions linked to their textual triggers in discharge summaries, emphasizing the spectrum of inquiry types encountered by clinicians during handoff and documentation (Lehman et al., 2022).
• FHIRPath-QA: 14,295 question-query pairs mapping natural questions to executable FHIRPath logic, focusing on patient-specific queries over EHRs (Frew et al., 26 Feb 2026).

Typical distributions are highlighted below:

Dataset	# Single-turn	# Multi-turn	Domain Spread (Primary)	Key Notes
QuarkMedBench	20,821	3,853	66% Clinical Care	Fine-grained rubrics
Real-POCQi (OE)	620	–	30 specialties	Expert-rated
DiSCQ	2,029	–	Handoff/summary focused	Linked text “triggers”
RealMedQA	230	–	NICE disease guidelines	Low lexical overlap
FHIRPath-QA	14,295	–	EHR patient questions	FHIRPath execution

3. Evaluation Frameworks and Methodologies

Robust evaluation of Real-POCQi demands adaptation well beyond multiple-choice or free-form metrics:

Automated Rubric-Based Scoring

• QuarkMedBench employs dynamic, multi-stage rubric generation and scoring:
  1. Candidate Response Pooling: Five LLMs × three prompt perspectives generate a wide response pool.
  2. Ground Truth Fusion: Reference answers synthesized by fusing retrieved evidence from guidelines/literature (retrieval agent $R$ aggregates $D(q)$).
  3. Hierarchical Rubric Extraction: Responses dissected into Essential, Important, Highlight, and Pitfall elements, each weighted (default 2:1:1:2 for w_acc : w_kp : w_risk).
  4. Safety Constraints: Truncation and saturation mechanisms hard-cap total and sub-dimension scores—critical for preventing unsafe output and “reward hacking.”
  5. Dynamic Knowledge Calibration: Novel or low-score queries trigger deeper retrieval and, if needed, human expert intervention for rubric revision.
  6. Concordance Testing: Automated rubric scoring achieves 91.8% concordance with blinded tertiary-hospital clinicians, establishing validity (Wu et al., 14 Mar 2026).

Expert and Multi-Dimensional Human Judging

• Real-POCQi benchmark assesses accuracy, clinical utility, source quality, verifiability, and completeness, with each axis rated by specialty-matched practicing physicians (Feng et al., 27 Jun 2026).
• Pairwise win differences ($\Delta D_{A,B}$), one-vs-rest win rates, and rigorous statistical controls ensure reproducibility.
• Inter-rater agreement, time-to-score, and stratification by answer length and citation display are employed to audit fairness and reliability.

Query–Logic Mapping and Programmatic Evaluation

• FHIRPath-QA evaluates text-to-FHIRPath synthesis, defining execution accuracy (EA), exact match (EMA), and failure rates (FR) on executable queries over real EHR data (Frew et al., 26 Feb 2026).
• Evaluations underline model-specific brittleness in the face of ambiguity, layperson phrasing, and resource schema variability.

4. State-of-the-Art Performance and Model Disparities

Real-POCQi-centric benchmarks reveal persistent gaps between SOTA LLM performance and real clinical needs:

• Exam Scores vs. Real-World Utility: High exam-style test scores (e.g., MedQA) do not correlate with strong performance on Real-POCQi; models exhibit sensitivity to out-of-domain phrasing, ambiguous context, and incomplete information (Wu et al., 14 Mar 2026).
• Model Ranking Under Real-POCQi: OE (a specialized clinical tool) outperformed GPT-5.5, Gemini 3.1 Pro, and Claude Opus 4.8 across all human-rated axes by 25–39 percentage points ((Feng et al., 27 Jun 2026) Table 2, primary analysis), highlighting the benefits of domain-specific engineering.
• Dimension-Specific Gaps: While Essential (medical-accuracy) scores converge for top models (>88%), coverage of Highlight (secondary, nuanced elements) and minimization of Pitfall (risk) errors vary widely (Highlight: 56% vs. 30–45%; Pitfall penalty: –4% to –18%) (Wu et al., 14 Mar 2026).
• Chain-of-Thought (CoT) Architectures: CoT-enabled models (Qwen3-30B-Thinking) reduce safety violations and improve nuanced coverage.
• Length Bias: Models relying on verbosity lose 5–10% score when responses are curtailed to 1,000 words.

5. System Integration and Dynamic Update Mechanisms

Translating Real-POCQi handling into practical clinical informatics systems requires:

• Dynamic Rubric and Knowledge Update: Queries for which LLMs or retrieval agents fail to provide high-quality answers are automatically routed into a DeepResearch pipeline for new evidence retrieval, regenerating reference answers and rubrics until stable concordance is achieved. Persistently unsolvable queries are escalated to human-in-the-loop (HITL) review for guideline and rubric refinement (Wu et al., 14 Mar 2026).
• Real-Time EHR Integration: Systems such as Conquery and PHKG provide sub-second distributed querying for large-scale, structured EHRs, enabling clinicians to select cohorts or extract recent measurements at point-of-care intervals (Kovacs et al., 2020, Bloor et al., 2023).
• PIORS Frameworks: LLM-driven reception assistant architectures (e.g., PIORS) simulate complex, personalized outpatient interactions, integrating multi-agent dialogue, HIS API access, and service-flow modelling for end-to-end reception and patient triage (Bao et al., 2024).
• Hybrid Document and Evidence Retrieval: PICOs-RAG improves upon standard retrieval-augmented generation by normalizing queries to professional clinical phrasing and decomposing them into explicit PICO fields, boosting evidence retrieval efficiency and answer relevance (+6.2% – +8.8% by LLM-based evaluation, Chinese-language setting) (Sun et al., 28 Oct 2025).

6. Limitations, Challenges, and Future Directions

While significant progress has been made, Real-POCQi research faces inherent challenges:

• Ambiguity, Answerability, and Coverage: No current model fully solves the “answerability gap” driven by ambiguous, under-specified, or entirely unanswerable real-world queries. Models must incorporate clarification sub-dialogue (e.g., Medical Information Retrieval QA Loop (Sinhababu et al., 2021)) and answerability detection modules (Lehman et al., 2022).
• Dataset Limitations: Even large benchmarks (QuarkMedBench, Real-POCQi) cover only a fraction of full clinical practice, and template-driven questions (BioASQ, MedQA, FHIRPath-QA) under-represent linguistic diversity. Generation and verification at scale require sophisticated de-identification and audit controls (Kell et al., 2024, Wu et al., 14 Mar 2026).
• Evaluation Rigidity and Overfitting: Supervised fine-tuning on narrow schema or synthetic queries risks overfitting, degrading generalization to unseen query and EHR resource types (Frew et al., 26 Feb 2026).
• Human Factors and Clinical Integration: Safety, usability, and satisfaction (both patient and provider) must be monitored in prospective clinical trials before large-scale deployment. Ultimate generalizability will require multilingual, regionally diverse seed data and ongoing human-in-the-loop refinement (Bao et al., 2024).
• Continual Knowledge Update: Point-of-care systems must maintain up-to-date clinical knowledge via continuous evidence surveillance and automated rubric revision in response to new literature and guideline updates (Wu et al., 14 Mar 2026).

---

• "QuarkMedBench: A Real-World Scenario Driven Benchmark for Evaluating LLMs" (Wu et al., 14 Mar 2026)
• "Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries" (Feng et al., 27 Jun 2026)
• "RealMedQA: A pilot biomedical question answering dataset containing realistic clinical questions" (Kell et al., 2024)
• "Learning to Ask Like a Physician" (Lehman et al., 2022)
• "Towards a Personal Health Knowledge Graph Framework for Patient Monitoring" (Bloor et al., 2023)
• "Conquery: an open source application to analyze high content healthcare data" (Kovacs et al., 2020)
• "FHIRPath-QA: Executable Question Answering over FHIR Electronic Health Records" (Frew et al., 26 Feb 2026)
• "PIORS: Personalized Intelligent Outpatient Reception based on LLM with Multi-Agents Medical Scenario Simulation" (Bao et al., 2024)
• "PICOs-RAG: PICO-supported Query Rewriting for Retrieval-Augmented Generation in Evidence-Based Medicine" (Sun et al., 28 Oct 2025)
• "Medical Information Retrieval and Interpretation: A Question-Answer based Interaction Model" (Sinhababu et al., 2021)