ArchEHR-QA 2026: Modular Clinical QA
- ArchEHR-QA 2026 is a clinical benchmark that focuses on patient-authored questions over EHR notes, emphasizing sentence-level evidence and clinician interpretation.
- It modularizes the QA task into four subtasks—question interpretation, evidence identification, answer generation, and answer–evidence alignment—to pinpoint system weaknesses.
- The framework uses diverse evaluation metrics and methodologies, including cascaded LLM pipelines and local-only systems, to advance grounded clinical QA research.
ArchEHR-QA 2026 is a shared-task formulation of grounded clinical question answering over electronic health records that operationalizes patient-centered information seeking as a sequence of tightly coupled inference problems: clinician-style question interpretation, evidence sentence identification, grounded answer generation, and answer–evidence alignment. In contrast to earlier EHR-QA benchmarks centered on structured databases or executable queries, ArchEHR-QA focuses on patient-authored questions about hospitalization, MIMIC-derived clinical note excerpts, sentence-level support, and clinician-authored answers, making grounding and attribution first-class evaluation targets rather than auxiliary diagnostics (Soni et al., 4 Jun 2025).
1. Historical formation and benchmark identity
The immediate precursor to ArchEHR-QA 2026 is the ArchEHR-QA dataset introduced for patient-centered, EHR-grounded question answering about the clinical course of hospitalization. That dataset contains 134 patient cases, comprising 104 ICU cases using MIMIC-III notes and 30 ED cases using MIMIC-IV notes. Each case links a patient question, a clinician-interpreted question, a clinical note excerpt with sentence-level relevance labels, and a clinician-authored answer (Soni et al., 4 Jun 2025).
The 2026 shared task expands this benchmark into four subtasks: question interpretation, evidence identification, answer generation, and answer-evidence alignment. In the shared-task descriptions, the benchmark is framed as grounded question answering over hospitalization-related EHR notes, with patient-authored questions serving as the entry point and explicit sentence-level support serving as the grounding substrate (Hosen et al., 29 Apr 2026).
This four-part factorization is significant because it converts what had been a single grounded QA benchmark into a modular evaluation suite. The earlier ArchEHR-QA formulation already measured both answer quality and evidence overlap; the 2026 shared task externalizes those latent competencies into separate tracks. A plausible implication is that ArchEHR-QA 2026 is designed not merely to rank end-to-end answer generators, but to diagnose where a system fails: in semantic normalization, evidence recall, answer synthesis, or citation alignment.
2. Data schema, task interfaces, and supervision structure
ArchEHR-QA examples are organized around a patient information need grounded in a clinical note excerpt. The canonical fields are a patient question, a clinician-interpreted question, a clinical note excerpt segmented into numbered sentences, sentence-level relevance labels—Essential, Supplementary, and Not-Relevant—and a clinician-authored answer of approximately 75 words / 5 sentences with citations to essential note sentences (Soni et al., 4 Jun 2025).
The 2026 task descriptions define the four subtasks as follows. Subtask 1 converts a patient-authored narrative into a concise clinician-style question, constrained to at most 15 words. Subtask 2 identifies the minimal supporting evidence at sentence granularity. Subtask 3 generates a grounded answer of at most 75 words. Subtask 4 aligns answer sentences to supporting note sentences via many-to-many links (Hosen et al., 29 Apr 2026).
Neural1.5 reports a Development set: 20 cases, Test set for Subtasks 1–3: 47 cases, and Test set for Subtask 4: 147 cases, attributing the larger ST4 range to the staged data release schedule (Majeedi et al., 11 May 2026). This indicates that the 2026 task was not only modular but also staged in its release mechanics.
| Subtask | Input focus | Official ranking signal |
|---|---|---|
| ST1 | Patient-authored question | Overall from ROUGE, BERTScore, AlignScore, MEDCON |
| ST2 | Question plus numbered note sentences | Strict Micro F1 |
| ST3 | Questions plus note evidence/context | Overall from BLEU, ROUGE, SARI, BERTScore, AlignScore, MEDCON |
| ST4 | Answer sentences plus numbered note sentences | Micro F1 |
This supervision regime is unusually explicit for clinical QA. Earlier dataset construction already required clinician review of sentence labels and answers, with clinicians agreeing with existing sentence labels 77.1% of the time in one review round and DDF agreeing with clinician-modified labels 84.7% of the time in a second round (Soni et al., 4 Jun 2025). That annotation history matters because ArchEHR-QA 2026 inherits a benchmark culture in which disagreement is expected at the boundary between essential and supplementary evidence.
3. Evaluation philosophy: grounding, factuality, and answer quality
The evaluation logic of ArchEHR-QA derives from the original benchmark’s distinction between Factuality and Relevance. In the dataset paper, factuality is measured as an F1 score between the note sentences cited by the system and the gold sentence relevance annotations, while relevance is measured by comparing generated answers to clinician-authored reference answers using BLEU, ROUGE, SARI, BERTScore, AlignScore, and MEDCON. The overall score is defined as (Soni et al., 4 Jun 2025).
ArchEHR-QA 2026 separates this compound objective into subtasks, but the underlying philosophy remains the same. ST2 isolates sentence-level evidence quality under strict and lenient matching, with strict micro F1 used for ranking. ST3 scores answer generation with the multi-metric relevance bundle. ST4 evaluates answer–evidence links directly with Micro F1 (Hosen et al., 29 Apr 2026).
A central misconception the benchmark counters is that grounded clinical QA can be treated as ordinary answer generation. The dataset paper’s manual error analysis found recurring problems such as omitted key clinical evidence, contradictory or hallucinated content, citation mismatch, and answer incompleteness (Soni et al., 4 Jun 2025). ArchEHR-QA 2026’s design turns those failure modes into measurable targets. ST2 tests whether the right note sentences were found at all; ST4 tests whether answer claims are attached to the correct evidence rather than merely accompanied by plausible-looking citations.
This evaluation philosophy also explains why some apparently strong systems perform unevenly across subtasks. A system can be fluent in ST3 yet weak in ST2, or precise in ST4 yet recall-limited. ArchEHR-QA 2026 therefore treats grounding as a structured, decomposable property rather than as a side effect of careful prompting.
4. Dominant methodological paradigms in 2026
The 2026 submissions converged on a small number of architectural patterns, each representing a distinct theory of how grounded clinical QA should be decomposed.
Cascaded LLM pipelines treat the task as a serial workflow in which each stage reduces ambiguity for the next. HealthNLP_Retrievers is the clearest example: a four-stage cascaded LLM pipeline built around Gemini 2.5 Pro with a few-shot query reformulation unit, a heuristic-based evidence scorer using Likert-style relevance score from 1 to 5, a grounded response generator capped at 75 words, and a many-to-many alignment module producing a JSON array mapping answer sentences to evidence sentence indices. Its evidence filter prefers scores at least 4 and backs off to at least 3 if necessary, embodying an explicitly recall-biased retrieval policy (Hosen et al., 29 Apr 2026).
Stage-specific prompt optimization treats each subtask as a separate prompting problem with its own objective. Neural1.5 defines each stage as a DSPy program and uses MIPROv2 to optimize instructions and few-shot demonstrations separately for ST1–ST3. It then adds stage-specific reliability mechanisms: majority-vote self-consistency for ST2 with with , claim-level consolidation for ST3, and a three-stage alignment pipeline for ST4 with self-reflection, chain-of-verification, and a confidence threshold (Majeedi et al., 11 May 2026).
Prompt-only low-resource orchestration emphasizes zero-shot or few-shot adaptation without weight updates. BIT.UA-AAUBS systematically evaluated proprietary and open-source LLMs with task decomposition, Chain-of-Thought, in-context learning, majority voting, and LLM-as-a-judge ensembling. Its study is especially important because it directly compares proprietary systems with locally deployable open-weight alternatives under GDPR and HIPAA-motivated privacy constraints, concluding that proprietary models are generally more robust to prompt variation while domain-adapted open models such as MedGemma 27B can be highly competitive with the right prompt (Jonker et al., 5 May 2026).
Local-only deployment pipelines ask how far the task can be pushed without external APIs. The sebis system ran all experiments locally on Apple MacBook M4 Pro and Apple Mac Studio M3 Max hardware, using Qwen3-Embedding-8B for evidence tasks, quantized local generators for ST1 and ST3, and prompted alignment models for ST4. Its results support the claim that privacy-preserving ArchEHR-QA systems running fully locally are feasible on commodity hardware, though with uneven performance across subtasks (Yurt et al., 14 Mar 2026).
Deterministic grounding with multi-pass alignment is exemplified by Yale-DM-Lab. For ST2–ST4 it uses Azure-hosted ensembles, few-shot prompting, vote aggregation, and post-processing, while ST4 adds the full clinician answer paragraph as prompt context and performs recall-oriented augmentation after initial alignment. The system reports development performance of 88.81 micro F1 on ST4, supporting the paper’s claim that alignment accuracy is largely limited by reasoning rather than formatting alone (Irankhah et al., 8 Apr 2026).
Across these paradigms, a shared pattern is visible: early stages often bias toward recall, later stages toward precision. This suggests that ArchEHR-QA 2026 is not naturally solved by a monolithic prompt. The benchmark structurally rewards systems that separate semantic normalization, evidence recall, answer drafting, and explicit verification.
5. Competitive landscape and reported results
The 2026 leaderboard landscape is best understood as subtask-specialized rather than dominated by a single universal architecture.
| System | Standout result | Broader profile |
|---|---|---|
| HealthNLP_Retrievers | 1st in ST1, Overall = 31.2 | Also 5th ST3, 7th ST2, 9th ST4 |
| Neural1.5 | 1st in ST2, Strict Micro F1 = 63.7 | 4th ST1, 4th ST3, 7th ST4; mean rank 4.00 |
| BIT.UA-AAUBS | 1st in ST4, Micro F1 = 81.5 | 3rd ST3 with 35.6 |
HealthNLP_Retrievers achieved rank 1 in question interpretation, rank 5 in answer generation, rank 7 in evidence identification, and rank 9 in answer-evidence alignment. Its official ST1 score was Overall = 31.2 with BERTScore = 46.8; its ST2 score was Strict Micro F1 = 60.2 and Lenient Micro F1 = 68.3; and its ST3 overall score was 34.6 (Hosen et al., 29 Apr 2026).
Neural1.5 was the top ST2 system, with Strict Micro F1 = 63.7, Strict Micro Precision = 60.2, Strict Micro Recall = 67.6, and Lenient Micro F1 = 72.4. It also reported Overall = 28.9 in ST1, Overall = 35.2 in ST3, and Micro F1 = 78.6 in ST4. Its mean rank across the four subtasks was 4.00, which the authors describe as second overall among teams that participated in all four subtasks (Majeedi et al., 11 May 2026).
BIT.UA-AAUBS was strongest on the downstream grounded-output tasks. It placed 3rd in ST3 with 35.6 and 1st in ST4 with 81.5 Micro F1. Its best ST4 submission was a 3-model majority voting ensemble using Gemini 2.5 Flash, Claude Opus 4.6, and Gemini 2.0 Flash with Prompt 5 (Jonker et al., 5 May 2026).
The local-only sebis study did not top the leaderboard, but it showed that smaller local systems can remain competitive. In ST2, Qwen3-Embedding-8B reached 51.61 on test, outperforming the paper’s fine-tuned Bio_ClinicalBERT-based alternative. In ST4, Qwen3.5-35B list-wise prompting reached 74.84, while a fine-tuned Bio_ClinicalBERT model collapsed to 8.33 on test, which the authors attribute to distribution shift and overfitting (Yurt et al., 14 Mar 2026).
An important empirical pattern is that answer generation and answer–evidence alignment favored strong prompting and ensemble control, while evidence identification remained more brittle. This is consistent with participant analyses that retrieval-like sentence selection is less naturally aligned with pure prompting than answer drafting or structured citation alignment.
6. Research significance, limitations, and relation to adjacent EHR-QA traditions
ArchEHR-QA 2026 occupies a distinctive position within the broader EHR-QA literature. Earlier structured EHR-QA benchmarks emphasized natural-language-to-query translation, including NLQ SQL and NLQ SPARQL over MIMIC-derived relational or knowledge-graph representations (Park et al., 2020). Subsequent work argued for NLQ2Program over graph-based EHRs, using typed executable operations such as gen_entset_down, gen_litset, intersect_entsets, and count_entset, and showed that data uncertainty is most indicative of the ambiguity in the input question (Kim et al., 2022). A different line extended EHR-QA to table–image multimodality through EHRXQA, which combines structured EHR tables with chest X-rays and executable NeuralSQL (Bae et al., 2023). ArchEHR-QA 2026 instead focuses on patient-authored, note-grounded, sentence-attributed QA, making it complementary rather than redundant.
The benchmark also inherits lessons from ArchEHR-QA 2025. UTSA-NLP showed that sentence selection is the main bottleneck, that self-consistency plus thresholding materially improves evidence classification, and that LLaMA 3.1 8B outperformed 70B for sentence identification in that setting (Shields-Menard et al., 5 Jun 2025). ArgHiTZ showed that a two-step divide-and-conquer pipeline with reranking-based sentence selection achieved overall score of 0.44, ranked 8th out of 30, and secured the top position in overall factuality, reinforcing the primacy of evidence selection over purely end-to-end answer drafting (Cuadrón et al., 15 Jun 2025). These findings align closely with the 2026 shared task’s decomposition.
Several limitations are now well characterized. First, annotation boundaries between essential and supplementary evidence remain difficult, and multiple papers report over-selection or under-selection around those categories (Soni et al., 4 Jun 2025). Second, error propagation is intrinsic to cascaded pipelines: an over-compressed clinician question can suppress evidence recall, which can then distort both generation and alignment (Hosen et al., 29 Apr 2026). Third, prompt-only systems remain fragile on extraction-heavy tasks, even when they are strong on generation and citation alignment (Jonker et al., 5 May 2026). Fourth, local-only systems are feasible but still sensitive to distribution shift and low-resource overfitting (Yurt et al., 14 Mar 2026).
A broader implication is that ArchEHR-QA 2026 is less a single task than a research program about grounded clinical communication. It tests whether a system can normalize noisy patient language, retrieve minimal support, write a concise answer, and prove that the answer is supported. This suggests that future work will likely move toward cross-stage feedback, ambiguity detection, calibration, and perhaps interactive clarification. The benchmark’s design already points in that direction by making grounding failures visible rather than collapsing them into a single end-to-end score.