---
title: 'ArchEHR-QA 2026: Modular Clinical QA'
url: https://www.emergentmind.com/topics/archehr-qa-2026
type: topic
---

# ArchEHR-QA 2026: Modular Clinical QA

ArchEHR-QA 2026 is a shared-task formulation of grounded clinical question answering over electronic health records that operationalizes patient-centered information seeking as a sequence of tightly coupled inference problems: clinician-style question interpretation, evidence sentence identification, grounded answer generation, and answer–evidence alignment. In contrast to earlier EHR-QA benchmarks centered on structured databases or executable queries, ArchEHR-QA focuses on patient-authored questions about hospitalization, MIMIC-derived clinical note excerpts, sentence-level support, and clinician-authored answers, making grounding and attribution first-class evaluation targets rather than auxiliary diagnostics [2506.04156].

## 1. Historical formation and benchmark identity

The immediate precursor to ArchEHR-QA 2026 is the ArchEHR-QA dataset introduced for patient-centered, EHR-grounded question answering about the clinical course of hospitalization. That dataset contains **134 patient cases**, comprising **104 ICU cases** using **MIMIC-III** notes and **30 ED cases** using **MIMIC-IV** notes. Each case links a patient question, a clinician-interpreted question, a clinical note excerpt with sentence-level relevance labels, and a clinician-authored answer [2506.04156].

The 2026 shared task expands this benchmark into **four subtasks**: **question interpretation**, **evidence identification**, **answer generation**, and **answer-evidence alignment**. In the shared-task descriptions, the benchmark is framed as grounded question answering over hospitalization-related EHR notes, with patient-authored questions serving as the entry point and explicit sentence-level support serving as the grounding substrate [2604.26880].

This four-part factorization is significant because it converts what had been a single grounded QA benchmark into a modular evaluation suite. The earlier ArchEHR-QA formulation already measured both answer quality and evidence overlap; the 2026 shared task externalizes those latent competencies into separate tracks. A plausible implication is that ArchEHR-QA 2026 is designed not merely to rank end-to-end answer generators, but to diagnose where a system fails: in semantic normalization, evidence recall, answer synthesis, or citation alignment.

## 2. Data schema, task interfaces, and supervision structure

ArchEHR-QA examples are organized around a patient information need grounded in a clinical note excerpt. The canonical fields are a **patient question**, a **clinician-interpreted question**, a **clinical note excerpt** segmented into numbered sentences, **sentence-level relevance labels**—**Essential**, **Supplementary**, and **Not-Relevant**—and a **clinician-authored answer** of approximately **75 words / 5 sentences** with citations to essential note sentences [2506.04156].

The 2026 task descriptions define the four subtasks as follows. **Subtask 1** converts a patient-authored narrative into a concise clinician-style question, constrained to **at most 15 words**. **Subtask 2** identifies the minimal supporting evidence at sentence granularity. **Subtask 3** generates a grounded answer of **at most 75 words**. **Subtask 4** aligns answer sentences to supporting note sentences via many-to-many links [2604.26880].

Neural1.5 reports a **Development set: 20 cases**, **Test set for Subtasks 1–3: 47 cases**, and **Test set for Subtask 4: 147 cases**, attributing the larger ST4 range to the **staged data release schedule** [2605.10877]. This indicates that the 2026 task was not only modular but also staged in its release mechanics.

| Subtask | Input focus | Official ranking signal |
|---|---|---|
| ST1 | Patient-authored question | Overall from ROUGE, BERTScore, AlignScore, MEDCON |
| ST2 | Question plus numbered note sentences | Strict Micro F1 |
| ST3 | Questions plus note evidence/context | Overall from BLEU, ROUGE, SARI, BERTScore, AlignScore, MEDCON |
| ST4 | Answer sentences plus numbered note sentences | Micro F1 |

This supervision regime is unusually explicit for clinical QA. Earlier dataset construction already required clinician review of sentence labels and answers, with clinicians agreeing with existing sentence labels **77.1%** of the time in one review round and **DDF** agreeing with clinician-modified labels **84.7%** of the time in a second round [2506.04156]. That annotation history matters because ArchEHR-QA 2026 inherits a benchmark culture in which disagreement is expected at the boundary between essential and supplementary evidence.

## 3. Evaluation philosophy: grounding, factuality, and answer quality

The evaluation logic of ArchEHR-QA derives from the original benchmark’s distinction between **Factuality** and **Relevance**. In the dataset paper, factuality is measured as an **F1 score** between the note sentences cited by the system and the gold sentence relevance annotations, while relevance is measured by comparing generated answers to clinician-authored reference answers using **BLEU**, **ROUGE**, **SARI**, **BERTScore**, **AlignScore**, and **MEDCON**. The overall score is defined as  
$$
\text{Overall Score} = \frac{\text{Overall Factuality} + \text{Overall Relevance}}{2}
$$  
[2506.04156].

ArchEHR-QA 2026 separates this compound objective into subtasks, but the underlying philosophy remains the same. ST2 isolates sentence-level evidence quality under **strict** and **lenient** matching, with **strict micro F1** used for ranking. ST3 scores answer generation with the multi-metric relevance bundle. ST4 evaluates answer–evidence links directly with **Micro F1** [2604.26880].

A central misconception the benchmark counters is that grounded clinical QA can be treated as ordinary answer generation. The dataset paper’s manual error analysis found recurring problems such as **omitted key clinical evidence**, **contradictory or hallucinated content**, **citation mismatch**, and **answer incompleteness** [2506.04156]. ArchEHR-QA 2026’s design turns those failure modes into measurable targets. ST2 tests whether the right note sentences were found at all; ST4 tests whether answer claims are attached to the correct evidence rather than merely accompanied by plausible-looking citations.

This evaluation philosophy also explains why some apparently strong systems perform unevenly across subtasks. A system can be fluent in ST3 yet weak in ST2, or precise in ST4 yet recall-limited. ArchEHR-QA 2026 therefore treats grounding as a structured, decomposable property rather than as a side effect of careful prompting.

## 4. Dominant methodological paradigms in 2026

The 2026 submissions converged on a small number of architectural patterns, each representing a distinct theory of how grounded clinical QA should be decomposed.

**Cascaded LLM pipelines** treat the task as a serial workflow in which each stage reduces ambiguity for the next. HealthNLP_Retrievers is the clearest example: a **four-stage cascaded LLM pipeline** built around **Gemini 2.5 Pro** with a **few-shot query reformulation unit**, a **heuristic-based evidence scorer** using **Likert-style relevance score from 1 to 5**, a grounded response generator capped at **75 words**, and a many-to-many alignment module producing a **JSON array mapping answer sentences to evidence sentence indices**. Its evidence filter prefers scores **at least 4** and backs off to **at least 3** if necessary, embodying an explicitly recall-biased retrieval policy [2604.26880].

**Stage-specific prompt optimization** treats each subtask as a separate prompting problem with its own objective. Neural1.5 defines each stage as a DSPy program and uses **MIPROv2** to optimize **instructions** and **few-shot demonstrations** separately for ST1–ST3. It then adds stage-specific reliability mechanisms: **majority-vote self-consistency** for ST2 with  
$$
v_i = \sum_{r=1}^{R} \hat{y}_i^{(r)}, \quad \hat{y}_i = \begin{cases} 1 & \text{if } v_i \ge \lceil R/2 \rceil, \\ 0 & \text{otherwise} \end{cases}
$$  
with \(R=5\), claim-level consolidation for ST3, and a three-stage alignment pipeline for ST4 with self-reflection, chain-of-verification, and a confidence threshold \(\tau_c = 0.9\) [2605.10877].

**Prompt-only low-resource orchestration** emphasizes zero-shot or few-shot adaptation without weight updates. BIT.UA-AAUBS systematically evaluated proprietary and open-source LLMs with **task decomposition**, **Chain-of-Thought**, **in-context learning**, **majority voting**, and **LLM-as-a-judge** ensembling. Its study is especially important because it directly compares proprietary systems with locally deployable open-weight alternatives under **GDPR** and **HIPAA**-motivated privacy constraints, concluding that proprietary models are generally more robust to prompt variation while domain-adapted open models such as **MedGemma 27B** can be highly competitive with the right prompt [2605.03618].

**Local-only deployment pipelines** ask how far the task can be pushed without external APIs. The sebis system ran all experiments locally on **Apple MacBook M4 Pro** and **Apple Mac Studio M3 Max** hardware, using **Qwen3-Embedding-8B** for evidence tasks, quantized local generators for ST1 and ST3, and prompted alignment models for ST4. Its results support the claim that privacy-preserving ArchEHR-QA systems running fully locally are feasible on commodity hardware, though with uneven performance across subtasks [2603.13962].

**Deterministic grounding with multi-pass alignment** is exemplified by Yale-DM-Lab. For ST2–ST4 it uses Azure-hosted ensembles, few-shot prompting, vote aggregation, and post-processing, while ST4 adds the **full clinician answer paragraph** as prompt context and performs recall-oriented augmentation after initial alignment. The system reports development performance of **88.81 micro F1 on ST4**, supporting the paper’s claim that alignment accuracy is largely limited by reasoning rather than formatting alone [2604.07116].

Across these paradigms, a shared pattern is visible: early stages often bias toward recall, later stages toward precision. This suggests that ArchEHR-QA 2026 is not naturally solved by a monolithic prompt. The benchmark structurally rewards systems that separate semantic normalization, evidence recall, answer drafting, and explicit verification.

## 5. Competitive landscape and reported results

The 2026 leaderboard landscape is best understood as subtask-specialized rather than dominated by a single universal architecture.

| System | Standout result | Broader profile |
|---|---|---|
| HealthNLP_Retrievers | **1st** in ST1, **Overall = 31.2** | Also **5th** ST3, **7th** ST2, **9th** ST4 |
| Neural1.5 | **1st** in ST2, **Strict Micro F1 = 63.7** | **4th** ST1, **4th** ST3, **7th** ST4; mean rank **4.00** |
| BIT.UA-AAUBS | **1st** in ST4, **Micro F1 = 81.5** | **3rd** ST3 with **35.6** |

HealthNLP_Retrievers achieved **rank 1 in question interpretation**, **rank 5 in answer generation**, **rank 7 in evidence identification**, and **rank 9 in answer-evidence alignment**. Its official ST1 score was **Overall = 31.2** with **BERTScore = 46.8**; its ST2 score was **Strict Micro F1 = 60.2** and **Lenient Micro F1 = 68.3**; and its ST3 overall score was **34.6** [2604.26880].

Neural1.5 was the top ST2 system, with **Strict Micro F1 = 63.7**, **Strict Micro Precision = 60.2**, **Strict Micro Recall = 67.6**, and **Lenient Micro F1 = 72.4**. It also reported **Overall = 28.9** in ST1, **Overall = 35.2** in ST3, and **Micro F1 = 78.6** in ST4. Its mean rank across the four subtasks was **4.00**, which the authors describe as **second overall among teams that participated in all four subtasks** [2605.10877].

BIT.UA-AAUBS was strongest on the downstream grounded-output tasks. It placed **3rd** in ST3 with **35.6** and **1st** in ST4 with **81.5 Micro F1**. Its best ST4 submission was a **3-model majority voting ensemble** using **Gemini 2.5 Flash**, **Claude Opus 4.6**, and **Gemini 2.0 Flash** with **Prompt 5** [2605.03618].

The local-only sebis study did not top the leaderboard, but it showed that smaller local systems can remain competitive. In ST2, **Qwen3-Embedding-8B** reached **51.61** on test, outperforming the paper’s fine-tuned Bio_ClinicalBERT-based alternative. In ST4, **Qwen3.5-35B list-wise** prompting reached **74.84**, while a fine-tuned Bio_ClinicalBERT model collapsed to **8.33** on test, which the authors attribute to distribution shift and overfitting [2603.13962].

An important empirical pattern is that answer generation and answer–evidence alignment favored strong prompting and ensemble control, while evidence identification remained more brittle. This is consistent with participant analyses that retrieval-like sentence selection is less naturally aligned with pure prompting than answer drafting or structured citation alignment.

## 6. Research significance, limitations, and relation to adjacent EHR-QA traditions

ArchEHR-QA 2026 occupies a distinctive position within the broader EHR-QA literature. Earlier structured EHR-QA benchmarks emphasized natural-language-to-query translation, including **NLQ \(\rightarrow\) SQL** and **NLQ \(\rightarrow\) SPARQL** over MIMIC-derived relational or knowledge-graph representations [2010.09394]. Subsequent work argued for **NLQ2Program** over graph-based EHRs, using typed executable operations such as `gen_entset_down`, `gen_litset`, `intersect_entsets`, and `count_entset`, and showed that **data uncertainty is most indicative of the ambiguity in the input question** [2203.06918]. A different line extended EHR-QA to table–image multimodality through **EHRXQA**, which combines structured EHR tables with chest X-rays and executable **NeuralSQL** [2310.18652]. ArchEHR-QA 2026 instead focuses on patient-authored, note-grounded, sentence-attributed QA, making it complementary rather than redundant.

The benchmark also inherits lessons from ArchEHR-QA 2025. UTSA-NLP showed that **sentence selection is the main bottleneck**, that **self-consistency plus thresholding** materially improves evidence classification, and that **LLaMA 3.1 8B** outperformed **70B** for sentence identification in that setting [2506.05589]. ArgHiTZ showed that a **two-step divide-and-conquer** pipeline with reranking-based sentence selection achieved **overall score of 0.44**, ranked **8th out of 30**, and secured the **top position in overall factuality**, reinforcing the primacy of evidence selection over purely end-to-end answer drafting [2506.12886]. These findings align closely with the 2026 shared task’s decomposition.

Several limitations are now well characterized. First, annotation boundaries between **essential** and **supplementary** evidence remain difficult, and multiple papers report over-selection or under-selection around those categories [2506.04156]. Second, error propagation is intrinsic to cascaded pipelines: an over-compressed clinician question can suppress evidence recall, which can then distort both generation and alignment [2604.26880]. Third, prompt-only systems remain fragile on extraction-heavy tasks, even when they are strong on generation and citation alignment [2605.03618]. Fourth, local-only systems are feasible but still sensitive to distribution shift and low-resource overfitting [2603.13962].

A broader implication is that ArchEHR-QA 2026 is less a single task than a research program about grounded clinical communication. It tests whether a system can normalize noisy patient language, retrieve minimal support, write a concise answer, and prove that the answer is supported. This suggests that future work will likely move toward cross-stage feedback, ambiguity detection, calibration, and perhaps interactive clarification. The benchmark’s design already points in that direction by making grounding failures visible rather than collapsing them into a single end-to-end score.

Source: https://www.emergentmind.com/topics/archehr-qa-2026