MedIQA: Medical NLP & Imaging Benchmarks

• MedIQA is a benchmark ecosystem defined by evolving tasks in medical NLP and multimodal clinical AI, covering inference, summarization, correction, and extraction.
• It began in 2019 with challenges in natural language inference, question entailment, and answer ranking, later expanding to dialogue summarization, error detection, and multimodal answer generation.
• The ecosystem now includes a distinct prompt-driven foundation model for medical image quality assessment, underscoring its dual role in clinical text analysis and imaging evaluation.

Searching arXiv for recent and foundational MEDIQA-related papers to ground the article. First, I’ll look for MEDIQA overview and shared-task papers spanning 2019–2025. Searching arXiv for “MEDIQA shared task medical question answering”. MedIQA denotes an evolving benchmark and shared-task ecosystem in medical natural language processing and multimodal clinical AI, originally organized around medical natural language inference, recognizing question entailment, and medical question answering, and later extended to dialogue summarization, clinical note error correction, multilingual and multimodal answer generation in dermatology, wound-care visual question answering, and medical order extraction from doctor–patient consultations (Bandyopadhyay et al., 2019, Mathur et al., 2023, Toma et al., 2024, Saeed, 2024, Karim et al., 12 Oct 2025, Corbeil et al., 30 Oct 2025). In a distinct later usage, “MedIQA” also names a 2025 prompt-driven foundation model for medical image quality assessment (Xun et al., 25 Jul 2025). This suggests that the term functions both as the name of a shared-task lineage and, separately, as the title of a medical imaging model.

1. Historical scope and evolution

The 2019 MEDIQA shared task defined three connected subtasks: medical Natural Language Inference, Recognizing Question Entailment, and medical Question Answering (Bandyopadhyay et al., 2019). In that setting, NLI tests whether one medical sentence entails, contradicts, or is neutral with respect to another; RQE asks whether one medical question entails another question; and QA uses such semantic understanding to filter and rerank candidate answers retrieved by CHiQA (Bandyopadhyay et al., 2019). Early MEDIQA systems therefore treated clinical inference, question understanding, and answer selection as parts of one medical NLU stack rather than as isolated benchmarks (Pugaliya et al., 2019, Xu et al., 2019).

Later shared tasks widened the operational scope. MEDIQA-Chat 2023 introduced Dialogue2Note summarization, with Task A for section-specific clinical summarization and section header classification, and Task B for full-note summarization (Mathur et al., 2023). MEDIQA-CORR 2024 shifted the focus to clinical note auditing through error flag prediction, error sentence detection, and corrected sentence generation (Toma et al., 2024, Gundabathula et al., 2024). MEDIQA-M3G 2024 addressed multilingual and multimodal medical answer generation in dermatology (Saeed, 2024, Xie et al., 2024). MEDIQA-WV 2025 formulated wound-care medical visual question answering as joint free-text response generation plus structured wound-attribute prediction (Karim et al., 12 Oct 2025). MEDIQA-OE 2025 moved further toward workflow automation by extracting medical orders from doctor–patient consultations into structured JSON records (Corbeil et al., 30 Oct 2025).

Year	MEDIQA usage	Core problem
2019	MEDIQA shared task	NLI, RQE, QA
2023	MEDIQA-Chat	Dialogue2Note summarization
2024	MEDIQA-CORR	Error detection and correction
2024	MEDIQA-M3G	Multilingual multimodal dermatology QA
2025	MEDIQA-WV	Wound-care multimodal VQA
2025	MEDIQA-OE	Medical order extraction
2025	“MedIQA” model	Medical image quality assessment

This chronology indicates a steady migration from sentence-pair semantics toward clinically actionable structured generation over longer, noisier, and more multimodal inputs.

2. Canonical task families

The earliest MedIQA tasks are sentence-pair inference problems. MedNLI is a three-way classification problem over entailment, contradiction, and neutral labels (Kearns et al., 2019, Bandyopadhyay et al., 2019). RQE is a binary entailment problem over medical question pairs, where a question $A$ entails question $B$ if every answer to $B$ is also a complete or partial answer to $A$ (Bandyopadhyay et al., 2019). In this formulation, question entailment is a retrieval-validation layer for medical QA rather than a standalone semantic similarity task. The “Medical Knowledge-enriched Textual Entailment framework” makes this explicit by positioning MEDIQA-RQE as a filtering step for consumer health questions matched to FAQ-style questions (Yadav et al., 2020).

The 2019 QA task is not free-form generation. It is a ranking problem over candidate answers produced by CHiQA, with answer quality supervision in four levels and official metrics including Accuracy, Spearman’s Rank Correlation Coefficient, MRR, and Precision (Bandyopadhyay et al., 2019, Pugaliya et al., 2019). Pentagon’s system shows how Task 3 can be reframed as a composition of auxiliary inference signals: RQE retrieves entailed questions, NLI compares candidate answers against retrieved evidence answers, and a multi-task architecture jointly learns answer filtering and pairwise reranking (Pugaliya et al., 2019).

Subsequent MedIQA tasks expanded the output space. MEDIQA-Chat 2023 recast the problem as note generation from clinical conversations, with section-wise and full-note summarization (Mathur et al., 2023). MEDIQA-CORR 2024 defined a three-stage correction problem over clinical notes: detect whether an error exists, identify the sentence containing the error, and generate the corrected sentence (Toma et al., 2024, Saeed, 2024). MEDIQA-M3G 2024 required multilingual multimodal answer generation in dermatology from images, user queries, and contextual text, with outputs in English, Chinese, and Spanish (Saeed, 2024). MEDIQA-WV 2025 required systems to generate both a patient-facing free-text response and a structured tuple of wound attributes from one or more wound images plus a bilingual query (Karim et al., 12 Oct 2025). MEDIQA-OE 2025 required a JSON list of medical orders, each containing description, order_type, reason, and provenance, extracted from consultation transcripts (Corbeil et al., 30 Oct 2025).

A common misconception is that MedIQA refers only to question answering in the narrow sense of answer retrieval. The benchmark lineage now includes inference, summarization, correction, multimodal generation, and structured extraction. A second misconception is conflation with “MeDiaQA,” which is a distinct 2021 question answering dataset on real online medical dialogues containing 22k multiple-choice questions for over 11k dialogues with 120k utterances (Suri et al., 2021).

3. Datasets and evaluation regimes

The benchmark family is characterized by heterogeneous data regimes. In MEDIQA 2019, the IITP systems report 14,049 MedNLI training pairs and 405 test pairs for NLI, 8,588 training pairs, 302 validation pairs, and 230 test pairs for RQE, and QA training data from LiveQAMed and Alexa with 104 questions in each file, plus a 25-question validation set and a test set described as 150 question pairs with an average of 8.5 answers per question (Bandyopadhyay et al., 2019). UW-BHI’s MedNLI analysis further notes that all 405 test hypotheses were manually annotated for semantic focus and tense (Kearns et al., 2019).

Dialogue summarization introduced a markedly different scale profile. MEDIQA-Chat 2023 used 1,200 Task A conversations and only 67 Task B conversations, which helps explain the appeal of retrieval-augmented in-context learning over large-scale supervised training (Mathur et al., 2023). MEDIQA-CORR 2024 used 2,189 MS training texts, 574 MS validation texts, 160 UW validation texts, and a 925-text test set spanning MS and UW, with each note containing either one or no medical error (Gundabathula et al., 2024). MEDIQA-M3G 2024 used 842 training instances, 56 validation instances, and 100 test instances in DermaVQA, with multilingual training data machine-translated for non-English languages and validation/test data human-translated (Saeed, 2024). MEDIQA-WV 2025 is smaller still, with 279 training encounters, 105 validation encounters, and 93 test encounters, many with multiple images and severe label imbalance such as traumatic wounds dominating at 85.9% and not_infected at 84.6% (Karim et al., 12 Oct 2025). MEDIQA-OE 2025 used 64 training samples, 100 development samples, and 100 test samples, with 255 medical orders in the test set (Corbeil et al., 30 Oct 2025).

Evaluation varies accordingly. Early MEDIQA relied heavily on accuracy for NLI and RQE, and on Accuracy, Spearman’s Rho, MRR, and Precision for QA (Bandyopadhyay et al., 2019, Pugaliya et al., 2019). MEDIQA-Chat 2023 used ROUGE, BERTScore, and BLEURT for summary generation (Mathur et al., 2023). MEDIQA-CORR 2024 evaluated correction with an aggregate over ROUGE-1 F1, BERTScore F1, and BLEURT-20, alongside accuracy for error flagging and sentence localization (Gundabathula et al., 2024, Gema et al., 2024). MEDIQA-M3G 2024 used DeltaBLEU and BERTScore (Saeed, 2024, Xie et al., 2024). MEDIQA-WV 2025 combined deltaBLEU, ROUGE variants, two BERTScore variants, and three LLM-as-judge metrics into an overall Avg score (Karim et al., 12 Oct 2025). MEDIQA-OE 2025 first matched predicted and reference orders based on description field word overlap, then scored description and reason with unigram ROUGE F1, type with accuracy, provenance with provenance-label F1, and ranked systems by the arithmetic average of description, reason, type, and provenance (Corbeil et al., 30 Oct 2025).

These dataset and metric choices have methodological consequences. Small, noisy, or skewed datasets favor transfer, retrieval, and prompting; structured outputs require strong schema control; and leaderboard behavior depends not only on semantic correctness but also on output format, provenance, and overlap-sensitive metrics.

4. Methodological patterns

A persistent methodological theme is transfer learning from adjacent domains. DoubleTransfer combines MT-DNN and SciBERT so that one source contributes general NLU competence and the other contributes scientific-domain language representations; its multi-source transfer and multi-task fine-tuning strategy uses a mixture ratio $\alpha = 0.5$ for external mini-batches (Xu et al., 2019). IITP’s MEDIQA 2019 systems similarly show the importance of domain-adapted PLMs: moving from bert-base-uncased to BioBERT improved NLI accuracy from 71.7% to 77.1%, and further to 80.3% when trained on the full 14,049 MedNLI pairs (Bandyopadhyay et al., 2019). UW-BHI’s MedNLI comparison fixed the ESIM backbone and varied only the representation layer, showing complementary behaviors for ClinicalBERT, ESP, and Cui2Vec (Kearns et al., 2019).

A second theme is explicit knowledge integration. Sem-KGN augments BERT with a graph-expanded knowledge encoder built from a vocabulary graph enriched with SNOMED-CT, ICD-10, UMLS, and Clinical Trials concepts, then fuses local and global representations with a multi-headed self-attention aggregator (Yadav et al., 2020). In practical terms, this targets two failure modes emphasized throughout MedIQA: synonymy and latent medical relations, and mismatches in question focus or intent.

A third theme is retrieval-augmented prompting. SummQA retrieves semantically similar dialogues using MiniLM embeddings and cosine similarity, then inserts the top-$k$ dialogue-summary pairs as in-context examples for GPT-4, with $k=7$ for Task A and $k=1$ for Task B due to context limits (Mathur et al., 2023). WangLab’s MEDIQA-CORR system uses DSPy to optimize prompt programs and few-shot examples, pairing a retrieval-based MS pipeline with a detect-localize-correct UW pipeline (Toma et al., 2024). MasonNLP’s MEDIQA-WV system uses FAISS over MiniLM text embeddings and CLIP image embeddings, retrieves top-2 exemplars, and inserts them into a multimodal prompt for meta-llama/Llama-4-Scout-17B-16E-Instruct without extra training (Karim et al., 12 Oct 2025).

A fourth theme is agentic and ensemble control. PromptMind adds explicit error categorization to MEDIQA-CORR, then uses GPT-4 self-consistency and a GPT-4/Claude-3 Opus agreement rule to improve reliability (Gundabathula et al., 2024). Edinburgh Clinical NLP guides GPT-4 with hints from a fine-tuned BioLinkBERT-large span detector and also studies a multiple-choice correction formulation (Gema et al., 2024). IryoNLP builds a four-agent architecture—MedReAct, MedEval, MedReFlex, and MedFinalParser—around ClinicalCorp and MedWiki, turning clinical note correction into an iterative retrieval, reflection, and review loop (Corbeil, 2024). In dermatology, WangLab’s MEDIQA-M3G system splits answer generation into two Claude 3 Opus calls—first generating a differential diagnosis from images, then compressing that differential to a single diagnosis label—while a second system recasts the task as CLIP-style image–disease retrieval (Xie et al., 2024). ImageCLEF MEDIQA-MAGIC 2025 extends this reasoning-centric pattern with a structured adjudication layer and agentic RAG over an American Academy of Dermatology corpus (Thakrar et al., 7 Jul 2025).

Across these systems, MedIQA has favored compositional designs: retrieval plus prompting, knowledge graphs plus sequence encoders, auxiliary discriminators plus LLMs, and structured post-processing to enforce ontology and schema constraints.

5. Empirical findings and benchmark dynamics

The benchmark history yields several stable empirical patterns. In MEDIQA 2019, BioBERT-based and multi-source transfer systems established strong baselines. IITP reported 81.8% accuracy for NLI, 53.2% for RQE, and 71.7% for QA (Bandyopadhyay et al., 2019). DoubleTransfer ranked first on the QA task with 78.0 accuracy, 81.91 precision, and 0.937 MRR, while Pentagon achieved the highest ranking-oriented Task 3 metrics with Spearman’s Rho 0.338 and MRR 0.9622 by combining answer filtering and reranking with NLI and RQE signals (Xu et al., 2019, Pugaliya et al., 2019). In MedNLI representation analysis, UW-BHI found 81.2% accuracy with BERT, 77.8% with ESP, and 65.2% with Cui2Vec, while also observing that contradiction was the easiest label for all three representations (Kearns et al., 2019). On MEDIQA-RQE, Sem-KGN reached 56.17 accuracy versus 47.90 for BERT and 51.30 for a prior BERT plus linear projection baseline listed in the paper (Yadav et al., 2020).

For generation tasks, retrieval and prompting repeatedly displaced small fine-tuned baselines. SummQA’s GPT-4 retrieval pipeline placed 3rd on Task A section-wise summarization and 4th on Task B division-wise summarization, and outperformed a fine-tuned T5-small baseline on Task B validation metrics (Mathur et al., 2023). In MEDIQA-CORR 2024, WangLab ranked first on all three subtasks with 86.5% error-flag accuracy, 83.6% error sentence detection accuracy, and 0.789 Aggregate-Score for correction generation (Toma et al., 2024). PromptMind’s ablation showed that adding error categories to GPT-3.5 improved Task 1 from 48.75% to 58.44% and Task 2 from 22.5% to 38.55% on the combined validation sets, while its best system reached 2nd place on Task 3 and top-3 performance on Task 2 (Gundabathula et al., 2024). Edinburgh Clinical NLP showed that span hints from a smaller model could sharply improve aggregate correction quality, with 8-shot plus Brief CoT plus hints yielding the best test Scoreagg of 0.6634 among its runs and ranking 6th overall (Gema et al., 2024). IryoNLP’s MedReAct’N’MedReFlex placed 9th with an aggregation score of 0.581 (Corbeil, 2024). MediFact, by contrast, was strongest on structured detection, ranking 2nd in Error Flag accuracy and 8th in Error Sentence detection but 14th in NLG aggregate score (Saeed, 2024). This pattern suggests that correction generation and structured detection are separable competencies.

Multimodal benchmarks revealed more task-specific behavior. WangLab’s English MEDIQA-M3G system ranked 1st with a two-stage image-only Claude 3 Opus pipeline scoring dBLEU 10.415, while its CLIP-style retrieval system ranked 2nd (Xie et al., 2024). The same paper argues that the evaluation setup rewarded short disease-label outputs more than fuller answers. This suggests a metric-task mismatch rather than a solved clinical dialogue problem. MediFact-M3G achieved 7th rank in English and 3rd rank in Chinese and Spanish among 75 participants through a modular VGG16-CNN-SVM, QA, and CLIP-based pipeline (Saeed, 2024). In MEDIQA-WV 2025, MasonNLP’s lightweight multimodal RAG system ranked 3rd among 19 teams with Avg 41.37%, outperforming zero-shot and few-shot prompting substantially and reducing hallucinated infection assertions from 31 of 93 zero-shot answers to 6 of 93 in the image+text RAG setting (Karim et al., 12 Oct 2025). In MEDIQA-OE 2025, closed-weight LLM prompting with constrained JSON generation led the leaderboard: WangLab scored AVG 60.2, silver_shaw 60.1, and the best open-weight result, MISo KeaneBeanz, scored 53.4 (Corbeil et al., 30 Oct 2025).

Several controversies recur across the literature. RQE exhibits dataset-shift sensitivity: DoubleTransfer reported excellent dev accuracy but much lower test accuracy on RQE, attributing the drop to train/dev/test mismatch (Xu et al., 2019). MEDIQA-CORR exposes domain heterogeneity: WangLab treated MS and UW as sufficiently different to require different architectures (Toma et al., 2024). MEDIQA-M3G exposed metric sensitivity to lexical form and short-output strategies (Xie et al., 2024). MEDIQA-OE showed that provenance is much harder than order-type classification and that small train size pushed all teams toward prompting rather than fine-tuning (Corbeil et al., 30 Oct 2025). These issues do not negate the benchmarks, but they do constrain the interpretation of leaderboard scores.

6. Terminological ambiguity and later extensions

The term “MedIQA” became polysemous in 2025. In one branch, it continued to denote the benchmark lineage summarized above. In another, “MedIQA: A Scalable Foundation Model for Prompt-Driven Medical Image Quality Assessment” defined a no-reference IQA problem over 2D images and 3D volumes, introduced a dataset of approximately 15k scans or cases, and proposed a prompt-driven architecture with dimension, modality, region, and type prompts plus a salient slice assessment module for 3D inputs (Xun et al., 25 Jul 2025). The model uses upstream pretraining on physical acquisition parameters such as CT dose and MRI field strength, downstream expert-annotation fine-tuning, a MANIQA backbone, seven salient slices for 3D volumes, and evaluation by SRCC, PLCC, and RMSE (Xun et al., 25 Jul 2025). In that distinct sense, MedIQA is a foundation model rather than a shared-task series.

A related but orthographically distinct line uses “Med-IQA” for medical image quality assessment more broadly. MedQ-UNI formulates an assess-then-restore paradigm in which a quality assessment expert first generates structured natural-language descriptions of modality, anatomy, degradation, technical attribution, diagnostic impact, and Good/Usable/Reject judgment, then a restoration expert conditions on those descriptions to restore CT, MRI, and PET images (Liu et al., 19 Mar 2026). MedQ-Engine reframes Med-IQA as an iterative data-engine problem, using failure prototype clustering, retrieval from a million-scale image pool, progressive human-in-the-loop annotation, and fine-tuning to improve MLLMs on both perception and description tasks across MRI, CT, Endoscopy, Fundus photography, and Histopathology (Liu et al., 20 Mar 2026). These works are not part of the original MedIQA shared-task series, but they show how the underlying concern with medically grounded quality assessment has expanded into adjacent multimodal evaluation and restoration settings.

The result is a term with two legitimate technical usages. In shared-task literature, MedIQA names a benchmark ecosystem for medical NLU, generation, and multimodal clinical reasoning. In imaging literature, MedIQA can denote a specific prompt-driven IQA foundation model, while Med-IQA denotes the broader field of medical image quality assessment. Any precise use of the term therefore requires local disambiguation by task family, modality, and publication context.