BioASQ 2025 Task13b Challenge

• BioASQ 2025 Task13b Challenge is a comprehensive biomedical QA benchmark that uses PubMed literature and expert curation to evaluate retrieval and answer generation.
• It is organized into three phases—retrieval, direct answering, and evidence-based summarization—to distinctly measure system performance.
• The challenge supports longitudinal comparisons with over 5,000 curated questions and leverages LLM-centric RAG methods for improved precision and recall.

BioASQ 2025 Task13b Challenge is the 2025 edition of BioASQ’s long-running Task b on large-scale biomedical question answering. It evaluates systems that understand natural language biomedical questions posed by experts, retrieve relevant literature from PubMed, extract or generate exact answers, and produce ideal answers in the form of short, coherent summaries. Within BioASQ 2025, Task13b is one of the two established question-answering tasks, alongside Synergy13, and it continues a 13-year benchmark line intended to reflect real biomedical information needs while supporting longitudinal comparison of QA systems (Nentidis et al., 28 Aug 2025).

1. Position within the BioASQ 2025 ecosystem

BioASQ 2025 consisted of six tasks: Task13b, Synergy13, MultiClinSum, BioNNE-L, ELCardioCC, and GutBrainIE. Across these six tasks, 83 competing teams participated with more than 1000 distinct submissions in total. Task13b is the classic biomedical semantic QA track, whereas Synergy13 targets developing topics with iterative interaction, MultiClinSum addresses multilingual clinical summarization, and the remaining tasks emphasize entity linking, clinical coding, or information extraction in specialized biomedical settings (Nentidis et al., 28 Aug 2025).

Task13b is explicitly English-only and PubMed-centered. Its domain scope is broad biomedical literature rather than a single specialty, and its operational emphasis is end-to-end QA over MEDLINE/PubMed with expert-created questions, evidence, and answers. This makes it distinct from the multilingual and task-specific expansions introduced elsewhere in BioASQ 2025, while still occupying a central role in the overall benchmark suite (Nentidis et al., 28 Aug 2025).

The 2025 edition also sits in a recent line of task redesign. Task12b in 2024 introduced the three-phase structure of retrieval, direct answering without gold evidence, and answering with curated evidence; Task13b preserves and extends that design, which now functions as the canonical evaluation protocol for contemporary BioASQ QA systems (Nentidis et al., 28 Aug 2025).

2. Task design, phases, and question typology

Task13b is organized into three phases, all using English biomedical questions. Phase A evaluates information retrieval; Phase A+ evaluates end-to-end answering without curated evidence; Phase B evaluates answering with expert-provided gold documents and snippets. The structure makes it possible to separate retrieval quality from answer generation quality and to compare systems operating with and without expert-selected evidence (Nentidis et al., 28 Aug 2025).

Phase	Input	Required outputs
Phase A	Question text	Ranked PubMed documents and ranked snippets
Phase A+	Question text only	Exact answers and ideal answers
Phase B	Question text + gold documents/snippets	Exact answers and ideal answers

The question taxonomy remains the standard BioASQ schema. Yes/No questions require an exact answer of “yes” or “no”. Factoid questions require a single entity, with one or more acceptable surface forms of that entity. List questions require a set of entities. Summary questions require a narrative answer, and for these the exact answer field is essentially the same as the ideal answer. For all four types, systems must also produce an ideal answer: a paragraph-sized, well-formed summary that directly addresses the question and integrates information from the literature (Nentidis et al., 28 Aug 2025).

The core information source is MEDLINE/PubMed. Documents are PubMed articles, typically represented by titles and abstracts, although full text may be used by systems when available; the benchmark itself is defined at the PubMed level. Snippets are short text spans, usually a few sentences or less, manually curated by experts as supporting evidence. In Phase B, BioASQ experts supply gold-standard relevant documents and snippets for each question (Nentidis et al., 28 Aug 2025).

3. Dataset construction, annotation, and benchmark growth

Task13b is built around expert curation. Biomedical experts formulate realistic questions, search PubMed, select gold documents and snippets, and provide exact and ideal answers. Exact answers are “yes” or “no” for yes/no questions and entity strings for factoid and list questions; ideal answers are paragraph-sized summaries synthesizing the evidence. The resulting benchmark is therefore grounded in published literature and expert judgment rather than distant supervision or automatic answer harvesting (Nentidis et al., 28 Aug 2025).

The 2025 training set, unchanged from previous editions, contains 5,389 biomedical questions with gold documents, snippets, exact answers, and ideal answers. Its type distribution is 1,459 Yes/No, 1,047 List, 1,600 Factoid, and 1,283 Summary questions. Average evidence density is 9.74 documents and 12.78 snippets per question. The 2025 test set adds about 340 new questions, split into four bi-weekly batches of 85 questions each, producing 5,729 questions overall with averages of 9.33 documents and 12.23 snippets per question across train and test (Nentidis et al., 28 Aug 2025).

The cumulative resource has expanded steadily over recent years. The corresponding total question counts reported for earlier editions are 5,049 in 2024 and 4,719 in 2023, which places the 2025 benchmark within a clear trajectory of incremental enlargement and ongoing expert curation (Nentidis et al., 28 Aug 2025, Nentidis et al., 2023).

Batch	Type counts (Y/N, List, Factoid, Summary)	Avg docs / snippets
Test 1	17, 23, 26, 19	2.68 / 3.74
Test 2	17, 19, 27, 22	2.71 / 3.06
Test 3	22, 22, 20, 21	3.00 / 3.66
Test 4	26, 19, 22, 18	3.15 / 3.92

The lower average number of gold documents and snippets per test question, relative to training, is explicitly attributed to the fact that test annotations are newer and sometimes sparser, especially for questions authored by new experts. This matters operationally because official scores are preliminary when first reported. After system submission, BioASQ experts manually assess answers and enrich the ground truth by adding missing relevant documents or snippets discovered by systems and by adding answer synonyms and alternative entity forms. Final scores are updated after this enrichment process (Nentidis et al., 28 Aug 2025).

The data format follows previous BioASQ b editions. Each question includes fields such as id, body, and type; for training data and for Phase B, the data also includes documents, snippets, exact_answer, and ideal_answer. Snippets carry document identifiers and offsets, supporting both evidence retrieval and character-based overlap evaluation (Nentidis et al., 28 Aug 2025).

4. Evaluation framework

Phase A document retrieval is evaluated with Mean Average Precision. For a question $q$ with relevant document set $R_q$, Average Precision is defined as

$$AP(q) = \frac{1}{|R_q|} \sum_{k=1}^{N} P(k) \cdot \text{rel}(k),$$

where $P(k)$ is precision at rank $k$, $\text{rel}(k)$ indicates whether the document at rank $k$ is relevant, and $N$ is the length of the retrieved list. Mean Average Precision is then

$MAP = \frac{1}{|Q|} \sum_{q \in Q} AP(q),$

with $Q$ the set of questions (Nentidis et al., 28 Aug 2025).

Snippet retrieval is evaluated by character-overlap F1. Gold and system snippets are interpreted as sets of character offsets within source documents, with precision and recall defined over overlapping characters rather than over snippet identities. This design is intended to avoid distortions caused by multiple overlapping system snippets aligning to the same gold snippet (Nentidis et al., 28 Aug 2025).

Exact-answer evaluation is type-specific. Yes/No questions use macro-averaged F1 over “yes” and “no”. Factoid questions use Mean Reciprocal Rank: $R_q$0 where $R_q$1 is the rank of the first correct answer after accounting for gold synonyms. List questions use F1 over set overlap between predicted and gold entities, after synonym resolution. Ideal answers in Phases A+ and B are evaluated manually by experts on four criteria: Precision, Recall, Repetition, and Readability. The exact aggregation formula for these manual scores is not given in the overview, but the criteria themselves are explicit (Nentidis et al., 28 Aug 2025).

Task13b and Synergy13 share the same evaluation measures, but Synergy13 applies residual collection evaluation for retrieval, meaning that only new documents since previous rounds are considered relevant. This creates methodological continuity between the established QA tracks while preserving Synergy’s iterative character (Nentidis et al., 28 Aug 2025).

5. Systems, methods, and methodological lineage

Task13b attracted 46 teams and 734 submissions from 146 distinct systems across all batches and phases. Phase A involved 34 teams and 95 systems, Phase A+ 20 teams and 79 systems, and Phase B 26 teams and 88 systems; 11 teams participated in all three phases. A baseline for Phase B exact answers was provided by OAQA, a CMU system that had been state-of-the-art in earlier BioASQ editions (Nentidis et al., 28 Aug 2025).

The dominant 2025 methodological pattern is LLM-centric retrieval-augmented generation. Retrieval stacks commonly begin with BM25 over PubMed indexes, optionally combined with dense retrieval using bi-encoder architectures and models such as BGE-M3 and MiniLM, followed by cross-encoder or LLM-based reranking. Hybrid sparse-dense retrieval is common. For answer generation, teams widely used Llama, Gemma, GPT, Claude, and Mistral families in RAG pipelines, direct prompting setups, self-feedback or self-refinement loops, ensembles, and ontology- or knowledge-graph-assisted normalization for entity-oriented questions (Nentidis et al., 28 Aug 2025).

Challenge notebooks from 2025 illustrate several concrete variants of this general pattern. One line of work investigated self-feedback in agentic RAG, using LLMs to generate, critique, and refine both Boolean-style retrieval queries and answer drafts, but reported that self-feedback improved some tasks and harmed others, especially retrieval MAP (Ateia et al., 7 Aug 2025). Another line explored question-type-specific multi-model aggregation over open-source LLMs, using majority voting for yes/no questions and unions of candidate answers for factoid and list questions (Panou et al., 2 Aug 2025). A retrieval-focused system combined dense retrieval over approximately 34 million PubMed abstracts with finetuned cross-encoders and GPT reranking, reporting MAP@10 of 0.1581 and rank 10 in the retrieval task, alongside macro-F1 0.95 for yes/no, MRR 0.64 for factoid, mean-F1 0.63 for list, and ROUGE-SU4 F1 0.29 for ideal answers in its answer-generation runs (Verma et al., 8 Jul 2025). A further study on zero-shot ensemble RAG emphasized the role of context length, reporting that focused snippets often outperformed longer abstract-based contexts and that an ensemble could reach state-of-the-art performance on yes/no questions without fine-tuning (Galat et al., 10 Sep 2025).

The 2025 methodological profile also has a clear BioASQ lineage. Earlier BioASQ work treated ideal-answer generation as query-focused multi-document summarization, including first-$R_q$2-snippet baselines, sentence scoring by cosine similarity, and regression to ROUGE-SU4 (Molla-Aliod, 2017). Later work on BioASQ8b refined this extractive framing with BERT or BioBERT token encoders, BiLSTM sentence encoders, and sentence-level classification versus regression objectives, finding that classification was generally stronger than direct ROUGE regression and that BERT followed by an LSTM gave the best summarization results among the tested variants (Molla et al., 2020). For factoid and list answering, earlier neural QA systems adapted extractive span models pretrained on SQuAD, added biomedical embeddings, and modified answer layers to support multiple spans so that the same system could handle both factoid and list questions (Wiese et al., 2017). Task13b therefore represents not a break with prior BioASQ methodology but a shift from extractive QA and summarization pipelines toward LLM-based RAG, prompt engineering, and multi-model orchestration.

6. Results, limitations, and future directions

Phase A retrieval results in the official overview show substantial batch variability. Batches 1 through 3 have broadly similar retrieval profiles, while Batch 4 is markedly harder. Average document MAP by batch is 0.231, 0.283, 0.175, and 0.072; top document MAP is 0.425, 0.442, 0.324, and 0.180. Average snippet F1 is 0.053, 0.084, 0.052, and 0.024; top snippet F1 is 0.120, 0.179, 0.110, and 0.079 (Nentidis et al., 28 Aug 2025).

Batch	Documents MAP (avg / top)	Snippets F1 (avg / top)
1	0.231 / 0.425	0.053 / 0.120
2	0.283 / 0.442	0.084 / 0.179
3	0.175 / 0.324	0.052 / 0.110
4	0.072 / 0.180	0.024 / 0.079

The overview explicitly suggests that Batch 4 may be more difficult because many Batch-4 questions were authored by new BioASQ experts, potentially making them less aligned with the historical training distribution or centered on more difficult topics. This suggests that expert diversity, while desirable for realism and coverage, may increase distribution shift and expose weaknesses in retrieval models (Nentidis et al., 28 Aug 2025).

For exact answers, the overview reports a clear contrast between Phase A+ and Phase B. With gold evidence in Phase B, top Task13b systems achieve scores comparable to or higher than Task12b for all question types. In Phase A+, performance is lower, as expected, but for yes/no questions the top performance is very close to Phase B. The organizers interpret this as evidence that modern LLMs can answer many yes/no questions from background knowledge or through implicit retrieval. By contrast, factoid and list questions show a larger gap between A+ and B, indicating the continuing importance of explicit evidence for entity-centric answering. The organizers also note that preliminary scores probably underestimate final performance because not all synonyms or alternative forms submitted by systems have yet been incorporated into the gold standard (Nentidis et al., 28 Aug 2025).

Ideal-answer evaluation remained incomplete at the time of the overview. No numeric table for manual ideal-answer scores is reported, and final results were scheduled to appear later. A cautious interpretation is therefore necessary: fluent LLM-generated summaries were already common in 2025, but the official overview does not yet quantify their relative quality in Task13b itself (Nentidis et al., 28 Aug 2025).

The main reported technical challenges fall into four classes. First, retrieval variability remains large, as shown by the strong batch-to-batch swings in MAP and snippet F1. Second, entity-centric questions remain difficult: systems struggle to identify all relevant entities for list questions and to match gold entities under synonymy and abbreviation variation, especially without curated evidence. Third, complex reasoning and synthesis remain challenging even with LLMs, particularly when evidence must be integrated across multiple articles while preserving factual consistency. Fourth, BioASQ’s domain specificity means that rare diseases, complex molecular interactions, and niche subdomains may be poorly represented in generic LLM pretraining data (Nentidis et al., 28 Aug 2025).

The benchmark itself has acknowledged limitations. Ground truth can initially be incomplete, especially with respect to relevant documents, snippets, and answer synonyms. Manual evaluation of ideal answers is labor-intensive. The benchmark is also centered on PubMed abstracts, even though clinically relevant information may reside in clinical records, registries, full-text articles, or guidelines. Team reports from 2025 add two further operational cautions: self-feedback in agentic RAG does not consistently improve either retrieval or answer generation, and expanding context length can dilute evidence and degrade question answering, particularly for factoid and list questions (Ateia et al., 7 Aug 2025, Galat et al., 10 Sep 2025).

Future directions named in the overview are correspondingly broad. They include expanding the QA benchmark with more questions and possibly more varied question types or multi-hop reasoning tasks, bringing additional biomedical experts into question creation and Synergy processes, broadening resource coverage beyond PubMed to include material such as clinical guidelines or clinical notes, incorporating multiple languages as BioASQ already does in other tasks, and defining evaluation setups that more directly stress-test LLM factuality, hallucination control, and source attribution. In that sense, Task13b functions simultaneously as a mature benchmark for biomedical QA and as a moving testbed for the methodological transition from retrieval-plus-extraction pipelines to evidence-grounded LLM systems (Nentidis et al., 28 Aug 2025).