HIVMedQA: HIV QA Benchmark
- HIVMedQA is an open benchmark for evaluating LLMs in HIV care, emphasizing multistep clinical reasoning, adherence challenges, and treatment complexity.
- It utilizes clinically curated, open-ended questions across four categories to assess factual recall, comprehension, and vulnerability to cognitive bias.
- The framework integrates lexical similarity and LLM-as-a-judge scoring to uncover gaps in safe, coherent, and bias-resistant clinical decision support.
Searching arXiv for the HIVMedQA paper and closely related medical LLM benchmarking work. Search query: HIVMedQA benchmark LLMs HIV medical decision support HIVMedQA is an open-source benchmark for open-ended medical question answering in HIV care, introduced to evaluate LLMs in a domain characterized by treatment heterogeneity, comorbidity, adherence challenges, and safety-sensitive decision-making (Antolin et al., 24 Jul 2025). Its design reflects the fact that HIV management routinely requires integration of multiple endpoint structures, adherence-mediated treatment effects, longitudinal virologic dynamics, and reservoir measurement frameworks, as illustrated respectively by multi-endpoint analysis of ACTG 175 (Cantarini et al., 2023), path-specific mediation in the Nigeria PEPFAR program (Miles et al., 2014), dynamic Bayesian modeling of adherence, resistance, and covariates (Huang et al., 2011), and combined dilution-assay plus sequencing methods for reservoir quantification (Lotspeich et al., 2023). Within that setting, HIVMedQA functions as a structured testbed for assessing whether LLM outputs are not merely factually similar to reference answers, but clinically coherent, safe, and resistant to bias (Antolin et al., 24 Jul 2025).
1. Clinical rationale and problem setting
HIVMedQA was motivated by the observation that HIV management is a compelling use case for clinical AI because of its complexity, including diverse treatment options, comorbidities, and adherence challenges (Antolin et al., 24 Jul 2025). The benchmark addresses a gap identified by its authors: AI applications in HIV care remain underexplored, and LLM benchmarking studies are scarce (Antolin et al., 24 Jul 2025).
The underlying clinical complexity is consistent with broader HIV research. In ACTG 175, treatment efficacy could not be reduced to a single endpoint, because disease progression was tracked through a primary composite endpoint consisting of a decline in CD4 count from baseline, development of AIDS-defining illness, or death from any cause, with semi-competing risks across outcomes (Cantarini et al., 2023). In the Harvard PEPFAR Nigeria analysis, regimen performance depended not only on pharmacologic effects but also on adherence pathways disentangled from toxicity-mediated effects (Miles et al., 2014). Long-term virologic response modeling has also required mechanism-based differential equation systems that integrate viral load, MEMS adherence, drug resistance, and baseline covariates (Huang et al., 2011). This suggests that a benchmark in HIV medicine must stress not only recall of guideline facts but also multistep clinical reasoning under interacting constraints.
A common misconception is that HIV question answering can be evaluated adequately through straightforward fact retrieval. HIVMedQA was explicitly constructed against that assumption: the study reports that reasoning and comprehension were more challenging than factual recall, and that performance declined as question complexity increased (Antolin et al., 24 Jul 2025).
2. Benchmark construction and dataset composition
The HIVMedQA dataset was developed by a multidisciplinary team comprising an infectious-disease clinician, a general practitioner, AI researchers, and computational biologists, who compiled and iterated on clinical questions in five rounds (Antolin et al., 24 Jul 2025). An infectious-disease physician reviewed every item, wrote gold-standard answers based on current guidelines, including EACS 12.0, and validated question validity and clinical relevance (Antolin et al., 24 Jul 2025). All questions and gold answers are publicly available at Zenodo (Antolin et al., 24 Jul 2025).
The benchmark is partitioned into four categories designed to stratify question complexity and failure modes. Category 1 contains 11 open questions on basic HIV knowledge, such as diagnosis and modes of transmission. Category 2 contains 10 patient-level USMLE Step 1 style vignettes reformatted as open-ended questions. Category 3 contains 21 complex clinical vignettes at USMLE Step 2/3 level requiring multistep decision making with laboratory and imaging data. Category 4 contains 21 modified versions of Category 3 vignettes that introduce recency, frequency, or status-quo bias in order to test susceptibility to cognitive bias (Antolin et al., 24 Jul 2025).
The benchmark’s emphasis on curated, clinically relevant, open-ended questions distinguishes it from multiple-choice-oriented evaluation regimes. A plausible implication is that HIVMedQA targets the operational regime most relevant to clinician-facing decision support, where models must generate explanatory responses rather than select from fixed options.
3. Prompting strategy and scoring framework
HIVMedQA evaluated ten LLMs—seven general-purpose and three medically specialized models—and each model generated five independent answers per question (Antolin et al., 24 Jul 2025). Performance tuning relied on prompt refinement and alignment through system messages only; no few-shot examples or chain-of-thought prompts were employed (Antolin et al., 24 Jul 2025).
The common system prompt cast the model as a senior physician specializing in HIV, instructed it to keep answers brief and clear, and explicitly prohibited guessing or inventing false information when uncertain (Antolin et al., 24 Jul 2025). This prompt design was intended to standardize model behavior while preserving open-ended generation.
Evaluation combined two complementary components. The first was MedSynF1, a lexical similarity metric that extracts medical entities using SciSpacy and UMLS mapping, then computes precision and recall over reference and predicted entities with synonym expansion (Antolin et al., 24 Jul 2025). HIVMedQA extended standard entity-overlap scoring by including synonyms from SNOMED CT, WordNet, and a GPT-4o-generated dictionary, and by lemmatizing tokens with Stanza (Antolin et al., 24 Jul 2025). The authors characterized MedSynF1 as the strict factual overlap metric, and reported that on rephrased gold answers it achieves an upper-bound of approximately $0.53$ (Antolin et al., 24 Jul 2025).
The second component was MedGPT, an LLM-as-a-judge procedure using GPT-4o to score answers from 1 to 5 along five dimensions: reading comprehension, logical reasoning, knowledge recall, demographic bias, and potential harm (Antolin et al., 24 Jul 2025). The scoring prompt supplied explicit rubrics, evidence-based justification requirements, and point-deduction rules, including penalties for logical fallacies and heavily penalized unsafe advice (Antolin et al., 24 Jul 2025). Supervised validation with a gold answer versus unsupervised validation without a gold answer showed that unsupervised scores inflate by approximately $0.7$, which the authors interpreted as evidence for the necessity of reference-based evaluation (Antolin et al., 24 Jul 2025).
4. Empirical performance across models
Gemini 2.5 Pro was the highest-performing model across most dimensions, with mean SD scores of for comprehension, for reasoning, for knowledge, for bias, for harm, and for MedSynF1 (Antolin et al., 24 Jul 2025). Claude 3.5 Sonnet v2 and MedGemma 27B were the next strongest performers, and the top three models therefore included two proprietary systems (Antolin et al., 24 Jul 2025). At the other extreme, Llama 3.2-1B had markedly lower scores, including $0.53$0 for comprehension, $0.53$1 for reasoning, $0.53$2 for knowledge, $0.53$3 for bias, $0.53$4 for harm, and $0.53$5 for MedSynF1 (Antolin et al., 24 Jul 2025).
Several aggregate findings define the benchmark’s central conclusions. Knowledge recall averaged $0.53$6, exceeding comprehension at $0.53$7 and reasoning at $0.53$8 (Antolin et al., 24 Jul 2025). Medically specialized models did not uniformly outperform general-purpose models (Antolin et al., 24 Jul 2025). Model size was not predictive of performance: MedGemma 27B outperformed both Llama 3.3-70B and NVLM 72B despite having fewer parameters (Antolin et al., 24 Jul 2025). As task complexity increased from Category 1 to Category 3, eight of ten models showed performance declines, and only Gemini 2.5 Pro remained stable (Antolin et al., 24 Jul 2025).
MedSynF1 behaved more harshly than judge-based scores. Under complex vignettes in Category 3, all models showed substantial drops in lexical overlap, with top scores around $0.53$9 compared with the reported upper bound of about $0.7$0 on rephrased gold answers (Antolin et al., 24 Jul 2025). This is not evidence that the better models were uniformly incorrect; rather, it indicates that strict lexical overlap underestimates acceptable clinical paraphrase and partial-credit reasoning.
5. Bias, safety, and clinically salient failure modes
Category 4 was designed specifically to probe recency, frequency, and status-quo biases by modifying Category 3 vignettes (Antolin et al., 24 Jul 2025). Performance drops from Category 3 to Category 4 were interpreted as evidence of cognitive-bias vulnerability, with NVLM 72B showing a score drop of up to $0.7$1 (Antolin et al., 24 Jul 2025). The benchmark therefore does not treat bias as a purely sociotechnical abstraction; it operationalizes it as susceptibility to contextually induced diagnostic or therapeutic distortion.
Potential harm was also treated as a first-class evaluation axis. The average MedGPT harm score was $0.7$2, but isolated scores below $0.7$3 occurred when models omitted answers or generated unsafe advice (Antolin et al., 24 Jul 2025). The authors emphasized that even rare harmful responses pose unacceptable risk in real-world HIV care (Antolin et al., 24 Jul 2025).
These findings qualify another common misconception: strong factual recall does not guarantee safe clinical decision support. HIVMedQA reports that reasoning and comprehension were harder than recall, and that unsafe failures can occur even when average scores appear high (Antolin et al., 24 Jul 2025). In a domain where treatment selection, adherence support, drug toxicity, and virologic monitoring interact, that distinction is clinically material.
6. Methodological significance and relation to HIV clinical decision support
HIVMedQA’s significance lies partly in its evaluation philosophy. The benchmark argues against sole reliance on lexical metrics and in favor of reference-based LLM-as-a-judge or expert-in-the-loop assessment to capture clinical nuance and avoid over-optimism (Antolin et al., 24 Jul 2025). That position aligns with the structure of HIV decision problems documented elsewhere in the literature provided here. ACTG 175 showed that pooling information across CD4 decline, AIDS, and death could increase power and reduce the risk of type II error relative to separate endpoint analyses (Cantarini et al., 2023). The Nigeria PEPFAR study showed that clinically relevant effects may traverse specific causal pathways, such as the adherence pathway $0.7$4 that excludes toxicity-mediated mechanisms (Miles et al., 2014). The dynamic Bayesian nonlinear mixed-effects model of virologic response likewise treated adherence as a temporally indexed quantity whose predictive power depends on the choice of summary metric, with $0.7$5 yielding the lowest DIC among the tested adherence summaries (Huang et al., 2011). This suggests that benchmark design for HIV LLMs benefits from explicitly targeting multistep, pathway-sensitive, and temporally structured reasoning.
The benchmark is therefore not simply an exercise in generic medical QA. It is a domain-specific attempt to evaluate whether LLMs can operate in a setting where clinically acceptable answers may require integration across guidelines, laboratory data, treatment history, adverse-event tradeoffs, and cognitive-bias resistance.
7. Developmental directions and broader implications
The authors proposed several directions for model development and deployment. These include chain-of-thought prompting, self-consistency, tool-augmented workflows, and retrieval-augmented generation using up-to-date HIV guidelines; a shift in medical fine-tuning from static knowledge injection toward explicit training for clinical reasoning and bias mitigation; adoption of LLM-as-a-judge or expert-in-the-loop evaluation; enrichment of training corpora with real-world, region-specific HIV scenarios, including underrepresented languages and settings; development of multi-turn interactive benchmarks; and continuous monitoring of model drift with regular updates to align systems with evolving HIV treatment guidelines (Antolin et al., 24 Jul 2025).
Some of these recommendations exceed the methods used in the benchmark itself. No few-shot examples or chain-of-thought prompts were employed in the reported experiments, so the recommendation to use such techniques is prospective rather than validated within HIVMedQA (Antolin et al., 24 Jul 2025). This suggests that the benchmark should be read as a baseline and diagnostic instrument rather than a claim that current systems are deployment-ready.
In that sense, HIVMedQA occupies a dual role. It is a benchmark for open-ended HIV medical question answering, and it is also a measurement framework for identifying where present-day LLMs fail: not primarily in surface-level recall, but in comprehension, reasoning, bias resistance, and harm avoidance under clinically realistic complexity (Antolin et al., 24 Jul 2025).