---
title: 'HIVMedQA: HIV QA Benchmark'
url: https://www.emergentmind.com/topics/hivmedqa
type: topic
---

# HIVMedQA: HIV QA Benchmark

Searching arXiv for the HIVMedQA paper and closely related medical LLM benchmarking work.
Search query: HIVMedQA benchmark large language models HIV medical decision support
HIVMedQA is an open-source benchmark for open-ended medical question answering in HIV care, introduced to evaluate large language models (LLMs) in a domain characterized by treatment heterogeneity, comorbidity, adherence challenges, and safety-sensitive decision-making [2507.18143]. Its design reflects the fact that HIV management routinely requires integration of multiple endpoint structures, adherence-mediated treatment effects, longitudinal virologic dynamics, and reservoir measurement frameworks, as illustrated respectively by multi-endpoint analysis of ACTG 175 [2311.15410], path-specific mediation in the Nigeria PEPFAR program [1411.6028], dynamic Bayesian modeling of adherence, resistance, and covariates [1104.3464], and combined dilution-assay plus sequencing methods for reservoir quantification [2302.00516]. Within that setting, HIVMedQA functions as a structured testbed for assessing whether LLM outputs are not merely factually similar to reference answers, but clinically coherent, safe, and resistant to bias [2507.18143].

## 1. Clinical rationale and problem setting

HIVMedQA was motivated by the observation that HIV management is a compelling use case for clinical AI because of its complexity, including diverse treatment options, comorbidities, and adherence challenges [2507.18143]. The benchmark addresses a gap identified by its authors: AI applications in HIV care remain underexplored, and LLM benchmarking studies are scarce [2507.18143].

The underlying clinical complexity is consistent with broader HIV research. In ACTG 175, treatment efficacy could not be reduced to a single endpoint, because disease progression was tracked through a primary composite endpoint consisting of a $\ge 50\%$ decline in CD4 count from baseline, development of AIDS-defining illness, or death from any cause, with semi-competing risks across outcomes [2311.15410]. In the Harvard PEPFAR Nigeria analysis, regimen performance depended not only on pharmacologic effects but also on adherence pathways disentangled from toxicity-mediated effects [1411.6028]. Long-term virologic response modeling has also required mechanism-based differential equation systems that integrate viral load, MEMS adherence, drug resistance, and baseline covariates [1104.3464]. This suggests that a benchmark in HIV medicine must stress not only recall of guideline facts but also multistep clinical reasoning under interacting constraints.

A common misconception is that HIV question answering can be evaluated adequately through straightforward fact retrieval. HIVMedQA was explicitly constructed against that assumption: the study reports that reasoning and comprehension were more challenging than factual recall, and that performance declined as question complexity increased [2507.18143].

## 2. Benchmark construction and dataset composition

The HIVMedQA dataset was developed by a multidisciplinary team comprising an infectious-disease clinician, a general practitioner, AI researchers, and computational biologists, who compiled and iterated on clinical questions in five rounds [2507.18143]. An infectious-disease physician reviewed every item, wrote gold-standard answers based on current guidelines, including EACS 12.0, and validated question validity and clinical relevance [2507.18143]. All questions and gold answers are publicly available at Zenodo [2507.18143].

The benchmark is partitioned into four categories designed to stratify question complexity and failure modes. Category 1 contains 11 open questions on basic HIV knowledge, such as diagnosis and modes of transmission. Category 2 contains 10 patient-level USMLE Step 1 style vignettes reformatted as open-ended questions. Category 3 contains 21 complex clinical vignettes at USMLE Step 2/3 level requiring multistep decision making with laboratory and imaging data. Category 4 contains 21 modified versions of Category 3 vignettes that introduce recency, frequency, or status-quo bias in order to test susceptibility to cognitive bias [2507.18143].

The benchmark’s emphasis on curated, clinically relevant, open-ended questions distinguishes it from multiple-choice-oriented evaluation regimes. A plausible implication is that HIVMedQA targets the operational regime most relevant to clinician-facing decision support, where models must generate explanatory responses rather than select from fixed options.

## 3. Prompting strategy and scoring framework

HIVMedQA evaluated ten LLMs—seven general-purpose and three medically specialized models—and each model generated five independent answers per question [2507.18143]. Performance tuning relied on prompt refinement and alignment through system messages only; no few-shot examples or chain-of-thought prompts were employed [2507.18143].

The common system prompt cast the model as a senior physician specializing in HIV, instructed it to keep answers brief and clear, and explicitly prohibited guessing or inventing false information when uncertain [2507.18143]. This prompt design was intended to standardize model behavior while preserving open-ended generation.

Evaluation combined two complementary components. The first was MedSynF1, a lexical similarity metric that extracts medical entities using SciSpacy and UMLS mapping, then computes precision and recall over reference and predicted entities with synonym expansion [2507.18143]. HIVMedQA extended standard entity-overlap scoring by including synonyms from SNOMED CT, WordNet, and a GPT-4o-generated dictionary, and by lemmatizing tokens with Stanza [2507.18143]. The authors characterized MedSynF1 as the strict factual overlap metric, and reported that on rephrased gold answers it achieves an upper-bound of approximately $0.53$ [2507.18143].

The second component was MedGPT, an LLM-as-a-judge procedure using GPT-4o to score answers from 1 to 5 along five dimensions: reading comprehension, logical reasoning, knowledge recall, demographic bias, and potential harm [2507.18143]. The scoring prompt supplied explicit rubrics, evidence-based justification requirements, and point-deduction rules, including penalties for logical fallacies and heavily penalized unsafe advice [2507.18143]. Supervised validation with a gold answer versus unsupervised validation without a gold answer showed that unsupervised scores inflate by approximately $0.7$, which the authors interpreted as evidence for the necessity of reference-based evaluation [2507.18143].

## 4. Empirical performance across models

Gemini 2.5 Pro was the highest-performing model across most dimensions, with mean $\pm$ SD scores of $4.12 \pm 0.03$ for comprehension, $4.03 \pm 0.06$ for reasoning, $4.49 \pm 0.04$ for knowledge, $4.99 \pm 0.02$ for bias, $4.91 \pm 0.02$ for harm, and $0.22 \pm 0.01$ for MedSynF1 [2507.18143]. Claude 3.5 Sonnet v2 and MedGemma 27B were the next strongest performers, and the top three models therefore included two proprietary systems [2507.18143]. At the other extreme, Llama 3.2-1B had markedly lower scores, including $2.18 \pm 0.09$ for comprehension, $1.83 \pm 0.09$ for reasoning, $2.13 \pm 0.13$ for knowledge, $4.97 \pm 0.02$ for bias, $3.41 \pm 0.16$ for harm, and $0.15 \pm 0.01$ for MedSynF1 [2507.18143].

Several aggregate findings define the benchmark’s central conclusions. Knowledge recall averaged $4.06$, exceeding comprehension at $3.75$ and reasoning at $3.59$ [2507.18143]. Medically specialized models did not uniformly outperform general-purpose models [2507.18143]. Model size was not predictive of performance: MedGemma 27B outperformed both Llama 3.3-70B and NVLM 72B despite having fewer parameters [2507.18143]. As task complexity increased from Category 1 to Category 3, eight of ten models showed performance declines, and only Gemini 2.5 Pro remained stable [2507.18143].

MedSynF1 behaved more harshly than judge-based scores. Under complex vignettes in Category 3, all models showed substantial drops in lexical overlap, with top scores around $0.18$ compared with the reported upper bound of about $0.53$ on rephrased gold answers [2507.18143]. This is not evidence that the better models were uniformly incorrect; rather, it indicates that strict lexical overlap underestimates acceptable clinical paraphrase and partial-credit reasoning.

## 5. Bias, safety, and clinically salient failure modes

Category 4 was designed specifically to probe recency, frequency, and status-quo biases by modifying Category 3 vignettes [2507.18143]. Performance drops from Category 3 to Category 4 were interpreted as evidence of cognitive-bias vulnerability, with NVLM 72B showing a score drop of up to $0.32$ [2507.18143]. The benchmark therefore does not treat bias as a purely sociotechnical abstraction; it operationalizes it as susceptibility to contextually induced diagnostic or therapeutic distortion.

Potential harm was also treated as a first-class evaluation axis. The average MedGPT harm score was $4.67$, but isolated scores below $2$ occurred when models omitted answers or generated unsafe advice [2507.18143]. The authors emphasized that even rare harmful responses pose unacceptable risk in real-world HIV care [2507.18143].

These findings qualify another common misconception: strong factual recall does not guarantee safe clinical decision support. HIVMedQA reports that reasoning and comprehension were harder than recall, and that unsafe failures can occur even when average scores appear high [2507.18143]. In a domain where treatment selection, adherence support, drug toxicity, and virologic monitoring interact, that distinction is clinically material.

## 6. Methodological significance and relation to HIV clinical decision support

HIVMedQA’s significance lies partly in its evaluation philosophy. The benchmark argues against sole reliance on lexical metrics and in favor of reference-based LLM-as-a-judge or expert-in-the-loop assessment to capture clinical nuance and avoid over-optimism [2507.18143]. That position aligns with the structure of HIV decision problems documented elsewhere in the literature provided here. ACTG 175 showed that pooling information across CD4 decline, AIDS, and death could increase power and reduce the risk of type II error relative to separate endpoint analyses [2311.15410]. The Nigeria PEPFAR study showed that clinically relevant effects may traverse specific causal pathways, such as the adherence pathway $E \to M \to Y$ that excludes toxicity-mediated mechanisms [1411.6028]. The dynamic Bayesian nonlinear mixed-effects model of virologic response likewise treated adherence as a temporally indexed quantity whose predictive power depends on the choice of summary metric, with $M_{2}^{2}$ yielding the lowest DIC among the tested adherence summaries [1104.3464]. This suggests that benchmark design for HIV LLMs benefits from explicitly targeting multistep, pathway-sensitive, and temporally structured reasoning.

The benchmark is therefore not simply an exercise in generic medical QA. It is a domain-specific attempt to evaluate whether LLMs can operate in a setting where clinically acceptable answers may require integration across guidelines, laboratory data, treatment history, adverse-event tradeoffs, and cognitive-bias resistance.

## 7. Developmental directions and broader implications

The authors proposed several directions for model development and deployment. These include chain-of-thought prompting, self-consistency, tool-augmented workflows, and retrieval-augmented generation using up-to-date HIV guidelines; a shift in medical fine-tuning from static knowledge injection toward explicit training for clinical reasoning and bias mitigation; adoption of LLM-as-a-judge or expert-in-the-loop evaluation; enrichment of training corpora with real-world, region-specific HIV scenarios, including underrepresented languages and settings; development of multi-turn interactive benchmarks; and continuous monitoring of model drift with regular updates to align systems with evolving HIV treatment guidelines [2507.18143].

Some of these recommendations exceed the methods used in the benchmark itself. No few-shot examples or chain-of-thought prompts were employed in the reported experiments, so the recommendation to use such techniques is prospective rather than validated within HIVMedQA [2507.18143]. This suggests that the benchmark should be read as a baseline and diagnostic instrument rather than a claim that current systems are deployment-ready.

In that sense, HIVMedQA occupies a dual role. It is a benchmark for open-ended HIV medical question answering, and it is also a measurement framework for identifying where present-day LLMs fail: not primarily in surface-level recall, but in comprehension, reasoning, bias resistance, and harm avoidance under clinically realistic complexity [2507.18143].

Source: https://www.emergentmind.com/topics/hivmedqa