Medical-Diff VQA: Longitudinal Chest X-Ray Analysis

• Medical-Diff-VQA is a longitudinal visual question answering task that compares paired radiological studies over time to identify clinically meaningful changes.
• It employs advanced methods such as expert graph models, saliency-guided encoder-decoders, and attention-consistent frameworks to capture subtle anatomical differences.
• Benchmarks leverage paired chest X-ray datasets like MIMIC-CXR, utilizing rigorous extraction and registration techniques for precise spatial and temporal alignment.

Searching arXiv for papers on Medical-Diff-VQA and closely related longitudinal medical VQA benchmarks/models. Medical-Diff-VQA denotes a longitudinal medical visual question answering setting in which the input is not a single image but a paired study of the same patient at different time points, together with a difference-focused question whose answer must describe clinically meaningful change. In the chest X-ray setting emphasized by recent work, the task requires comparing a current image with a prior image, identifying anatomically corresponding regions, and reasoning about progression, improvement, new findings, or stability rather than only static abnormalities (Hu et al., 2023). Subsequent work uses “Medical-Diff-VQA” both for the general difference-VQA problem and for a large-scale chest X-ray benchmark derived from MIMIC-CXR, with free-text answers evaluated by generation metrics such as BLEU, METEOR, ROUGE-L, and CIDEr (Wu et al., 29 Sep 2025).

1. Emergence of difference-aware medical VQA

The explicit formulation of chest X-ray difference visual question answering was introduced as a new task in which a model receives a main image, a reference image, and a natural-language question, and must produce an answer conditioned on both images and the question (Hu et al., 2023). The main and reference images are from the same patient but different visits, with the reference study preceding the main study (Hu et al., 2023). This setup was presented as closer to radiologists’ actual workflow, since radiologists routinely compare a current study against a prior study before concluding whether a finding has improved, worsened, newly appeared, or resolved (Hu et al., 2023).

The earliest large-scale dataset in this line is MIMIC-Diff-VQA, which contains 700,703 QA pairs from 164,324 pairs of main and reference images and was explicitly designed around the Assessment-Diagnosis-Intervention-Evaluation treatment procedure used by clinical professionals (Hu et al., 2023). Its question taxonomy includes abnormality, presence, view, location, type, level, and difference, where only the difference type explicitly asks about change relative to the reference image (Hu et al., 2023). A later chest X-ray benchmark, referred to as Medical-Diff-VQA, is derived from MIMIC-CXR / MIMIC-CXR-JPG and contains 164,223 total samples, with 131,556 for training, 16,278 for validation, and 16,389 for testing (Wu et al., 29 Sep 2025). In that benchmark, each sample consists of a pair of studies, a difference-focused question, and a free-text answer describing clinically meaningful changes (Wu et al., 29 Sep 2025).

This progression suggests a shift from adapting single-image MedVQA methods to constructing dedicated longitudinal reasoning benchmarks. A plausible implication is that Medical-Diff-VQA crystallized a distinct subfield because the clinically relevant signal lies in temporal change rather than in absolute appearance.

2. Benchmark construction and task structure

MIMIC-Diff-VQA was built from MIMIC-CXR by excluding patients with only one radiology visit and pairing studies from the same patient so that the reference study is earlier and the main study is later (Hu et al., 2023). The authors describe an Extract–Check–Fix process: biomedical entities were extracted with ScispaCy, manually reviewed to curate abnormalities and attributes, converted into a structured Key-Info dataset of positive and negative findings, and then used to generate question-answer pairs (Hu et al., 2023). Human verification of 1,700 randomly sampled QA pairs yielded 1,657 correct pairs, corresponding to 97.4% correctness (Hu et al., 2023).

The later Medical-Diff-VQA benchmark for longitudinal chest radiograph VQA is also MIMIC-based but is framed directly as a free-text generation problem (Wu et al., 29 Sep 2025). Questions are explicitly about longitudinal changes rather than single-image status, and answers are open-ended sentences or phrases such as change descriptions, new findings, or statements of stability (Wu et al., 29 Sep 2025). Model selection in one study on this benchmark uses the scalar $$0.6 \cdot \frac{\mathrm{CIDEr}}{1 + \mathrm{CIDEr}} + 0.4 \cdot \mathrm{METEOR}$$, intended to emphasize clinically informative tokens and semantic alignment (Wu et al., 29 Sep 2025).

The task design differs materially from standard medical VQA. Standard MedVQA typically maps one image and one question to a short answer, whereas longitudinal Medical-Diff-VQA requires comparison of paired studies, temporal consistency of visual focus, and interpretation of directional change (Wu et al., 29 Sep 2025). This suggests that architectures built only for single-image recognition or global image-text matching are structurally mismatched to the longitudinal setting.

3. Core modeling paradigms

The first baseline tailored to the difference-VQA setting is an expert knowledge-aware image-difference graph model (Hu et al., 2023). It extracts anatomical structure features and disease features using Faster-RCNN detectors trained on Chest ImaGenome and VinDr-CXR, builds a multi-relationship graph with spatial, semantic, and implicit edges, encodes the question with a BiGRU plus self-attention, and computes an image-difference representation by subtracting graph features of the reference image from those of the main image (Hu et al., 2023). Answer generation is then performed with an LSTM-based decoder (Hu et al., 2023).

A second paradigm is saliency-guided longitudinal encoder-decoder modeling (Wu et al., 29 Sep 2025). In this formulation, the model performs a lightweight near-identity affine pre-alignment to reduce nuisance motion between visits, then runs a within-epoch two-step loop: first, it extracts a clinically relevant keyword from the answer and generates keyword-conditioned Grad-CAM maps on both images; second, it applies a shared saliency mask to both time points and generates the final answer (Wu et al., 29 Sep 2025). The shared mask is defined as

$S = \max(S_{\text{main}},\, S_{\text{ref}})$

and is applied multiplicatively to both the registered main image and the reference image (Wu et al., 29 Sep 2025). The model uses ResNet-50 pretrained on ImageNet-1K as image encoder, a 6-layer text encoder, and GPT-2 as decoder, with no radiology-specific pretraining (Wu et al., 29 Sep 2025).

A related but more foundation-model-oriented paradigm is attention-consistent longitudinal VQA guided by vision foundation models (Wu et al., 3 Jun 2026). This method also uses lightweight affine registration, but replaces keyword-conditioned Grad-CAM with a frozen DINO-based mask generator plus a trainable adaptive mask generator (Wu et al., 3 Jun 2026). The resulting shared mask is applied to both time points, the masked images are re-encoded, and the representations are concatenated with question features and decoded by GPT-2 (Wu et al., 3 Jun 2026). Additional auxiliary objectives include a mask rebuilding loss, a pairwise Gram-style consistency loss, and a KoLeo uniformity loss (Wu et al., 3 Jun 2026).

A fourth paradigm emphasizes pretraining rather than downstream architecture. “Location-Aware Pretraining for Medical Difference Visual Question Answering” pretrains a SigLIP-style ViT encoder on MIMIC-CXR and Chest ImaGenome with captioning plus three location-aware tasks: automatic referring expressions (AREF), grounded captioning (GCAP), and conditional automatic referring expressions (CAREF) (Musinguzi et al., 5 Mar 2026). After pretraining, the vision encoder is frozen, paired images are encoded with shared weights and temporal embeddings, projected to the GPT-2 latent space, concatenated with question tokens, and decoded autoregressively (Musinguzi et al., 5 Mar 2026).

These paradigms share a common premise: difference reasoning requires representations that are spatially grounded, temporally aligned, and sensitive to subtle local changes. They differ mainly in whether that structure is imposed via graphs, saliency, foundation-model masks, or location-aware pretraining.

4. Alignment, saliency, and spatial grounding

Mild geometric pre-alignment is a recurring design principle. In saliency-guided longitudinal VQA, the model predicts affine parameters $$\Theta = [A\;\mathbf{t}] \in \mathbb{R}^{2\times 3}$$ and regularizes them with

$$\mathcal{L}_{\text{reg}} = w_{\text{small}}\|\Theta - I\|_F^{2} + w_{\det}(\det(A) - 1)^{2} + w_{\text{trans}}\|\mathbf{t}\|_2^{2},$$

using $w_{\text{small}} = 10^{-4}$, $w_{\det} = 10^{-5}$, and $w_{\text{trans}} = 10^{-6}$ (Wu et al., 29 Sep 2025). The intent is to reduce nuisance pose and scale variation without erasing genuine anatomical change (Wu et al., 29 Sep 2025). The attention-consistent model uses the same near-identity affine principle, again with a “small registration regularizer” (Wu et al., 3 Jun 2026).

Shared attention across time is another recurring principle. In the saliency-guided model, answer-derived keywords define the target concept for Grad-CAM, and the same keyword-conditioned shared mask is applied to both images (Wu et al., 29 Sep 2025). In the attention-consistent model, the final mask is a convex combination of a DINO union prior and an adaptive fusion output,

$M = \lambda\,U + (1-\lambda)\,F,$

where $U = \max(A_{\text{main}}, A_{\text{ref}})$ is the union of frozen RAD-DINO attention maps (Wu et al., 3 Jun 2026). This mask is then applied symmetrically to both time points (Wu et al., 3 Jun 2026).

Location-aware pretraining pushes spatial grounding even earlier in the pipeline. In that framework, bounding box coordinates are represented as regular decoder tokens over 448×448 images, and the encoder-decoder is trained to generate text and coordinates jointly for AREF, GCAP, and CAREF (Musinguzi et al., 5 Mar 2026). The joint objective is

$$\mathcal{L} = \mathcal{L}_{\text{CAP}} + \mathcal{L}_{\text{AREF}} + \mathcal{L}_{\text{GCAP}} + \mathcal{L}_{\text{CAREF}},$$

with tasks balanced by sampling equal amounts per batch (Musinguzi et al., 5 Mar 2026). The authors argue that standard contrastive or classification pretraining captures coarse global semantics but not the fine-grained, spatially grounded visual representations required for change detection (Musinguzi et al., 5 Mar 2026).

This body of work indicates a broad consensus that explicit spatial grounding is not optional in Medical-Diff-VQA. A plausible implication is that the main modeling challenge is not generic multimodal fusion but preserving lesion-level correspondences under longitudinal variation.

5. Performance trends on the benchmark

The original MIMIC-Diff-VQA paper reports that its expert knowledge-aware graph model outperforms MMQ on the six non-difference question types, achieving 26.4% open-ended accuracy, 79.9% closed-ended accuracy, and 52.5% total accuracy, versus 11.5%, 10.8%, and 11.5% for MMQ (Hu et al., 2023). On the difference subset, the same paper compares against MCCFormers and IDCPCL and reports BLEU-4 scores of 0.434 for the proposed model, 0.414 for IDCPCL, and 0.153 for MCCFormers (Hu et al., 2023).

On the later Medical-Diff-VQA benchmark, the saliency-guided encoder-decoder reports the following comparison: MCCFormers achieves BLEU-4 0.153, METEOR 0.319, ROUGE-L 0.340, CIDEr 0.000; IDCPCL 0.414, 0.303, 0.582, 0.703; EKAID 0.434, 0.339, 0.557, 1.027; RegioMix 0.517, 0.381, 0.651, 1.804; PLURAL 0.520, 0.381, 0.653, 1.832; ReAl 0.530, 0.395, 0.736, 2.409; VED 0.537, 0.389, 0.670, 2.119; and the saliency-guided model 0.341, 0.651, 0.627, 1.263 (Wu et al., 29 Sep 2025). Its authors emphasize the unusually high METEOR score of 0.651 and interpret that as strong semantic adequacy despite lower BLEU-n (Wu et al., 29 Sep 2025).

The attention-consistent model guided by vision foundation models reports stronger lexical metrics on the same benchmark: MCCFormers 0.214 BLEU-1, 0.319 METEOR, 0.340 ROUGE-L, 0.000 CIDEr; IDCPCL 0.614, 0.303, 0.582, 0.703; EKAID 0.628, 0.339, 0.557, 1.027; RegioMix 0.705, 0.381, 0.651, 1.804; PLURAL 0.704, 0.381, 0.653, 1.832; VED 0.716, 0.389, 0.670, 2.119; and the proposed model 0.747, 0.700, 0.703, 2.011 (Wu et al., 3 Jun 2026). The same paper shows that removing the frozen-encoder training stage, removing the DINO-inspired unsupervised objectives, or removing attention masks each lowers BLEU-1, BLEU-4, METEOR, ROUGE-L, and CIDEr (Wu et al., 3 Jun 2026).

The location-aware pretraining paper reports the strongest gains among the provided sources on the difference subset of MIMIC-Diff-VQA. It compares global contrastive pretraining, regional contrastive pretraining, CapPa, BLIP-2, RG-AG, ReAL, and PLURAL, and reports for its own method BLEU-4 0.594, METEOR 0.425, ROUGE-L 0.747, CIDEr 2.997, and BERTScore 0.972 (Musinguzi et al., 5 Mar 2026). The same study reports that removing all location-aware tasks and keeping only captioning reduces BLEU-4 to 0.350, METEOR to 0.317, ROUGE-L to 0.529, and CIDEr to 0.950; removing AREF is especially damaging, with BLEU-4 0.283 and CIDEr 0.850 (Musinguzi et al., 5 Mar 2026).

These results suggest that benchmark performance has improved along two axes: stronger temporal alignment and stronger spatial grounding. They also suggest that CIDEr and METEOR can diverge substantially across methods, reflecting a persistent mismatch between lexical overlap and clinical semantic correctness.

6. Robustness, calibration, and open issues

Although Medical-Diff-VQA is a longitudinal benchmark, robustness problems familiar from general MedVQA remain relevant. A 2025 study on robust medical VQA constructs RoMed, a perturbation benchmark with 144k question variants spanning word-level, sentence-level, and semantic-level perturbations, and shows that state-of-the-art Med-VLMs exhibit up to a 40% decline in Recall under rephrasings (Jiang et al., 26 Aug 2025). That work proposes Consistency and Contrastive Learning and reports 50% improvement in answer consistency on RoMed (Jiang et al., 26 Aug 2025). This suggests that longitudinal VQA systems, which already depend on subtle comparative reasoning, are likely especially sensitive to linguistic perturbations unless explicitly regularized for invariance.

Confidence reliability is another open issue. A 2026 empirical study on medical VQA calibration reports that elicited confidence often misaligns with actual accuracy and that a method combining Multi-Strategy Fusion-Based Interrogation with auxiliary expert LLM assessment reduces Expected Calibration Error by an average of 40% across three medical VQA datasets (Du et al., 18 Jun 2026). A related 2026 study finds that overconfidence persists across Qwen3-VL, InternVL3, and LLaVA-NeXT families, that simple post-hoc calibration approaches such as Platt scaling reduce calibration error, and that hallucination-aware calibration improves both calibration and AUROC, with the largest gains on open-ended questions (Byun et al., 2 Apr 2026). Since Medical-Diff-VQA uses free-text answers about clinically meaningful changes, these findings imply that reliability cannot be assessed by generation metrics alone.

Evaluation itself remains contested. The saliency-guided and attention-consistent longitudinal papers both note that standard generation metrics may not fully capture medical answer quality, especially when key clinical entities or directional change are correct but phrased differently (Wu et al., 29 Sep 2025, Wu et al., 3 Jun 2026). The original MIMIC-Diff-VQA work likewise highlights large answer diversity, with 9,252 distinct possible answers even excluding difference questions, making fixed-vocabulary classification unrealistic (Hu et al., 2023). This suggests that future Medical-Diff-VQA evaluation will likely need stronger semantic or clinically grounded metrics.

Current scope is also narrow. The strongest Medical-Diff-VQA systems in the provided literature are all centered on chest X-ray longitudinal reasoning (Wu et al., 29 Sep 2025, Wu et al., 3 Jun 2026, Musinguzi et al., 5 Mar 2026). A plausible implication is that extending the paradigm to CT, MRI, ultrasound, or more than two time points will require new annotation resources comparable to Chest ImaGenome or new ways of obtaining fine-grained spatial supervision.