Diff-VQA: Longitudinal Medical Visual QA
- Diff-VQA is a specialized task that evaluates temporal changes in paired medical images to facilitate clinical assessments.
- Models leverage innovative methods like temporal token fusion, report-grounded reasoning, and saliency-conditioned comparison for precise longitudinal analysis.
- Benchmarks from datasets such as MIMIC-CXR show improved performance in detecting progression and regression, though challenges with spatial alignment persist.
Longitudinal medical visual question answering, commonly termed Diff‑VQA, is the task of answering natural-language questions about clinically meaningful change between temporally separated medical images of the same patient. In chest radiography, the canonical setting pairs a current or main chest X‑ray with an earlier reference study and asks whether a finding has appeared, resolved, increased, decreased, or remained stable. The central difficulty is that the informative signal is not the absolute content of either image alone, but the medically relevant difference between them, together with anatomically consistent comparison across time (Wu et al., 29 Sep 2025). Recent work has treated this problem as a distinct subfield of medical VQA, developed dedicated datasets from MIMIC‑CXR, and extended the paradigm toward prognosis, report-grounded reasoning, and 3D temporal diagnosis (Cho et al., 2024).
1. Task definition and clinical scope
Diff‑VQA differs from conventional single-image medical VQA in three ways that recur across the literature. First, the question is explicitly about temporal change rather than static appearance. Second, the model must compare corresponding anatomy across visits while discounting nuisance variation such as positioning, scale, or acquisition differences. Third, the answer is often generative, requiring a clinically meaningful description of change rather than a one-time categorical label (Wu et al., 29 Sep 2025).
This framing mirrors routine radiological workflow. Multiple papers emphasize that radiologists rarely interpret a chest X‑ray in isolation: they compare current and prior exams to assess treatment response, distinguish chronic from acute disease, identify subtle new abnormalities, and judge whether a lesion has progressed, regressed, or remained stable (Jiang et al., 24 Feb 2026). In this sense, Diff‑VQA is not simply “multi-image VQA”; it is longitudinal reasoning over anatomically corresponding evidence.
The broader medical-VQA literature had already identified the lack of multi-image input as a major gap. The survey literature notes that medical studies often include multiple views, sequential slices, and contextual information, yet most VQA datasets and models historically reduced the problem to a single image and a single question (Lin et al., 2021). Diff‑VQA can therefore be understood as a response to that structural mismatch between benchmark design and clinical practice.
2. Datasets and task formulations
The dominant benchmark family is Medical‑Diff‑VQA, derived from MIMIC‑CXR. One formulation describes 164,223 samples consisting of paired chest radiographs from the same patient, each paired with a difference-focused question and answer, split into 131,556 training, 16,278 validation, and 16,389 test samples (Wu et al., 29 Sep 2025). Another line of work describes Medical‑Diff‑VQA as 700,703 QA pairs built from longitudinal CXR pairs, including seven categories—abnormality, presence, view, location, level, type, and difference—with the difference category defining the core Diff‑VQA task (Serra et al., 22 May 2025). This suggests that published works count either paired-image samples or all generated QA instances, depending on whether they focus on the longitudinal subset alone or the full mixed-task corpus.
LUMEN makes this split explicit by distinguishing Diagnostic VQA from Difference / Diff‑VQA and further adding Prognostic VQA. In that framework, diagnostic questions depend on a single image, difference questions compare a main and reference image, and prognostic questions ask about expected future changes using paired images plus temporal metadata injected as text tokens in the prompt (Jiang et al., 24 Feb 2026). The longitudinal extension is thus not limited to retrospective comparison; it can also be cast as present-to-future prediction.
Longitudinal reasoning has also been formalized beyond 2D chest radiography. The 3D‑RAD dataset for chest CT introduces Static Temporal Diagnosis and Longitudinal Temporal Diagnosis. Task 5 requires inference of one of four temporal categories—Refractory Lesion, Resolved Lesion, New Lesion, No Abnormality—from a single current 3D scan, while Task 6 adds a symbolic history of prior labels in the question, making it a label-sequence–augmented analogue of Diff‑VQA (Gai et al., 11 Jun 2025). By contrast, UCSF‑PDGM‑VQA, although motivated by neuro-oncology workflows that are inherently longitudinal, is explicitly single-time-point and filters out questions about progression, recurrence, prior scans, or temporal comparison (Ghosh et al., 16 May 2026).
3. Architectural paradigms
The principal architectural divide in Diff‑VQA is between models that make temporal comparison an explicit structural object and models that rely on a general multimodal transformer to infer change from paired inputs. Representative systems are summarized below.
| Approach | Core mechanism | Representative paper |
|---|---|---|
| Temporal token fusion | Shared image encoder, temporal embeddings, encoder–decoder generation | PLURAL (Cho et al., 2024) |
| Report-grounded reasoning | Longitudinal Projection Module plus Report Generation → Answer Generation | RG‑AG (Serra et al., 22 May 2025) |
| Saliency-conditioned comparison | Micro affine registration and shared saliency mask across time | Saliency Guided Diff‑VQA (Wu et al., 29 Sep 2025) |
| Foundation-model-guided masking | DINO prior, adaptive mask generator, auxiliary geometric losses | Attention Consistent Diff‑VQA (Wu et al., 3 Jun 2026) |
| Instruction-tuned VLM | Multi-image prompt, multi-task fine-tuning for diagnostic, difference, prognostic QA | LUMEN (Jiang et al., 24 Feb 2026) |
| Location-aware encoder pretraining | AREF, GCAP, CAREF for fine-grained spatial grounding | Location-Aware Pretraining (Musinguzi et al., 5 Mar 2026) |
PLURAL is architecturally minimal: a shared ResNet‑101 encodes both past and current radiographs, trainable temporal encodings distinguish the two streams, and a 6-layer Transformer encoder with a 6-layer decoder autoregressively generates the answer (Cho et al., 2024). The model does not handcraft a difference module; instead, self-attention over past-image tokens, current-image tokens, and question tokens is expected to learn the comparative representation.
RG‑AG adopts a more structured longitudinal representation. Each image is converted into “finding-aware anatomical tokens” via a Faster R‑CNN trained on Chest ImaGenome. A Longitudinal Projection Module fuses current and prior anatomical-region tokens, and a two-stage pipeline first generates a radiology report and then conditions answer generation on the question plus the predicted report (Serra et al., 22 May 2025). In this formulation, the report acts as an explicit intermediate textual representation of clinically salient evidence.
Saliency-guided models place anatomical consistency at the center of the design. “Saliency Guided Longitudinal Medical Visual Question Answering” uses a lightweight near-identity affine registration to coarsely align the main image to the reference image, then runs a within-epoch two-step loop: a keyword is extracted from the answer, keyword-conditioned Grad‑CAM is computed on both images, the two saliency maps are merged into a shared mask, and the masked image pair is re-encoded for answer generation (Wu et al., 29 Sep 2025). “Attention Consistent Longitudinal Medical Visual Question Answering Guided by Vision Foundation Models” replaces Grad‑CAM with a frozen DINO-based prior, a trainable adaptive mask generator, and auxiliary objectives such as mask rebuilding, pairwise Gram-style consistency, and KoLeo uniformity loss, again applying a single shared mask to both time points (Wu et al., 3 Jun 2026). In both systems, the core architectural claim is that shared attention across time is necessary for reliable comparison.
LUMEN moves in a different direction. It fine-tunes NVILA‑8B end-to-end and feeds multiple images into the VLM as distinct visual token blocks, with time interval information represented as text. There is no explicit temporal transformer or explicit difference embedding; difference reasoning emerges from instruction-tuned supervision on diagnostic, difference, and prognostic tasks (Jiang et al., 24 Feb 2026). A plausible implication is that LUMEN prioritizes scalability and prompt-level flexibility over explicit spatial correspondence modeling.
4. Supervision, pretraining, and intermediate reasoning
A defining feature of the field is the use of nonstandard supervision to compensate for limited manual longitudinal annotation. PLURAL uses a three-stage pipeline: pretrained OFA weights from natural images and text; longitudinal chest X‑ray adaptation using paired images, radiology Findings, Impression, and difference QA; and final fine-tuning on the difference subset of MIMIC‑Diff‑VQA (Cho et al., 2024). Its ablations show that both natural-image pretraining and longitudinal chest X‑ray pretraining contribute materially, and that using past images during stage‑2 medical pretraining is beneficial.
Report grounding is another major strategy. RG‑AG first generates Findings and Impression sections from the imaging input and then provides the predicted report as explicit evidence to the answer generator (Serra et al., 22 May 2025). The paper explicitly interprets this as analogous to Chain‑of‑Thought reasoning: images are translated into a structured clinical text artifact, and that artifact is then used to support final answer generation.
Location-aware pretraining attacks the representation problem at the encoder level. The proposed pretraining tasks—Automatic Referring Expressions (AREF), Grounded Captioning (GCAP), and Conditional Automatic Referring Expressions (CAREF)—teach a SigLIP-style ViT to associate chest X‑ray regions, bounding-box coordinates, and clinical phrases (Musinguzi et al., 5 Mar 2026). The downstream Diff‑VQA model itself remains simple: the pretrained encoder is frozen, a small adapter projects reference and main image tokens into GPT‑2 space, temporal embeddings disambiguate time order, and GPT‑2 medium generates the answer.
Saliency-guided systems reinterpret explanation as supervision. In the saliency-guided encoder–decoder, post-hoc Grad‑CAM is converted into actionable training structure: the keyword extracted from the answer determines where the model looks, and that shared saliency mask then modulates the image pixels used for learning (Wu et al., 29 Sep 2025). The attention-consistent extension replaces answer-conditioned Grad‑CAM with a frozen DINO prior and explicitly optimizes both supervised and unsupervised objectives simultaneously, illustrating a paradigm in which image foundation models contribute not only features but training constraints (Wu et al., 3 Jun 2026).
Instruction tuning broadens the supervisory envelope even further. LUMEN constructs a longitudinal instruction-following corpus by converting short Medical‑Diff‑VQA answers into fluent responses and generating prediction-oriented questions with Llama‑3.2‑11B‑Vision‑Instruct. It also injects expert model predictions from TorchXRayVision into prompts and uses multi-image, multi-task instruction fine-tuning to jointly train diagnostic, difference, and prognostic behavior (Jiang et al., 24 Feb 2026). This suggests that supervision in Diff‑VQA is increasingly hybrid: report-derived, model-generated, instruction-tuned, and weakly grounded rather than purely hand-labeled.
5. Evaluation and empirical findings
Evaluation protocols reflect the mixed generative and categorical nature of the task. On chest X‑ray Diff‑VQA, the standard metrics are BLEU‑1/2/3/4, METEOR, ROUGE‑L, and CIDEr (Wu et al., 29 Sep 2025). LUMEN additionally uses Llama Score for open-ended difference and prognostic evaluation (Jiang et al., 24 Feb 2026). Outside free-text CXR settings, 3D‑RAD evaluates temporal diagnosis with macro-averaged accuracy on the four-way temporal classes (Gai et al., 11 Jun 2025).
Performance figures vary markedly by architecture and benchmark slice. On the Medical‑Diff‑VQA difference subset, PLURAL reports BLEU‑4 $0.520$, METEOR $0.381$, ROUGE‑L $0.653$, and CIDEr $1.832$, improving over MCCFormers, IDCPCL, and EKAID (Cho et al., 2024). RG‑AG reports BLEU‑4 $0.551$, METEOR $0.384$, ROUGE‑L $0.668$, and CIDEr $2.198$, with additional gains when ground-truth reports are supplied instead of predicted reports (Serra et al., 22 May 2025). Location-aware pretraining reports BLEU‑4 $0.594$, METEOR $0.425$, ROUGE‑L $0.381$0, CIDEr $0.381$1, and BERTScore $0.381$2 on the difference category, outperforming Global Contrastive Pretraining, Regional Contrastive Pretraining, BLIP‑2, CapPa, RG‑AG, ReAL, and PLURAL in that comparison (Musinguzi et al., 5 Mar 2026).
The saliency-centered line reports a different profile. “Saliency Guided Longitudinal Medical Visual Question Answering” achieves BLEU‑4 $0.381$3, METEOR $0.381$4, ROUGE‑L $0.381$5, and CIDEr $0.381$6, with notably high METEOR relative to the baselines listed in that paper (Wu et al., 29 Sep 2025). “Attention Consistent Longitudinal Medical Visual Question Answering Guided by Vision Foundation Models” reports BLEU‑1 $0.381$7, METEOR $0.381$8, ROUGE‑L $0.381$9, and CIDEr $0.653$0, and its ablations attribute gains to initial encoder freezing, DINO-inspired unsupervised objectives, and the use of attention masks (Wu et al., 3 Jun 2026). These results indicate that saliency-conditioned generation can improve semantic alignment even when exact lexical overlap remains method-dependent.
Instruction-tuned longitudinal VLMs show a different strength profile. On open-ended difference questions, LUMEN† reports BLEU $0.653$1, ROUGE $0.653$2, and Llama Score $0.653$3, outperforming pretrained NVILA‑8B and diagnostic-only NVILA‑8B† with improvements that are reported as highly significant under the Wilcoxon signed rank test with $0.653$4 (Jiang et al., 24 Feb 2026). On prognostic questions it reports BLEU $0.653$5, ROUGE $0.653$6, and Llama Score $0.653$7, with the paper explicitly noting that absolute scores remain modest because prognostic reasoning is intrinsically difficult.
The 3D setting remains more challenging. In 3D‑RAD, zero-shot performance on temporal tasks is poor: the best zero-shot result on Static Temporal Diagnosis is 44.11% and on Longitudinal Temporal Diagnosis 42.99%, both from RadFM. Fine-tuning M3D models on 3D‑RAD‑T raises performance to 51.20% and 74.78% for one model variant and to 49.30% and 74.77% for another, showing that temporally structured supervision is learnable but not yet trivialized (Gai et al., 11 Jun 2025).
6. Limitations, misconceptions, and future directions
A recurrent misconception is that feeding two images into a VLM is sufficient for longitudinal reasoning. Several papers explicitly contest this. LUMEN states that it “does not explicitly model spatial correspondence” and instead relies on generic cross-attention to discover anatomy alignment (Jiang et al., 24 Feb 2026). RG‑AG likewise notes that temporal change is modeled implicitly through anatomical token fusion and report grounding rather than through explicit time intervals or linear trend modeling (Serra et al., 22 May 2025). Saliency-guided and attention-consistent models were motivated precisely by the view that consistent attention across time must be enforced rather than assumed (Wu et al., 29 Sep 2025).
Another misconception is that strong text-generation metrics guarantee clinically grounded reasoning. The UCSF‑PDGM‑VQA study, although not longitudinal, is important here because it shows that current VLMs can exhibit modality collapse and over-reliance on language priors: Lingshu‑32B performs 66.04% with a blank image, higher than with MRI, and a text-only Qwen3‑8B reaches 52.57% on the benchmark (Ghosh et al., 16 May 2026). The medical hallucination benchmark reinforces the same concern from a different angle: FAKE, NOTA, and SWAP scenarios show that models may provide confident answers when the image does not support the question, when the correct answer is “None of the above,” or when the image–question pairing is mismatched (Wu et al., 2024). For Diff‑VQA, a plausible implication is that longitudinal systems require explicit stress tests for temporal mismatch, abstention, and image grounding.
Limitations reported by the chest X‑ray literature are technically specific and consistent across papers. Saliency-guided approaches note computational overhead from Grad‑CAM in the loop, the possibility of generic or incorrect keywords from a general-purpose LLM, the limited expressiveness of simple affine registration, lack of cross-dataset validation, and the absence of radiologist user studies (Wu et al., 29 Sep 2025). The attention-consistent extension adds that DINO-based priors can be noisy, the framework increases training complexity, and metrics may still fail to capture clinical correctness (Wu et al., 3 Jun 2026). LUMEN emphasizes that only two time points are used, prognostic labels are synthetically generated, and non-imaging confounders such as treatment changes and comorbidities remain unobserved (Jiang et al., 24 Feb 2026).
The broader trajectory of the field is nevertheless clear. Survey work had already called for multiple-image input, integration of extra medical information, and clinically grounded reasoning beyond static single-image benchmarks (Lin et al., 2021). Current longitudinal work has operationalized that agenda in chest X‑ray Diff‑VQA, extended it to prognosis and 3D temporal diagnosis, and exposed the remaining challenges: explicit spatial correspondence, multi-timepoint modeling, robust clinical evaluation, and resistance to hallucination and modality collapse. This suggests that the next phase of Diff‑VQA research will likely be defined less by whether models can process paired images at all, and more by how reliably they can ground temporal claims in anatomically matched evidence across richer clinical contexts.