---
title: 'Diff-VQA: Longitudinal Medical Visual QA'
url: https://www.emergentmind.com/topics/longitudinal-medical-visual-question-answering-diff-vqa
type: topic
---

# Diff-VQA: Longitudinal Medical Visual QA

Longitudinal medical visual question answering, commonly termed Diff‑VQA, is the task of answering natural-language questions about clinically meaningful change between temporally separated medical images of the same patient. In chest radiography, the canonical setting pairs a current or *main* chest X‑ray with an earlier *reference* study and asks whether a finding has appeared, resolved, increased, decreased, or remained stable. The central difficulty is that the informative signal is not the absolute content of either image alone, but the medically relevant difference between them, together with anatomically consistent comparison across time [2509.25374]. Recent work has treated this problem as a distinct subfield of medical VQA, developed dedicated datasets from MIMIC‑CXR, and extended the paradigm toward prognosis, report-grounded reasoning, and 3D temporal diagnosis [2402.08966].

## 1. Task definition and clinical scope

Diff‑VQA differs from conventional single-image medical VQA in three ways that recur across the literature. First, the question is explicitly about temporal change rather than static appearance. Second, the model must compare corresponding anatomy across visits while discounting nuisance variation such as positioning, scale, or acquisition differences. Third, the answer is often generative, requiring a clinically meaningful description of change rather than a one-time categorical label [2509.25374].

This framing mirrors routine radiological workflow. Multiple papers emphasize that radiologists rarely interpret a chest X‑ray in isolation: they compare current and prior exams to assess treatment response, distinguish chronic from acute disease, identify subtle new abnormalities, and judge whether a lesion has progressed, regressed, or remained stable [2602.21142]. In this sense, Diff‑VQA is not simply “multi-image VQA”; it is longitudinal reasoning over anatomically corresponding evidence.

The broader medical-VQA literature had already identified the lack of multi-image input as a major gap. The survey literature notes that medical studies often include multiple views, sequential slices, and contextual information, yet most VQA datasets and models historically reduced the problem to a single image and a single question [2111.10056]. Diff‑VQA can therefore be understood as a response to that structural mismatch between benchmark design and clinical practice.

## 2. Datasets and task formulations

The dominant benchmark family is Medical‑Diff‑VQA, derived from MIMIC‑CXR. One formulation describes 164,223 samples consisting of paired chest radiographs from the same patient, each paired with a difference-focused question and answer, split into 131,556 training, 16,278 validation, and 16,389 test samples [2509.25374]. Another line of work describes Medical‑Diff‑VQA as 700,703 QA pairs built from longitudinal CXR pairs, including seven categories—abnormality, presence, view, location, level, type, and difference—with the *difference* category defining the core Diff‑VQA task [2505.16624]. This suggests that published works count either paired-image samples or all generated QA instances, depending on whether they focus on the longitudinal subset alone or the full mixed-task corpus.

LUMEN makes this split explicit by distinguishing **Diagnostic VQA** from **Difference / Diff‑VQA** and further adding **Prognostic VQA**. In that framework, diagnostic questions depend on a single image, difference questions compare a *main* and *reference* image, and prognostic questions ask about expected future changes using paired images plus temporal metadata injected as text tokens in the prompt [2602.21142]. The longitudinal extension is thus not limited to retrospective comparison; it can also be cast as present-to-future prediction.

Longitudinal reasoning has also been formalized beyond 2D chest radiography. The 3D‑RAD dataset for chest CT introduces **Static Temporal Diagnosis** and **Longitudinal Temporal Diagnosis**. Task 5 requires inference of one of four temporal categories—**Refractory Lesion**, **Resolved Lesion**, **New Lesion**, **No Abnormality**—from a single current 3D scan, while Task 6 adds a symbolic history of prior labels in the question, making it a label-sequence–augmented analogue of Diff‑VQA [2506.11147]. By contrast, UCSF‑PDGM‑VQA, although motivated by neuro-oncology workflows that are inherently longitudinal, is explicitly single-time-point and filters out questions about progression, recurrence, prior scans, or temporal comparison [2605.17140].

## 3. Architectural paradigms

The principal architectural divide in Diff‑VQA is between models that make temporal comparison an explicit structural object and models that rely on a general multimodal transformer to infer change from paired inputs. Representative systems are summarized below.

| Approach | Core mechanism | Representative paper |
|---|---|---|
| Temporal token fusion | Shared image encoder, temporal embeddings, encoder–decoder generation | PLURAL [2402.08966] |
| Report-grounded reasoning | Longitudinal Projection Module plus Report Generation → Answer Generation | RG‑AG [2505.16624] |
| Saliency-conditioned comparison | Micro affine registration and shared saliency mask across time | Saliency Guided Diff‑VQA [2509.25374] |
| Foundation-model-guided masking | DINO prior, adaptive mask generator, auxiliary geometric losses | Attention Consistent Diff‑VQA [2606.06534] |
| Instruction-tuned VLM | Multi-image prompt, multi-task fine-tuning for diagnostic, difference, prognostic QA | LUMEN [2602.21142] |
| Location-aware encoder pretraining | AREF, GCAP, CAREF for fine-grained spatial grounding | Location-Aware Pretraining [2603.04950] |

PLURAL is architecturally minimal: a shared ResNet‑101 encodes both past and current radiographs, trainable temporal encodings distinguish the two streams, and a 6-layer Transformer encoder with a 6-layer decoder autoregressively generates the answer [2402.08966]. The model does not handcraft a difference module; instead, self-attention over past-image tokens, current-image tokens, and question tokens is expected to learn the comparative representation.

RG‑AG adopts a more structured longitudinal representation. Each image is converted into “finding-aware anatomical tokens” via a Faster R‑CNN trained on Chest ImaGenome. A Longitudinal Projection Module fuses current and prior anatomical-region tokens, and a two-stage pipeline first generates a radiology report and then conditions answer generation on the question plus the predicted report [2505.16624]. In this formulation, the report acts as an explicit intermediate textual representation of clinically salient evidence.

Saliency-guided models place anatomical consistency at the center of the design. “Saliency Guided Longitudinal Medical Visual Question Answering” uses a lightweight near-identity affine registration to coarsely align the main image to the reference image, then runs a within-epoch two-step loop: a keyword is extracted from the answer, keyword-conditioned Grad‑CAM is computed on both images, the two saliency maps are merged into a shared mask, and the masked image pair is re-encoded for answer generation [2509.25374]. “Attention Consistent Longitudinal Medical Visual Question Answering Guided by Vision Foundation Models” replaces Grad‑CAM with a frozen DINO-based prior, a trainable adaptive mask generator, and auxiliary objectives such as mask rebuilding, pairwise Gram-style consistency, and KoLeo uniformity loss, again applying a single shared mask to both time points [2606.06534]. In both systems, the core architectural claim is that shared attention across time is necessary for reliable comparison.

LUMEN moves in a different direction. It fine-tunes NVILA‑8B end-to-end and feeds multiple images into the VLM as distinct visual token blocks, with time interval information represented as text. There is no explicit temporal transformer or explicit difference embedding; difference reasoning emerges from instruction-tuned supervision on diagnostic, difference, and prognostic tasks [2602.21142]. A plausible implication is that LUMEN prioritizes scalability and prompt-level flexibility over explicit spatial correspondence modeling.

## 4. Supervision, pretraining, and intermediate reasoning

A defining feature of the field is the use of nonstandard supervision to compensate for limited manual longitudinal annotation. PLURAL uses a three-stage pipeline: pretrained OFA weights from natural images and text; longitudinal chest X‑ray adaptation using paired images, radiology **Findings**, **Impression**, and difference QA; and final fine-tuning on the *difference* subset of MIMIC‑Diff‑VQA [2402.08966]. Its ablations show that both natural-image pretraining and longitudinal chest X‑ray pretraining contribute materially, and that using past images during stage‑2 medical pretraining is beneficial.

Report grounding is another major strategy. RG‑AG first generates *Findings* and *Impression* sections from the imaging input and then provides the predicted report as explicit evidence to the answer generator [2505.16624]. The paper explicitly interprets this as analogous to Chain‑of‑Thought reasoning: images are translated into a structured clinical text artifact, and that artifact is then used to support final answer generation.

Location-aware pretraining attacks the representation problem at the encoder level. The proposed pretraining tasks—**Automatic Referring Expressions (AREF)**, **Grounded Captioning (GCAP)**, and **Conditional Automatic Referring Expressions (CAREF)**—teach a SigLIP-style ViT to associate chest X‑ray regions, bounding-box coordinates, and clinical phrases [2603.04950]. The downstream Diff‑VQA model itself remains simple: the pretrained encoder is frozen, a small adapter projects reference and main image tokens into GPT‑2 space, temporal embeddings disambiguate time order, and GPT‑2 medium generates the answer.

Saliency-guided systems reinterpret explanation as supervision. In the saliency-guided encoder–decoder, post-hoc Grad‑CAM is converted into actionable training structure: the keyword extracted from the answer determines where the model looks, and that shared saliency mask then modulates the image pixels used for learning [2509.25374]. The attention-consistent extension replaces answer-conditioned Grad‑CAM with a frozen DINO prior and explicitly optimizes both supervised and unsupervised objectives simultaneously, illustrating a paradigm in which image foundation models contribute not only features but training constraints [2606.06534].

Instruction tuning broadens the supervisory envelope even further. LUMEN constructs a longitudinal instruction-following corpus by converting short Medical‑Diff‑VQA answers into fluent responses and generating prediction-oriented questions with Llama‑3.2‑11B‑Vision‑Instruct. It also injects expert model predictions from TorchXRayVision into prompts and uses multi-image, multi-task instruction fine-tuning to jointly train diagnostic, difference, and prognostic behavior [2602.21142]. This suggests that supervision in Diff‑VQA is increasingly hybrid: report-derived, model-generated, instruction-tuned, and weakly grounded rather than purely hand-labeled.

## 5. Evaluation and empirical findings

Evaluation protocols reflect the mixed generative and categorical nature of the task. On chest X‑ray Diff‑VQA, the standard metrics are **BLEU‑1/2/3/4**, **METEOR**, **ROUGE‑L**, and **CIDEr** [2509.25374]. LUMEN additionally uses **Llama Score** for open-ended difference and prognostic evaluation [2602.21142]. Outside free-text CXR settings, 3D‑RAD evaluates temporal diagnosis with macro-averaged accuracy on the four-way temporal classes [2506.11147].

Performance figures vary markedly by architecture and benchmark slice. On the Medical‑Diff‑VQA difference subset, PLURAL reports BLEU‑4 \(0.520\), METEOR \(0.381\), ROUGE‑L \(0.653\), and CIDEr \(1.832\), improving over MCCFormers, IDCPCL, and EKAID [2402.08966]. RG‑AG reports BLEU‑4 \(0.551\), METEOR \(0.384\), ROUGE‑L \(0.668\), and CIDEr \(2.198\), with additional gains when ground-truth reports are supplied instead of predicted reports [2505.16624]. Location-aware pretraining reports BLEU‑4 \(0.594\), METEOR \(0.425\), ROUGE‑L \(0.747\), CIDEr \(2.997\), and BERTScore \(0.972\) on the difference category, outperforming Global Contrastive Pretraining, Regional Contrastive Pretraining, BLIP‑2, CapPa, RG‑AG, ReAL, and PLURAL in that comparison [2603.04950].

The saliency-centered line reports a different profile. “Saliency Guided Longitudinal Medical Visual Question Answering” achieves BLEU‑4 \(0.341\), METEOR \(0.651\), ROUGE‑L \(0.627\), and CIDEr \(1.263\), with notably high METEOR relative to the baselines listed in that paper [2509.25374]. “Attention Consistent Longitudinal Medical Visual Question Answering Guided by Vision Foundation Models” reports BLEU‑1 \(0.747\), METEOR \(0.700\), ROUGE‑L \(0.703\), and CIDEr \(2.011\), and its ablations attribute gains to initial encoder freezing, DINO-inspired unsupervised objectives, and the use of attention masks [2606.06534]. These results indicate that saliency-conditioned generation can improve semantic alignment even when exact lexical overlap remains method-dependent.

Instruction-tuned longitudinal VLMs show a different strength profile. On open-ended difference questions, LUMEN† reports BLEU \(0.375\), ROUGE \(0.656\), and Llama Score \(4.611\), outperforming pretrained NVILA‑8B and diagnostic-only NVILA‑8B† with improvements that are reported as highly significant under the Wilcoxon signed rank test with \(p < 0.001\) [2602.21142]. On prognostic questions it reports BLEU \(0.095\), ROUGE \(0.303\), and Llama Score \(4.866\), with the paper explicitly noting that absolute scores remain modest because prognostic reasoning is intrinsically difficult.

The 3D setting remains more challenging. In 3D‑RAD, zero-shot performance on temporal tasks is poor: the best zero-shot result on **Static Temporal Diagnosis** is 44.11% and on **Longitudinal Temporal Diagnosis** 42.99%, both from RadFM. Fine-tuning M3D models on 3D‑RAD‑T raises performance to 51.20% and 74.78% for one model variant and to 49.30% and 74.77% for another, showing that temporally structured supervision is learnable but not yet trivialized [2506.11147].

## 6. Limitations, misconceptions, and future directions

A recurrent misconception is that feeding two images into a VLM is sufficient for longitudinal reasoning. Several papers explicitly contest this. LUMEN states that it “does not explicitly model spatial correspondence” and instead relies on generic cross-attention to discover anatomy alignment [2602.21142]. RG‑AG likewise notes that temporal change is modeled implicitly through anatomical token fusion and report grounding rather than through explicit time intervals or linear trend modeling [2505.16624]. Saliency-guided and attention-consistent models were motivated precisely by the view that consistent attention across time must be enforced rather than assumed [2509.25374].

Another misconception is that strong text-generation metrics guarantee clinically grounded reasoning. The UCSF‑PDGM‑VQA study, although not longitudinal, is important here because it shows that current VLMs can exhibit modality collapse and over-reliance on language priors: Lingshu‑32B performs 66.04% with a blank image, higher than with MRI, and a text-only Qwen3‑8B reaches 52.57% on the benchmark [2605.17140]. The medical hallucination benchmark reinforces the same concern from a different angle: FAKE, NOTA, and SWAP scenarios show that models may provide confident answers when the image does not support the question, when the correct answer is “None of the above,” or when the image–question pairing is mismatched [2401.05827]. For Diff‑VQA, a plausible implication is that longitudinal systems require explicit stress tests for temporal mismatch, abstention, and image grounding.

Limitations reported by the chest X‑ray literature are technically specific and consistent across papers. Saliency-guided approaches note computational overhead from Grad‑CAM in the loop, the possibility of generic or incorrect keywords from a general-purpose LLM, the limited expressiveness of simple affine registration, lack of cross-dataset validation, and the absence of radiologist user studies [2509.25374]. The attention-consistent extension adds that DINO-based priors can be noisy, the framework increases training complexity, and metrics may still fail to capture clinical correctness [2606.06534]. LUMEN emphasizes that only two time points are used, prognostic labels are synthetically generated, and non-imaging confounders such as treatment changes and comorbidities remain unobserved [2602.21142].

The broader trajectory of the field is nevertheless clear. Survey work had already called for multiple-image input, integration of extra medical information, and clinically grounded reasoning beyond static single-image benchmarks [2111.10056]. Current longitudinal work has operationalized that agenda in chest X‑ray Diff‑VQA, extended it to prognosis and 3D temporal diagnosis, and exposed the remaining challenges: explicit spatial correspondence, multi-timepoint modeling, robust clinical evaluation, and resistance to hallucination and modality collapse. This suggests that the next phase of Diff‑VQA research will likely be defined less by whether models can process paired images at all, and more by how reliably they can ground temporal claims in anatomically matched evidence across richer clinical contexts.

Source: https://www.emergentmind.com/topics/longitudinal-medical-visual-question-answering-diff-vqa