Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models

Abstract: Large Vision LLMs (LVLMs) show promise in medical applications, but their inability to faithfully ground responses in visual evidence raises serious concerns about clinical trustworthiness. While visual attribution methods are widely used to explain LVLM predictions, whether these explanations actually reflect the visual evidence underlying the model's decision is largely unverified, since ground-truth annotations for internal model reasoning are typically unavailable. We address this question for chest X-ray (CXR) reasoning by developing a causal evaluation framework that retains only CXR-VQA samples for which the expert-annotated region is verified, via counterfactual editing, to be causally responsible for the model's prediction. Using this framework across 11 attribution methods, six open-source LVLMs, and two output modes (direct answer and step-by-step reasoning), we find that existing attribution methods often fail to identify the evidence used by LVLMs. To address this failure, we propose MedFocus, a concept-based attribution method that localizes clinically meaningful anatomical regions via unbalanced optimal transport and measures their causal effect on model outputs through targeted interventions. MedFocus produces spatial, concept-level, and token-level attributions and substantially outperforms prior methods, taking a step toward more trustworthy attribution for medical LVLMs. Our data and code are available at https://github.com/gzxiong/medfocus/.

• The paper introduces MedFocus, a novel concept-guided attribution method that links model outputs to clinically meaningful anatomical regions using causal intervention.
• It presents MedGround-Bench, a rigorous evaluation framework employing counterfactual edits to verify that visual attributions accurately reflect model reasoning.
• Comparative analyses show that traditional attribution methods underperform, while MedFocus achieves significantly higher IoU, F1, and balanced precision-recall metrics.

Causal Evaluation and Concept-Guided Attribution for Chest X-ray Reasoning in LVLMs

Introduction and Motivation

Large Vision LLMs (LVLMs) have demonstrated robust performance in medical imaging tasks, but their inability to provide reliable visual attributions remains a barrier to clinical adoption due to trust and safety concerns. Existing attribution methods—gradient-based, attention-based, perturbation-based, and prompting-based—are commonly used to explain predictions of these multimodal models. However, the extent to which such attributions genuinely reflect model-internal evidence use, especially in medical contexts with high stakes, has been largely unverified due to the absence of objective ground truth for model reasoning. The work under review addresses this critical gap by proposing a causal framework to evaluate attribution methods and introducing MedFocus, a concept-based method grounded in anatomical regions.

MedGround-Bench: Causal Evaluation Framework

The core methodological contribution is the introduction of MedGround-Bench, a benchmark for evaluating attribution faithfulness using chest X-ray (CXR) data. Samples from three expert-annotated datasets (ImaGenome, VinDR-CXR, PadChest-GR) are reformulated as binary visual question answering (VQA) tasks, each paired with human-attributed bounding boxes. Since traditional evaluation relies on the assumption that ground-truth boxes reflect the model’s reasoning, this work proposes a more principled approach: samples are rigorously filtered through three steps—requiring correct model answers, foreground region counterfactual editing (removing the attribute), and background editing (ensuring only the foreground drives answer changes). The resulting benchmark contains only those samples for which model predictions are causally linked to the annotated region, permitting objective faithfulness evaluation for attribution methods.

Figure 1: Overview of the construction of MedGround-Bench for CXR attribution evaluation.

MedFocus: Concept-based Causal Attribution

MedFocus is introduced as an attribution technique that links model decisions to clinically meaningful anatomical regions. MedFocus first segments images into 11 anatomical regions using unbalanced optimal transport (UOT) to align the target image with references, refined by MedSAM for mask quality. For each concept, a counterfactual image is produced by zero-masking the region, and the causal effect on model output is quantified by measuring the change in output log-probability sequence, conditioned on the original answer. The maximal probability drop identifies the region most causally relevant to the decision. MedFocus’s design yields spatial attributions, interpretable concept-level textual explanations, and token-level rationales for reasoning outputs.

Figure 2: Overview of the proposed MedFocus attribution pipeline. Words significantly affected by the perturbation are highlighted in red.

Comparative Analysis of Attribution Methods

An extensive evaluation across 11 attribution methods demonstrates that existing approaches—spanning attention, gradient, prompting, and perturbation—reliably fail to recover the ground-truth visual evidence as defined by MedGround-Bench. For instance, while some gradient-based attention methods reach high recall, their precision is low, leading to overly broad attributions. GradCAM and Integrated Gradients underperform in both spatial localization and recall compared to their use in unimodal or classification settings. Prompting-based methods, either directly or via MedSAM segmentation, offer improved precision but their recall is not competitive.

In contrast, MedFocus consistently attains the highest Intersection-over-Union (IoU), F1, and balanced precision-recall on all datasets and model families evaluated. For instance, on ImaGenome-Direct, MedFocus attains 54.24% IoU and 67.54% F1, substantially outperforming all baselines. These trends persist across both direct and step-by-step reasoning tasks and for both generalist and medical-specialized model architectures.

Figure 3: Reasoning attribution evaluation on MedGround-Bench-Reason. Metrics are averaged across all models.

Representative visualizations further emphasize the qualitative difference: other methods generate diffuse or misplaced heatmaps, whereas MedFocus highlights more precise, clinically interpretable evidence.

Figure 4: Qualitative comparison on three MedGround-Bench-Direct examples. Ground-truth evidence is shown in red and predicted attributions are in yellow.

MedFocus also uniquely enables token-level attributions for reasoning chains, revealing the progression from coarse to specific evidence use within the generated explanation.

Figure 5: Token-level concept attribution for a MedGround-Bench-Reason example.

Model and Group-wise Attribution Behavior

Analysis across models and filtering stages confirms that both medical-domain pretraining and increased model scale improve attribution faithfulness. Consistently higher IoU and lower failure rates are observed for MedGemma and MedGemma1.5 variants compared to general-purpose Gemma3 and Qwen2.5-VL. Moreover, generating step-by-step reasoning is found to decrease the likelihood of models relying on non-anatomical visual cues, further evidenced by decreasing attribution failure rates in the reasoning mode relative to direct answers.

Figure 6: Comparison of MedFocus attributions across models and sample groups.

Qualitative and Finer-grained Model Comparison

Comparison across multiple model variants on challenging examples (e.g., small focal lesions or presence of metal) shows that even medical-specialized models can produce broad attributions, but are qualitatively tighter and more relevant than those from generalist architectures.

Figure 7: Qualitative comparison of MedFocus spatial attributions across Gemma3 and MedGemma variants on three representative MedGround-Bench examples. Ground-truth evidence is shown in red and predicted attributions are shown in yellow.

Token-level attributions for complex reasoning questions further highlight the superiority of medical-adapted models in grounding token probabilities in meaningful visual evidence.

Figure 8: Token-level concept attribution for a reasoning example about osteosynthesis material. Colored words indicate tokens whose probabilities are most affected by concept intervention, and the corresponding highlighted regions show the attributed evidence.

Implications, Limitations, and Future Directions

This work provides strong empirical evidence that commonly deployed attribution techniques are unreliable in the context of medical LVLM reasoning when evaluated under causal criteria, challenging assumptions underlying their widespread clinical use. MedFocus, by directly connecting model output changes to explainable anatomical concepts, provides more faithful and interpretable attribution. This approach could serve as a foundation for future explainability standards in clinical AI, where both faithfulness and clinical interpretability are non-negotiable.

Practically, this method establishes a new bar for deployment of explainability in high-stakes applications, and the public release of the MedGround-Bench resource sets a framework for further methodological advances and cross-modality extensions. Theoretically, the demonstrated failure of saliency- and attention-based heuristics under causal scrutiny suggests the need to rethink attributions as inherently interventional and concept-driven. Limitations include restriction to CXR due to available annotation and editing models, and binary VQA format, though both can be extended as tooling and more sophisticated datasets emerge.

Anticipated future research includes broadening the causal framework to richer diagnostic and report-like tasks and diverse modalities (CT, MRI), and investigating human-AI alignment in evidential reasoning using causal benchmarks. Further, advances in LVLM architectures and domain-specific pretraining are expected to synergistically improve both attribution reliability and clinical utility.

Conclusion

This work establishes that robust, clinically meaningful visual attribution in medical LVLMs cannot be reliably achieved via traditional post-hoc heuristics. The causal sample construction of MedGround-Bench and the concept-driven, intervention-based design of MedFocus together demonstrate a path toward faithful, interpretable, and actionable model explanations essential for safe and trustworthy deployment in clinical decision support systems.