Predictive Entropy Links Calibration and Paraphrase Sensitivity in Medical Vision-Language Models

Abstract: Medical Vision LLMs VLMs suffer from two failure modes that threaten safe deployment mis calibrated confidence and sensitivity to question rephrasing. We show they share a common cause, proximity to the decision boundary, by benchmarking five uncertainty quantification methods on MedGemma 4BIT across in distribution MIMIC CXR and outof distribution PadChest chest X ray datasets, with cross architecture validation on LLaVA RAD7B. For well calibrated single model methods, predictive entropy from one forward pass predicts which samples will flip under rephrasing AUROC 0.711 on MedGemma, 0.878 on LLaVARAD p 10 4, enabling a single entropy threshold to flag both unreliable and rephrase sensitive predictions. A five member LoRA ensemble fails under the MIMIC PadChest shift 42.9 ECE, 34.1 accuracy, though LLaVA RAD s ensemble does not collapse 69.1. MC Dropout achieves the best calibration ECE 4.3 and selective prediction coverage 21.5 at 5 risk, yet total entropy from a single forward pass outperforms the ensemble for both error detection AUROC 0.743 vs 0.657 and paraphrase screening. Simple methods win.

• The paper shows that a single-model softmax entropy effectively links calibration quality with paraphrase sensitivity by measuring decision-boundary proximity.
• Empirical results across MIMIC-CXR and PadChest reveal that targeted LoRA fine-tuning and MC Dropout improve error detection and risk management under distribution shifts.
• The study suggests that using an entropy threshold enables unified abstention policies, offering a practical, efficient approach for safe clinical deployments.

Predictive Entropy as a Unified Signal for Calibration and Paraphrase Sensitivity in Medical Vision-LLMs

Introduction

This paper studies the failure modes of Medical Vision-LLMs (VLMs), focusing on overconfident miscalibration and paraphrase sensitivity—both critical for safe clinical deployment. The authors demonstrate empirically that these two phenomena are not independent but are mathematically and geometrically tied to proximity to the decision boundary, as measured by predictive entropy. Through comprehensive benchmarking across uncertainty quantification (UQ) methods, datasets (MIMIC-CXR in-distribution; PadChest out-of-distribution), model architectures (MedGemma-4B-IT, LLaVA-RAD-7B), and multiple distribution shifts (including synthetic corruptions), the work concludes that softmax entropy from a single fine-tuned model is a sufficient and efficient proxy for both calibration and paraphrase sensitivity detection.

Methods

The study evaluates five uncertainty estimation strategies applied to VLMs rendered as binary classifiers by extracting only “Yes” and “No” logits from autoregressively generated responses. The following UQ methods are benchmarked:

• Softmax Entropy: Baseline single forward pass entropy.
• Temperature Scaling: Scalar reparameterization of logits post-hoc on a calibration split.
• MC Dropout: $K$ stochastic passes (dropout enabled) with entropy/mutual information decomposition.
• Deep Ensemble: Probability-averaged predictions of independently fine-tuned LoRA adapters.
• Margin Magnitude: Absolute decision margins.

The central technical analysis connects paraphrase instability (“flips” under rephrasing) with entropy. For each input, paraphrase-consistent (stable) and inconsistent (flip-prone) predictions are compared via entropy statistics and AUROC, showing that higher entropy correlates strongly with paraphrase flips.

Empirical Results

Model Calibration and Distribution Shift

LoRA fine-tuning is the dominant factor in calibration quality. The Targeted LoRA model (modifying layers 15–19 with 0.1% of parameters) achieves ECE 6.1% on PadChest, compared to the base MedGemma at 44.1%. MC Dropout with $K=10$ further improves ECE to 4.3%. The Full LoRA (all layers adapted) degrades OOD calibration (ECE 51.6%), indicating that focused adaptation is essential.

Calibration is robust under synthetic distribution shifts—a notable outcome given the variety of image corruptions (Gaussian noise, blur, contrast, brightness, JPEG), with ECE only rising for mild Gaussian noise (severity 1) and generally stable or improving for other corruptions.

Figure 1: Reliability diagrams under Gaussian noise at three severity levels. Severity 1 produces the largest deviation from the diagonal (highest ECE), paradoxically worse than severity 3 and 5.

Ensemble Failure and Selective Prediction

While deep ensembles of LoRA adapters are known to provide robust uncertainty out-of-distribution, this work shows that for MedGemma-4B-IT under MIMIC-to-PadChest shift, ensemble performance collapses (ECE 42.9%, accuracy 34.1%). Per-member diagnosis reveals only one functional adapter, with the majority diluting performance—unlike the LLaVA-RAD-7B ensemble, which retains OOD generalization.

Selective prediction is most effective with MC Dropout, covering 21.5% of samples at a 5% risk target, compared to 7.3% for softmax entropy and 0% for ensembles. Thus, MC Dropout is practically recommended for operation points requiring strict low-risk guarantees.

Figure 2: Risk-coverage curves on PadChest. MC-Drop achieves the widest coverage at the 5% risk threshold (vertical dotted line). The Ensemble curve sits near the top, unable to reach low risk at any coverage.

Entropy Decomposition

Deep ensembles display meaningful epistemic uncertainty (mutual information 0.082 nats, elevated for errors), but total entropy from a single model still yields superior AUROC for error detection (0.743), indicating that more complex decomposition provides little utility in these settings.

The Predictive Entropy–Paraphrase Sensitivity Bridge

A key finding is that flip-prone samples have significantly higher entropy, consistent across all well-calibrated single-model UQ methods. On PadChest, the softmax entropy's AUROC for paraphrase flip detection is 0.711 ($p < 10^{-20}$); the bridge generalizes to LLaVA-RAD-7B (AUROC 0.878 on MIMIC). The entropy margin separating flip-prone and stable predictions is consistent (~0.09 nats), supporting a geometric interpretation: high-entropy predictions are closer to the decision boundary and more susceptible to label flip under minor perturbations.

Figure 3: Mean paraphrase margin variance for flip-prone versus stable predictions on PadChest. Flip-prone samples show higher margin variance, confirming that the bridge reflects genuine decision-boundary proximity.

Thresholding on entropy enables simultaneous reduction in both error and flip rates; e.g., at 40% coverage, error rate drops to 10.4% and flip rate to 26.4%, both substantially below the rates at full coverage.

Reliability, Visualization, and Method Comparison

Reliability diagrams demonstrate that well-finetuned LoRA models after temperature scaling and MC Dropout closely match the ideal diagonal on both in-distribution and OOD data. By contrast, ensembles succeed in-distribution but fail OOD, reinforcing that ensemble failure is not universal but model/deployment-specific.

Figure 4: Reliability diagrams on clean test data. Targeted LoRA methods (center) track the diagonal on both datasets. The deep ensemble (right) succeeds on in-distribution MIMIC but fails on OOD PadChest.

Practical and Theoretical Implications

Clinical Risk Management: The results provide a pragmatic deployment recipe: a single, well-finetuned model with softmax entropy screening delivers robust calibration, error detection, and paraphrase vulnerability identification in one fast forward pass. For operating points demanding stricter risk control, MC Dropout adds significant coverage (up to 21.5% at 5% risk) at a moderate computational cost.

Ensembles in Clinical Domain Adaptation: The findings contradict the general machine learning lore that ensembles extrapolate well under shift—demonstrating that LoRA ensembles may fail catastrophically on subtle near-distribution shifts (e.g., MIMIC to PadChest) due to aggregation over poorly adapted members, whereas other architectures (LLaVA-RAD) remain robust. This indicates the necessity of member-level OOD validation prior to deployment.

Unified Abstention Policies: By establishing that predictive entropy identifies both error-prone and paraphrase-unstable predictions, the work validates single-threshold entropy-based abstention as a theoretically justified and empirically effective criterion for safe, selective AI-assisted clinical use.

Generalization Across Architectures and Shifts: The entropy-paraphrase sensitivity bridge holds across multiple model families and diverse distribution shifts, supporting the geometric hypothesis that uncertainty, as measured by entropy, corresponds to decision-boundary proximity irrespective of model specifics.

Future Directions

• Semantic Uncertainty for Open-Ended Outputs: Extending the geometric analysis to semantic entropy for free-text VQA is a natural next step for generalizing paraphrase sensitivity findings to unconstrained clinical queries.
• Ensemble Member Diversity: Exploring training diversifications and member selection strategies may resolve ensemble pathologies seen in MedGemma; more robust aggregation mechanisms are needed for domain adaptation.
• Distribution-Free Guarantees: Integration of distribution-free conformal prediction frameworks can provide statistical guarantees for selective prediction under OOD shift and subgroup variability.
• Clinical Multi-site Validation: Real-world, multi-center deployment studies are required to further stress-test the calibration and abstention protocols under authentic, complex shifts.

Conclusion

The paper provides compelling evidence that, with focused fine-tuning, the predictive entropy of a single VLM not only resolves both calibration and paraphrase sensitivity but does so robustly across distribution shift and architectures. This supports the adoption of simple, entropy-thresholded abstention as a unifying, risk-controlling mechanism for medical VLM deployment. Nevertheless, ensemble-based uncertainty estimation in LoRA-adapted models requires rigorous cross-site validation, as failure modes are architecture- and adaptation-specific. The unified entropy-based abstention strategy is immediately actionable for practitioners seeking safe and interpretable AI support in clinical decision-making.