Assessing VLM Reliability for Medical Image Quality Evaluation Under Corruption and Bias

Abstract: Vision-LLMs (VLMs) are increasingly applied in medical tasks such as pathology description, report generation, and visual question answering. Medical Image Quality Assessment (MIQA) supports diagnostic accuracy and patient safety by determining whether images meet the standards required for clinical decision-making. Automating MIQA with VLMs may reduce workload, but their behavior under real-world conditions, where images may be degraded or textual context may affect judgments, should be further explored before deployment. We benchmark VLMs on medical image quality using the MediMeta-C dataset zero-shot across seven corruption types and five severity levels. We evaluate sensitivity to degradation patterns, the effect of corruptions on embedding geometry, and whether textual attributes (demographics, expertise, infrastructure, institution) alter scores. Across 16 VLMs and seven modalities, pixelation produced the largest score reductions (mean -20.58%, up to -34.4% for OCT), whereas brightness had limited effect (-0.81%). Embedding displacement was associated with score changes. Same-family models showed correlations of 0.67-0.83; some produced increases up to +31% for corrupted mammography. Textual attributes affected scores: institutional prestige raised them +17.15%, and equipment age lowered them -14.7%. The largest changes were +95.62% (InternVL-8B) and -37.7% (MedGemma). Current VLMs show limitations for medical image quality assessment. Pixelation, a privacy-preserving transformation, reduces performance, indicating a trade-off between patient privacy and reliability. Sensitivity to contextual metadata indicates limited objectivity and marks metadata as a privacy and bias source. Privacy protection and objective quality assessment are related requirements for use.

• The paper systematically evaluates 16 VLMs on the MediMeta-C benchmark, revealing significant sensitivity to diverse image corruptions.
• The paper employs embedding analysis via PCA to correlate latent representation shifts with quality score drops, emphasizing robustness challenges.
• The paper exposes pronounced biases, showing that contextual metadata (e.g., institutional prestige) can dramatically alter MIQA scores.

Assessing the Reliability of Vision-LLMs for Medical Image Quality Evaluation Under Corruption and Bias

Introduction

Vision-LLMs (VLMs) have garnered significant attention in medical AI due to their multimodal capabilities, extending the foundational architecture of LLMs to integrate visual content for tasks including image captioning, report generation, and visual question answering. Given the high stakes of medical decision-making, the reliability of VLM-based Medical Image Quality Assessment (MIQA) is a critical prerequisite for their safe integration into clinical pipelines. This work presents a systematic evaluation of 16 open-source VLMs, both general purpose and medical-specialized, focusing on their robustness to image corruptions and contextual biases using the MediMeta-C benchmark. The study investigates: (1) sensitivity of MIQA scores to various degradations, (2) latent representation shifts in response to corruptions, and (3) susceptibility to textual prompt biases reflecting realistic metadata.

Figure 1: Pipeline of analyzing the behavior of VLM-as-a-Judge for medical image quality assessment on corrupted images of MediMeta-C.

Experimental Design

Diverse Model and Corruption Coverage

The evaluation includes 11 generalist and 5 medical-specific VLMs, covering a spectrum of parameter scales and architectures (e.g., MedGemma, Lingshu, Qwen-VL, LLava, InternVL). Zero-shot MIQA is performed on the MediMeta-C dataset, encompassing seven medical imaging modalities (cell microscopy, breast imaging, chest X-ray, funduscopy, retinal OCT) and seven realistic corruption types (motion blur, zoom blur, Gaussian noise, impulse noise, brightness, contrast, and pixelation) each at five severity levels.

Quality is rated on a 1–5 scale using standardized prompts, with modifications to metadata for the bias analysis.

Embedding-Based Robustness Analysis

Model behavior is further interrogated via geometric analysis of latent joint image embeddings using PCA and centroid distance metrics to elucidate the sensitivity of latent space structure to input perturbations.

Figure 2: Two-dimensional PCA visualization of Llava-v1.6-mistral-7B embeddings for clean and corrupted fundus images, illustrating clear separation in the latent space.

Systematic Bias Injection

Metadata is incorporated into prompts to simulate four classes of textual context: operator expertise, institutional prestige, equipment quality, and provider demography, enabling measurement of systematic biases in VLM scoring behavior.

Results

Baseline Quality Scoring and Modality Differences

Clean image scoring demonstrates significant inter-model and inter-modality variability. MedGemma-27B yields the highest average scores (3.994/5), while mammography consistently scores lower, reflecting either intrinsic modality attributes or model limitations in processing low-contrast and subtle-detail diagnostic features.

Figure 3: Comparative radar chart showing average MIQA scores across VLMs on clean images for each modality.

Medical-specialist VLMs generally assign higher quality scores to clean images than non-medical models, suggesting that medical domain adaptation confers improved alignment with clinical image distribution.

Sensitivity to Image Corruptions

Pixelation induces the most aggressive MIQA score reductions, averaging -20.58% across models and -34.4% for OCT, compared to negligible changes for brightness manipulations (-0.81%). This reflects prioritization of fine spatial detail and textural integrity in model quality evaluation functions. MedGemma models display the steepest consistency in penalizing spatially disruptive corruptions.

Figure 4: Radar chart illustrating model-specific and corruption-specific average MIQA score changes.

Notably, anomalous score increases for corrupted mammography images are apparent in certain VLMs (e.g., InternVL-14B, +31.4%), suggesting misinterpretation of noise artifacts as potential pathology—a nontrivial risk in automated MIQA deployment.

Embedding Displacement and Score Correlation

Latent centroid analysis shows that quality score drops closely track increased Euclidean distance between clean and corrupted image embeddings; strongest for pixelation, weakest for brightness. For OCT-pixelation, $d^c=69.99$, reflecting severe semantic shift in the representation space underlying pronounced MIQA downgrades.

Figure 5: Average distance $d^{c}$ between clean and corrupted image embedding centroids per corruption and modality.

Within-family model correlations for MIQA scores are high (0.67–0.83), indicating strong architectural/training imprint on evaluation heuristics. Outlier models with negligible correlation demonstrate divergent, possibly misaligned, quality assessment criteria.

Impact of Prompt Metadata and Contextual Bias

Systematic prompt manipulation exposes major bias vulnerabilities: institutional prestige metadata causes average MIQA increases of +17.15%; negative equipment descriptions degrade scores up to -14.7%. Extreme outliers include InternVL-8B (+95.6% on MAMMO_CALC with context bias) and MedGemma models (-37.7% on "old device" prompts). These artifacts are independent of visual input, indicating that MIQA outputs are highly susceptible to irrelevant textual context.

Figure 6: Heatmap of average MIQA percentage change across bias types and modalities, highlighting cases of severe context-driven evaluation skew.

Discussion and Implications

Clinical Reliability and Failure Modes

Robust MIQA requires VLM sensitivity to technical image degradation, not mislabeling of artifact patterns as diagnostic features. The consistent penalizing of spatial/signal-corrupting artifacts aligns with clinician expectations, yet the occasional score increases for “noisier” images underscore a pathological shortcut—likely a result of data or annotation confounds in pretraining, especially for mammography.

Misalignment between embedding space geometry and output invariance further highlights the need for stricter constraints in learning objectives if VLMs are to be deployed for quality control in safety-critical workflows.

Metadata Sensitivity and Privacy Risks

The observed magnitude of text-induced scoring bias demonstrates that VLMs inherently conflate non-visual contextual information with objective image quality: provider race, institutional reputation, and device age directly alter scores despite identical imaging data. This presents dual threats: (1) technical—undermining algorithmic fairness and objectivity in downstream MIQA-based automation, potentially propagating health disparities and institutional bias; (2) privacy—sensitive metadata can be leveraged to systematically alter VLM decision outputs, paralleling privacy risks associated with DICOM metadata leakage.

Future Research Trajectories

The results motivate several research directions:

• Debiasing and Fairness: Systematic removal or separate modeling of metadata effects, integrating fairness constraints into MIQA objectives.
• Adversarial Robustness: Training and evaluation against adversarial corruption and contextual attacks; strong regularization of context dependence.
• Latent Space Supervision: Introducing explicit constraints linking geometric shifts in the latent embedding space to known physical degradation metrics.
• Benchmarks and Transparency: Public MIQA benchmarks encompassing realistic clinical artifacts and diverse metadata, fostering reproducible, trustworthy development.

Conclusion

The study rigorously demonstrates that state-of-the-art VLMs exhibit substantial reliability limitations for Medical Image Quality Assessment in both robustness to technical corruptions and objectivity under contextual bias. The strong link between embedding displacement and scoring behavior, combined with severe susceptibility to metadata-induced bias, underscores the necessity of targeted debiasing and robustness measures prior to clinical deployment. Privacy, fairness, and accuracy are tightly coupled in VLM-based quality control, and future research must systematically address these axes to enable safe integration of multimodal models into healthcare workflows.

---

Reference: "Assessing VLM Reliability for Medical Image Quality Evaluation Under Corruption and Bias" (2607.01973)