MedQ-Reasoning: Structured Clinical IQA

• MedQ-Reasoning is a paradigm that evaluates multi-modal LLMs by requiring detailed, structured clinical explanations for medical image quality assessments.
• It integrates no-reference and comparison reasoning tasks, prompting models to identify image modalities, evaluate degradation severity, and justify clinical judgments.
• The framework employs multi-dimensional scoring metrics and human–AI alignment studies to highlight model limitations and guide improvements in diagnostic workflows.

MedQ-Reasoning is a structured evaluation paradigm within MedQ-Bench, created to assess the capacity of multi-modal LLMs (@@@@1@@@@) to produce transparent, human-like clinical reasoning for medical image quality assessment (IQA). Unlike traditional IQA, which relies on scalar score prediction, MedQ-Reasoning requires models to generate explicit, context-aware explanations for image quality degradations and their clinical impact, closely aligning with the qualitative reporting standards adopted by expert radiologists (Liu et al., 2 Oct 2025).

1. Task Definition and Reasoning Paradigm

MedQ-Reasoning is composed of two primary sub-tasks:

• No-reference Reasoning: The model receives a single medical image and must autonomously generate a structured response covering:
  • Image modality and anatomical region identification,
  • Enumeration and severity assessment of all observable quality degradations (e.g., blur, noise, artifacts),
  • Causal explanation for each degradation,
  • A clinical judgment (e.g., “good”, “usable”, “reject”) rooted in the visual attributes alone.
• Comparison Reasoning: The model is provided with a pair of images (e.g., original vs. reconstructed) and must:
  • Conduct a fine-grained comparison,
  • Decide which image is of higher clinical quality,
  • Justify this decision with a natural language explanation that analyses and prioritizes observed quality differences.

These tasks simulate radiologists’ workflow, demanding both perceptual acumen and an ability to reason about how visual degradations affect interpretability.

2. Multi-dimensional Judging Protocol and Evaluation Metrics

MedQ-Reasoning introduces a multi-dimensional scoring system to account for the subjectivity and multi-faceted nature of explanatory judgments. Model outputs (𝒪) are compared to expert-crafted references (𝑅) and evaluated along four discrete axes (scores in {0, 1, 2}):

Metric	Definition	Formula (LaTeX notation)
Completeness	Captures coverage of all key degradation cues	$$C(𝒪, 𝑅) = \frac{1}{|\mathcal{K}_R|}\sum_{k \in \mathcal{K}_R} \mathbf{1}[k \in \mathcal{K}_O]$$
Preciseness	Penalizes hallucinations and contradictions	$$P(𝒪, 𝑅) = 1 - \frac{1}{|\mathcal{K}_O|} \sum_{k \in \mathcal{K}_O} \mathbf{1}[\text{contradict}(k, R)]$$
Consistency	Alignment between reasoning trace and final judgment	$$S(𝒪, 𝑅) = f_\text{consistency}(\text{reasoning}(𝒪), \text{conclusion}(𝒪), R)$$
Quality Accuracy	Exact match of final quality or comparison decision	$$Q(𝒪, 𝑅) = \mathbf{1}[\text{comparison}(𝒪) = \text{comparison}(𝑅)]$$

$\mathcal{K}_R$ and $\mathcal{K}_O$ denote the sets of key visual elements in the reference and model output, respectively. $f_\text{consistency}$ is a task-specific function verifying logical alignment; $P$ and $C$ account for lexical phrasing as long as semantic equivalence is maintained.

This protocol, by disaggregating scoring across distinct axes, enables granular analysis of model performance and avoids rewarding superficial or flawed explanations. Quality Accuracy explicitly validates whether the model’s final judgment matches that of an expert.

3. Human–AI Alignment and Validation

To ensure the validity of the evaluation protocol, MedQ-Bench includes a robust human–AI alignment paper. In this process:

• 200 randomly chosen cases were evaluated independently by three board-certified medical imaging experts, scoring outputs on completeness, preciseness, and consistency.
• LLM-based automated judgments (using GPT-4o) were compared to human scores, calculating the quadratic weighted Cohen’s kappa ($\kappa_w$) to assess inter-rater agreement:

$$\kappa_w = 1 - \frac{\sum_{i,j} w_{ij} O_{ij}}{\sum_{i,j} w_{ij} E_{ij}}$$

where $w_{ij} = \frac{(i - j)^2}{(k - 1)^2}$ penalizes more severe disagreements.

• Results: Automated scoring showed strong concordance, with classification accuracies (83–91%) and high weighted kappa values (0.774–0.985), confirming the protocol reliably mirrors expert judgment.

4. Model Evaluation Results and Observed Limitations

MedQ-Bench assessed 14 contemporary MLLMs—spanning open-source, medical-specialized, and commercial models—across these dimensions. Key findings include:

• The best-performing models achieved only moderate scores on reasoning axes: completeness, preciseness, and consistency did not exceed roughly 1.1–1.3 on a 2-point scale.
• Models performed better on coarse, easily-distinguished quality differences than on subtle degradations, particularly in paired comparison tasks.
• Several specialized medical MLLMs underperformed compared to generalist systems on low-level perceptual analysis, indicating that current domain adaptation strategies prioritize diagnostic reasoning over raw perceptual sensitivity.
• In free-text tasks, incomplete or prematurely truncated generations (often caused by token window limits) further reduced performance, particularly for the most complex reasoning prompts.

5. Significance and Implications for Clinical AI

The findings from MedQ-Reasoning highlight several crucial insights:

• Current MLLMs lack the robustness and perceptual fidelity necessary for first-line clinical IQA deployment, as they frequently miss subtleties that may impact patient safety and diagnostic accuracy.
• The protocol reveals a disconnect between high-level clinical decision-making and low-level visual discernment—a gap not apparent with scalar IQA metrics alone.
• Optimizing future models for enhanced detection of subtle, clinically relevant quality degradations, and for more reliable explanatory reasoning, is essential.
• There is a demonstrated need for training paradigms that simultaneously target perceptual acuity and structured, verifiable justification of clinical judgments.

6. Future Directions and Research Challenges

MedQ-Reasoning sets a new standard for the evaluation of image quality judgment in medical AI, surfacing important development challenges:

• Integrating targeted pre-training or curriculum learning on diverse artifact patterns and their clinical implications may improve both perception and reasoning ability.
• Advancing methods to balance diagnostic reasoning with low-level assessment—without overfitting to high-level diagnostic cues at the expense of perceptual granularity—is a priority.
• Expanding beyond English-centric benchmarks and accommodating multi-modal, multi-lingual clinical explanations will be vital for universal deployment.
• The framework opens opportunities for research into neural architectures and learning objectives that explicitly align perceptual, causal, and workflow-based reasoning.

Conclusion

MedQ-Reasoning, as defined within MedQ-Bench, represents a principled benchmark for structured clinical reasoning in medical IQA, replacing unitary metrics with multidimensional, radiologist-aligned assessment. Through its detailed task design, rigorous scoring protocol, and demonstration of present model shortcomings, MedQ-Reasoning establishes an open roadmap for enhancing the reliability, transparency, and safety of medical image quality assessment in AI-driven clinical care (Liu et al., 2 Oct 2025).