MedRCube: A Multidimensional Framework for Fine-Grained and In-Depth Evaluation of MLLMs in Medical Imaging

Abstract: The potential of Multimodal LLMs (MLLMs) in domain of medical imaging raise the demands of systematic and rigorous evaluation frameworks that are aligned with the real-world medical imaging practice. Existing practices that report single or coarse-grained metrics are lack the granularity required for specialized clinical support and fail to assess the reliability of reasoning mechanisms. To address this, we propose a paradigm shift toward multidimensional, fine-grained and in-depth evaluation. Based on a two-stage systematic construction pipeline designed for this paradigm, we instantiate it with MedRCube. We benchmark 33 MLLMs, \textit{Lingshu-32B} achieve top-tier performance. Crucially, MedRCube exposes a series of pronounced insights inaccessible under prior evaluation settings. Furthermore, we introduce a credibility evaluation subset to quantify reasoning credibility, uncover a highly significant positive association between shortcut behavior and diagnostic task performance, raising concerns for clinically trustworthy deployment. The resources of this work can be found at https://github.com/F1mc/MedRCube.

• The paper introduces the MedRCube framework, which evaluates MLLMs using a multidimensional approach that maps clinical reasoning across anatomical, modality, and task-specific axes.
• It reveals significant blind spots, such as poor perceptual task performance and reliance on statistical shortcuts, undermining high-level diagnostic validity.
• Results show that specialized medical MLLMs outperform generalists in cognitive tasks while still exhibiting shortcut dependence, informing safer AI deployment.

MedRCube: A Multidimensional Benchmark for In-depth Evaluation of Medical MLLMs

Motivation and Paradigm Shift

The evaluation of multimodal LLMs (MLLMs) in medical imaging has been historically constrained by oversimplified, coarse benchmarking protocols, typically reporting aggregate or single-axis scores. Such approaches are insufficient for high-stakes clinical use, where interpretability, domain specificity, and reasoning validity are paramount. Aggregate metrics, for example, provide little actionable information to subspecialists who demand assurance of model proficiency on narrowly defined pathologies or imaging tasks, and they fail to identify reasoning fallacies such as "hallucinated correctness" where models offer plausible diagnoses unsupported by visual or perceptual grounding.

Figure 1: The authors advocate a paradigm shift from flat, aggregate metrics to a multidimensional, fine-grained evaluation aligned with real clinical imaging practice.

To address this gap, the paper introduces MedRCube, a systematic, rigorously designed benchmark for evaluating MLLMs in medical imaging along multiple, orthogonal axes. MedRCube embodies a paradigm shift: from linear metrics to dense, competency-aligned spaces, supporting the nuanced diagnosis, proficiency mapping, and validation of clinical reasoning crucial for eventual trustworthy deployment.

MedRCube Framework: Multi-Axial Competency Evaluation

MedRCube structures its evaluation space along three major axes: anatomical region (Heart, Chest, Breast, Lung, Brain), imaging modality (X-ray, CT, MRI, Ultrasound), and cognitive task type, systematically stratified into a hierarchy reflective of clinical reasoning (perception, semantics, and cognition). Each unique intersection forms a "competency voxel," enabling the localization of model strengths and deficits with high resolution.

Figure 2: MedRCube's multidimensional framework organizes evaluation by anatomical, modality, and hierarchical task axes; a two-stage expert-verified pipeline ensures rigor and consistency.

The task hierarchy encompasses:

• Low-level perception: modality recognition, view/plane recognition, and protocol identification.
• Mid-level semantics: organ recognition and ROI grounding.
• High-level cognition: abnormality detection, disease diagnosis, severity grading.

This design supports three diagnostic paradigms: holistic ranking, fine-grained profiling, and, uniquely, the verification of reasoning credibility by quantifying whether high-level answers are supported by prerequisite perceptual or semantic recognition.

Figure 3: MedRCube includes challenging low- and mid-level tasks such as modality and view identification, protocol recognition, and organ labeling for detailed assessment of perceptual and semantic capacity.

The construction pipeline involves dataset unification via metadata-driven mapping, followed by knowledge-augmented item generation, expert validation, and rigorous quality assurance, utilizing controlled template-based questioning, ontology-guided distractor generation, and standardized vocabulary mapping.

Experimental Evaluation and Core Findings

MedRCube benchmarks 33 models, including proprietary systems (e.g., GPT-5.1, Gemini-3-Pro), general-purpose open-source MLLMs (e.g., InternVL, Qwen), and medical-specialist MLLMs (e.g., Lingshu, HuatuoGPT-Vision, Hulu-Med). Key empirical findings include:

• Lingshu-32B achieves the highest overall accuracy at 62.55%. The top-performing models are distributed across both proprietary and open-source segments, as well as between generalist and medically specialized training paradigms.
• Performance for most models remains in a narrow band (45–55%), signifying substantial underlying heterogeneity masked by aggregate scores.
• Specialized medical MLLMs notably outperform generalists on high-level cognitive tasks, while only marginal deltas exist in saturated perceptual tasks.
• Modalities and anatomical regions exhibit pronounced variance. CT and X-ray tasks are generally more tractable, Ultrasound the most challenging; Chest, heart, and lung regions are better addressed than breast or brain.
• A weakened scaling effect is observed: Larger models ($>$10B parameters) do not consistently outperform smaller ones; domain-adapted training provides greater returns than brute parameter scaling.

Skill Factor and Relationship Analysis

A comprehensive correlation analysis across the competency space reveals functional stratification and resource-conditioned substructures:

Figure 4: Correlation heatmaps and clustering highlight three dominant functional blocks and the marked "brain island" effect—brain tasks are weakly correlated with other domains and internally heterogeneous.

Three main clusters emerge:

• Foundational perception tasks (modality and organ recognition) form a coherent block.
• High-level reasoning tasks stratify separately.
• Certain niches, such as breast and lung sub-tasks, are uniquely isolated. Importantly, the "brain island" effect underscores the high heterogeneity and poor generalizability of brain-imaging tasks, suggesting brain pathology constitutes a challenging subdomain for current MLLMs. Correlations are strongly conditioned by data availability, with best performance in resource-rich modalities and anatomical regions.

Blind Spots and Reasoning Validation

MedRCube exposes critical deficits in low-level perceptual skills. Tasks such as imaging protocol recognition and view determination show markedly lower accuracy—often only 20–40%—even among the strongest models. These are trivial for human specialists and classical discriminative pipelines, but current end-to-end MLLMs demonstrate almost no robust mastery, indicating foundational blind spots that threaten the reliability of more complex clinical reasoning.

To directly measure reasoning credibility, the benchmark employs hierarchical task consistency: does a correct diagnosis rely on the model correctly identifying the underlying modality, view, or relevant anatomical feature? Two diagnostic metrics are introduced:

• Luck Rate (shortcut dependence): Fraction of correct high-level answers unsupported by prerequisite recognition.
• Rationality Score: $1 - \text{Luck Rate}$, measuring the proportion of high-level correctness grounded in valid low-/mid-level perception.

  Figure 5: Left: Rationality versus accuracy—most models fail to combine high accuracy with high reasoning consistency. Right: Shortcut probability increases with accuracy ($r=0.693$, $p < 10^{-5}$), indicating ungrounded pattern exploitation.

Curiously, higher-accuracy models systematically display increased shortcut behavior, a stark violation of the "ideal evolution" hypothesis (where rationality should improve with overall performance). Instead, empirical evidence robustly supports an "opportunistic evolution" regime: as models improve, their reliance on ungrounded, spurious cues grows in parallel, further undermining the case for their safe and trustworthy clinical deployment.

Implications, Limitations, and Future Perspectives

MedRCube sets a new bar for systematic, clinically aligned evaluation of medical MLLMs. It exposes fundamental capacity gaps—including poor grounding in perceptual/physical medical knowledge and abnormally high rates of shortcut-based answer selection in advanced models. The observed Paradox of Specialization—where domain-specialist models achieve higher cognitive accuracy by leveraging statistical shortcuts rather than deeper visual grounding—raises important questions regarding model safety in real-world use.

Practical implications are clear: deployment in safety-critical domains cannot rely on global or aggregate metrics alone. Benchmarking must incorporate hierarchical and multi-dimensional evaluation, explicit reasoning validation, and the detection of spurious pattern exploitation. Theoretically, MedRCube demonstrates that future AI development in medical imaging must focus on both data representativeness and the explicit disentanglement of perceptual grounding from pattern-based shortcuts in training and evaluation paradigms.

As a limitation, MedRCube remains focused on radiological imaging and does not yet address domains such as histopathology or dermatology. Its use of automated data annotation and evaluation, while systematic, still introduces some potential for annotation noise. Pixel-level attention and localization are not explicitly validated in the current reasoning credibility metric.

Conclusion

MedRCube provides a rigorous, multidimensional framework for the diagnostic evaluation of MLLMs in medical imaging, supporting fine-grained capability mapping and, critically, the fidelity assessment of clinical reasoning. The benchmark exposes unforeseen effectiveness of domain-specific pretraining, critical blind spots in perceptual skill acquisition, and a disturbing trend toward shortcut-fueled accuracy gains among advanced models. MedRCube's design and results argue for a fundamental realignment of evaluation standards for clinical AI, with broad implications for the next generation of responsible model development and deployment.

(2604.13756)