MedRCube: In-Depth Medical Imaging Evaluation

• MedRCube is a multidimensional evaluation framework that assesses multimodal LLMs in medical imaging by benchmarking diagnostic competencies across anatomy, modality, and task hierarchy.
• It employs a voxel-based method to measure specific clinical capabilities, uncovering gaps in perceptual, semantic, and reasoning performance.
• The framework integrates rigorous quality assurance and credibility assessments to ensure that diagnostic accuracy aligns with underlying evidence-based reasoning.

Searching arXiv for MedRCube and closely related work to ground the article and clarify naming ambiguity. MedRCube is a multidimensional, fine-grained, in-depth evaluation framework for multimodal LLMs (MLLMs) in medical imaging. It was introduced to address the mismatch between coarse aggregate reporting and the requirements of clinical practice by organizing evaluation along Anatomy, Modality, and a cognitive Task Hierarchy, and by assessing not only answer accuracy but also whether high-level diagnostic outputs are supported by prerequisite lower-level competencies on the same image. In its initial instantiation, MedRCube curates 7,626 samples from 36 datasets, benchmarks 33 MLLMs, and introduces a dedicated credibility subset that operationalizes shortcut behavior and reasoning rationality (Bao et al., 15 Apr 2026).

1. Motivation and problem formulation

MedRCube is motivated by two deficiencies in prevailing medical visual question answering evaluation. First, coarse metrics lack clinical granularity. Healthcare practice is compartmentalized, and broad averages over a modality or anatomical category do not establish competence on specific pathologies, protocols, or prerequisite perceptual skills. The framework explicitly rejects the assumption that a single aggregate score, such as an “MRI score,” is sufficient for specialist-facing deployment. Second, prior evaluations do not assess reasoning reliability. MedRCube adopts the view that clinical diagnosis follows a logical hierarchy from perception to semantics to cognition, and that a correct diagnosis without correct recognition of modality, organ, or related prerequisites constitutes “hallucinated correctness” rather than clinically acceptable reasoning (Bao et al., 15 Apr 2026).

This design places MedRCube within a broader shift from flat benchmarking to structured competency analysis. Rather than treating medical imaging performance as a one-dimensional quantity, it models competence as a space of localized abilities and inconsistencies. A central implication is that a model can appear strong under aggregate accuracy while remaining weak in clinically foundational primitives such as imaging protocol recognition or view recognition.

2. Competency space and hierarchical evaluation design

The framework defines a three-axis competency space. The Anatomy axis comprises Heart, Chest, Breast, Lungs, and Brain. The Modality axis comprises X-ray, CT, MRI, and Ultrasound. The Task Hierarchy axis comprises low-level perception tasks, mid-level semantic tasks, and high-level cognition tasks. The low-level tasks are Modality Recognition, View Recognition, and Imaging Protocols Recognition. The mid-level tasks are Organ Recognition and Region-of-Interest Grounding. The high-level tasks are Abnormality Diagnosis, Disease Diagnosis, and Severity Grading (Bao et al., 15 Apr 2026).

The atomic unit of evaluation is the Competency Voxel, defined as the intersection of one anatomy, one modality, and one task. This voxelization is not merely organizational. It enables the benchmark to localize strengths and weaknesses at the level of concrete clinical capabilities, such as protocol recognition in a specific modality or disease diagnosis in a specific anatomical target.

MedRCube implements three levels of evaluation. The first is holistic ranking, intended to summarize overall competence with balanced coverage across axes. The second is capability profiling, which slices the benchmark by task type, modality type, anatomical region, or individual voxels to expose granular failure modes. The third is credibility and consistency assessment, which uses the task hierarchy to test whether a correct high-level cognition output is grounded in correct lower-level performance on the same image. This third level is the distinctive element of the framework, because it treats clinical trustworthiness as a property of hierarchical consistency rather than of answer accuracy alone.

3. Systematic construction pipeline and quality assurance

MedRCube is built with a two-stage systematic construction pipeline. Stage I performs data curation and metadata-driven competency mapping. The framework collects 35 datasets across X-ray, CT, MRI, and Ultrasound and reports 7,626 samples from 36 datasets, including ROCOv2 used only for supplementary analysis. In this stage, each image and its metadata are treated as clinical evidence, dataset-specific task labels are ignored, and samples are remapped into MedRCube’s standardized taxonomy. Metadata determine task support: modality or acquisition tags support perceptual tasks, masks or boxes support semantic tasks, and diagnostic annotations support cognition tasks. Each sample is then assigned to one or more Competency Voxels according to the evidence it contains (Bao et al., 15 Apr 2026).

Stage II performs knowledge-augmented item production. Question templates follow National Board of Medical Examiners item-writing principles so that each item targets a single competency, remains deterministic, and is answerable from the image and prompt alone. All templates undergo clinician review. Distractor generation depends on answer-space type. Disease-oriented distractors use ontology-guided retrieval based on HPO plus controlled LLM generation with difficulty control. Abnormality-oriented distractors use constrained LLM generation under modality- and region-aware constraints. Closed-set tasks use predefined, expert-verified candidate pools. Terminology is standardized to RadLex for radiology and anatomy terms and to ICD-11, with ICD-10 where noted in the Appendix, for diseases (Bao et al., 15 Apr 2026).

Quality assurance is a formal component rather than a post hoc filter. The benchmark applies NBME-style Question Review for unambiguity and clinical determinism, together with Option Review for homogeneity, mutual exclusivity, and plausibility. It also includes LLM-based quality review audited against a clinical expert. Reported agreement exceeds 90% for every rule, and sample-level agreement is 0.84. These values situate MedRCube as an explicitly curated evaluation framework rather than a simple aggregation of heterogeneous pre-existing datasets.

4. Metrics, credibility protocol, and shortcut analysis

MedRCube uses micro-averaged accuracy as its primary performance metric. Overall accuracy is defined as

$$ACC_{overall} = \frac{N_{correct}(\mathcal{D}_{all})}{N_{total}(\mathcal{D}_{all})} \times 100\%.$$

For any subset, including task type, modality type, anatomical region, or voxel, type-specific accuracy is

$$ACC_{type} = \frac{N_{correct}(\mathcal{D}_{type})}{N_{total}(\mathcal{D}_{type})} \times 100\%.$$

No macro-averages or composite weighted indices are used (Bao et al., 15 Apr 2026).

The credibility analysis is based on paired outcomes on the same image. A prerequisite task $L$ and a high-level cognition task $H$ yield four groups: $A=(1,1)$, coherent correct; $B=(1,0)$, broken reasoning; $C=(0,0)$, both incorrect; and $D=(0,1)$, shortcut or incoherent correct. From these groups, MedRCube defines

$\mathrm{LuckRate}=\frac{N(D)}{N(A)+N(D)},$

with Rationality Score defined as $1-\mathrm{LuckRate}$, and

$$ACC_{type} = \frac{N_{correct}(\mathcal{D}_{type})}{N_{total}(\mathcal{D}_{type})} \times 100\%.$$0

The Appendix also defines

$$ACC_{type} = \frac{N_{correct}(\mathcal{D}_{type})}{N_{total}(\mathcal{D}_{type})} \times 100\%.$$1

and

$$ACC_{type} = \frac{N_{correct}(\mathcal{D}_{type})}{N_{total}(\mathcal{D}_{type})} \times 100\%.$$2

To instantiate this protocol, the framework constructs a dedicated subset of 300 images, each paired with at least one prerequisite question and one high-level cognition question, yielding 900 total items. The purpose is image-level verification of grounded reasoning: a cognition-level correct answer is counted as coherent only if the model also succeeds on the lower-level prerequisite for that same image (Bao et al., 15 Apr 2026).

The framework further formalizes two developmental hypotheses. Under Ideal Evolution $$ACC_{type} = \frac{N_{correct}(\mathcal{D}_{type})}{N_{total}(\mathcal{D}_{type})} \times 100\%.$$3, improvements increase $$ACC_{type} = \frac{N_{correct}(\mathcal{D}_{type})}{N_{total}(\mathcal{D}_{type})} \times 100\%.$$4 while not increasing $$ACC_{type} = \frac{N_{correct}(\mathcal{D}_{type})}{N_{total}(\mathcal{D}_{type})} \times 100\%.$$5, and $$ACC_{type} = \frac{N_{correct}(\mathcal{D}_{type})}{N_{total}(\mathcal{D}_{type})} \times 100\%.$$6. Under Opportunistic Evolution $$ACC_{type} = \frac{N_{correct}(\mathcal{D}_{type})}{N_{total}(\mathcal{D}_{type})} \times 100\%.$$7, improvements increase both $$ACC_{type} = \frac{N_{correct}(\mathcal{D}_{type})}{N_{total}(\mathcal{D}_{type})} \times 100\%.$$8 and $$ACC_{type} = \frac{N_{correct}(\mathcal{D}_{type})}{N_{total}(\mathcal{D}_{type})} \times 100\%.$$9, and $L$0. Empirically, MedRCube supports $L$1: Shortcut Probability and high-level accuracy are strongly positively correlated, with $L$2 and $L$3, indicating that higher-performing models may also exhibit more shortcut behavior.

5. Benchmark setup and empirical findings

The benchmark evaluates 33 MLLMs. Proprietary models include GPT-5.1, Claude Opus 4.5, Gemini-3-Pro, and Qwen3-VL-Plus. Medical-specific models include MedVLM-R1, the HealthGPT series, LLaVA-Med-7B, HuatuoGPT-Vision, Lingshu, Hulu-Med, and MedDr-40B. General-purpose open-source models include InternVL 3/3.5 variants, Qwen 2.5-VL and 3-VL variants, Llama-3.2-11B-Vision, Phi-3.5-Vision, Janus-Pro-7B, MiniCPM-o 2.6, and LLaVA-v1.5-7B. All models are evaluated with the unified zero-shot prompt, “Question: {question} Options: {options} Answer with the option’s letter from the given choices directly.” Decoding is greedy with temperature=0, top_p=0.0001, and max_tokens=8192. Reasoning modes are disabled, explicit chain-of-thought is not used, responses are restricted to the option letter, and no external tools or retrieval are used (Bao et al., 15 Apr 2026).

On overall accuracy, Lingshu-32B ranks first at 62.55. It is followed by Lingshu-7B at 59.86, Gemini-3-Pro at 59.35, and InternVL3.5-38B at 58.14. MedRCube’s more granular reporting shows that leadership is highly axis-dependent. By task type, the best reported scores are: Modality Recognition, Lingshu-7B at 91.38; View Recognition, Gemini-3-Pro at 71.33; Imaging Protocols Recognition, Hulu-Med-32B at 40.71; Organ Recognition, Lingshu-32B at 84.72; ROI Grounding, InternVL3.5-38B at 55.90; Disease Diagnosis, Lingshu-32B at 65.77; Abnormality Diagnosis, Hulu-Med-4B at 43.77; and Severity Grading, Qwen3-VL-Plus at 44.72. By modality, Lingshu-32B records the highest CT score at 77.83 and the highest MRI score at 70.00, while InternVL3.5-38B leads on X-ray at 64.20 and Lingshu-7B leads on Ultrasound at 57.40. By anatomical region, the highest scores are Brain 65.42 for Lingshu-32B, Chest 67.73 for Lingshu-32B, Heart 88.81 for Gemini-3-Pro, Lungs 74.88 for InternVL3.5-38B, and Breast 55.06 for Qwen2.5-VL-7B (Bao et al., 15 Apr 2026).

Several benchmark-level findings are central to the framework’s significance. First, MedRCube exposes foundational blind spots in basic perceptual skills. Imaging Protocols Recognition commonly remains in the 20–40% range even for top models, and View Recognition peaks at 71.33. Second, the correlation analysis identifies a brain “isolated island” effect: brain tasks have unusually low correlations with other anatomies and even among themselves, with average within-brain correlation reported as $L$4 before stratifying by modality. Third, the benchmark reports a weakened scaling effect: larger models above 10B do not consistently dominate smaller counterparts, and performance gaps narrow to less than 2.5% in some comparisons, indicating that medical domain adaptation can outweigh parameter scale. Fourth, performance aligns with public-data availability, with CT and X-ray generally stronger and Ultrasound remaining challenging. Fifth, a text-only baseline collapses performance toward random, with Gemini-3-Pro dropping by 28.99 points, which confirms reliance on visual evidence under the evaluation setup (Bao et al., 15 Apr 2026).

The credibility subset extends these observations from capability to trustworthiness. Rationality scores span 24.1%–99.3%. Leading models including Lingshu, GPT-5.1, Qwen, and HuatuoGPT exceed 90% rationality, whereas some medically fine-tuned systems, including LLaVA-Med-7B and MedDr-40B, show LuckRates above 60%. Even top-tier systems can exhibit high Shortcut Probability; Gemini-3-Pro is reported at approximately 57.9%. These results make explicit that strong diagnostic accuracy and grounded reasoning are not interchangeable properties.

6. Limitations, significance, and nomenclature

MedRCube is explicitly limited to radiology modalities: CT, MRI, X-ray, and Ultrasound. Optical domains such as histopathology and dermatology are not covered, and the framework does not claim generalization to those domains. The authors also note that, as a data-driven evaluation, separating intrinsic capability from dataset priors and distribution artifacts remains challenging. Its notion of rationality is a hierarchical consistency proxy rather than an explicit pixel-level grounding audit, and automated generation can inherit noise from the source datasets despite the systematic construction and review pipeline (Bao et al., 15 Apr 2026).

Within those limits, the framework contributes a clinically aligned evaluation paradigm. Its main contribution is not only broader coverage, but the restructuring of model assessment around clinically meaningful axes and a task hierarchy that can distinguish evidence-backed reasoning from correct-by-accident responses. This suggests a shift in benchmark design from aggregate leaderboard optimization toward capability localization and credibility auditing.

The term “MedRCube” has, however, been used ambiguously across arXiv contexts. In gamma-ray instrumentation, the CubeSat Compton telescope concept MeVCube has been noted as being referenced as “MedRCube” in informal queries, although that work concerns MeV astronomy rather than medical imaging (Lucchetta et al., 2022). A related MeVCube technical note likewise discusses the same 6U CubeSat concept and not a medical benchmark (Lucchetta, 2021). Separately, Med-R$L$5 is explicitly pronounced “Med R-cube,” but denotes a retrieval-augmented reasoning framework for medical LLMs rather than an imaging evaluation benchmark (Lu et al., 31 Jul 2025). In room acoustics, the MIRaGe room impulse response database has also been described as a basis for a “MedRCube” style cube-shaped measurement concept, again unrelated to MLLM evaluation in radiology (Čmejla et al., 2019). In strict bibliographic terms, the title-bearing entity named MedRCube is the medical-imaging evaluation framework introduced in 2026 (Bao et al., 15 Apr 2026).