Medical Data Standardization Benchmark

• The paper introduces MDS-Bench, a benchmark that evaluates vision-language models on converting raw medical images and texts into structured outputs.
• It decomposes the standardization process into sub-tasks like source identification, image conversion, text extraction, and structured pairing with precise metrics.
• Empirical results reveal high schema compliance but significant challenges in achieving complete end-to-end content fidelity.

Searching arXiv for the benchmark paper and closely related context. Medical Data Standardization Benchmark (MDS-Bench) is a benchmark for evaluating whether a vision-LLM can standardize raw, heterogeneous, and fragmented medical data into structured image–text pairs suitable for downstream medical AI. It is motivated by the gap between conventional benchmark settings, which typically assume that standardized medical images, texts, or question-answer pairs are already available, and real clinical practice, where data are distributed across raw dataset folders containing diverse file formats and local organizational conventions. MDS-Bench studies this “missing first step” by giving models raw dataset folders and evaluating their ability to identify source formats, convert raw medical images into VLM-compatible visual inputs, extract relevant textual information, and organize the results into structured outputs (Chen et al., 6 Jul 2026).

1. Benchmark scope and problem definition

MDS-Bench is defined around raw medical data standardization rather than diagnosis. The benchmark asks whether an agentic VLM can automatically identify the correct source files in a raw folder, convert them into a uniform visual format, extract the relevant textual and annotation information, and assemble the result into a structured image–text pair (Chen et al., 6 Jul 2026). In this framing, the core object of evaluation is not clinical prediction quality but the ability to recover a standardized representation from heterogeneous archival data.

The motivating setting is explicit. In clinical practice, digital medical data rarely come as a single model-ready image plus label. Hospitals and research centers instead maintain file systems containing DICOM volumes, TIFF slides, NIfTI scans, masks, metadata tables, and reports, organized according to local conventions. Downstream VLMs usually assume that a single image, its mask or bounding box, and a report or label are already aligned. MDS-Bench isolates the upstream standardization step needed before such assumptions can hold (Chen et al., 6 Jul 2026).

The benchmark’s target schema is also narrowly specified. Given a raw dataset folder and a target schema for a single sample, the model is expected to recover a standardized image, such as a PNG or JPEG, a per-image JSON annotation grounded in the original files, and a dataset-level JSON summary. This suggests that MDS-Bench operationalizes standardization as a source-grounded transformation problem with explicit structural outputs rather than as an informal preprocessing heuristic.

2. Task formulation

MDS-Bench decomposes raw standardization into four core sub-tasks with defined inputs and outputs (Chen et al., 6 Jul 2026).

First, source format identification takes as input a directory tree containing heterogeneous files, including DICOM (.dcm), NIfTI (.nii, .nii.gz), TIFF, BMP, MATLAB arrays (.mat), and possibly nested manifest CSV or JSON files. The required output is the file-path or record identifier, such as a DICOM SeriesInstanceUID or TIFF stem, that corresponds to the target sample. This selection must match the manually traced ground truth source.

Second, image conversion into VLM-compatible inputs takes the raw visual source selected in the first stage, for example a 3D NIfTI volume, a DICOM series, or a multipage TIFF. The required output is a single 2D image in PNG or JPEG form that preserves the medically relevant slice, projection, or montage needed for downstream VLMs. The conversion must reproduce the correct orientation, window-level, or color channels as documented by the original dataset.

Third, textual information extraction operates over all auxiliary files associated with the source, including metadata JSON or CSV, segmentation masks, bounding-box files, label tables, file-name conventions, directory structure, and dataset README. The output is a JSON dictionary recording task-type, modality, anatomical context, class labels, mask or box coordinates, provenance links back to the source identification step, and any other required fields prescribed by the target schema.

Fourth, structured image–text pair organization combines the converted image with the annotation JSON. The output is a folder or naming convention that pairs each standardized image with its JSON, together with a dataset-level JSON summarizing global dataset metadata such as number of samples, modalities, and class names.

A plausible implication is that the benchmark evaluates a composite capability spanning file-system navigation, format recognition, image transformation, annotation recovery, and schema-constrained packaging. The difficulty arises not from any single operation in isolation, but from the requirement that these operations remain mutually consistent.

3. Dataset composition and annotation protocol

MDS-Bench comprises 1,939 annotated standardization tasks drawn from 100 public medical imaging datasets (Chen et al., 6 Jul 2026). These tasks span three canonical vision tasks: 1,708 segmentation samples, 178 classification samples, and 53 detection samples. The covered domains include radiology modalities such as CT, MRI, X-ray, and ultrasound, as well as pathology and microscopy, ophthalmic OCT, endoscopy, hyperspectral imaging, and microwave imaging.

The raw-source diversity is central to the benchmark design. Included formats comprise DICOM, NIfTI (.nii / .nii.gz), mha, TIFF in both single- and multi-page forms, BMP, MATLAB arrays (.mat), NumPy arrays (.npy / .npz), videos (AVI), HDF5, and additional formats. Directory organizations vary from flat folders to nested study-series structures with implicit links in filenames. Gold-standard annotations also vary substantially, including pixel-level masks stored as separate TIFF or NIfTI files, bounding-box CSVs, class labels in JSON or CSV, and metadata tables with series UIDs. Some datasets expose explicit manifest files, whereas others rely on naming conventions or directory hierarchy.

The annotation protocol is explicitly source-grounded. For each sample, model-assisted scripts first draft a candidate standardized JSON and image. Human annotators then verify every field against the original raw sources; if a draft field conflicts with the data evidence, it is corrected or left empty. The stated purpose is to ensure that ground truth is fully source-grounded rather than inferred from image appearance alone (Chen et al., 6 Jul 2026).

This annotation design is methodologically significant because it distinguishes visual plausibility from archival fidelity. A model may generate a syntactically valid record or an apparently reasonable image without tracing the actual provenance of the sample. MDS-Bench treats such cases as failures when they deviate from the manually verified source-grounded target.

4. Evaluation framework and metrics

MDS-Bench uses eleven metrics grouped into Structure, Semantic, Content, Metadata, and Joint categories, with two primary overall metrics: end-to-end success rate and per-subtask accuracy (Chen et al., 6 Jul 2026). Let $N$ be the total number of target samples. For sample $i$, the benchmark defines:

• $p_i \in \{0,1\}$: valid image+JSON pair exists
• $s_i \in \{0,1\}$: source identification correct
• $g_i \in [0,1]$: schema-validity score
• $q_i \in [0,1]$: semantic correctness score
• $f_i \in [0,1]$: content fidelity score

The end-to-end success rate is

$$\mathrm{E2E} \;=\; \frac{1}{N}\sum_{i=1}^{N} \mathbb{I}\Bigl[p_i=1,\;s_i=1,\;g_i\ge0.85,\;q_i\ge0.5,\;f_i\ge0.5\Bigr].$$

For sub-task $j$, with binary indicator $a_i^{\,j}=1$ if sub-task $i$0 is correct on sample $i$1 and $i$2 otherwise, per-subtask accuracy is

$i$3

The benchmark also reports additional metrics:

Grouping	Metric
Structure / Semantic	Schema Validity (SV) $i$4, Schema-Semantic Composite (SSC) $i$5
Semantic / Content	Semantic Correctness (SC) $i$6 for $i$7; Information Completeness (IC), Validity (IV), Non-Redundancy (INR), Content Fidelity (CF) $i$8 for $i$9
Metadata / Joint	Metadata Semantic Correctness (MSC), Meta-Sample Joint Score (MSJ), Source-Content Joint Score (SCJ) $p_i \in \{0,1\}$0

The metric design makes the benchmark intentionally strict. Small errors in source identification, conversion, annotation extraction, or organization can invalidate the end-to-end result even when isolated components appear competent. This suggests that MDS-Bench emphasizes pipeline integrity over partial correctness.

5. Empirical findings

Nine state-of-the-art agentic VLMs were evaluated under identical prompts and tool access across all 1,939 samples (Chen et al., 6 Jul 2026). The reported results indicate a marked gap between schema compliance and full source-grounded standardization.

Structure metrics remain relatively high, with Schema Validity reaching up to 88.2%, showing that most models can generate a syntactically valid JSON schema. Semantic Correctness peaks at 65.2% for Claude Opus 4.6, indicating moderate recovery of key fields. Content metrics—Information Completeness, Validity, Non-Redundancy, and Content Fidelity—fall into the 45–60% range, reflecting difficulty in faithful annotation extraction.

The strict end-to-end results are substantially lower. Gemini 3 Flash is the best performer at 48.6% E2E and 56.3% SCJ. Claude Sonnet 4.6, the next best model, achieves 33.8% E2E (Chen et al., 6 Jul 2026). For Gemini 3 Flash, per-subtask accuracies illustrate the main bottlenecks: source identification is approximately 88%, image conversion approximately 68%, textual information extraction approximately 64%, and structured image–text pairing approximately 49%.

Error-attribution analysis further localizes the failure modes. Only 1–13% of datasets fail the Structure group, while over 50% fail Content and Joint groups. When the full pipeline is enforced, small errors in conversion or annotation accumulate rapidly. In other words, syntactic adherence to a target schema is comparatively tractable, whereas faithful recovery of clinically relevant content and cross-stage consistency remains difficult.

6. Interpretation, bottlenecks, and prospective directions

The benchmark identifies raw medical data standardization as a critical bottleneck for medical AI diagnosis in real practice (Chen et al., 6 Jul 2026). The central conclusion is that modern VLMs can often adhere to a JSON schema, yet preserving source-grounded semantics and content fidelity across heterogeneous raw sources remains a major unsolved challenge.

Several concrete observations are reported. Implicit links among images, masks, metadata, and directory structure often confuse even the strongest models. Converting diverse imaging formats into a consistent 2D view without losing clinical context is error-prone. Extracting task-specific annotations, including ROI masks and bounding boxes, from arbitrary files remains brittle. End-to-end consistency—requiring source selection, schema correctness, semantics, and fidelity all to hold simultaneously—drops below 50% even for the best model (Chen et al., 6 Jul 2026).

The paper also discusses validation-based inference strategies. Self-refinement and verification-guided candidate selection can increase E2E scores from approximately 48.6% to above 63%, but do not eliminate the core bottleneck of faithful content recovery. Future needs identified in the work include specialized tools for format conversion, such as DICOM-to-PNG conversion, stronger symbolic tracing of file provenance, tighter integration of medical ontologies for semantic grounding, and improved multiagent workflows that can cross-validate each other’s source mappings.

A plausible implication is that MDS-Bench reframes medical VLM deployment as a systems problem extending well beyond multimodal reasoning over already-curated inputs. In this view, clinical readiness depends not only on downstream diagnostic competence but also on reliable upstream mediation between heterogeneous medical archives and model-consumable representations. That emphasis is encapsulated in the benchmark’s characterization of raw medical data standardization as the “missing first step” (Chen et al., 6 Jul 2026).