SpineBench: Multimodal Spine Evaluation Framework

• SpineBench is a clinically salient, multimodal, level-aware benchmarking framework built to rigorously evaluate AI models in spine diagnosis and planning.
• It leverages the SpineMed-450k corpus, a large, clinician-curated dataset linking imaging modalities with vertebral-level reasoning for precise assessment.
• The framework’s structured tasks and composite scoring system ensure high-fidelity evaluation of model performance and clinical applicability.

SpineBench is a clinically salient, multimodal, level-aware benchmarking framework developed to advance and rigorously assess AI-assisted diagnosis, reporting, and planning in spine disorders. It is powered by the SpineMed-450k corpus—a large, highly curated instruction dataset designed in collaboration with practicing spine surgeons to encode fine-grained, vertebral-level clinical reasoning across X-ray, CT, and MRI modalities. The framework addresses the historical bottleneck in AI for spine care: the absence of robust, high-fidelity, traceable instruction data and standardized, domain-specific evaluation targets for vertebral-level and pathology-aware tasks.

1. Purpose and Components

SpineBench is purpose-built to evaluate AI models—particularly large vision-LLMs (LVLMs)—on anatomical and clinical reasoning tasks that mirror real-world diagnostic scenarios in spine medicine. Its essential components are:

• Benchmarked Tasks: 487 high-quality, multiple-choice, close-ended QA items and 87 clinical report generation prompts, encompassing 14 subconditions (e.g., herniated disc, spinal stenosis, vertebral fracture).
• Vertebral-Level Reasoning: All items require not only pathology identification but also precise localization to specific spinal levels (cervical, thoracic, lumbar), which is crucial for surgical planning and prognosis.
• Traceable Clinical Provenance: Evaluation data is drawn from SpineMed-450k, whose entries link directly to textbooks, guidelines, practice datasets, and nearly 1,000 de-identified hospital cases. Every item is vetted through a clinician-in-the-loop curation pipeline.

2. Multimodal and Level-Aware Instruction Design

Unlike generic benchmarks, SpineBench incorporates multimodal stimuli (text, X-ray, CT, MRI) and mandates level-aware reasoning:

• Close-Ended QA involves both text-only and multimodal items, forcing models to align and integrate visual findings with textual context for each vertebral level.
• Medical Report Generation Tasks follow a five-section protocol:
  1. Structured Imaging Findings (SIP)
  2. AI-Assisted Diagnosis (AAD)
  3. Treatment Recommendations (TR)
  4. Risk and Prognosis Management (RPM)
  5. Reasoning and Disclaimer (RD) Each section requires coverage, relevance, granularity, and explanation tailored to clinical standards.

This multimodal, anatomy-centric focus exposes deficiencies in models that lack high-fidelity image and level-specific alignment, as shown by systematic drops in model performance when transitioning from text to image-based questions.

3. Evaluation Metrics and Scoring

To enable rigorous, multifaceted model appraisal, SpineBench employs a composite quantitative scoring scheme:

$$\text{Score}_{\text{total}} = \sum_{k=1}^{3} w_k \cdot P_k$$

where $P_1$, $P_2$, and $P_3$ represent scores for text-only QA, multimodal QA, and report generation, respectively, and weights $w_k$ are set proportional to the number of instances in each category:

$w_k = \frac{N_k}{\sum_{i=1}^{3} N_i}$

Report generation is scored section-wise:

$$P_3 = 20 \times \sum_{i=1}^{5} \left(\frac{1}{n_i} \sum_{j=1}^{n_i} s_{ij}\right)$$

where $s_{ij}$ denotes the score on the $j$th sub-dimension of section $i$, and $n_i$ counts the relevant sub-dimensions.

All evaluation items are traceably linked to the SpineMed-450k instance from which they derive, ensuring external validity and clinical grounding.

4. Construction of the SpineMed-450k Dataset

SpineMed-450k is the supporting corpus for SpineBench:

• Scale and Diversity: Over 450,000 instruction instances systematically cover pathology, level, and modality, making it the largest and most comprehensive resource for this application space.

• Clinician-in-the-Loop Pipeline: Data is curated with explicit domain guidance—physicians define inclusion criteria, refine taxonomy, and vet annotation quality.
• Automated-Human Hybrid Generation: The two-stage LLM generation process involves a “draft” phase for high-volume automatic content creation, followed by a “revision” phase for clinician-guided correction, with manual logging for every adjustment.
• Traceability and Contextual Integrity: Each sample encodes its provenance (dataset ID, case reference), supported by automated OCR and the “Picture Context Matching” algorithm, which ensures that each multimodal item is appropriately bound to its textual scenario.

5. Task Taxonomy and Clinical Salience

By design, SpineBench’s evaluation spectrum encompasses:

• Level Identification: Precise vertebral labeling, required in pathologies like disc herniation or vertebral collapse.
• Pathology Assessment: Differentiation among high-overlap diseases (e.g., spondylosis vs. stenosis) with imaging findings and anatomical constraints.
• Surgical Planning: Encompassing treatment strategy, affected levels, and risk/prognosis calculations, consistent with guidelines for patient-specific technical feasibility.
• Consultation Emulation: Multi-turn interactions reflect the workflow of clinical reasoning and shared decision-making.

All items are structured to simulate decision points that are encountered in real clinical practice, rather than synthetic or decontextualized academic tasks.

6. Model Benchmarking and Observed Insights

Systematic benchmarking on SpineBench revealed:

• Multimodal Weaknesses: State-of-the-art LVLMs experience significant degradation in performance for image-based, level-specific reasoning—underscoring a persistent inability to align complex imaging with textual cues at clinically meaningful granularity.
• Fine-Tuning Advances: Models fine-tuned on SpineMed-450k (e.g., SpineGPT) achieve close-ended QA accuracy near 88% and produce more detailed, clinically coherent reports, with clinician raters confirming greater diagnostic utility.
• Clinical Impact: The integration of level-aware, multimodal benchmarking identifies and quantifies progress in AI’s ability to support actual clinical workflows, serving simultaneously as a stress test and roadmap for future model development.

7. Significance and Future Directions

SpineBench establishes a new, robust gold standard for AI model evaluation in spine medicine:

• Clinical Traceability: Eliminates ambiguity and hallucination risk by enforcing provable linkage to guideline-based, real-patient or expert-curated source material.
• Level-Awareness: Ensures that evaluation addresses the true complexity of clinical decision-making, including surgical planning that relies on precise anatomical localization.
• Community Contribution: By setting a unified, clinically-relevant target, SpineBench enables reproducible benchmarking, transparent comparison, and iterative improvement of foundation models in medical imaging.

A plausible implication is that the SpineBench paradigm could be extended to other organ systems and domains where fine-grained, context-aware, and multimodal reasoning is indispensable for AI safety, adoption, and regulatory approval. The approach underscores the necessity of clinician-in-the-loop, provenance-driven data curation, and standardized, transparent evaluation for effective integration of AI into high-stakes clinical practice.