Learning from Medical Entity Trees: An Entity-Centric Medical Data Engineering Framework for MLLMs

Abstract: Multimodal LLMs (MLLMs) have shown transformative potential in medical applications, yet their performance is hindered by conventional data curation strategies that rely on coarse-grained partitioning by modality or department. Such fragmented approaches fail to capture the hierarchical and interconnected nature of clinical medical knowledge, limiting the models' ability to perform fine-grained recognition and complex reasoning. In this paper, we propose a novel Entity-Centric Medical Data Engineering framework. We automatically extract entities from authoritative medical literature to construct a Medical Entity Tree (MET), a hierarchical structure that systematically encodes diseases, anatomical structures, modalities, and symptoms into a unified knowledge repository. Building upon the MET, we propose an advanced data engine that includes: (1) node-guided retrieval to anchor raw data to specific medical concepts, (2) a two-stage hybrid filtering and alignment pipeline to ensure precise visual-semantic correspondence, and (3) knowledge-aware data synthesis to generate enriched captions and targeted reasoning VQA pairs, leveraging structural constraints. Extensive evaluations across six medical benchmarks demonstrate that our approach significantly enhances the medical capabilities of general-purpose MLLMs, improving their ability to handle complex clinical queries and achieve state-of-the-art performance in diverse medical contexts.

• The paper introduces a Medical Entity Tree that organizes multimodal medical data into a hierarchical structure to improve fine-grained clinical reasoning.
• The paper details a multi-stage framework combining entity extraction, semantic clustering, and LLM-based conflict resolution for scalable, knowledge-aligned data retrieval.
• The paper demonstrates significant performance gains over state-of-the-art models with enhanced coverage of rare entities and improved accuracy across multiple VQA benchmarks.

Entity-Centric Medical Data Engineering for MLLMs: Structured Medical Knowledge as a Foundation for Multimodal Clinical AI

Introduction

The paper "Learning from Medical Entity Trees: An Entity-Centric Medical Data Engineering Framework for MLLMs" (2604.25296) establishes a paradigm shift in the curation, alignment, and synthesis of medical multimodal data for MLLMs by departing from the prevailing practice of partitioning datasets by visual modality or clinical department. The authors introduce a Medical Entity Tree (MET), automatically constructed from open-source medical literature, that provides a hierarchical, systematic knowledge structure encoding diseases, anatomical structures, modalities, and symptoms as first-class entities. This framework underpins a data engineering pipeline that augments data retrieval, filtering, and synthesis to achieve high-fidelity knowledge alignment and greatly improved fine-grained reasoning and clinical coverage.

Figure 1: Comparison between traditional, modality-centric medical data curation and the proposed entity-centric, knowledge-anchored framework leveraging a hierarchical Medical Entity Tree (MET) as the organizing principle.

Limitations of Modality-Based Data Partitioning

Traditional data curation for medical MLLMs fragments multimodal datasets into flat modality- or department-centric partitions (e.g., X-ray vs. CT, or radiology vs. dermatology). This approach neglects the complex, interdependent structure of clinical reasoning, where entities and their relationships span across modalities and specialties. Such fragmentation results in:

• Reduced coverage of rare ("long tail") entities.
• Inability to capture and model causal and hierarchical knowledge.
• Limited performance on fine-grained expert-level tasks and in generalization to unseen medical questions.

  Figure 2: Schematic overview of the entity-centric pipeline: (1) Entity tree construction from literature, (2) node-guided retrieval and two-stage hybrid alignment of raw data, (3) knowledge-aware data synthesis for context-rich and logic-constrained training samples.

Construction and Structure of the Medical Entity Tree

The construction of the MET proceeds through a multi-stage pipeline:

1. Entity Extraction: LLM-based prompt engineering and batch chunking are used to extract entity candidates at scale from medical textbooks and papers, with output constraints to increase speed and precision.
2. Typing and Clustering: Entities are classified and grouped semantically via joint prompt engineering and K-means on LLM embeddings, producing a robust, multi-level core taxonomy.
3. Incremental Tree Attachment and Conflict Resolution: Candidate entities are grafted onto the core tree via LLM reasoning. A ReAct Agent framework (augmented with external search) is used to resolve ambiguities, enacting principles of etiological dominance and specificity to resolve multi-parent conflicts and enforce taxonomic rigor.

The final MET comprises 1.47 million entities over five hierarchical tiers, efficiently indexable via Aho-Corasick automation to enable large-scale tagging and mapping.

Entity-Driven Data Acquisition and Filtering

Data collection departs from blind scraping and is instead directed by the MET. Key processes include:

• Node-Guided Retrieval: Entities are used as high-recall anchors to acquire data that is inherently knowledge-aligned.
• Two-Stage Quality Filtering: The first stage employs a large MLLM as a teacher to annotate and filter a sample set; its outputs supervise a ViT classifier for scalable filtering on large datasets. Entity-visual consistency verifies correct alignment, leveraging the MET for both retrieval and semantic validation.

Knowledge-Aware Data Synthesis

The data synthesis stage enforces tight semantic and causal linkage between images, captions, and structured knowledge, via two tracks:

• Track 1 (Contextual Re-Captioning): The synthesis fuses original ("Alt Text") captions, visual evidence, and hierarchical entity context to produce enriched, precise captions with explicit linkage to pathophysiology and findings.

  Figure 3: Example of a contextually re-captioned data sample, illustrating the integration of visual evidence with expert-level entity terminology and reasoning.

• Track 2 (Structure-Constrained Reasoning VQA): Using entity-inference chains derived from the MET, the framework generates reasoning-intensive VQA samples—multiple-choice or judgment questions—where label and explanation are grounded in specific visual biomarkers and require stepwise causal justification, making answerability strictly image-dependent and aligned to rigorous clinical logic.

  Figure 4: Example of a synthesized judgment VQA sample enforcing structured reasoning along the entity-inference path.

  

  Figure 5: Example of a synthesized MCQ VQA sample with medically plausible distractors, requiring fine-grained visual differential diagnosis.

Benchmark Evaluation and Performance

Extensive evaluation across six medical VQA benchmarks (MMMU-Med, VQA-RAD, SLAKE, PathVQA, PMC-VQA, OmniMedVQA) demonstrates robust and consistent improvements over both generalist vision-LLMs and state-of-the-art specialized medical MLLMs. Key highlights:

• Average accuracy: 69.16%, outperforming Lingshu-7B (67.68%) and InternVL3-8B (60.59%).
• Expert reasoning (MMMU-Med): 73.77% vs. 54–56% for leading baselines.
• Massive-scale generalization (OmniMedVQA): 83.36%, outperforming all prior models.
• Long-tail entity coverage: Near-perfect recognition of rare and safety-aligned entities, where baselines often register zero accuracy due to data and alignment biases.
• Model generalization: Synthesis pipeline produces gains across different MLLM backbones, validating its general applicability.

Synergy of Declarative Knowledge and Reasoning Data

Ablation demonstrates that sole reliance on entity-enriched captions (declarative) or on logic-driven VQA samples (interrogative) is suboptimal; their combination achieves the best trade-off. Notably, even small additions of structure-constrained VQA data (10%) boost performance disproportionately, indicating the synergistic effect wherein captions act as a latent knowledge reservoir that the VQA data unlocks for complex reasoning and query answering.

Figure 6: Scaling and ablation study demonstrating performance gains as proportions of reasoning data are integrated into entity-enriched declarative supervision.

Implications and Future Directions

The introduction of the MET-centred, entity-driven pipeline imposes a data engineering regime that aligns with human medical pedagogy: knowledge is systematically acquired, refined, and operationalized at the level of entities and their relations, not merely at the level of image modalities or department tags. This approach both increases semantic coverage—including the clinical long tail—and enhances reasoning capabilities by constructing logic paths grounded in expert knowledge.

Practical implications include:

• Mitigating safety-alignment pitfalls: The entity-driven approach can retain and align clinically essential but safety-sensitive findings (e.g., reproductive organs), overcoming current biases in generalist models.
• Transferability: The pipeline is domain-agnostic and extensible to other knowledge-structured fields (e.g., law, finance, education).
• Scalability: The automation of taxonomy expansion and corpus mapping supports ongoing updates as new entities and knowledge emerge.

Conclusion

By reorganizing multimodal medical data around a structured, automatically curated Medical Entity Tree, the proposed framework sets a new technical standard for scalable, knowledge-grounded data engineering in medical MLLMs. The empirical performance gains, improvements in rare entity coverage, and robust reasoning benchmarks underscore the superiority of the entity-centric paradigm for the next generation of clinical AI models. Further research may extend these principles to broader domains or refine entity-reasoning synthesis for increasingly complex, real-world decision making.