- The paper introduces a novel end-to-end system integrating memory-augmented segmentation, structured vision-language inference, and retrieval-based generation to enhance TCM clinical decision support.
- It standardizes tongue ROI extraction with a 99.1% success rate and improves performance metrics such as BLEU-4 and TDEU.
- Retrieval-augmented prescription generation grounds diagnosis and treatment in both empirical case banks and canonical texts, bolstering clinical accuracy and interpretability.
MMIR-TCM: Memory-Integrated Multimodal Inference and Retrieval for TCM Clinical Decision Support
Overview of MMIR-TCM Architecture
MMIR-TCM is an end-to-end multimodal clinical decision support system tailored for Traditional Chinese Medicine (TCM). It addresses the intrinsic challenges of TCM tongue diagnosis—primarily practitioner subjectivity, acquisition variability in tongue images, and the semantic gap between visual features and textual clinical reasoning—by integrating a memory-augmented segmentation module, structured multimodal diagnosis generation, and a retrieval-augmented generation (RAG) prescription recommendation pipeline. The entire workflow emulates expert TCM diagnostic processes, leveraging a large, multi-center dataset (MedTCM) and a purpose-built evaluation metric (TDEU) that encapsulates both semantic understanding and clinical significance.
Data Standardization: Tongue Region-of-Interest Extraction
Consistent with longstanding findings in medical imaging, MMIR-TCM leverages input standardization to improve downstream reasoning by isolating semantically relevant tongue regions and mitigating background variance. The Memory-SAM module, a human-prompt-free architecture, retrieves relevant exemplars from a curated memory bank, generating explicit foreground/background prompts for SAM2-based segmentation. This pipeline enables robust, training-free tongue mask extraction under diverse and uncontrolled clinical conditions—achieving a 99.1% success rate across the MedTCM corpus.
(Figure 1)
Figure 1: Background removal and ROI standardization with Memory-SAM. For diverse in-the-wild inputs, extracted tongue masks produce clean foreground crops that reduce illumination and background variance, stabilizing the interface for report and prescription generation.
Ablation analysis demonstrates that mask-guided region-of-interest (ROI) extraction, when integrated into both training and inference, yields modest but consistent improvements in both lexical (BLEU/ROUGE) and domain-aware (TDEU) evaluation, supporting ROI standardization as a pragmatic design choice for robust multimodal learning.
Structured Multimodal Inference with Qwen3-VL
After spatial standardization, tongue images are processed by a LoRA-fine-tuned Qwen3-VL-30B, a vision-language foundation model. To address the notorious variability in practitioner-generated textual tongue diagnoses, the system enforces a structured, attribute-enumerative reporting style, synthesizing concise descriptions of tongue body color, shape, coating, and anatomical location in a fixed schema. This uniform output is critical for enabling reliable case retrieval, facilitating structured downstream reasoning, and supporting system interpretability.
Evaluation against public multimodal LLMs (e.g., GPT-4o, Gemini 2.5 Flash) and strong internal baselines underscores the superiority of domain-specific adaptation and input segmentation. The best configuration of MMIR-TCM achieves BLEU-4 of 83.57 and TDEU Overall of 0.627, in sharp contrast to the 20s BLEU and 0.34–0.35 TDEU observed for generic public MLLMs on the same MedTCM tongue diagnosis evaluation set.
Retrieval-Augmented Prescription Generation
MMIR-TCM’s core innovation is realized in the prescription generation stage, which models the TCM expert’s workflow by combining patient metadata, structured tongue diagnosis, and external retrieval from both the empirical Clinical Case Bank (124,593 records) and canonical LiuJing theoretical texts. The semantic embedding-based FAISS+Langchain architecture enables efficient and accurate similarity search, providing dense, evidence-anchored context for the Qwen3-based generation module. The pipeline outputs syndrome differentiation, biomedical diagnoses, herbal prescriptions (with dosages), and explicit clinical reasoning steps—all grounded in retrieved precedents.
Global system evaluation is conducted on the MedTCM test set and in blinded physician preference studies. The radar and score distribution analyses robustly demonstrate MMIR-TCM’s superiority over leading baselines (notably including GPT-4o and Gemini 2.5 Flash) in all TCM-specific evaluation axes: accuracy, fidelity, rationality, safety, and interpretability.
(Figure 2)
Figure 2: Overall performance comparison. (a) Radar chart shows MMIR-TCM’s consistent superiority. (b) Average results across tasks. (c) Score distributions highlight higher median and stability.
In controlled ablation studies, the absence of RAG memory (either empirical or theoretical, or both) produces dramatic (>500%) degradations in diagnostic and prescription metrics, underscoring the necessity of evidence-grounded generation for TCM clinical support.
Expert preference studies, leveraging 240 independent clinical evaluations, confirm consistent clinician favor for MMIR-TCM, with perceived gains in diagnostic safety and clinical reliability. However, failure analysis reveals persisting limitations in handling syndromic complexity, meridian attribution, and context-dependent combinatory patterns—critical avenues for subsequent research.
Implications, Limitations, and Future Prospects
Theoretically, MMIR-TCM validates that robust region-of-interest standardization and structured-output multimodal large models, when tightly coupled with retrieval from expert-curated knowledge bases, can surpass zero-shot generalist MLLMs in specialized, multimodal clinical tasks with complex reasoning demands. Practically, it delivers a reproducible, evidence-backed system for TCM decision support, offering visible gains in documentation efficiency and interpretability, while positioning itself as an assistive (not autonomous) agent that augments, rather than replaces, clinician judgment.
The major limitations identified are threefold: (1) reliance on retrospective, hospital-biased datasets and derived knowledge; (2) sensitivity to image quality, domain drift, and metadata completeness; and (3) limited integration of full multimodal and contextual cues beyond tongue images and structured metadata.
Prospective directions include: rigorous prospective validation and broader, more unbiased data acquisition; integration of additional diagnostic modalities (e.g., facial, pulse, and voice); investigation of full/model-specific fine-tuning beyond LoRA; uncertainty quantification and provenance-traceable evidence presentation for clinical accountability; and formalization of clinician-in-the-loop interfaces.
Conclusion
MMIR-TCM establishes an authoritative blueprint for memory-integrated, multimodal clinical decision support in TCM by uniting robust segmentation, structured vision-language reasoning, and retrieval-augmented evidence synthesis. Its validated gains in clinical fidelity, interpretability, and factual robustness represent an essential advance for translational AI in integrative medicine, with future research poised to expand its scope, reliability, and impact.