XMedFusion: A Knowledge-Guided Multimodal Perception and Reasoning Framework for Autonomous Medical Systems

Abstract: Autonomous medical and robotic systems increasingly rely on intelligent perception and reasoning capabilities to interpret visual data and support clinical decision making. Radiology report generation represents a critical component of such automated diagnostic workflows, yet existing end-to-end multimodal models often suffer from weak visual grounding, resulting in unreliable interpretations and omission of subtle clinical findings. This paper presents XMedFusion, a modular AI framework designed as an intelligent perception and reasoning module for autonomous medical systems. The proposed framework decomposes visual information into coordinated functional components that emulate expert-driven analysis, including a visual perception agent that extracts image-grounded evidence, a knowledge graph construction agent that structures clinically relevant findings, and a retrieval-guided drafting process that ensures a consistent reporting structure. A synthesis agent iteratively integrates visual and structured evidence through reasoning-driven verification to produce reliable and interpretable diagnostic outputs. Experimental evaluation on a public chest radiograph dataset demonstrates significant improvements over baseline vision-LLMs, achieving gains from 0.0493 to 0.3359 in BLEU-1, 0.0863 to 0.2440 in ROUGE-L, and 0.0829 to 0.1708 in METEOR, along with substantial improvements in semantic evaluation metrics such as Consistency (2.38 to 7.80) and Accuracy (2.34 to 6.93). The results highlight the effectiveness of structured multi-agent perception and reasoning for enhancing robustness, transparency, and automation in intelligent medical imaging systems, enabling integration into autonomous healthcare and robotic diagnostic workflows.

• The paper introduces a four-agent framework that combines visual evidence extraction, neuro-symbolic knowledge graphs, retrieval-based scaffolding, and evidence-prioritized synthesis for radiology report generation.
• XMedFusion substantially outperforms LLaVA-Med 1.5 on IU X-ray, improving BLEU-1 from 0.0493 to 0.3359 and LLM-judged Consistency from 2.38 to 7.80.
• The framework makes findings auditable by encoding positive and negative observations in RadGraph-compatible structures and linking knowledge-graph entities to anatomical image regions, although broader clinical validation remains necessary.

Motivation and problem statement

Radiology Report Generation (RRG) is a representative perception task for autonomous medical and robotic diagnostic systems, yet end-to-end multimodal LLMs (MLLMs) applied to it exhibit weak visual grounding: missed fine-grained abnormalities cause the model to fall back on learned priors, producing hallucinated findings. The authors position RRG not as a standalone text-generation problem but as a component of an intelligent perception module for autonomous healthcare workflows, and argue that reliable clinical interpretation requires externalized evidence, structured intermediate representations, and verification of reasoning steps rather than a single-pass image-to-text mapping.

Framework architecture

XMedFusion decomposes report generation into four coordinated agents operating over interpretable intermediates. Given an image $x$, the pipeline models the report as $$\hat{y} = \arg\max_y P_\theta(y \mid d_{\text{vis}}, G, D_k)$$ rather than directly $P(y \mid x)$:

• Vision Agent: produces a dense, image-grounded description $d_{\text{vis}}$ covering anatomical regions (lung fields, pleura, mediastinum, cardiac silhouette), constrained to directly observable findings to suppress speculation.
• Knowledge Graph (KG) Agent: builds a graph $G = (V, E)$ via a hybrid neuro-symbolic pipeline. BioMedCLIP-based detection runs at global and regional levels; a multi-label classifier supplies global pathology priors while zero-shot prompting localizes findings in region crops. A logical gating mechanism suppresses low-confidence regional detections when the global classifier confidently predicts absence of a pathology, reducing false positives. Validated findings are instantiated in a RadGraph-compliant schema with Anatomy and Observation nodes; negative findings are explicitly encoded so the graph acts as a strict control signal against unsupported statements.
• Retrieval and Draft Agent: retrieves top-$k$ ($k=3$) similar cases by cosine similarity in a shared embedding space following the RULE methodology, generating only a structural scaffold. Retrieved content is treated strictly as contextual guidance, never as evidence.
• Synthesis Agent: consolidates all inputs through iterative self-refinement in the ReAct style, prioritizing visual evidence and structured knowledge over retrieval priors during conflict resolution.

The central design claim is that assigning distinct evidential responsibilities—evidence extraction, structured representation, contextual scaffolding, evidence-prioritized synthesis—separates XMedFusion from retrieval-centric pipelines and general medical multi-agent systems such as MDAgents, MMedAgent, and MedRAX, which do not use structured clinical representations as control signals during synthesis.

Experimental setup

Evaluation uses IU X-ray with the standard preprocessed split of 2,068 training and 590 test image-report pairs, frontal view only, with no overlap between retrieval candidates and test samples. All generative modules use MedGemma 1.5:4B, chosen explicitly for deployability in resource-constrained environments. Metrics comprise BLEU, ROUGE-1/2/L, METEOR, and an LLM-as-a-Judge framework scoring Coverage, Consistency, Accuracy, Style, and Conciseness on a 1–10 scale. LLaVA-Med 1.5 serves as the baseline VLM.

Results

The quantitative gains over the baseline are large across every metric:

Metric	LLaVA-Med	XMedFusion
BLEU-1	0.0493	0.3359
ROUGE-1	0.1150	0.2385
ROUGE-2	0.0213	0.1328
ROUGE-L	0.0863	0.2440
METEOR	0.0829	0.1708

Semantic reliability shows even starker contrasts: Consistency rises from 2.38 to 7.80, Accuracy from 2.34 to 6.93, Conciseness from 2.16 to 7.33, Style from 3.22 to 7.00, and Coverage from 2.54 to 5.73. The authors attribute the outsized Consistency and Accuracy gains specifically to structured knowledge representation and evidence-driven synthesis, which supports their claim that explicit grounding—not merely modular orchestration—is what reduces hallucination. An explainability module maps KG entities back to anatomical regions as visual overlays, providing a traceable path from perception to final report that radiologists can audit.

Limitations and open questions

The paper concedes several constraints plainly. Evaluation is confined to a single dataset (IU X-ray) and a single modality (frontal-view chest radiographs); the framework's claimed multimodality via its routing mechanism is asserted but not empirically demonstrated. The comparison baseline is limited to LLaVA-Med 1.5—no comparison against specialized RRG systems such as MAIRA-1 or R2GenGPT is reported, despite these being discussed in related work, so the magnitude of improvement relative to the state of the art in RRG remains unestablished. The semantic evaluation relies entirely on LLM-as-a-Judge without clinician review or clinical efficacy measures (e.g., RadGraph-F1 or CheXbert-based factual consistency), leaving open whether judge scores correlate with expert assessment. Retrieval quality sensitivity—a weakness the authors themselves identify in prior retrieval-augmented work—is mitigated by design but not ablated. No ablation isolating each agent's contribution is provided, so the attribution of gains to individual components rests on qualitative argument.

Conclusion

XMedFusion demonstrates that decomposing medical image interpretation into evidence extraction, neuro-symbolic knowledge graph construction, retrieval-as-scaffolding, and iterative evidence-prioritized synthesis yields substantial improvements over single-pass VLM generation on IU X-ray, with particularly strong gains in Consistency and Accuracy under LLM-as-a-Judge evaluation. The principal open questions are validation beyond chest radiographs, comparison against dedicated RRG baselines, human clinical evaluation, and component-level ablations to confirm which elements of the pipeline drive the reported gains.