MaMe: Modular Multi-Agent Medical Diagnosis

• MaMe is a modular, role-specialized framework that decomposes clinical diagnosis into specialized LLM-based agents handling text, image, audio, and video inputs.
• It uses assigned-role prompting and iterative consensus through multi-agent discussion to achieve significant performance gains over monolithic models.
• The design allows targeted model updates and integration of external knowledge, aligning with clinical workflows from triage to final diagnosis synthesis.

MaMe denotes the Modular Multi-Agent Framework for Multi-Modal Medical Diagnosis (MAM), a modular, role-specialized, prompt-based architecture that decomposes medical diagnosis across multiple LLM-based agents rather than relying on a single unified multimodal medical LLM. Its stated objective is to make reasoning over heterogeneous clinical inputs—text, image, audio, and video—more accurate, updatable, and flexible, while better matching clinical workflows such as triage, specialist consultation, imaging interpretation, retrieval of external knowledge, multidisciplinary discussion, and final attending-level synthesis. The framework is evaluated on a broad set of public multimodal medical datasets and is reported to achieve relative improvements ranging from 18% to 365% over baseline models (Zhou et al., 24 Jun 2025).

1. Concept and design rationale

MaMe is motivated by three limitations attributed to monolithic multimodal medical LLMs: high knowledge update cost, limited flexibility, and limited comprehensiveness in practice. In the formulation underlying MaMe, a unified model must simultaneously handle general triage, radiology, cardiology, question answering about published studies, and other tasks, while remaining customizable for institutions, languages, and subdomains. The framework instead adopts a modular multi-agent design in which individual roles can be swapped, updated, or grounded by distinct tools and knowledge sources.

The immediate design rationale comes from two empirical findings. First, “Assigned Roles” prompting, as opposed to “Direct” prompting, yields large and consistent gains across 10 datasets and all modalities; reported examples include +38.5% on PubMedQA and +38.1% on NIH chest X-rays. Second, LLMs exhibit “diagnostic discernment”: when asked to choose among multiple candidate diagnoses they previously generated, they perform better than chance; one reported example is PathVQA, where “Reasoning” accuracy is 91.7% versus “Expectation” 50.0%. These observations motivate an architecture in which role context, opinion generation, synthesis, and adjudication are explicit rather than implicit (Zhou et al., 24 Jun 2025).

A central claim of the framework is therefore organizational as much as algorithmic. Each agent corresponds to a clinically legible function, and the overall diagnostic trajectory is structured as staged collaboration rather than a single forward pass. This suggests a diagnostic regime in which specialization and deliberation are treated as first-class primitives.

2. Agent decomposition and formal pipeline

MaMe defines five LLM-based roles: a General Practitioner, Specialist Team, Radiologist, Medical Assistant, and Director. The multimodal input is written as $M = \{m_1, m_2, ..., m_k\}$, where each $m_i$ is a modality such as text, image, audio, or video.

Role	Core function	Principal outputs
General Practitioner	Initial triage and disease type classification	$d$, specialist set $S$
Specialist Team	Problem decomposition, anonymization, diagnostic opinions, voting	$P$, $O_{s_i}$, $v_i$
Radiologist	Imaging-focused interpretation	$O_{Rad}$
Medical Assistant	Retrieval and summarization of external knowledge	$I_r$, $I_s$
Director	Synthesis, consensus management, final diagnosis	$m_i$0, $m_i$1

The formal pipeline begins with disease typing and referral selection:

$m_i$2

The Specialist Team then decomposes the case into sub-problems,

$m_i$3

and anonymizes case information before retrieval. The Medical Assistant retrieves and summarizes external material,

$m_i$4

Specialists and the Radiologist then produce diagnostic opinions:

$m_i$5

$m_i$6

The Director synthesizes these into a report,

$m_i$7

and, once consensus is reached, emits the final diagnosis,

$m_i$8

This decomposition is explicitly intended to support targeted model choice. A radiology LVLM can underwrite the Radiologist, an audio model can underwrite audio-specific reasoning, and a medical text LLM can underwrite text-heavy roles. A domain update therefore need not entail global retraining; the stated design principle is that updating a domain means updating only the corresponding agent (Zhou et al., 24 Jun 2025).

3. Multimodal reasoning and consensus formation

MaMe does not introduce a single learned cross-modal fusion layer. Instead, modalities are handled by specialized agents and fused at the language level through LLM reasoning. The textual context—patient question, modality descriptions, and retrieved knowledge—acts as the hub where heterogeneous evidence is integrated. In the described implementation, non-text modalities can be converted into textual descriptions through prompts such as “Please describe this image briefly in 100 words,” after which Director-level synthesis effectively performs multimodal fusion in natural language.

This yields a specific interpretation of multimodal reasoning. The Radiologist attends to imaging components; audio-oriented roles can use outputs from Qwen-Audio-Chat; video-LLMs can be incorporated when video is present; but the final cross-modal integration occurs in the shared linguistic space of agent opinions and summaries. The framework characterizes this as agent-wise specialization by modality and clinical function, followed by language-level fusion rather than explicit neural feature fusion.

Consensus is enforced through an iterative discussion loop. Specialists and the Radiologist present opinions; the Director synthesizes a report; specialists review that report and vote with $m_i$9; and total support is computed as

$d$0

If $d$1, consensus is reached; otherwise another round begins. The operational significance of this mechanism is conflict resolution. Different specialists may advance incompatible candidate diagnoses, but the Director’s synthesis plus specialist voting acts as an adjudication layer. The framework directly links this design to the earlier empirical observation on diagnostic discernment, treating deliberation and reevaluation as productive rather than purely redundant (Zhou et al., 24 Jun 2025).

4. External knowledge, model reuse, and update mechanics

A defining feature of MaMe is explicit reuse of existing best-in-class models by modality and task. The reported substrate includes Medichat-Llama3-8B for text tasks; image models such as LLaVA-7B, Qwen2-VL-7B, LLaVA-Med-7B, Qilin-Med-VL-13B, and HuatuoGPT-Vision-7B; Qwen-Audio-Chat for audio; and video models such as LLaVA-Next-Video-7B, Qwen2-VL-7B, and VideoLLaMA2-7B. The framework itself is prompt-based rather than trained end-to-end.

The Medical Assistant implements a retrieval-augmented generation pattern. Retrieval is performed over anonymized sub-problems $d$2 using Google API in the experiments, and the resulting material is summarized into $d$3 for use by Specialists and Director. The paper emphasizes privacy by noting that queries are formed from anonymized $d$4 to prevent leakage. This arrangement also serves the update objective: if radiology practice changes, the Radiologist agent can be replaced; if cardiology guidelines change, the cardiology specialist or retrieval prompts can be updated; if a stronger audio model becomes available, it can underwrite the relevant role.

The retrieval analysis is reported with two quantities. “Recall” is the proportion of cases in which retrieved documents include the correct answer, and “Answer Correct” is the final-answer accuracy when the correct answer is present in the retrieved material. Example values include Recall 12.1% and Answer Correct 46.2% for NIH, and Recall 34.0% and Answer Correct 58.8% for Heartbeat. The stated interpretation is that when retrieval captures the relevant knowledge, the framework can use it effectively, indicating synergy between retrieval and multi-agent reasoning (Zhou et al., 24 Jun 2025).

5. Evaluation across modalities

Evaluation spans four modality types and multiple public medical datasets. Text tasks use MedQA and PubMedQA; image tasks use Brain Tumor, DeepLesion, NIH Chest X-rays, PathVQA, and PMC-VQA; audio tasks use Heartbeat and SoundDr; and video uses MedVidQA. The primary metric is accuracy.

Modality	Public datasets	Reported MaMe accuracy
Text	MedQA, PubMedQA	40.0 / 84.0
Image	PathVQA, PMC-VQA, DeepLesion, NIH, Brain Tumor	47.6 / 32.5 / 35.1 / 58.6 / 97.9
Audio	Heartbeat, SoundDr	64.0 / 47.9
Video	MedVidQA	74.3

For text, Medichat-Llama3-8B scores 30.8 on MedQA and 48.5 on PubMedQA, AdaptLLM-7B scores 30.5 and 56.8, and MaMe scores 40.0 and 84.0. For image tasks, HuatuoGPT-Vision-7B reports 40.1 on PathVQA, 24.0 on PMC-VQA, 11.1 on DeepLesion, 12.6 on NIH, and 80.2 on Brain Tumor, while MaMe reports 47.6, 32.5, 35.1, 58.6, and 97.9. For audio, Qwen-Audio-Chat reports 34.9 on Heartbeat and 25.0 on SoundDr, while MaMe reports 64.0 and 47.9. For video, VideoLLaMA2-7B reports 55.3 on MedVidQA and MaMe reports 74.3 (Zhou et al., 24 Jun 2025).

The reported relative improvements include approximately 73% on PubMedQA versus Medichat-Llama3-8B, approximately 216% on DeepLesion versus HuatuoGPT-Vision-7B, approximately 365% on NIH versus HuatuoGPT-Vision-7B, approximately 83% on Heartbeat versus Qwen-Audio-Chat, and approximately 34% on MedVidQA versus VideoLLaMA2-7B. Within the paper’s framing, these results support the claim that orchestrating specialized agents plus retrieval is more effective than applying a single modality-specific LLM or LVLM to the same tasks.

6. Ablations, limitations, and clinical interpretation

The component ablation starts from a Direct baseline with a single model and no roles, no discussion, and no retrieval; then adds +Roles, +Discussion, and +Retrieval. The pattern reported across nearly all datasets is monotonic improvement. On NIH, the sequence is Direct 12.6, +Roles 36.0, +Discussion 38.6, and +Retrieval 58.6. This isolates the contribution of role conditioning, multi-agent discussion, and external knowledge. Earlier prompt-only experiments similarly show that assigning a physician role improves performance across all modalities.

Two additional analyses qualify the collaboration mechanism. Varying the number of discussion rounds up to 3 shows that performance often improves for 1–2 rounds, then saturates or fluctuates, and too many rounds can slightly hurt performance through over-discussion or noise. Varying the number of roles yields an inverted U-shape: 1 role is weakest, 3 roles is often best, and 5 roles can be slightly lower. The paper’s interpretation is that some role specialization is crucial, but excessive granularity can introduce overhead or redundancy. A plausible implication is that MaMe’s full five-role decomposition is a maximal organizational template rather than a universally optimal team size.

The framework’s stated limitations are threefold: dependence on the quality and biases of underlying LLMs and LVLMs; lack of real-world clinical validation beyond public datasets; and retrieval limitations arising from use of Google API rather than curated clinical databases or EHR systems. Real deployment is said to require prospective studies, regulatory review, and human-in-the-loop validation. The practical discussion nevertheless emphasizes interpretability and auditability: agent discussions, reasoning, and votes are visible as text; the Director’s summary lists agreements, disagreements, and conclusions; review prompts aim to detect “medical reasoning errors, redundant statements, or invalid outputs”; and an appendix-level “Overall Review” prompt asks whether the final answer is reasonable, with a “yes” or “no” output. The paper explicitly notes that these mechanisms are not sufficient for clinical-grade safety, but that the structure is well suited for adding further safeguards such as rule-based checks, constraint-based reasoning, and required human approval for high-risk decisions (Zhou et al., 24 Jun 2025).

7. Nomenclature and other uses of “MaMe”

The string “MaMe” is not unique in arXiv usage. In meta-reinforcement learning, MAME denotes “Model-Agnostic Meta-Exploration,” a framework that separates an exploration policy $d$5 from an exploitation policy $d$6 and uses a supervised or self-supervised inner-loop update of a task latent $d$7 rather than policy-gradient adaptation (Gurumurthy et al., 2019). In efficient State Space Models for vision, MaMe denotes “Mamba-aware Merging,” a token-merging method that uses the state transition parameter $d$8 as a token informativeness measure and preserves sequential order in Vision Mamba-style backbones (Park et al., 19 Aug 2025). In transformer acceleration, MaMe denotes “Matrix-based Token Merging,” a training-free, differentiable, GPU-friendly compression method defined by dense matrix operations, with MaRe as its restoration counterpart for visual perception and synthesis (Huo et al., 15 Apr 2026). In computer vision datasets, MAMe denotes the “Museum Artworks Medium” dataset, a 29-class benchmark emphasizing high resolution and variable image shape in medium classification for museum objects (Parés et al., 2020). In categorical logic, “MaMe” is used as shorthand for the Mariano–Mendes work “Towards a good notion of categories of logics,” which motivates flexible categories of logics and morphisms preserving algebraizing pairs (Pinto et al., 2014).

Within the medical-diagnosis literature, however, MaMe refers to the role-specialized modular multi-agent framework introduced as MAM. In that usage, the term designates not merely a prompting style but a full orchestration scheme for multimodal diagnosis, retrieval, deliberation, and consensus.