---
title: 'MoMA: Mixture-of-Multimodal Agents'
url: https://www.emergentmind.com/topics/mixture-of-multimodal-agents-moma
type: topic
---

# MoMA: Mixture-of-Multimodal Agents

A Mixture-of-Multimodal-Agents (MoMA) is a computational architecture that orchestrates multiple heterogeneous specialized agents—including but not limited to large language models (LLMs), domain-specific tools, and expert models—across diverse input modalities and tasks. Originating as a direct solution to both routing and interference challenges in multi-domain inference and multimodal reasoning, MoMA systems unify agent-level selection, model-level expert selection, and multi-agent coordination within a general, modular inference paradigm. MoMA abstracts across several instantiations: routing and orchestration of LLMs/tools in multi-domain QA and task completion [2509.07571], specialist agent cascades for clinical prediction from EHR data [2508.05492], and task-isolated Mixture-of-Experts Transformers in open-world embodied agents [2506.10357].

## 1. MoMA System Architectures

MoMA systems are typified by an explicit resource pool comprising both LLMs and specialized agents, and a routing+aggregation pipeline whose architecture varies by domain:

- **LLM-Agent Routing (Generalized Inference):** The MoMA router [2509.07571] consists of two stages. The first, "agent routing," uses embedded query-category retrieval and a context-aware finite state machine (CA-FSM) to identify and dynamically mask available domain agents (e.g., code generator, travel assistant). The second, "LLM routing," invokes a Mixture-of-Experts head that predicts model-specific utility vectors and applies Pareto-frontier+TOPSIS selection to balance accuracy/cost among LLMs of varying scale.

- **Specialist Aggregation (Clinical Prediction):** Each non-text modality is projected via a specialist LLM agent to produce structured summaries (e.g., CXR-LLAVA-v2 for chest X-rays, Llama-3 for lab tables) [2508.05492]. These are concatenated with textual clinical notes and compressed with a frozen aggregator LLM; a predictor agent produces final outputs from the aggregator's latent representation.

- **Task-Level Gated MoE Transformers (Open-World Agents):** In Optimus-3, each MoE layer incorporates K task-specific experts and one shared knowledge expert [2506.10357]. A lightweight router classifies the instruction and activates only the corresponding expert plus the shared expert, entirely avoiding cross-task gradient interference.

This modular design generalizes across domains, supporting pluggable agents and scalable resource expansion.

## 2. Mathematical Formulation and Routing Algorithms

MoMA routing and aggregation are formalized as multi-stage, multi-agent decision processes:

- **LLM Routing Optimization:** For a query $q_i$, with LLM set $M = \{1, ..., M\}$, and costs $c_k$ for each model $m_k$, the router predicts $r_k(q_i)$ for each model and applies score-cost normalization:
  $$
  c' = \frac{c_k - c_{min}}{c_{max} - c_{min}}, \quad s' = \frac{r_k(q_i) - s_{min}}{s_{max} - s_{min}}
  $$
  The ideal (0,1) and anti-ideal (1,0) points define closeness $\phi_k$ via the TOPSIS metric. The optimal model is selected as $\arg\max_k \phi_k$ [2509.07571].

- **Task-Level Expert Gating:** For instruction-identified task $t$, gating vector $g_i(u^{\ell})$ is binary: $g_s(u)=1$, $g_t(u)=1$, $g_i(u)=0$ elsewhere; the layer output is
  $$
  h^{\ell} = E_s(u^{\ell}) + E_t(u^{\ell})
  $$
  [2506.10357]

- **Modal Aggregation in Clinical MoMA:** The aggregator receives input
  $$
  m_i = t_i \oplus \left( \bigoplus_{j=1}^M s_{i,j} \right)
  $$
  and outputs $u_i = \mathcal{A}(m_i)$; a predictor applies $z_i = W u_i + b$, with softmax/sigmoid for classification [2508.05492].

Training is task-conditional and may use categorical cross-entropy, RL surrogates, or fine-tuning of only final predictor modules.

## 3. Training Data Construction and Optimization

- **Scale and Diversity:** MoMA for general inference is trained on $\approx 2.25$ million instances spanning science, writing, code, and more, automatically labeled for model-pair win/loss/equal outcomes via LLM-based judgment [2509.07571].

- **Knowledge-Enhanced Pipelines:** In open-world agents, knowledge graphs are constructed from domain wikis to synthesize planning and action data; expert models (DeepSeek-VL, Grounding DINO, GPT-4) annotate observational trajectories [2506.10357].

- **Clinical MoMA:** Specialist agents operate zero-shot (no gradient updates); only the final predictor agent is fine-tuned using LoRA on Llama-3, with AdamW optimizer and 8-bit quantization [2508.05492].

- **Data Augmentation:** For robust profiling, queries are diversified using BERT-based selection and response generation by multiple LLMs, assigning ordinal comparison labels and optional Elo rankings [2509.07571].

## 4. Inference-Time Routing and Execution Pipelines

MoMA's inference pipeline is fully modular and dynamically adapts per-query:

- **Generalized Routing (Pseudocode):**
  ```
  function MoMA_route(q):
    cats = retrieve_categories(q)
    state = CA_FSM.initial
    for input in [q, events]:
      state = δ(state, input)
    A_cand = filter_agents(state, cats)
    mask = build_mask(A_cand)
    agent_choice = LLM_decode_with_mask(q, mask)
    if agent_choice ≠ NONE:
      return invoke(agent_choice, q)
    hidden = LLM_encoder(q)
    r = MoE_router(hidden)
    m* = select_by_Pareto_COST_PERF(r, costs)
    return call_LLM(m*, q)
  ```
  [2509.07571]

- **Multimodal Aggregation:** Specialist agents summarize their modalities, followed by aggregation and final prediction. All specialist and aggregation steps are zero-shot; inference cost is minimized by restricting fine-tuning to the predictor [2508.05492].

- **Task-Routed MoE:** Task classifier activates only the matching expert and shared expert at each layer, ensuring forward passes are sparse and free from cross-task parameter updates [2506.10357].

This sequence ensures both cost-efficiency and broad applicability, adaptively leveraging deterministic tools, specialized models, and generalist LLMs as appropriate.

## 5. Empirical Results and Quantitative Comparisons

MoMA systems have demonstrated superior performance-cost tradeoffs and robustness across several domains.

- **Generalized Routing MoMA** [2509.07571]:
  - Achieves up to 90% cost reduction versus best single LLM baseline in cost-priority mode; 31–57% cost reduction in auto-routing and performance-priority settings.
  - Pareto-optimal model selection often chooses 8B–13B parameter models for typical queries, reserving 235B models for only the hardest cases.
  - Outperforms SFT and contrastive routers by matching or exceeding performance at lower cost.
  - Scales to sub-100 ms latency at agent-routing stages with only 10–20 ms router overhead.

- **Optimus-3 (Embodied Minecraft MoMA)** [2506.10357]:
  - Outperforms GPT-4o, Qwen2.5-VL, and prior task-MoE architectures across long-horizon, planning, captioning, VQA, grounding, and reflection benchmarks.
  - Yields up to 3.4× gain in grounding IoU@0.5 and substantially higher scores in planning and embodied QA.
  - Ablations confirm that task-level routing eliminates catastrophic interference, preserving prior tasks when new experts are added.

- **Clinical MoMA** [2508.05492]:
  - Macro-F1 for chest trauma prediction: 0.834 (vs. 0.802 best baseline).
  - Alcohol screening AUROC: 0.755 (vs. 0.714).
  - Remains robust across demographic subgroups.
  - Ablations show 0.834→0.778 F1 drop when specialist agent removed, highlighting the necessity of modality-specific summarization.

## 6. Limitations and Future Extensions

- **Generic Limitations:**
  - Intermediate LLM-generated summaries in the clinical MoMA are not directly validated and pose hallucination risk [2508.05492].
  - Even with sparse gating, MoE Transformers require increased GPU memory and introduce routing overhead [2506.10357].
  - No lifelong memory or self-improving meta-controller present in current open-world MoMA systems; experience is not accumulated across missions [2506.10357].

- **Modularity and Adaptability:**
  - Plug-and-play addition of new agents is supported; more advanced gating, iterative aggregation, or mutual critics among agents (multi-turn conversations) are not yet deployed in production [2508.05492].
  - A plausible implication is that extending MoMA to robotics or other embodied domains would primarily require definition and incorporation of new task experts via the same abstraction [2506.10357].

- **Validation and Deployment:**
  - Real-world deployment in healthcare and open-world environments will demand rigorous external validation and monitoring for input drift and fairness [2508.05492].

Planned future directions include differentiable memory banks, generative task annotation pipelines, domain-adaptive end-to-end training, and conversational agent aggregation [2506.10357, 2508.05492].

## 7. Conceptual and Practical Significance

MoMA represents a convergence point for several research threads: Mixture-of-Experts (MoE), tool-augmented LLMs, modular agent orchestration, and task-level gating for catastrophic interference avoidance. Its abstraction layer enables multi-domain, multimodal task execution at scale with efficiency unattainable for monolithic models.

By leveraging dedicated experts for modality transformation and agent aggregation, MoMA achieves strong cost-performance tradeoffs in both traditional ("text-to-label") and embodied ("perception–action–reflection") settings. The architecture's plug-and-play nature and minimal data-pairing requirements facilitate rapid extension to new domains, contingent on validation and continued methodological refinement [2509.07571, 2508.05492, 2506.10357].

Source: https://www.emergentmind.com/topics/mixture-of-multimodal-agents-moma