Multi-Modal Agents: Integration & Action

• Multi-Modal Agents are intelligent systems that fuse diverse sensory inputs into operational states for dynamic interaction across various domains.
• They employ iterative control loops to convert raw inputs into structured representations, enabling precise planning and robust action execution.
• These systems leverage specialized collaboration and reinforcement techniques to enhance adaptation, reliability, and performance in complex tasks.

A multi-modal agent is an agentic system in which a multimodal model, or a coordinated set of models and agents, receives heterogeneous observations such as screenshots, images, video, audio, speech, tables, documents, or robot state, reasons over them, and acts through tools, GUI operations, external expert models, or embodied controllers. In the surveyed literature, such agents appear in smartphone operation, desktop automation, biometric recognition, document question answering, pathology, human-robot interaction, mobile manipulation, video object segmentation, and multimedia content generation. What distinguishes them from a passive multimodal predictor is the presence of an interaction loop: perception is converted into an operational state, that state conditions planning, and planning is translated into executable actions over an external environment (Zhang et al., 2023, Wang et al., 2024, Zhu et al., 27 Mar 2026, Chen et al., 4 Jun 2025).

1. Conceptual scope and defining characteristics

The term multi-modal agent is used across the literature for systems that combine multimodal perception with explicit control. In some cases, the controller itself is a vision-language or multimodal LLM. T3-Agent, for example, is built around the claim that the controller should be a VLM able to inspect images and files directly while deciding which tools to call next, rather than a purely text-driven planner (Gao et al., 2024). SpeechAgents adopts the same principle in a different medium: each agent is governed by a multi-modal LLM, and the inter-agent message itself is speech plus style, not merely text (Zhang et al., 2024).

The concept also includes collaborative systems in which no single model is expected to solve the entire task. MAMMQA decomposes multimodal question answering into modality-specific evidence extraction, cross-modal synthesis, and final aggregation through two VLM agents and one LLM agent (Rajput et al., 27 May 2025). MMAC-Copilot similarly treats computer interaction as a team collaboration chain involving a Planner, Librarian, Programmer, Viewer, Video Analyst, and Mentor, with iterative replanning when execution feedback reveals mismatch (Song et al., 2024).

This body of work shows that “multimodal” does not denote vision alone. Some systems are image-centric or screenshot-centric, but others center speech communication, text-table-image reasoning, audio-text-video generation, or text-vision contradiction handling (Zhang et al., 2024, Rajput et al., 27 May 2025, Lu et al., 18 Feb 2026). Nor does “agent” denote a single architectural pattern: the surveyed systems include standalone controllers, tool-using reasoners, hierarchical multi-agent frameworks, and embodied multi-robot coordinators (Hasan et al., 24 Mar 2026, Ghafarollahi et al., 2024).

2. Perception, representation, and memory

A recurrent design choice is to convert raw multimodal input into a representation that is easier for the planner to use. AppAgent receives a real-time screenshot and an XML description of interactive UI elements, assigns each element a unique identifier, and overlays those identifiers on the screenshot so that the model can act over labeled interface targets instead of predicting exact pixel coordinates (Zhang et al., 2023). Mobile-Agent takes the opposite stance on system metadata: it works from screenshots alone, using OCR for text localization and Grounding DINO plus CLIP for icon localization, thereby avoiding XML or accessibility dependence (Wang et al., 2024).

In embodied settings, multimodal perception is often normalized into a textual or structured observation. In the human-multi-agent interaction framework, each robot fuses spoken input and onboard visual input through a VLM to create a textual structured observation used by the planner (Hasan et al., 24 Mar 2026). OWMM-Agent combines a language instruction, a pre-mapping pose graph, posed scene RGB images, current egocentric RGB and depth observations, and robot state, then maintains a robot-history memory so the agent can track whether it has already searched, picked, or should now place (Chen et al., 4 Jun 2025).

Long-horizon systems frequently supplement perception with explicit memory management. MDocAgent uses parallel retrieval pipelines—ColBERTv2 for text and ColPali for page images—so that both textual and visual evidence can be passed downstream to specialized reasoning agents (Han et al., 18 Mar 2025). MMA adds a Confidence Module over retrieved memories, assigning each memory item a reliability score that combines source credibility, temporal decay, and conflict-aware network consensus, then reweights evidence and abstains when support is insufficient (Lu et al., 18 Feb 2026).

A common implication of these designs is that multimodal agency depends as much on representation engineering as on model capacity. The environment is rarely presented to the planner in raw form; it is converted into identifiers, summaries, retrieved evidence, memory items, or compact state variables that make downstream planning tractable.

3. Planning, action spaces, and control loops

Most multimodal agents are organized around explicit iterative control loops. AppAgent uses an “Observe, Think, Act, and Summarize” loop in deployment, with the interaction summary fed into the next step as short-term memory. Its action space is intentionally reduced to six predefined actions—Tap, Long_press, Swipe, Text, Back, and Exit—to avoid the coordinate-prediction problem that degrades raw GUI control (Zhang et al., 2023). Mobile-Agent adopts a related structure inspired by ReAct, with Observation, Thought, and Action, but defines eight operations including Open App, Click the text, Click the icon, Type, Page up / down, Back, Exit, and Stop (Wang et al., 2024).

MMCTAgent generalizes the same pattern to complex visual reasoning. Its planner cycles through Thought, Action, and Observation, invokes OCR, transcript search, frame retrieval, or localized vision analysis as needed, and then passes the reasoning chain to a vision-based critic that checks answer completeness, reasoning comprehensiveness, and hallucination (Kumar et al., 2024). FusionAgent applies the agent loop to biometric recognition: the MLLM examines a query, chooses one expert model/tool, receives a predicted label and stored score vector, and then decides whether to call another tool or stop. The first selected model becomes the anchor model for ACT score fusion (Zhu et al., 27 Mar 2026).

In more modular computer-use systems, planning is explicitly separated from execution. InfantAgent-Next divides the workflow into Planning, Tool selection, Execution, and Completion checking, with role-specific prompts and a structured memory cache using tags such as <thought>, <task>, and execution tags for dialogue reconstruction across phases (Lei et al., 16 May 2025). In multi-robot interaction, planning is decentralized at the agent level but globally constrained by a centralized coordinator that computes response likelihood scores $s_t^i$ and selects agents whose scores exceed a threshold $\tau$, scheduling them sequentially if multiple agents qualify (Hasan et al., 24 Mar 2026).

These papers repeatedly treat action selection as a constrained interface design problem. Symbolic UI actions, expert-tool calls, function calls to planners, and ordered lists of parameterized embodied actions are all ways of narrowing a large, continuous control problem into an executable set of choices that a language-centered controller can manage.

4. Learning, adaptation, and data generation

The surveyed literature uses several distinct adaptation strategies. AppAgent does not require parameter fine-tuning for each app; instead, it builds an app-specific knowledge base through autonomous exploration or human demonstrations, updates that document when elements are encountered multiple times, and reuses the resulting app document plus interaction summaries during deployment (Zhang et al., 2023). By contrast, T3-Agent argues for direct trajectory tuning of the controller itself. Its MM-Traj dataset contains 20K verified multimodal tool-usage tasks distilled from 23.5K raw synthetic data points, and the tuned controller improves two open VLMs by about 20% over untuned versions on GTA (Gao et al., 2024).

SpeechAgents trains multi-agent communication capability through Multi-Agent Tuning. The benchmark dialogues are parsed into 751,691 agent trajectories, and the model is explicitly prompted to “Think Before You Speak,” while mix-tuning with general speech instruction data is used to preserve broader speech-to-speech ability (Zhang et al., 2024). FusionAgent learns sample-wise dynamic model selection by Reinforcement Fine-Tuning with GRPO, using a reward that combines format reward, tool success reward, answer accuracy reward, and a metric-based reward defined over Rank, mAP, TAR, and FNIR (Zhu et al., 27 Mar 2026).

Other systems center data curation around workflow quality and preference alignment. MultiMedia-Agent uses Skill Acquisition Theory to motivate a three-stage pipeline comprising base-plan finetune, success-plan finetune, and DPO-based preference optimization, with a two-stage correlation process that first self-corrects plan executability and then refines plans using model-based preference metrics (Zhang et al., 6 Jan 2026). OWMM-Agent introduces an agentic data synthesis pipeline in Habitat, producing 21,046 valid episodes and about 235K annotations across pick, place, navigate, and search-scene-frame subsets so that the VLM learns robot history tracking, multi-image reasoning, and function-calling outputs (Chen et al., 4 Jun 2025).

Not all multimodal agents are tuned. MAMMQA is explicitly zero-shot and prompt-driven, and its contribution lies in pipeline decomposition rather than parameter adaptation (Rajput et al., 27 May 2025). The literature therefore contains both train-free orchestration frameworks and heavily supervised or reinforcement-tuned controllers.

5. Multi-agent decomposition and specialist collaboration

A large fraction of multimodal-agent research treats specialization as a central design principle. MAMMQA uses a three-stage sequential pipeline: a modality expert agent extracts query-relevant information independently from text, table, and image inputs; a cross-modality synthesis agent reasons across modalities from anchored perspectives; and an aggregator agent decides the final answer from synthesized evidence without access to the raw multimodal inputs (Rajput et al., 27 May 2025). MDocAgent applies the same logic to documents through five agents—a general agent, a critical agent, a text agent, an image agent, and a summarizing agent—over a dual text/image retrieval substrate (Han et al., 18 Mar 2025).

In pathology, WSI-Agents extends this pattern into verification-heavy collaboration. A task agent identifies the question type and routes the task to a morphology, diagnosis, treatment planning, or report generation expert; candidate outputs from a WSI model zoo are then checked by a logic agent for internal consistency, a fact agent against a pathology knowledge base, a consensus agent against TITAN, CONCH, and Prism, and finally a summarizing agent with iterative reasoning-agent feedback (Lyu et al., 19 Jul 2025). PathFinder decomposes diagnosis into Triage, Navigation, Description, and Diagnosis agents, thereby emulating the pathologist’s process of scanning, noting, and synthesizing evidence from selected patches rather than making a single-shot bag-level prediction (Ghezloo et al., 13 Feb 2025).

Collaboration is not restricted to language-only or document-only tasks. The human-multi-agent interaction framework keeps each humanoid robot as an autonomous cognitive agent with perception → planning → action, but adds a centralized coordination layer to prevent overlapping speech and conflicting actions (Hasan et al., 24 Mar 2026). AtomAgents organizes alloy design as a hierarchical multi-agent workflow with User, Assistant, Scientist, Group chat manager, and a planning tool containing Admin, Planner, and Critic, plus specialized tools for retrieval, computation, coding, and plot or image analysis (Ghafarollahi et al., 2024).

These systems share a common thesis: multimodal reasoning benefits when retrieval, interpretation, verification, and synthesis are distributed across role-specific components instead of being forced into a single monolithic forward pass.

6. Domains, benchmarks, and reported performance

The empirical literature is highly domain-specific, and the reported metrics are not directly comparable across application areas. Nevertheless, the benchmark record shows that multimodal agents have been evaluated on GUI control, multimodal QA, document understanding, pathology, social reasoning, video segmentation, robotics, and computer-use tasks.

Domain	Representative system	Reported result
Smartphone apps	AppAgent	SR rises from around 2.2% with raw action/no document to 95.6% with manually crafted documents; 10 apps, 50 tasks (Zhang et al., 2023)
Mobile device control	Mobile-Agent	Success rates 91%, 82%, 82% across three instruction types; completion rates above 90% (Wang et al., 2024)
Text-table-image QA	MAMMQA	76.37 on MultiModalQA with GPT-4o-mini; 89.90 on ManyModalQA (Rajput et al., 27 May 2025)
Long-document QA	MDocAgent	12.1% average improvement over M3DocRAG under top-1 retrieval (Han et al., 18 Mar 2025)
Whole-slide pathology	WSI-Agents	0.703 on WSI-Bench and 60.0% on WSI-VQA (Lyu et al., 19 Jul 2025)
Melanoma diagnosis	PathFinder	74% accuracy; 9% above average pathologist performance (Ghezloo et al., 13 Feb 2025)
Multi-agent Theory of Mind	LIMP on MuMA-ToM	76.6% overall; human baseline 93.5%; GPT-4o 50.6% (Shi et al., 2024)
Referring video object segmentation	M²-Agent	46.1 mIOU on MeViS and 36.26 on Ref-AVS (Tran et al., 14 Aug 2025)
Open-world mobile manipulation	OWMM-VLM-38B	21.90% strict episodic success, 51.52% relaxed, 90% real-world total accuracy (Chen et al., 4 Jun 2025)
Desktop and multimodal computer use	InfantAgent-Next	35.3% on OSWorld and 56.97% average on GAIA validation (Lei et al., 16 May 2025)

Several papers also report strong gains from architectural choices rather than from larger base models alone. In AppAgent, the symbolic action space and app documents materially improve success rates (Zhang et al., 2023). In T3-Agent, verified synthetic trajectories improve tool accuracy and code executability on GTA and answer accuracy on GAIA (Gao et al., 2024). In MMAC-Copilot, the collaborative agent chain reaches 25.91 average on GAIA and 70.32 average on VIBench, surpassing the listed baselines in the reported setup (Song et al., 2024). In FusionAgent, dynamic model selection with ACT reduces tool usage by about 32% while improving Rank-1, mAP, TAR, and FNIR on CCVID, LTCC, and MEVID (Zhu et al., 27 Mar 2026).

7. Reliability, limitations, and unresolved issues

The literature repeatedly identifies reliability as a systems-level problem. MMA shows that similarity-based memory retrieval can surface stale, low-credibility, or conflicting items, and introduces dynamic reliability scoring plus abstention. Its MMA-Bench further exposes the “Visual Placebo Effect,” in which raw visual input can induce overconfident answers even in Type D unknowable cases; for GPT-4.1-mini, Type D CoRe drops from 0.85 in Text mode to 0.23 in Vision mode (Lu et al., 18 Feb 2026).

A separate line of work studies implementation bugs directly. The systematic study of 34 representative multimodal agents identifies 158 M-agent-specific bugs from 1,268 issue reports and organizes them into 6 global symptoms, 16 component-level symptoms, and 7 root causes. MATester, which analyzes runtime inter-component outputs, covers 61.4% of known open issues on 12 extra M-agents and discovers 31 additional bugs (Li et al., 6 Jul 2026). This formalizes a practical fact already visible in many application papers: multimodal-agent failure can originate in stale perception, malformed plans, inapplicable execution, or incorrect reflection rather than in model reasoning alone.

Application papers also note domain-specific constraints. AppAgent excludes advanced primitives such as multi-touch or irregular gestures (Zhang et al., 2023). T3-Agent currently uses multimodal information in queries but not in intermediate trajectories, which the paper identifies as a limitation for tasks such as image editing (Gao et al., 2024). The human-multi-agent interaction framework warns that centralized arbitration may become a scaling bottleneck for larger robot teams and that LLM/VLM delays affect interaction flow (Hasan et al., 24 Mar 2026). MMCTAgent reports computational overhead and notes that its critic can occasionally over-correct a correct answer into a hallucinated one (Kumar et al., 2024).

Social reasoning remains particularly difficult. MuMA-ToM shows that strong large multimodal models still lag far behind humans on multi-agent Theory of Mind, with standard models often confusing helping with hindering or missing small and occluded objects that determine latent intent (Shi et al., 2024). A plausible implication is that multimodal agency cannot be reduced to raw perception or tool invocation; it also depends on calibrated memory, verification, environment-aware execution, and, in some tasks, explicit models of latent state.

Taken together, these papers portray the multi-modal agent not as a single architecture but as a research program centered on closed-loop interaction under heterogeneous evidence. Its characteristic problems are multimodal representation, action grounding, specialist coordination, memory reliability, and error containment; its characteristic promise is the ability to convert multimodal observation into sustained, context-sensitive action across domains that were previously handled either by static predictors or by narrowly engineered pipelines.