Agentic MLLMs: Autonomous Multimodal AI

Updated 8 December 2025

Agentic MLLMs are AI models that couple autonomous decision-making, multimodal perception, and dynamic tool use through persistent memory and closed-loop planning.
They leverage hybrid training paradigms like supervised fine-tuning and reinforcement learning to achieve long-horizon reasoning and adaptive tool invocation.
Applications span robotics, sports analytics, medical imaging, and GUI agents, highlighting their potential to advance interactive and autonomous AI systems.

Agentic Multimodal LLMs (Agentic MLLMs) are a class of AI architectures that couple autonomous decision-making, multimodal perception, dynamic tool use, and interaction with external environments. Unlike standard MLLMs that passively generate outputs in response to static prompts, agentic MLLMs instantiate learned policies capable of iterative perception, reasoning, planning, and adaptive action. This agentic paradigm is foundational to advancements in long-horizon reasoning, interactive systems, robotics, recommender systems, general video and image understanding, GUI agents, and embodied AI (Yao et al., 13 Oct 2025, Huang et al., 20 Mar 2025, Hong et al., 7 Nov 2025, Deng et al., 31 Oct 2025, Li et al., 3 Nov 2025).

1. Conceptual Foundations and Distinguishing Features

Agentic MLLMs formalize task-solving as optimal control in a Markov Decision Process, parameterized by a policy $\pi(a | s)$ over states $s_t$ and actions $a_t$ :

$\pi^* = \arg\max_\pi \mathbb{E}_{\tau\sim\pi}\Bigl[\sum_{t=0}^T \gamma^t\, r(s_t, a_t)\Bigr]$

where $\tau$ is the trajectory, $\gamma$ is the discount factor, and $r(s_t, a_t)$ is the environment- or task-specified reward (Yao et al., 13 Oct 2025, Huang et al., 20 Mar 2025).

Core characteristics of agentic MLLMs relative to traditional LLM agents and static MLLMs include:

Persistent memory: The system maintains and retrieves long-term state beyond the immediate context window, enabling lifelong adaptation and personalization (Huang et al., 20 Mar 2025).
Closed-loop planning and action: The model decomposes tasks, optimizes over sequences of actions, executes via external tools or APIs, senses environment changes, and iteratively refines its policy (Yao et al., 13 Oct 2025).
Multimodal perception and fusion: Inputs span text, images, audio, video, and structured sensor data, processed via encoders and fused through cross-attention or fusion modules (Huang et al., 20 Mar 2025, Li et al., 3 Nov 2025).
Autonomous tool use: Explicit, policy-driven invocation of computational tools, image/search APIs, code execution, frame extraction tools, and external databases in the reasoning loop (Hong et al., 7 Nov 2025, Deng et al., 31 Oct 2025, Li et al., 3 Nov 2025).
Reflection and self-correction: Some architectures include step-wise or post-hoc self-criticism and revision modules (Yao et al., 13 Oct 2025, Fard et al., 28 Oct 2025).

2. Reference Architectures and Workflows

Canonical agentic MLLM architectures are modular, comprising:

Perception module: Encoders for each modality convert raw inputs to dense embeddings (e.g., ViT/CLIP for vision, transformer for language).
Fusion/reasoning layer: Cross-attention/fusion networks combine embeddings and facilitate higher-order multimodal inferences. Example: $z = \text{LayerNorm}(W_T e_t + W_V e_v + b)$ or cross-modal attention $A$ (Huang et al., 20 Mar 2025).
Memory subsystem: Implements working/long-term memory (RAG, memory banks), with read/write and retrieval mechanisms driven by the policy.
Planning and execution: Policies may operate over state-action sequences using MDPs or RL. Common loop is:
- Observe environment state $s_t$
- Plan next action $a_t = \pi_\theta(s_t)$
- Optionally call external tool/API and update context/memory
- Iterate until task termination (Huang et al., 20 Mar 2025, Hong et al., 7 Nov 2025, Jiang et al., 24 Oct 2025).
Tool interface: Special tokens or structured outputs invoke tools (e.g., code snippets, API wrappers), returning structured results to the agent for continued reasoning (Deng et al., 31 Oct 2025, Fard et al., 28 Oct 2025).

Notable frameworks include:

Two-stage SFT→RL pipelines (e.g., DeepEyesV2, ToolScope, DeepSport): Supervised fine-tuning for cold-start pattern acquisition, followed by RL for optimal policy/tool use (Hong et al., 7 Nov 2025, Deng et al., 31 Oct 2025, Zou et al., 17 Nov 2025).
ReAct-style reasoning: Alternating “Thought→Action→Observation” steps for flexible tool integration and recovery from tool failures (Li et al., 3 Nov 2025, Tran et al., 14 Aug 2025).
Agentic in-context learning: Dynamic construction and iterative refinement of multimodal contexts via retrieval, alignment, and workflow graphs (see ContextNav) (Fu et al., 6 Oct 2025).

3. Capabilities: Reasoning, Memory, Tool-Use, and Environment Interaction

Agentic MLLMs combine several key intelligence modules:

Multi-step reasoning: Chain-of-thought on fused multimodal state, with integration of tool outputs at each step (Huang et al., 20 Mar 2025, Hong et al., 7 Nov 2025).
Reflection: Critique and revision modules (either interleaved or post-hoc), trained via prompt scaffolding, RL with reflection-aware rewards, or separate critic/generator templates (Fard et al., 28 Oct 2025, Yao et al., 13 Oct 2025).
Temporal and cross-modal memory: Architectures offer both token-context extension (e.g., LongRoPE) and external memory banks, with selective recall, update, and deletion operations triggered by the agent (Yao et al., 13 Oct 2025).
Autonomous and adaptive tool invocation: Agentic models actively decide when to invoke perception, search, code, or computation tools, enabling long-horizon and compositional workflows (Hong et al., 7 Nov 2025, Deng et al., 31 Oct 2025).
Interaction with external environments: Environments include simulated GUIs (e.g., LightAgent (Jiang et al., 24 Oct 2025)), video frames (DeepSport (Zou et al., 17 Nov 2025)), robotics scenes, or physical control loops. Actions can be clicks/taps, code execution, or multimodal API calls.

Agentic MLLMs are distinguished from tool-using static pipelines by closed feedback loops, dynamic policy adaptation, and persistent state management.

4. Domain Applications and Evaluation Benchmarks

Representative application domains validated by empirical studies include:

Video understanding and sports analytics: Frame-level tool access, stepwise reasoning, and grounded tool rewards allow MLLMs to “think with videos” for fine-grained, multi-sport video QA (DeepSport (Zou et al., 17 Nov 2025)).
Recommendation systems: LLM-ARS agents leverage planning, memory, and tool-use for proactive and interactive recommendation, supporting multi-turn dialogues and lifelong personalization (Huang et al., 20 Mar 2025).
Medical image/text diagnosis: Agentic self-reflection mechanisms produce explainable, real-time inference with clinician-style assessment (FT-ARM (Fard et al., 28 Oct 2025)).
GUI/mobile agents: Real-time device-cloud orchestration, condensed memory summarization, and action-policy learning for mobile applications (LightAgent (Jiang et al., 24 Oct 2025)).
Visual reasoning and “thinking-with-images”: Explicit tool-manipulation modules enable advanced image transformation and manipulation for tasks in TIR-Bench (Li et al., 3 Nov 2025).

Evaluation protocols span traditional metrics (accuracy, NDCG, Recall@K), agentic-specific measures (Autonomy, Interaction Efficiency, Tool-use Correctness, Multimodal Grounding), and benchmarks focused on compositional agentic reasoning (AgentBench, RealX-Bench, TIR-Bench, AndroidLab, ScienceQA, MathVista) (Li et al., 3 Nov 2025, Hong et al., 7 Nov 2025, Deng et al., 31 Oct 2025, Jiang et al., 24 Oct 2025).

Application Domain	Core Agentic Capability	Key Benchmark/example
Sports Video Analysis	Frame tool, CoT, RL	DeepSport, MLVU, Video-MME Long
GUI Agents	Device-cloud, RL, CoT Mem.	LightAgent, AndroidLab
Medical Imaging	Reflection, Multimodal Fusion	FT-ARM, PIID
Recommendation	Planning, Memory, Tool-Use	LLM-ARS, RecMind
Visual Reasoning	Python tool calls, Fusion	TIR-Bench, o3-TU

5. Training Paradigms, Data, and Tool Integration

Training agentic MLLMs typically involves staged procedures:

Supervised fine-tuning (SFT): Imitation of tool-augmented traces, often leveraging strong teacher models, curated for complex reasoning and explicit tool usage (Hong et al., 7 Nov 2025, Li et al., 3 Nov 2025).
Reinforcement learning (RL; e.g., PPO/GRPO): Policy refinement to maximize task success, reward for correct answers, valid tool-use, and minimal redundant invocation (Hong et al., 7 Nov 2025, Deng et al., 31 Oct 2025).
Hybrid cold-start + RL: Essential for stable tool-use initialization and prevention of reward-hacking in RL-only regimes (Hong et al., 7 Nov 2025).
Uncertainty calibration and selection: For frameworks like SRICE, conformal prediction calibrates tool outputs and token-level uncertainty guides answer selection (Zhi et al., 11 Mar 2025).

Essential training datasets include multimodal QA with tool-and-reasoning traces (e.g., Mulberry-260K, Vision-R1-cold), GUI trajectories, memory/retrieval samples, and challenging “integration-required” tasks (RealX-Bench) (Hong et al., 7 Nov 2025, Deng et al., 31 Oct 2025).

6. Open Challenges, Limitations, and Research Directions

Key challenges identified by the literature are:

Autonomy versus safety and controllability: Agentic MLLMs may hallucinate tools/actions or execute unsafe code. External database grounding, real-time validation, and transparent explanation are necessary mitigations (Huang et al., 20 Mar 2025, Yao et al., 13 Oct 2025).
Scalability and efficiency: Multimodal fusion and iterative tool-use significantly increase latency and resource consumption. Distillation to lightweight models, quantization aware planning, and efficient memory summarization are current remedies (Jiang et al., 24 Oct 2025, Huang et al., 20 Mar 2025).
Long-horizon memory and lifelong learning: Avoiding catastrophic forgetting and supporting persistent, multimodal user profiles remain open problems. Episodic and prioritized retrieval, meta-learning, and external memory managers are active research topics (Huang et al., 20 Mar 2025, Yao et al., 13 Oct 2025).
Tool and environment diversity: Expanding action/tool spaces (hundreds of APIs), integrating heterogeneous sensory modalities, and orchestrating multi-step workflows.
Benchmarking and evaluation: Existing benchmarks are limited in revealing the tight integration of memory, planning, tool-use, and reaction to environment changes. Comprehensive, task-diverse, agentic benchmarks are being developed (RealX-Bench, TIR-Bench) (Li et al., 3 Nov 2025, Hong et al., 7 Nov 2025).
Alignment and explainability: Ensuring decisions remain interpretable and correct even with autonomous tool use and dynamic workflows (Fard et al., 28 Oct 2025).

Future work will likely address hierarchical planning, real-time safety and human-in-the-loop overrides, scalable and multimodal memory, and robust tool invocation under partial observability and evolving environments, as well as closing the loop for learning from continuous real-world deployment (Yao et al., 13 Oct 2025).

7. Resources, Frameworks, and Community Efforts

Open-source agentic MLLM development is supported by:

Training libraries: LLaMA-Factory, MS-Swift, AgentTuning, R1-V, RLFactory, VERL, rLLM (PPO/GRPO implementations with multimodal and tool support) (Yao et al., 13 Oct 2025).
Datasets: Mulberry-260K, MAVIS, Vision-R1-cold, DeepEyes traces, RealX-Bench, GUI-World, MemoryBank, Search-R1 (Hong et al., 7 Nov 2025, Deng et al., 31 Oct 2025).
Benchmarks: AgentBench, TIR-Bench, AndroidLab, RealX-Bench, MMBench, ScienceQA, MathVista (Li et al., 3 Nov 2025, Deng et al., 31 Oct 2025).
Public repositories: For up-to-date resource tracking, see the maintained collection at https://github.com/HJYao00/Awesome-Agentic-MLLMs (Yao et al., 13 Oct 2025).

These resources collectively provide the infrastructure for advancing agentic MLLMs toward autonomous, adaptive, and trustworthy multimodal agents across diverse domains and deployment scenarios.