MDP-Agent: Explicit Decision Modeling

• MDP-Agent is an architectural pattern that models decision-making through explicit MDPs and variants (e.g., POMDPs, Dec-MDPs) by detailing state, action, transition, and reward structures.
• It supports multiple optimization objectives—from discounted returns to average reward and lexicographic criteria—enabling adaptation to diverse application domains.
• Key implementations leverage communication-aware coordination, scalable planning techniques like symmetry factorization and tensor compression, and integration with foundation models for sim-to-real tasks.

MDP-Agent denotes a class of agent formulations in which decision making is made explicit through a Markov decision process, or through closely related models such as POMDPs, multi-agent MDPs, factored Dec-MDPs, and zero-sum Markov games. In the basic single-agent form, the agent is defined on $\mathcal{M}=(\mathcal{S},\mathcal{A},T,R,\gamma)$ and optimizes $$J(\pi)=\mathbb{E}\!\left[\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t)\right]$$; in other lines of work, the same design principle is extended to common-belief coordinators, local-policy message-passing systems, tensor-compressed planners, reward-free exploration schemes, and automated RL pipelines that first infer an MDP interface and then optimize within it (Liu et al., 5 Jun 2026, Wei et al., 16 Sep 2025). Taken together, the literature suggests that “MDP-Agent” functions less as a single canonical algorithm than as a reusable architectural pattern for making state, action, transition, and reward structure explicit across heterogeneous decision problems.

1. Formal scope and semantic range

The most general interpretation of MDP-Agent is an agent whose control loop is organized around explicit state abstractions and Bellman-style optimization. In the single-agent case, this is the standard discounted MDP; in partially observable settings, the agent operates on an observation model $O(o\mid s)$ and updates a belief $b_t\in\Delta(\mathcal{S})$ by Bayes’ rule, with policy conditioning on observations or beliefs (Liu et al., 5 Jun 2026). In broader formulations, the same idea appears as a fully observable cooperative MMDP with joint state and action spaces, a Dec-MDP with local observations that are jointly sufficient to reconstruct the system state, or a two-player zero-sum Markov game with minimax value recursion (Pol et al., 2021, Fu et al., 2022, Xiong et al., 2022).

A foundational precursor is the feature-based view in which histories $h_t$ are compressed into finite states through a mapping $s_t=\phi(h_t)$, with the induced process required to be predictive enough for control. In that formulation, the feature map is selected by minimizing a coding-cost criterion that trades off the compressibility of induced state transitions against the predictability of rewards, thereby turning state construction itself into part of the agent design problem (0812.4580). This perspective is important because many later MDP-Agent variants differ primarily in how they construct or approximate the effective state.

The objective optimized by an MDP-Agent is not fixed across the literature. Besides the standard discounted return, one finds average reward in tree-structured multi-agent systems, undiscounted finite-horizon utilities in healthcare allocation, stochastic shortest-path objectives for multi-target cover time, lexicographic quantile criteria over terminal states, and reward-free exploration in which the exploration phase is decoupled from the extrinsic reward used at planning time (Qu et al., 2019, Hosseini et al., 2014, Nawaz et al., 2022, Li et al., 2017, Qiu et al., 2021). A common misconception is therefore that the term implies a single optimization criterion; the literature instead spans discounted, average-reward, finite-horizon, and SSP-style formulations.

2. Common-information coordination and communication-aware control

One prominent meaning of MDP-Agent is a coordinator-based architecture for multi-agent systems that must jointly decide when to communicate and how to control. In the formulation of dynamically chosen communication, agents observe local states $X_t^i$, choose communication actions before control, and either share the full global state or observe a null symbol $\phi$ depending on whether communication occurs. The key structural result is that each agent can ignore its entire private history without loss of optimality: optimal communication and control strategies depend only on the current local state and common information, yielding simplified strategies of the form $M_t^i=\bar f_t^i(X_t^i,C_t)$ and $U_t^i=\bar g_t^i(X_t^i,C_{t^+})$ (Sudhakara et al., 2021).

This reduction enables a coordinator reformulation. A fictitious coordinator observes only common information and selects prescriptions $$J(\pi)=\mathbb{E}\!\left[\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t)\right]$$0 for communication and $$J(\pi)=\mathbb{E}\!\left[\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t)\right]$$1 for control, where each prescription is a mapping from local state to a binary communication decision or a control action. The original decentralized problem is thereby recast as a single-agent POMDP over a common belief $$J(\pi)=\mathbb{E}\!\left[\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t)\right]$$2, and the optimal joint communication-control strategy is characterized by a dynamic program on that belief state (Sudhakara et al., 2021).

The resulting Bellman recursion makes the tradeoff explicit. Communication incurs an immediate cost, typically $$J(\pi)=\mathbb{E}\!\left[\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t)\right]$$3 or a state-dependent cost $$J(\pi)=\mathbb{E}\!\left[\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t)\right]$$4, while state sharing collapses the common belief to delta distributions and can reduce future control cost. The formulation also supports communication budgets and timing constraints through auxiliary state variables such as remaining communications, time since last sharing, and total shares so far; erasure channels are handled by modifying the observation probabilities while leaving the dynamic-programming structure intact (Sudhakara et al., 2021).

A related decentralized formulation appears in the Locally Interdependent Multi-Agent MDP, where agents observe exactly the joint state of their current visibility component, and the reward decomposes over visibility-connected groups because the visibility radius exceeds the dependence radius. That model yields three closed-form group-decentralized policies—Amalgam $$J(\pi)=\mathbb{E}\!\left[\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t)\right]$$5, Cutoff $$J(\pi)=\mathbb{E}\!\left[\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t)\right]$$6, and the first-step finite-horizon policy $$J(\pi)=\mathbb{E}\!\left[\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t)\right]$$7—whose value is exponentially close to the fully observable optimum as visibility grows (DeWeese et al., 2024). This suggests a broader MDP-Agent interpretation in which coordination is mediated not by full centralization, but by structured common information.

3. Structural scalability: locality, symmetry, low-rank compression, and approximate independence

A second major strand treats MDP-Agent as a scalable planner that exploits structure in large joint state-action spaces. For directed-tree multi-agent MDPs with local binary states and actions, the LLPS procedure constructs approximate local rewards from truncated root-to-node paths of length $$J(\pi)=\mathbb{E}\!\left[\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t)\right]$$8. Under a $$J(\pi)=\mathbb{E}\!\left[\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t)\right]$$9 fast-decaying property, the approximation error in average reward decays as $O(o\mid s)$0, and LLPS computes an exact maximizer of the surrogate objective in time $O(o\mid s)$1, giving a near-optimal local policy whose per-node gap is also controlled by $O(o\mid s)$2 (Qu et al., 2019).

A different scalability mechanism is symmetry factorization. Multi-Agent MDP Homomorphic Networks begin from cooperative MMDPs with global symmetry groups acting on states and actions, then decompose global transformations into local observation transforms, edge transforms, and consistent agent and edge permutations. The resulting message-passing architecture is equivariant by construction, supports decentralized execution with only local information and local communication, and improves data efficiency on symmetric tasks such as wildlife monitoring and traffic light control under CTDE with PPO (Pol et al., 2021).

Tensor compression gives a third route. CP-MDP represents the transition model as a low-rank CANDECOMP/PARAFAC decomposition of the transition tensor, replacing tabular Bellman backups by contractions through CP factors. In the paper’s complexity accounting, Cp-Mdp-VI runs in $O(o\mid s)$3 and Cp-Mdp-PI in $O(o\mid s)$4, with empirical memory reductions above $O(o\mid s)$5 for problems beyond $O(o\mid s)$6 states and successful solution of instances up to $O(o\mid s)$7 states that exceeded the memory budget of tabular baselines (Kuinchtner et al., 2021).

Approximate independence yields a fourth route. In scalable planning under $O(o\mid s)$8-transition dependence, each agent’s next-state law may depend on the rest of the system, but only up to a bounded total-variation deviation $O(o\mid s)$9 from transition independence. With monotone increasing and submodular rewards in the joint action, local MDP improvements over a TI surrogate deliver a polynomial-time algorithm whose performance on the true MMDP is bounded by an additive term linear in $b_t\in\Delta(\mathcal{S})$0 together with the local-search slack (Sahabandu et al., 2021). Here, MDP-Agent is best viewed as a local best-response planner operating against a factored surrogate while controlling approximation error by a perturbation metric.

4. Specialized instantiations across application domains

In healthcare resource allocation, MDP-Agent denotes a decomposition in which each resource consumer is an independent MDP with health and per-resource utilization state, while global coordination is imposed by an iterative auction based on expected regret. Resource access is represented through an average model $b_t\in\Delta(\mathcal{S})$1 and health progression through $b_t\in\Delta(\mathcal{S})$2, both parameterized by Dirichlet priors. The global planner allocates each resource to the patient with highest current regret, and under realistic time constraints this coordinated MDP approach outperformed heuristic baselines and a time-limited UCT planner, while total computation time for complete allocations was less than $b_t\in\Delta(\mathcal{S})$3 seconds for $b_t\in\Delta(\mathcal{S})$4 agents and scaled approximately linearly with the numbers of agents and resources (Hosseini et al., 2014).

In scientific computing, a factored Dec-MDP instantiates MDP-Agent as a learned numerical method for hyperbolic PDEs. Each agent is tied to a grid interface, observes a local stencil, and outputs reconstruction parameters such as WENO convex weights; the joint transition is the deterministic PDE integrator, and rewards measure local approximation quality against a reference trajectory. A homogeneous policy trained by Backpropagation Through Time and Space generalizes across grid resolutions, episode lengths, dimensions, and even equation types, and the learned RL-WENO agent matches standard WENO to within plotting or roundoff precision on the reported 1D Euler tests while transferring to Burgers’ equation and 2D Euler (Fu et al., 2022).

In multi-target path planning, MDP-Agent appears as a planner minimizing expected cover time of a target set. The exact formulation lifts the state space to a product MDP $b_t\in\Delta(\mathcal{S})$5 and has time complexity exponential in the number of targets, $b_t\in\Delta(\mathcal{S})$6. The proposed suboptimal Algorithm 1 instead recomputes a discounted value function over the original state space only when a new target is reached, yielding polynomial per-step complexity and exact optimality on path graphs with terminal start state, cycle graphs, and complete graphs; the multi-agent extension adds a target-partition heuristic with provable optimality for clustered target scenarios (Nawaz et al., 2022).

In HPC resource management, four MDP design variants—Dense, Compact, Sparse, and Compact + Sparse + Reduced—were compared for job scheduling with PPO and MLP approximators. The compact representation fixes input dimensionality across environments, the sparse variant invokes the agent only at decision-relevant times, and the reduced variant computes reward only from jobs in the visible window. Under this formulation, transferred compact agents outperformed specialized agents in $b_t\in\Delta(\mathcal{S})$7 of tested scenarios even without retraining, and the compact models used a fixed $b_t\in\Delta(\mathcal{S})$8 parameters versus up to $b_t\in\Delta(\mathcal{S})$9 for dense image-like variants (Cunha et al., 2021).

The term also appears in decision criteria that depart from scalar expectation. In lexicographic multi-objective planning, FLMDP solves finite-horizon MDPs with reward vectors under lexicographic preference and is used to optimize multiple quantiles of the terminal-state distribution by reducing each quantile objective to an expectation problem with indicator rewards (Li et al., 2017). This emphasizes that an MDP-Agent need not be tied to a single scalar reward when the application requires strict priority across objectives.

5. Foundation-model agents, automated construction, and explanation interfaces

Recent work applies the MDP-Agent label directly to foundation-model systems. In the sim-to-real setting, the agent loop is formalized as an MDP or POMDP whose observations are textual or multimodal context, whose actions are schema-aware tool or API calls, and whose transition and reward kernels incorporate tool outcomes, latency, and cost. The central proposal is to decompose sim-to-real discrepancy into observation, action, transition, and reward gaps—$h_t$0—and to adapt classical methods such as domain randomization, robust optimization, system identification, and constraint-aware decoding to multilingual and tool-using FM agents (Liu et al., 5 Jun 2026).

Agent$h_t$1 pushes this one step further by making the MDP-Agent itself a generated artifact. A Generator Agent first normalizes an environment into an MDP with wrappers $h_t$2, $h_t$3, optional $h_t$4, and a chosen discount factor, then materializes a Target Agent together with algorithm selection, network design, hyperparameters, and configuration. This decomposition into MDP modeling and algorithmic optimization is implemented on the Model Context Protocol and evaluated on MuJoCo, MetaDrive, MPE, and SMAC, where the system reports up to $h_t$5 performance improvement and stage-wise gains attributable to both MDP modeling and subsequent optimization (Wei et al., 16 Sep 2025).

Learning-theoretic treatments focus on the optimization backend rather than the wrapper layer. Offline linear MDP agents use pessimism with uncertainty decomposition via a reference function to obtain nearly minimax-optimal guarantees under linear function approximation, and reward-free agents use optimistic exploration bonuses with kernel or neural approximators to collect offline data that later supports planning for arbitrary extrinsic rewards with $h_t$6 sample complexity (Xiong et al., 2022, Qiu et al., 2021). These works broaden the meaning of MDP-Agent from a software architecture to a statistically characterized planning-and-learning object.

Explanation interfaces provide another extension. In multi-objective navigation, tradeoff-focused contrastive explanations compare the chosen policy to Pareto-optimal foil policies and decompose the scalarized preference difference into objective-wise gains and losses grounded in domain concepts such as obstacle density and private-area traversal. In a human-subjects experiment, this explanation mechanism improved correctness with odds ratio approximately $h_t$7, increased confidence by $h_t$8 on a $h_t$9–$s_t=\phi(h_t)$0 scale, and improved Reliable Confidence Score by $s_t=\phi(h_t)$1 (Sukkerd et al., 2020). Here the MDP-Agent is not only an optimizer but also an explainer of its own tradeoff structure.

6. Limitations, misconceptions, and open problems

A common misconception is that MDP-Agent implies model-free learning by default. Much of the literature is explicitly model-based: the common-information communication framework assumes known dynamics and known communication cost, LLPS assumes known local kernels and rewards on a directed tree, CP-MDP assumes a transition tensor to compress, and the healthcare coordination framework depends on explicit transition and reward models (Sudhakara et al., 2021, Qu et al., 2019, Kuinchtner et al., 2021, Hosseini et al., 2014). Even when learning enters the picture, the MDP abstraction is usually specified before optimization begins.

Another misconception is that MDP-Agent necessarily requires centralized execution. Distributed execution using only local information is explicit in homomorphic networks, in LI-MAMDP visibility-group policies, and in the PDE Dec-MDP where each agent acts on a local stencil with a shared homogeneous policy (Pol et al., 2021, DeWeese et al., 2024, Fu et al., 2022). What varies is not whether execution is centralized, but where the state abstraction, coupling assumptions, and optimization are placed.

The recurring technical limitation is scalability. Exact dynamic programming is feasible only for small spaces in common-belief communication control; prescription spaces are exponential in local state size; tree-based locality methods degrade as $s_t=\phi(h_t)$2 grows; large or continuous symmetry groups complicate equivariant architectures; and tensor compression helps only when the transition model is sufficiently low rank (Sudhakara et al., 2021, Qu et al., 2019, Pol et al., 2021, Kuinchtner et al., 2021). The literature repeatedly addresses this by factorization, approximation, message passing, or reduced decision epochs rather than by removing the underlying combinatorial difficulty.

Open problems are correspondingly diverse. For foundation-model agents, unresolved issues include quantifying high-dimensional observation gaps, defining tractable uncertainty sets over large tool ecosystems, handling reward misspecification, reconciling POMDP belief tracking with LLM memory, and standardizing robustness benchmarks (Liu et al., 5 Jun 2026). For structured multi-agent systems, open directions include asynchronous communication, richer noisy-channel models, model-free RL with locality guarantees on nontrivial graphs, and principled treatment of approximate rather than exact symmetries (Sudhakara et al., 2021, Qu et al., 2019, Pol et al., 2021). A plausible implication is that future uses of “MDP-Agent” will continue to be plural: the term will likely remain a shared architectural vocabulary for explicit sequential decision models rather than converging to a single universally adopted formalism.