Deterministic Dec-POMDPs & IDPP

• Det-Dec-POMDPs are formal models for multi-agent planning with deterministic transitions and observations, where all uncertainty arises from the initial state.
• The IDPP algorithm reduces joint policy search to sequential single-agent planning, iteratively computing best responses until achieving a local Nash equilibrium.
• Empirical benchmarks validate IDPP's scalable performance in complex domains like multi-robot navigation and structured resource collection.

A Deterministic Decentralized Partially Observable Markov Decision Process (Det-Dec-POMDP) is a formal model for sequential decision-making in multi-agent systems characterized by a shared environment, local partial observability, and deterministic transitions and observations. In Det-Dec-POMDPs, all uncertainty arises solely from the initial state distribution. This class of models is well-suited for high-level multi-agent planning domains, such as multi-robot navigation and structured resource collection, where environmental transitions and percepts are deterministic given the underlying state and agents’ joint actions (You et al., 29 Aug 2025).

1. Formal Model

A finite-horizon Det-Dec-POMDP is defined by the tuple

$$D = \langle\,\mathcal I,\;\mathcal S,\;\{\mathcal A_i\}_{i\in\mathcal I},\;\mathcal T,\;\mathcal Z,\;\mathcal R,\;h,\;b_0\,\rangle$$

with components:

• $\mathcal I = \{1,\dots,N\}$: finite set of agents.
• $\mathcal S$: finite set of world states.
• $\mathcal A_i$: finite action set of agent $i$; $\mathcal A = \bigtimes_{i\in\mathcal I} \mathcal A_i$ is the joint action space.
• $\mathcal Z_i$: finite observation set of agent $i$; $\mathcal Z = \bigtimes_{i\in\mathcal I} \mathcal Z_i$ is the joint observation space.
• $$\mathcal T: \mathcal S \times \mathcal A \to \mathcal S$$: deterministic transition function.
• $\mathcal I = \{1,\dots,N\}$0: deterministic observation function.
• $\mathcal I = \{1,\dots,N\}$1: shared immediate reward function.
• $\mathcal I = \{1,\dots,N\}$2: finite decision horizon.
• $\mathcal I = \{1,\dots,N\}$3: initial belief over world states.

Each agent $\mathcal I = \{1,\dots,N\}$4 follows a local policy $\mathcal I = \{1,\dots,N\}$5, mapping its own action-observation trajectory to actions. Given deterministic $\mathcal I = \{1,\dots,N\}$6 and $\mathcal I = \{1,\dots,N\}$7, all uncertainty in system evolution is driven by the initial belief $\mathcal I = \{1,\dots,N\}$8. As execution proceeds, agents’ local histories eliminate world states inconsistent with their observations and actions.

2. Relationship to General Dec-POMDPs and Computational Complexity

General (stochastic) Dec-POMDPs extend this framework with transition and observation functions $\mathcal I = \{1,\dots,N\}$9 and $\mathcal S$0. In the deterministic subclass, both functions reduce to mappings with outputs in $\mathcal S$1, substantially simplifying simulation and planning. Despite this reduction in complexity, the theoretical challenge remains severe: finding an optimal finite-horizon policy is NEXP-complete, even in the deterministic setting. This computational hardness arises because the space of joint policies is double-exponential in the number of agents $\mathcal S$2 and the horizon $\mathcal S$3 (You et al., 29 Aug 2025).

3. The Iterative Deterministic POMDP Planning (IDPP) Algorithm

The Iterative Deterministic POMDP Planning (IDPP) algorithm is a scalable technique for computing Nash equilibrium policies in Det-Dec-POMDPs. IDPP is a specialized variant of the Joint Equilibrium Search for Policies (JESP) framework, taking advantage of deterministic structure to reduce each best-response computation to a sequence of single-agent deterministic POMDPs.

IDPP operates as follows:

• Initialization: A centralized MDP policy $\mathcal S$4 ignoring observability constraints is computed. For each agent $\mathcal S$5, the other agents are fixed to $\mathcal S$6 to create an initial deterministic POMDP, and $\mathcal S$7 is derived by solving this instance.
• Best Response Iteration: For each agent $\mathcal S$8 in turn, the remaining agents' policies $\mathcal S$9 are fixed. The best-response Det-POMDP $\mathcal A_i$0 is constructed and optimally solved for $\mathcal A_i$1.
• Repeat: The process repeats until convergence to a local Nash equilibrium, where no agent can improve its expected return by unilateral deviation. Given the finite horizon and determinism, convergence occurs after finitely many iterations.

A summary of IDPP’s conceptual workflow is given in the table:

Step	Purpose	Mechanism/Tool
Centralized policy init	Heuristic policy bootstrap	Centralized MDP planning
Best response construction	Reduce to single-agent Det-POMDP	Fix other agents' policies
Solve Det-POMDP	Compute best response	Standard Det-POMDP solver
Iterative updating	Achieve Nash equilibrium	JESP iteration

Finite convergence is guaranteed under deterministic transitions, observations, and state/action spaces (You et al., 29 Aug 2025).

4. Empirical Results and Benchmarks

IDPP demonstrates strong scalability and performance on large Det-Dec-POMDP benchmarks not tractable for prior solvers:

Multi-Agent Canadian Traveler Problem (MACTP):

• Grid of size $\mathcal A_i$2, $\mathcal A_i$3 agents, $\mathcal A_i$4 stochastic edges.
• State space size $\mathcal A_i$5.
• Example: MACTP$\mathcal A_i$6 ($\mathcal A_i$7), IDPP achieved average return $\mathcal A_i$8 in $\mathcal A_i$9 s, outperforming MCJESP and proving infeasible for InfJESP and MAA*.

Collecting Task:

• $i$0 grid, $i$1 boxes/goals, $i$2 agents.
• State space $i$3.
• Example: Collecting$i$4 ($i$5), IDPP mean return $i$6 in $i$7 s, outperforming MCJESP and deep reinforcement learning baselines.

Memory consumption under IDPP remains stable (e.g., $i$8200 MB) where alternative methods exhaust resources.

5. Illustrative Example

A canonical two-agent navigation instance elucidates the model’s dynamics. Consider two robots on a $i$9 grid separated by an initially unknown blocked door. The state is specified as $\mathcal A = \bigtimes_{i\in\mathcal I} \mathcal A_i$0, where $\mathcal A = \bigtimes_{i\in\mathcal I} \mathcal A_i$1 encodes the blockage. Each robot's actions consist of cardinal moves and wait; observations reveal local blockages upon adjacency; transitions are deterministic, and the initial belief is uniform over $\mathcal A = \bigtimes_{i\in\mathcal I} \mathcal A_i$2. One IDPP iteration for agent 1 proceeds by fixing agent 2’s policy, constructing a Det-POMDP on $\mathcal A = \bigtimes_{i\in\mathcal I} \mathcal A_i$3, solving for $\mathcal A = \bigtimes_{i\in\mathcal I} \mathcal A_i$4, then updating agent 2 accordingly (You et al., 29 Aug 2025).

6. Extensions and Open Challenges

• Partial Determinism: The Quasi-Det-Dec-POMDP relaxes deterministic observations to allow stochasticity while transitions remain deterministic; IDPP can be adapted via stochastic Det-POMDP solvers.
• Online Planning: Incorporating IDPP within receding-horizon or Monte Carlo Tree Search enables online deployment.
• Parallelization: Simultaneous best-response updates across agents present a potential efficiency gain.
• Theoretical Refinement: Opportunities exist to refine complexity and convergence analyses, especially under horizon and state-space constraints (You et al., 29 Aug 2025).

7. Significance and Research Impact

Det-Dec-POMDPs provide a mathematically rigorous formalism for multi-agent planning tasks where all process uncertainty is rooted in initial conditions. IDPP supplies a tractable and scalable algorithmic strategy, reducing joint policy search to a series of deterministic single-agent planning problems and attaining Nash equilibria for large-scale instances. These advances enable practical solution of decentralized planning domains previously intractable due to computational and memory bottlenecks inherent in general Dec-POMDP solvers (You et al., 29 Aug 2025).