Multi-Agent Semi-Markov Decision Process (MSMDP)

• MSMDP is a formal framework for modeling trajectory-level decision-making among agents using temporally extended macro-actions.
• It integrates hierarchical reinforcement learning with mean-field actor-critic methods to enable scalable strategy selection in complex environments like robotic soccer.
• The framework employs high-level and low-level policies to optimize team rewards through spatial awareness and temporal abstraction, promoting advanced cooperation.

A Multi-Agent Semi-Markov Decision Process (MSMDP) is a formal framework for modeling trajectory-level decision-making among multiple agents where macro-actions (options) are temporally extended. In the hierarchical reinforcement learning (HRL) architecture proposed by Taourirte & Mia (2024), the MSMDP captures high-level strategy selection for teams of homogeneous agents, specifically applied to robotic soccer in adversarial, real-time, multi-agent virtual environments (Taourirte et al., 2 Dec 2025).

1. Mathematical Formulation of MSMDP

The MSMDP is structured as a tuple $(S, \{O^n\}_{n=1}^N, P^{(0)}, R^{(0)}, \gamma)$, defining the joint state, options, transition kernel, reward function, and discount factor. At each high-level decision index $i$, the global state $s_i^T$ aggregates agents' and ball attributes:

• $P \in \mathbb{R}^{N \times 2}$: Teammates' $(x, y)$ positions
• $P_{\text{goal}} \in \mathbb{R}^2$: Ball position
• $if_c \in \{0,1\}^N$: Indicator vector for ball control per agent
• $P^{oppo} \in \mathbb{R}^{N \times 2}$: Opponents’ positions

Each agent $n$ selects a high-level option from $O^n = \{o^n_k \mid k=1...8\}$, corresponding to eight coarse trajectory directions over the next $i$0 low-level steps. Each option $i$1 is defined by a triple $i$2:

• Initiation set $i$3 (all states permit any directional option)
• Intra-option policy $i$4, implemented via PPO at the low level
• Termination $i$5 (fixed $i$6 step duration)

The semi-Markov kernel is $i$7 with $i$8 determined after $i$9 steps of low-level interaction, reflecting the options' temporal extension.

2. High-Level Reward Design

The high-level reward $s_i^T$0 aggregates instantaneous low-level team rewards $s_i^T$1 accrued over $s_i^T$2 steps for each option:

$s_i^T$3

Where:

• $s_i^T$4
• $s_i^T$5, $s_i^T$6 being robot-to-ball distance
• $s_i^T$7, $s_i^T$8 denoting non-holder’s distance to opponent goal, $s_i^T$9 the neighborhood of agent $P \in \mathbb{R}^{N \times 2}$0

Hence, for an option $P \in \mathbb{R}^{N \times 2}$1 taken at state $P \in \mathbb{R}^{N \times 2}$2 and ending at $P \in \mathbb{R}^{N \times 2}$3,

$P \in \mathbb{R}^{N \times 2}$4

This design incentivizes scoring, ball proximity, and control, with explicit spatial awareness embedded.

3. Value Functions and Hierarchical Policy Objective

Let $P \in \mathbb{R}^{N \times 2}$5 denote the high-level stochastic policy over options. The HRL scheme aims to maximize the expected discounted sum of rewards every $P \in \mathbb{R}^{N \times 2}$6 high-level steps:

$P \in \mathbb{R}^{N \times 2}$7

This is equivalently expressed in low-level timesteps:

$P \in \mathbb{R}^{N \times 2}$8

Bellman equations for SMDP option and state value functions are:

$P \in \mathbb{R}^{N \times 2}$9

$(x, y)$0

These express the modular temporal abstraction underpinning high-level policy learning.

4. Mean-Field Actor-Critic Integration for Scalability

To scale the architecture for large $(x, y)$1, agent $(x, y)$2 approximates the collective impact of peers via the mean option $(x, y)$3. This simplifies multi-agent interactions to agent-vs-population averages and permits stable learning as shown in robotic soccer experiments (Taourirte et al., 2 Dec 2025).

The high-level mean-field Q-update:

$(x, y)$4

where

$(x, y)$5

The mean-field policy gradient update:

$(x, y)$6

Concurrently, PPO optimizes each intra-option policy $(x, y)$7 at the low level.

5. Learning and Execution Procedure

Training and execution of the MSMDP follows a hierarchical protocol. Each episode begins with environment reset and state observation. For each high-level time index $(x, y)$8, all agents sample options conditioned on both state and neighborhood mean. These options execute for $(x, y)$9 steps, during which low-level actions are decided via PPO, and cumulative team reward is collected.

After execution, agents update high-level value and policy parameters:

• High-level TD error:

$P_{\text{goal}} \in \mathbb{R}^2$0

• Critic gradient:

$P_{\text{goal}} \in \mathbb{R}^2$1

• Actor gradient:

$P_{\text{goal}} \in \mathbb{R}^2$2

• Mean-field Q-table (or network) update:

$P_{\text{goal}} \in \mathbb{R}^2$3

The algorithm cycles through episodes, yielding performance metrics superior to non-hierarchical MARL and non-mean-field approaches (e.g., 5.93 avg. goals, 89.1% ball control, 92.3% passing accuracy in 4v4 Webots simulations).

6. Significance in Multi-Agent Reinforcement Learning

The MSMDP architecture directly addresses the curse of dimensionality in multi-agent RL, supports multi-granular temporal abstraction (macro-strategy vs. micro-execution), and is validated in adversarial stochastic domains. The mean-field actor-critic module provides a scalable trajectory planning solution, enabling robust cooperation and strategic behavior amongst many agents. A plausible implication is that this MSMDP-HRL paradigm generalizes to other domains characterized by frequent interaction, real-time constraints, and large homogeneous cohorts.

7. Summary Table: MSMDP Components (as formulated by Taourirte & Mia, 2024)

Component	Symbol/Structure	Description
Joint State Space	$P_{\text{goal}} \in \mathbb{R}^2$4	Concatenated agent, ball, and opponent data
Option Set	$P_{\text{goal}} \in \mathbb{R}^2$5	Eight spatial directions per agent
Option Definition	$P_{\text{goal}} \in \mathbb{R}^2$6	Initiation, intra-option PPO, fixed duration
High-Level Reward	$P_{\text{goal}} \in \mathbb{R}^2$7	Sum of low-level team rewards over $P_{\text{goal}} \in \mathbb{R}^2$8 steps
Mean-Field Update	$P_{\text{goal}} \in \mathbb{R}^2$9	Aggregated population impact in TD learning

All components are instantiated to facilitate hierarchical, mean-field multi-agent RL as validated in robotic soccer simulation research (Taourirte et al., 2 Dec 2025).