Temporal Macro-action IGM in Asynchronous MARL

• The paper introduces a temporal extension to IGM, enabling decentralized decision-making by factoring in macro-action execution durations.
• The framework enhances credit assignment through temporal positional encoding and a CTDE architecture integrated with macro-action segmented replay.
• Empirical results on benchmarks like BoxPushing and Overcooked demonstrate improved convergence, stability, and robust adaptability over traditional methods.

Searching arXiv for the cited papers and closely related work on To-Mac-IGM and macro-action value factorization. Temporal Macro-action based IGM (To-Mac-IGM) is a consistency condition for cooperative asynchronous multi-agent reinforcement learning in which agents execute temporally extended macro-actions rather than synchronized one-step actions. It is formalized in the MacDec-POMDP setting by "ToMacVF : Temporal Macro-action Value Factorization for Asynchronous Multi-Agent Reinforcement Learning" (Zhang et al., 14 Jul 2025). In that formulation, the joint value function depends not only on joint macro-actions and macro-histories, but also on the execution time of each ongoing macro-action, so that centralized greedy selection over joint temporal macro-actions is consistent with decentralized greedy selection over individual temporal macro-action values for agents whose current macro-actions terminate, while other agents continue their ongoing macro-actions unchanged. The concept is motivated by the mismatch between synchronous value factorization and asynchronous macro-action execution, and it underpins a CTDE architecture for fine-grained temporal credit assignment in asynchronous MARL (Zhang et al., 14 Jul 2025).

1. Problem setting and conceptual lineage

The immediate formal context of To-Mac-IGM is a cooperative Dec-POMDP

$$\langle \mathcal{I}, \mathcal{S}, \mathcal{U}, T_s, R, \mathcal{O}, T_\mathcal{O}, \gamma \rangle,$$

together with its MacDec-POMDP extension for temporally extended macro-actions. In this extension, each agent $i$ has a macro-action set $M^i$, macro-observation space $\hat O^i$, and macro-action histories $\hat h^i$. A macro-action is defined as

$\langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$

where $I_{m^i}$ is the initiation set, $\pi_{m^i}$ is a fixed low-level policy, and $\beta_{m^i}$ is a stochastic termination condition. The high level learns only macro-action policies $\pi_{M^i}(m^i\mid \hat h^i)$; the primitive controllers $i$0 are pre-trained and not learned in this work (Zhang et al., 14 Jul 2025).

The need for a temporal macro-action analogue of IGM arises because existing asynchronous methods based on MacDec-POMDP typically collect data only at macro-action termination timesteps and then apply synchronous value-factorization algorithms unchanged. The ToMacVF paper identifies three resulting pathologies: duplicated macro-actions, inconsistent macro-observations, and incorrect cumulative rewards leading to biased credit assignment. These failures are traced to endpoint-only replay and to ignoring how long a macro-action has been running and what happened during its execution (Zhang et al., 14 Jul 2025).

This problem sits within a broader macro-action literature. In online POMDP planning, temporally extended actions reduce planning depth linearly and planning complexity exponentially, and learned macro-actions can improve long-horizon planning under partial observability (Lee et al., 2020). In hierarchical MDPs, macro-actions have also been treated as local policies over regions, with abstract MDPs defined on peripheral states (Hauskrecht et al., 2013). For asynchronous cooperative control, event-driven MacDec-POMDPs emphasize that stochastic macro-action durations induce asynchronous decision points and require temporally aware credit assignment (Menda et al., 2017). In temporal planning, sequential temporal macro-actions have been defined so that the encapsulated sequence of durative actions is always executable, even under concurrency constraints and shared resources (Bortoli et al., 2023). To-Mac-IGM specializes this general macro-action perspective to CTDE value factorization in asynchronous MARL (Zhang et al., 14 Jul 2025).

2. Formal definition of To-Mac-IGM

To-Mac-IGM extends the classical IGM condition from synchronized primitive actions to asynchronous macro-actions with explicit execution time. The required objects are a joint macro-history $i$1, a set of agents whose current macro-actions terminate at the current decision point,

$i$2

an execution timestep space

$i$3

per-agent elapsed execution times $i$4, the joint elapsed-time vector $i$5, and the set $i$6 of macro-actions that are still ongoing (Zhang et al., 14 Jul 2025).

The central joint value is

$i$7

and the individual temporal macro-action values are

$i$8

To-Mac-IGM is satisfied when the greedy joint macro-action decision decomposes as follows: $i$9

The operational interpretation is precise. Only agents whose current macro-actions terminate choose a new macro-action, and they do so by maximizing their local temporal macro-action value at execution time $M^i$0. Agents whose macro-actions are still running do not choose anew; their current macro-actions remain fixed, and their elapsed execution times continue to evolve. This is the asynchronous generalization of classical IGM: decentralized greedy execution is preserved, but only for the subset of agents at a decision boundary, while the value functions remain conditioned on the temporal state of all ongoing macro-actions (Zhang et al., 14 Jul 2025).

This definition encodes duration, start and stop structure, and asynchrony directly in the value factorization interface. The temporal argument $M^i$1 lets the same macro-action have different values at different stages of execution. The conditioning on $M^i$2 ensures that a new decision by one agent is evaluated relative to the still-running macro-actions of others. The paper treats this as the principled consistency requirement for asynchronous value factorization with temporally extended actions (Zhang et al., 14 Jul 2025).

3. Relation to classical IGM, Mac-IGM, and advantage-based variants

Classical synchronous value factorization uses the IGM condition

$M^i$3

which presumes that every agent selects an action at every step. Mac-IGM extends the action space from primitive actions to macro-actions, but still omits explicit temporal progress. To-Mac-IGM enlarges the domain further by making execution time part of both joint and individual value functions (Zhang et al., 14 Jul 2025).

The corresponding function classes satisfy the strict inclusions

$M^i$4

The argument is structural. Macro-action sets satisfy $M^i$5, so macro-action factorization has a larger domain than primitive-action factorization, and adding $M^i$6 enlarges the domain again. Standard synchronous IGM is therefore a special case of To-Mac-IGM obtained when macro-actions collapse to one-step primitives and $M^i$7 always (Zhang et al., 14 Jul 2025).

The paper also defines an advantage-based equivalent, To-MacAdv-IGM. With

$M^i$8

and

$M^i$9

the same asynchronous greedy-consistency condition can be written purely in terms of advantages because the state-value terms cancel under $\hat O^i$0. The paper concludes that

$\hat O^i$1

This gives To-Mac-IGM the most general representational class among these variants (Zhang et al., 14 Jul 2025).

A nearby theoretical issue is the critique that IGM decomposition is lossy under insufficient observation, and that the resulting decomposition error accumulates in hypernetwork-based methods when Bellman updates are performed through local utilities (Hong et al., 2022). The ToMacVF paper does not frame To-Mac-IGM as a remedy for that lossiness. A plausible implication is narrower: asynchronous temporal execution introduces an additional representational mismatch that is already outside the function classes of synchronous IGM and Mac-IGM, and To-Mac-IGM addresses that temporal mismatch explicitly (Hong et al., 2022).

4. Mac-SJERT and temporal replay representation

To make To-Mac-IGM trainable, ToMacVF replaces endpoint-only replay with the Macro-action Segmented Joint Experience Replay Trajectory (Mac-SJERT). For each agent $\hat O^i$2, the buffer is a pair $\hat O^i$3, where $\hat O^i$4 is a finite set of segmented joint trajectories and $\hat O^i$5 orders them by macro-action termination time. Each $\hat O^i$6 corresponds to one full execution of a macro-action $\hat O^i$7 and stores, at every timestep during that execution,

$\hat O^i$8

where $\hat O^i$9 is global state, $\hat h^i$0 is the joint macro-observation with temporal encoding, $\hat h^i$1 is the joint macro-action with temporal encoding, and $\hat h^i$2 is the immediate joint reward. At the termination timestep of $\hat h^i$3, the segment additionally stores the cumulative discounted reward

$\hat h^i$4

The construction is per-agent and segmented by that agent’s macro-action duration, which removes the reward-reset pathology of Mac-JERT (Zhang et al., 14 Jul 2025).

Temporal positional encoding distinguishes the same macro-action or macro-observation at different global times and at different execution-progress stages. The paper uses sinusoidal encodings $\hat h^i$5 and $\hat h^i$6, for example

$\hat h^i$7

and, for finer execution progress,

$\hat h^i$8

This lets the value function distinguish a macro-action at its start from the same macro-action after several timesteps of execution (Zhang et al., 14 Jul 2025).

Mac-SJERT induces two transition types. A micro-transition is a one-timestep sample

$\hat h^i$9

and a macro-transition for agent $\langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$0 is

$\langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$1

where the state and joint macro-observation/action at the start of $\langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$2 are paired with the cumulative reward over that macro-action and the state and joint macro-observation at termination. Both use temporal positional encoding (Zhang et al., 14 Jul 2025).

The role of Mac-SJERT is not incidental. To-Mac-IGM requires value functions $\langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$3 and $\langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$4 that depend on execution stage, accumulated reward, and the joint configuration of starting and ongoing macro-actions. Endpoint-only replay cannot provide that information. Mac-SJERT provides complete sequences, per-agent segmentation, and temporally encoded joint context, which together support the learning of time-conditioned values and fine-grained temporal credit assignment (Zhang et al., 14 Jul 2025).

5. ToMacVF architecture and learning dynamics

ToMacVF is the neural architecture that realizes To-Mac-IGM under CTDE. It comprises temporal agent networks $\langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$5, an Agent-oriented Temporal Parameter Generation module (ATPG), and a mixing network $\langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$6 (Zhang et al., 14 Jul 2025).

Each temporal agent network receives the temporally encoded local macro-observation $\langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$7, the temporally encoded previous macro-action $\langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$8, and an RNN hidden state encoding the local macro-history $\langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$9. It outputs the fine-grained temporal individual macro-action value

$I_{m^i}$0

which is the decentralized quantity used at execution time (Zhang et al., 14 Jul 2025).

ATPG generates time- and agent-dependent non-negative mixing parameters. It first applies temporal self-attention over historical joint trajectories $I_{m^i}$1 with a causal mask $I_{m^i}$2: $I_{m^i}$3 with

$I_{m^i}$4

It then applies agent-oriented self-attention,

$I_{m^i}$5

with attention weights

$I_{m^i}$6

and final encoding

$I_{m^i}$7

From $I_{m^i}$8 and $I_{m^i}$9, ATPG generates the non-negative weights and biases of the mixing network. The resulting gradient contribution satisfies

$\pi_{m^i}$0

so temporal and inter-agent dependence enters the factorization while preserving monotonicity (Zhang et al., 14 Jul 2025).

The joint value takes the conceptual form

$\pi_{m^i}$1

Non-negativity of $\pi_{m^i}$2 provides the sufficiency condition for monotonic factorization, exactly as monotonicity does for QMIX in the synchronous case, but now over temporal macro-action values (Zhang et al., 14 Jul 2025).

Learning uses two TD objectives. The micro-TD loss over one-timestep transitions is

$\pi_{m^i}$3

and the macro-TD loss over a complete macro-action of agent $\pi_{m^i}$4 is

$\pi_{m^i}$5

The first objective assigns per-timestep temporal credit; the second refines long-horizon macro-action credit at macro termination. During execution, the mixing network and ATPG are discarded, and each agent acts greedily with its local temporal agent network, selecting a new macro-action only when its termination condition is met (Zhang et al., 14 Jul 2025).

6. Empirical behavior, significance, and limitations

The empirical evaluation uses three asynchronous MacDec-POMDP benchmarks: BoxPushing, Warehouse Tool Delivery (WTD), and Overcooked. Baselines include Mac-Dec-MADDRQN, Mac-Cen-DDRQN, Mac-CAC, Mac-IAICC, Mac-NIACC, and AVF. Across Figures 3 to 5, ToMacVF is reported to show consistently strong performance across all tasks, with higher returns, faster and more stable convergence, and strong adaptability and robustness to different asynchronous patterns (Zhang et al., 14 Jul 2025).

The ablations isolate the roles of Mac-SJERT and ATPG. Removing Mac-SJERT and reverting to Mac-JERT significantly degrades performance, which the paper interprets as evidence that fine-grained temporally segmented replay is necessary for To-Mac-IGM. Removing ATPG also harms performance, indicating that temporally aware parameter generation and inter-agent attention are important for realizing To-Mac-IGM in practice (Zhang et al., 14 Jul 2025).

The architecture is modular and can integrate different value-factorization back ends such as QMIX, QTRAN, and QPLEX. In the reported experiments, however, ToMacVF$\pi_{m^i}$6 is consistently strong and robust, whereas ToMacVF$\pi_{m^i}$7 and ToMacVF$\pi_{m^i}$8 show poor performance in some scenarios, high variance, and unstable training. The paper therefore treats QMIX’s monotonic mixing as especially well aligned with the temporal credit assignment strategy used in ToMacVF (Zhang et al., 14 Jul 2025).

The broader implication is that conventional value factorization is inadequate for asynchronous, temporally abstracted MARL if applied naively. To-Mac-IGM is presented as the consistency principle needed to model macro-action durations and asynchronous termination while preserving decentralized greedy execution under CTDE. The strict inclusion

$\pi_{m^i}$9

is used to argue that Temporal Macro-action based IGM forms a richer representational class than synchronous IGM and non-temporal Mac-IGM (Zhang et al., 14 Jul 2025).

The formulation also has clear limitations. Macro-actions are predefined and low-level controllers are fixed; only high-level macro-action policies are learned. Centralized training requires access to global state and joint macro-history. Although ATPG improves coordination, computational cost grows with the number of agents and trajectory length. The instability observed with QTRAN and QPLEX under the ToMacVF interface is left unresolved. The paper lists future directions including more scalable temporal encodings, better integration with other value factorization methods, joint optimization of macro-action definitions and value factorization, and application to heterogeneous systems with more varied macro-action duration distributions (Zhang et al., 14 Jul 2025).

Within the broader macro-action literature, these limitations are consistent with longstanding trade-offs. Macro-actions improve tractability by compressing decision depth or state-space structure, but their semantics, parameterization, and planner dependence remain central design issues (Lee et al., 2020, Hauskrecht et al., 2013). For temporal and asynchronous multi-agent settings, To-Mac-IGM is the specific formal device that makes those trade-offs explicit at the level of value factorization: it treats temporal progress as part of the factorized value representation and constrains decentralized greediness to the subset of agents that are actually at macro-action decision boundaries (Zhang et al., 14 Jul 2025).