Temporal Macro-action Value Factorization

• The paper introduces ToMacVF, a framework that resolves bias in asynchronous macro-action credit assignment through micro and macro temporal updates.
• It features Mac-SJERT, a decentralized replay buffer storing complete trajectories with temporal encodings to accurately capture macro-action execution.
• It generalizes the IGM condition into To-Mac-IGM, ensuring consistency between centralized training and decentralized execution in cooperative MARL.

Searching arXiv for the ToMacVF paper and closely related macro-action/value-factorization work. Temporal Macro-action Value Factorization (ToMacVF) is a value-factorization framework for cooperative asynchronous multi-agent reinforcement learning in Macro-action Decentralized Partially Observable Markov Decision Processes (MacDec-POMDPs). It was introduced to address a specific failure mode of prior asynchronous MARL pipelines: training buffers constructed only at macro-action endpoints provide an incomplete and inaccurate representation of macro-action execution, and therefore induce unsuitable credit assignments. ToMacVF instead combines fine-grained temporal credit assignment, a centralized training buffer called Macro-action Segmented Joint Experience Replay Trajectory (Mac-SJERT), a temporal generalization of the Individual-Global-Max consistency condition called Temporal Macro-action based IGM (To-Mac-IGM), and a modular CTDE architecture that can incorporate existing value factorization methods (Zhang et al., 14 Jul 2025). In the broader literature, it sits at the intersection of two earlier lines of work: value factorization for cooperative MARL, which formalized the role of factorized joint value classes in CTDE (Wang et al., 2020), and macro-action MARL in MacDec-POMDPs, which developed SMDP-style asynchronous value definitions and learning algorithms over temporally extended options (Xiao, 2022).

1. Asynchronous macro-actions and the problem ToMacVF addresses

ToMacVF is formulated for cooperative tasks in which agents choose temporally extended macro-actions with stochastic durations and asynchronous start and termination times. In the underlying Dec-POMDP, the objective is to maximize

$$\mathcal{J}(\boldsymbol{\pi}) = \mathbb{E}_{s_0,\,(s_t,\boldsymbol{u}_t)} \Big[ \sum_{t=0}^\infty \gamma^t R(s_t,\boldsymbol{u}_t)\Big].$$

MacDec-POMDPs extend this setting by introducing a joint macro-action space and macro-histories. Each macro-action is defined as

$$m^i = \langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$$

where $I_{m^i}$ is the initiation set, $\pi_{m^i}$ is a fixed low-level policy, and $\beta_{m^i}$ is the termination condition. During high-level learning, only the macro-action policy $\pi_{M^i}(m^i \mid \hat{h}^i)$ is trained; the low-level $\pi_{m^i}$ is fixed (Zhang et al., 14 Jul 2025).

The asynchronous aspect is essential rather than incidental. At a macro-termination event, only a subset of agents selects new macro-actions; the remaining agents continue executing their current ones. Earlier MacDec-POMDP work already treated this as a central modeling issue and formulated asynchronous Bellman equations and learning schemes over macro-observation-action histories, but without the ToMacVF terminology (Xiao, 2022).

The immediate motivation for ToMacVF is a critique of existing asynchronous MARL methods based on MacDec-POMDPs, including Mac-Dec-MADDRQN, Mac-CAC, and AVF. These methods use a buffer called Mac-JERT, which stores joint transitions only at macro-action termination times and then applies synchronous value factorization methods such as QMIX or VDN to the resulting compressed trajectory. The ToMacVF paper identifies three concrete problems with that practice: duplicated macro-actions, inconsistent macro-observations, and incorrect cumulative rewards. A macro-action executed for multiple steps is represented identically at all times; macro-observations become outdated between endpoints; and cumulative rewards are reset when any agent terminates, which underestimates other agents’ ongoing macro-actions (Zhang et al., 14 Jul 2025).

These objections are not merely representational. They imply that prior endpoint-only compression yields biased trajectories, incompletely represents macro-action execution, and makes value estimation and credit assignment coarse and often wrong. A common simplification is therefore to treat asynchronous macro-actions as if they were synchronized one-step actions; ToMacVF is explicitly designed to reject that simplification.

2. Temporal value factorization

The core idea of ToMacVF is to extend synchronous value factorization into an explicitly temporal macro-action setting. In synchronous MARL, a standard factorized form is

$$Q_{total}(s_t,\boldsymbol{u}_t) = \mathcal{F}_{mix}(Q_1(h_t^1,u_t^1),\dots,Q_n(h_t^n,u_t^n),s_t).$$

ToMacVF generalizes this by making each agent’s local utility depend on macro-history, macro-action, and current execution time $t_{m^i}$, while the joint value function combines these temporally extended utilities together with temporal context (Zhang et al., 14 Jul 2025).

This yields a different notion of credit assignment from endpoint-based macro-action methods. Fine-grained temporal credit assignment means that ToMacVF records transitions at every environment timestep, computes micro-TD updates at each step, and also performs macro-level TD updates at macro boundaries. Rewards accrued during the life of a macro-action are therefore observed repeatedly during execution, rather than only as a terminal lump sum. The model can learn how value evolves as a function of macro-action progress, not merely which macro-action was initiated (Zhang et al., 14 Jul 2025).

This design is naturally related to the general role of factorization in cooperative MARL. Earlier theoretical work showed that value factorization is the structural constraint that makes joint Q-learning scalable and compatible with decentralized execution, but that its properties depend strongly on the geometry of the chosen factorized value class and on the data distribution used for Bellman regression (Wang et al., 2020). ToMacVF inherits that general perspective while shifting the factorization domain from synchronized primitive actions to asynchronous temporally extended macro-actions. This suggests that ToMacVF should be understood not as a replacement for value factorization, but as its temporal and asynchronous extension.

A second conceptual precursor comes from macro-action RL under partial observability. Earlier MacDec-POMDP work defined SMDP-style macro-action values with duration-dependent discounting and cumulative reward over macro-action execution windows, including asynchronous conditional targets that keep unfinished agents’ macro-actions fixed (Xiao, 2022). ToMacVF differs by explicitly introducing temporal execution-time dependence into factorized value learning and by coupling per-timestep and per-macro updates within a CTDE architecture.

3. Mac-SJERT and temporal representation of macro-action execution

Mac-SJERT, the Macro-action Segmented Joint Experience Replay Trajectory, is the data structure that makes ToMacVF’s temporal factorization operational. It is designed to eliminate the duplicated-macro-action, inconsistent-macro-observation, and incorrect-cumulative-reward issues attributed to Mac-JERT, while providing a complete and accurate representation of each macro-action’s execution over time (Zhang et al., 14 Jul 2025).

Its storage scheme is decentralized per agent. Each agent $i$ has its own buffer $$m^i = \langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$$0, containing segmented trajectories $$m^i = \langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$$1, where each segment corresponds to one full execution of a macro-action $$m^i = \langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$$2. For every environment step from macro-action initiation to termination, Mac-SJERT stores the real information

$$m^i = \langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$$3

with temporally encoded joint macro-observations and joint macro-actions. At the termination of $$m^i = \langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$$4, it additionally stores the joint cumulative reward for the whole duration,

$$m^i = \langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$$5

Crucially, cumulative rewards are accumulated decentrally per agent; reset is local to the macro-action rather than tied to any global termination event (Zhang et al., 14 Jul 2025).

Temporal positional encoding is added to both macro-actions and macro-observations. For timestep $$m^i = \langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$$6, the encoded time is $$m^i = \langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$$7, and the representation is augmented as

$$m^i = \langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$$8

The additional $$m^i = \langle I_{m^i}, \pi_{m^i}, \beta_{m^i} \rangle,$$9 term encodes progress within the macro-action. As a result, the same macro-action identity at different execution phases is no longer duplicated in representation space (Zhang et al., 14 Jul 2025).

Mac-SJERT supports two kinds of training transitions. Micro-transitions are per-timestep objects,

$I_{m^i}$0

while macro-transitions are per-macro-action execution objects for a specific agent,

$I_{m^i}$1

This dual representation is central to ToMacVF’s learning rule: the micro scale captures value evolution during execution, and the macro scale captures total return over the whole temporally extended action (Zhang et al., 14 Jul 2025).

A common misconception is that Mac-SJERT is simply a larger replay buffer. In the ToMacVF formulation, it is more specific: it is the mechanism that makes temporal phase information, real-time macro-observation updates, and agent-local cumulative reward bookkeeping available to the value factorization architecture.

4. To-Mac-IGM and the generalization of decentralized consistency

ToMacVF’s theoretical centerpiece is To-Mac-IGM, the Temporal Macro-action based Individual-Global-Max condition. In synchronous Dec-POMDPs with primitive actions, standard IGM requires that greedy joint action selection under the centralized $I_{m^i}$2 agree with greedy local action selection under the individual $I_{m^i}$3. That condition is the formal bridge between centralized training and decentralized execution in value factorization (Wang et al., 2020).

In asynchronous MacDec-POMDPs, however, joint action selection is conditional on the fact that some agents terminate and some do not. To-Mac-IGM therefore defines the centralized macro-action-value function over joint macro-histories, joint macro-actions, and execution-time vectors,

$I_{m^i}$4

together with local macro-action-value functions

$I_{m^i}$5

At a decision point, terminating agents choose new macro-actions by locally maximizing $I_{m^i}$6, while non-terminating agents keep their current macro-actions fixed in $I_{m^i}$7. The joint greedy choice under $I_{m^i}$8 must coincide with those local greedy choices under this termination-conditioned rule (Zhang et al., 14 Jul 2025).

The paper proves a strict representational inclusion: $I_{m^i}$9 The interpretation is direct. Ordinary IGM is a special case of macro-action IGM when macro-actions collapse to single-step primitives; Mac-IGM is itself a special case of To-Mac-IGM when explicit execution time is omitted. By adding $\pi_{m^i}$0, To-Mac-IGM can represent value functions that depend on where the system is within ongoing macro-actions, not merely on which macro-actions are active (Zhang et al., 14 Jul 2025).

The appendix also defines an equivalent To-MacAdv-IGM condition on advantage functions and states that it is equivalent in representational power to To-Mac-IGM. This places ToMacVF within the same family of representational arguments that motivated richer factorized value classes in cooperative Q-learning. A plausible implication is that To-Mac-IGM plays, for asynchronous macro-actions, the role that IGM-complete classes were argued to play for primitive-action CTDE factorization.

5. Architecture and learning algorithm

The ToMacVF architecture is modular and CTDE-compliant. It has three main components: temporal agent networks, an Agent-oriented Temporal Parameter Generation (ATPG) module, and a mixing network (Zhang et al., 14 Jul 2025).

Temporal agent networks operate on each agent’s local inputs and output local macro-action utilities,

$\pi_{m^i}$1

Their inputs include macro-observation $\pi_{m^i}$2 with temporal encoding, previous macro-action $\pi_{m^i}$3 with temporal encoding, and optionally a recurrent hidden state. During decentralized execution, these networks alone are used, with each agent acting greedily via

$\pi_{m^i}$4

ATPG generates mixing parameters from joint temporal context. It consumes full joint histories and temporal encodings from Mac-SJERT, applies temporal self-attention to capture within-agent temporal dynamics and agent-oriented self-attention to model inter-agent dependencies, and outputs encodings $\pi_{m^i}$5 that parameterize the mixer. The generated weights and biases are constrained to be non-negative, extending the usual QMIX-style monotonicity condition to the To-Mac-IGM setting. The paper emphasizes that ATPG also modulates the gradient path from $\pi_{m^i}$6 back to each $\pi_{m^i}$7, thereby affecting credit assignment across agents and time (Zhang et al., 14 Jul 2025).

The mixing network computes the centralized joint value as

$\pi_{m^i}$8

The framework is intentionally backend-agnostic: the mixing network can instantiate QMIX, QTRAN, or QPLEX. In the reported experiments, the default implementation is ToMacVF with QMIX (Zhang et al., 14 Jul 2025).

Learning combines micro-level and macro-level TD objectives. The micro loss is

$\pi_{m^i}$9

and the macro loss for a macro-transition of agent $\beta_{m^i}$0 is

$\beta_{m^i}$1

Target networks are softly updated by

$\beta_{m^i}$2

The algorithm alternates between asynchronous data collection, per-agent Mac-SJERT updates, micro-transition storage, macro-transition storage at each agent’s termination, and periodic training on both losses (Zhang et al., 14 Jul 2025).

6. Empirical behavior, relation to prior work, and limitations

The empirical evaluation uses three standard MacDec-POMDP benchmarks: BoxPushing, Warehouse Tool Delivery (WTD), and Overcooked. These environments share the properties that motivate ToMacVF: strong asynchrony, partial observability, and coordinated macro-actions with nontrivial temporal overlap. Baselines include Mac-CAC, Mac-IAICC, Mac-NIACC, Mac-Dec-MADDRQN, Mac-Cen-DDRQN, and AVF, with ToMacVF integrated with QMIX by default and additionally tested with QTRAN and QPLEX mixing (Zhang et al., 14 Jul 2025).

The main results are consistent across domains. In BoxPushing, ToMacVF achieves the best performance across all map sizes, with higher returns, faster convergence, and lower variance; AVF can collapse to zero returns in some settings. In Warehouse, ToMacVF is consistently the top performer across WTD-A to WTD-E and shows strong adaptability to different human speeds and robot counts. In Overcooked, ToMacVF achieves high and stable returns on all maps A–C, whereas macro-based baselines generally underperform and AVF is unstable and high variance (Zhang et al., 14 Jul 2025).

Ablation studies isolate the contribution of Mac-SJERT and ATPG. Removing ATPG degrades performance, replacing Mac-SJERT with Mac-JERT also harms performance, and removing both causes a substantial drop approaching baseline performance. Additional mixing comparisons show that ToMacVF$\beta_{m^i}$3 is strong and stable across all tested Overcooked maps, whereas ToMacVF$\beta_{m^i}$4 and ToMacVF$\beta_{m^i}$5 are unstable and high-variance in some maps. This is noteworthy because the architecture is formally modular; empirically, not all mixing backends behave equally well in the asynchronous macro-action regime (Zhang et al., 14 Jul 2025).

Relative to earlier macro-action MARL, ToMacVF can be seen as a more explicitly temporal and factorized treatment of a problem that prior MacDec-POMDP methods already recognized. The earlier literature introduced asynchronous SMDP-style macro-action values, conditional targets for unfinished macro-actions, and CTDE variants with centralized critics or centralized Q-guided decentralized learning, but did not provide To-Mac-IGM, Mac-SJERT, or a per-timestep temporal factorization architecture of this form (Xiao, 2022). Relative to the general theory of factorized cooperative Q-learning, ToMacVF extends the logic of CTDE-compatible factorization into an asynchronous domain where execution time becomes part of the representational state (Wang et al., 2020).

The paper also states several limitations. Mac-SJERT stores trajectories at every timestep, increasing memory and compute cost. ATPG adds self-attention over time and agents, further increasing complexity. Macro-action sets and low-level policies are assumed to be predefined and pre-trained. The mixing network uses non-negative parameters to ensure To-Mac-IGM sufficiency, which imposes a monotonicity restriction even though the temporal version is more expressive than standard QMIX. Implementation is correspondingly more complex because segmented trajectories, cumulative rewards, temporal encodings, and asynchronous termination logic must all be maintained correctly (Zhang et al., 14 Jul 2025).

The authors suggest extensions to more general temporal abstraction beyond macro-actions, integration into hierarchical MARL where macro-actions or options are learned, combination with richer communication or belief modeling under partial observability, and adaptation to continuous-time or asynchronous event-driven systems. This suggests that ToMacVF is not only a specific algorithmic proposal, but also a formal template for bringing value factorization into temporally extended and asynchronous cooperative decision processes (Zhang et al., 14 Jul 2025).