OMDPG: Optimal Marginal Deterministic Policy Gradient

• OMDPG is a heterogeneous MARL approach that defines agents’ marginal contributions via an Optimal Marginal Q function, replacing sequential policy ratios.
• It employs a Generalized Q Critic and Centralized Critic Grouped Actor architecture to stabilize training and counteract baseline-drift in grouped parameter sharing.
• Empirical evaluations on SMAC and MAMuJoCo demonstrate that OMDPG achieves higher performance and more stable returns compared to existing MARL algorithms.

Optimal Marginal Deterministic Policy Gradient (OMDPG) is a heterogeneous multi-agent reinforcement learning method introduced to reconcile two objectives that are usually in tension in cooperative heterogeneous settings: monotonic policy improvement and partial parameter-sharing (ParPS). Proposed for Dec-POMDPs, OMDPG replaces HAPPO’s sequential policy-ratio machinery with an Optimal Marginal Q (OMQ) construction, introduces a Generalized Q Critic (GQC) trained with a pessimistic uncertainty-constrained loss, and implements a Centralized Critic Grouped Actor (CCGA) architecture so that grouped deterministic actors can coexist with accurate global Q-estimation (Yu et al., 14 Jul 2025).

1. Problem setting and motivation

OMDPG is formulated in heterogeneous cooperative MARL under the Dec-POMDP formalism

$$G=\left\langle S, A, O; \mathbb{P}, \Omega, R; \gamma, N, K, T \right\rangle,$$

where $N=\{1,\dots,n\}$ is the set of agents and $K=\{1,\dots,k\}$ is the set of agent groups. Agent $i$ belongs to group $k_i$, receives local observation $o_i^t$, selects $a_i^t\in A_i$, and contributes to the joint action

$$a^t=(a_1^t,\dots,a_n^t), \qquad a_{1:i}^t=(a_1^t,\dots,a_i^t).$$

The environment evolves according to $\mathbb{P}(s^{t+1}\mid s^t,a^t)$ and returns a global reward $R(s,a^t)$. Each agent only accesses its local history

$N=\{1,\dots,n\}$0

The immediate motivation is the conflict between HAPPO-style monotonic optimization and grouped parameter sharing. HAPPO’s sequential update relies on the ratio

$N=\{1,\dots,n\}$1

and approximates agent $N=\{1,\dots,n\}$2’s sequential advantage through

$N=\{1,\dots,n\}$3

This construction is compatible with No Parameter-sharing (NoPS), because updating one agent leaves the others’ policies fixed for the ratio calculation. In heterogeneous MARL, however, Partial Parameter-sharing is often introduced by grouping similar agents. The paper’s central claim is that directly combining ParPS with the sequential update scheme leads to the policy updating baseline drift problem, because updating one shared actor silently changes the baseline policy of other agents in the same group. The resulting drift is written explicitly as

$N=\{1,\dots,n\}$4

This conflict defines OMDPG’s scope. The method is not a generic deterministic policy gradient algorithm for single-agent continuous control; it is a heterogeneous MARL construction whose target is grouped actor learning without losing the monotonic-improvement logic associated with MAAD (Yu et al., 14 Jul 2025).

2. Optimal Marginal Q and marginal contribution estimation

OMDPG’s core replacement for sequential policy-ratio correction is the Optimal Marginal Q function. The starting point is the paper’s claim that, under MAAD, sequential advantage and sequential Q are equivalent: $N=\{1,\dots,n\}$5 This equivalence motivates direct optimization of a Q-style marginal contribution instead of sequential policy-ratio approximations.

Using global state $N=\{1,\dots,n\}$6, OMQ is defined as

$N=\{1,\dots,n\}$7

Formally, $N=\{1,\dots,n\}$8 measures the marginal contribution of $N=\{1,\dots,n\}$9, conditioned on preceding actions $K=\{1,\dots,k\}$0, under an optimal completion $K=\{1,\dots,k\}$1 of the remaining joint action. The baseline replaces agent $K=\{1,\dots,k\}$2’s action by a null action $K=\{1,\dots,k\}$3.

The practical implementation does not solve for $K=\{1,\dots,k\}$4 exactly. Instead, it uses policy-greedy continuation

$K=\{1,\dots,k\}$5

OMQ is therefore approximate in practice, but the paper’s argument is that the approximation avoids the baseline-drift pathology because it depends on critic-evaluated joint actions rather than on pre-update versus post-update policy ratios.

The actor objective is a pessimistic OMQ maximization: $K=\{1,\dots,k\}$6 This is why the method is described as deterministic-policy-gradient-based in the paper: actors deterministically output actions and are updated through critic values. The paper also states that it does not derive the classic Silver-style deterministic policy gradient formula and does not analyze off-policy DPG in that sense (Yu et al., 14 Jul 2025).

3. Generalized Q Critic and the CCGA architecture

OMDPG requires critic estimates not only for real sampled joint actions but also for counterfactual or out-of-distribution tuples created by greedy completion. GQC is introduced to provide these values with conservative calibration. The paper states that a unified Q-function estimates three kinds of quantities: the true $K=\{1,\dots,k\}$7, the counterfactual value with $K=\{1,\dots,k\}$8 and optimal continuation, and the counterfactual baseline with $K=\{1,\dots,k\}$9 instead of $i$0.

GQC uses $i$1 parallel critics. Uncertainty is quantified by ensemble standard deviation: $i$2

For real transitions, the critic target is

$i$3

with squared loss

$i$4

For OOD counterfactual tuples, greedy continuation is combined with a pessimistic uncertainty penalty: $i$5

$i$6

The full critic loss is

$i$7

Architecturally, OMDPG implements a Centralized Critic Grouped Actor design. Grouped actors partition agents into $i$8 groups and share parameters within each group, thereby realizing ParPS. The critic remains centralized and receives global state and joint-action information, which is necessary for evaluating OMQ under counterfactual completions. This separation is the mechanism by which OMDPG simultaneously targets grouped local policy networks and accurate global Q-function computation (Yu et al., 14 Jul 2025).

4. Theoretical role and place within deterministic and marginal policy-gradient research

Within the paper’s own theoretical narrative, OMDPG preserves MAAD’s monotonic-improvement logic while eliminating the conflict between sequential updates and ParPS through optimal joint action sequences instead of sequential policy-ratio calculations. The supplied material does not reproduce a full new theorem-and-proof pair for monotonicity, but it states that OMQ is designed as a surrogate for the MAAD-consistent sequential contribution and that the monotonic-improvement claim is inherited conceptually from this construction (Yu et al., 14 Jul 2025).

OMDPG therefore occupies a specific position in the literature. It is not the same object as earlier work on “marginal” policy gradients, deterministic deployment after stochastic training, or deterministic policy optimization in single-agent control. Several nearby lines of work are relevant but distinct.

Work	Relation to OMDPG	Key distinction
“Marginal Policy Gradients” (Eisenach et al., 2018)	Shares the term “marginal”	Stochastic PG under action transformations
“Learning Optimal Deterministic Policies with Stochastic Policy Gradients” (Montenegro et al., 2024)	Studies deterministic deployment after stochastic PG	No OMDPG algorithm or MARL construction
“Learning Deterministic Policies with Policy Gradients in Constrained Markov Decision Processes” (Montenegro et al., 6 Jun 2025)	Gives constrained deterministic-deployment guarantees	CMDP primal-dual setting, not heterogeneous MARL
“Truly Deterministic Policy Optimization” (Saleh et al., 2022)	Optimizes directly in deterministic policy space	Wasserstein trust-region method, not ParPS/OMQ
“Deterministic Policy Gradient Primal-Dual Methods for Continuous-Space Constrained MDPs” (Rozada et al., 2024)	Uses deterministic policy improvement under marginal visitation weighting	Single-agent constrained MDPs

These comparisons indicate that OMDPG’s novelty is not “deterministic policy gradient” in isolation, but the combination of deterministic grouped actors, optimal marginal contribution estimation, and baseline-drift resolution in heterogeneous MARL. This also clarifies a common misconception: OMDPG is not merely a renamed DDPG or TD3 variant for multi-agent settings (Yu et al., 14 Jul 2025).

5. Empirical evidence and ablation structure

The paper evaluates OMDPG on SMAC and MAMuJoCo. In SMAC, the metric is Winning Rate, and the baselines include HASAC, MAPPO, MATD3, HetGPPO, MADDPG, HAPPO, and Kaleidoscope. The reported mean winning rates show that OMDPG is strongest overall in the more difficult heterogeneous maps (Yu et al., 14 Jul 2025).

SMAC task	OMDPG	Selected comparator
MMM2	$i$9	MAPPO $k_i$0
6m2m_15m	$k_i$1	HASAC $k_i$2
8m3m_21m	$k_i$3	HASAC $k_i$4
12m4m_30m	$k_i$5	MATD3 $k_i$6

In MAMuJoCo, the metric is return. The heterogeneous variants are produced by modifying gear ratios, and ParPS groups are defined by limb structure. OMDPG attains the best reported return in all four settings.

MAMuJoCo task	OMDPG
HalfCheetah	$k_i$7
HalfCheetah-Het.	$k_i$8
Walker	$k_i$9
Walker-Het.	$o_i^t$0

The ablation studies attribute the largest improvement to OMQ itself. In MATD3-based ablations, ParPS alone gives little improvement over NoPS; OMQ produces the clearest performance gain and lower variance; GQC helps more in harder environments but can increase variance and network complexity; and OMQ appears to be the primary source of benefit, with GQC acting as a stabilizing support module. Hyperparameter sweeps report that $o_i^t$1 is usually best and that $o_i^t$2 performs best on several SMAC tasks. For MMM2, the wall-clock times are 18.9 h for OMDPG with $o_i^t$3, 19.6 h for OMDPG with $o_i^t$4, 14.2 h for HAPPO, and 10.5 h for MATD3, while converged wall time is reported as 12.6 h for $o_i^t$5 and 14.8 h for $o_i^t$6. These numbers show that OMDPG incurs additional ensemble-critic cost but does not eliminate practical tractability (Yu et al., 14 Jul 2025).

6. Limitations, misconceptions, and research significance

Several limitations are explicit. First, OMQ is only approximate in implementation because $o_i^t$7 is replaced by greedy continuation $o_i^t$8. Second, the monotonic-improvement claim is presented as conceptually inherited from MAAD rather than as a fully reproduced new theorem in the supplied material. Third, performance depends on critic quality for counterfactual OOD tuples, which is why GQC and pessimistic uncertainty regularization are central. Fourth, the centralized critic requires access to global state during training. Fifth, ensemble critics increase computation and memory, and the paper’s conclusion identifies scaling to very large agent counts as future work (Yu et al., 14 Jul 2025).

A second misconception concerns terminology. The phrase “marginal” in OMDPG does not match the usage in earlier single-agent “Marginal Policy Gradients,” where the marginalization is over transformed action distributions (Eisenach et al., 2018). Nor does “deterministic policy gradient” here mean the paper derives the canonical Silver-style off-policy DPG theorem. OMDPG is better understood as a MAAD-inspired deterministic actor-critic for heterogeneous MARL whose actor target is a critic-based marginal contribution,

$o_i^t$9

rather than a sequential policy-ratio surrogate (Yu et al., 14 Jul 2025).

This suggests a broader significance. OMDPG addresses a concrete incompatibility between grouped parameter sharing and monotonic-style sequential optimization, and it does so by moving the optimization target from fragile policy-version ratios to critic-based marginal action completion. Within heterogeneous MARL, that shift is the article’s defining technical contribution.