Metric-Decoupled Policy Optimization

• Metric-Decoupled Policy Optimization (MDPO) is a multi-objective reinforcement learning framework that optimizes discrete driving trajectories using independent safety, progress, and comfort metrics.
• It decouples vectorized rewards by applying per-metric softmax normalization and explicit weighting, enabling refined local candidate selection in hierarchical planning.
• MDPO improves performance over scalar reward methods, effectively balancing conflicting driving objectives and enhancing interpretability in end-to-end autonomous driving.

Searching arXiv for the cited MDPO-related papers and terminology to ground the article. Metric-Decoupled Policy Optimization (MDPO) is the reinforcement learning component of HAD, the end-to-end driving framework introduced in "HAD: Combining Hierarchical Diffusion with Metric-Decoupled RL for End-to-End Driving" (Yao et al., 4 Apr 2026). Within HAD, MDPO trains the second-stage local diffusion policy to prefer trajectories that are good along multiple safety and driving-quality dimensions, rather than optimizing a single coupled scalar reward. Its defining feature is that reward remains vectorized by metric: the local decoder predicts metric-specific RL logits, computes per-metric probabilities and normalized rewards within compact local candidate sets, and combines their contributions with explicit weights $\alpha^{(m)}$. In this sense, MDPO is a multi-objective RL scheme over a discrete set of diffusion-generated trajectories, integrated into a hierarchical diffusion planner.

1. Position within the HAD planning stack

HAD decomposes planning into a coarse-to-fine hierarchy. Stage 1, termed Driving Intention Establishment, uses a global diffusion policy to pick a small set of high-level anchors. A Structure-Preserved Trajectory Expansion module then generates local candidate maneuvers around each anchor. Stage 2, termed Local Trajectory Refinement, refines those local candidates and scores them with several heads, including distance, safety metrics, and RL logits. MDPO is applied only in this second stage (Yao et al., 4 Apr 2026).

The state is an encoded driving scene comprising RGB images, LiDAR or positional embeddings in HAD-L, and ego status, processed by a Transfuser-like encoder $\mathrm{Enc}_{env}$ to produce BEV and agent features $cond$. The action is a $T$-step ego-trajectory,

$\tau = \{(x_t, y_t)\}_{t=1}^T.$

The policy is the hierarchical diffusion policy $\pi_\theta(\tau\mid s)$, implemented by two transformer-based decoders, $\mathrm{Dec}_{gbl}$ and $\mathrm{Dec}_{lcl}$. MDPO specifically optimizes the metric-specific logits $\hat{s}_{\mathrm{lcl},j}^{(m\text{-}\mathrm{rl})}$ in $\mathrm{Dec}_{lcl}$.

The driving objectives are drawn from the NAVSIM family of metrics and are partitioned into multiplicative penalty metrics and additive average metrics.

Metric family	Metrics
Penalty metrics $\mathrm{Enc}_{env}$0	No Collisions (NC), Drivable Area Compliance (DAC), Driving Direction Compliance (DDC, v2), Traffic Light Compliance (TLC, v2)
Average metrics $\mathrm{Enc}_{env}$1	Ego Progress (EP), Time-to-Collision (TTC), Lane Keeping (LK, v2), Comfort variants (C/HC/EC)

This placement within HAD is consequential. Stage 1 narrows the search to a few intention anchors, and the expansion module constructs realistic local alternatives. MDPO then operates where metric variation is fine-grained enough to support meaningful local ranking among candidates.

2. Decoupling by driving metric

In the formulation used by HAD, “metric-decoupled” means that the reward is not collapsed into a single scalar such as PDMS or a weighted sum. Instead, HAD treats reward as a vector $\mathrm{Enc}_{env}$2, predicts per-metric RL logits, computes per-metric probabilities and advantages, and combines metric-specific contributions only at the final aggregation step via weights $\mathrm{Enc}_{env}$3 (Yao et al., 4 Apr 2026).

The motivation is explicitly tied to limitations of coupled scalar rewards in end-to-end driving. Existing approaches are described as mixing fundamentally different objectives such as collision, lane keeping, progress, and comfort; hiding trade-offs and making optimization coarse; and encouraging reward hacking by allowing improvement in an aggregate score through exploitation of one dimension at the expense of others. The paper further emphasizes that different metrics live on different scales and have different semantics, for example NC as a safety pass/fail signal versus EP as scalar progress.

MDPO addresses these issues by assigning each optimized metric its own RL head. Each head learns to discriminate trajectories with respect to a specific metric, rewards are standardized within each local candidate set, and the relative influence of objectives is controlled through $\mathrm{Enc}_{env}$4. The authors characterize the result as better signal structure, easier balancing of safety versus efficiency versus comfort, and greater interpretability of the optimization process.

A plausible implication is that MDPO uses the word “metric” in the sense of evaluation metric, not in the sense of a geometric divergence on policy space. That usage distinguishes it from other arXiv works that use the same acronym for different ideas, including "Mirror Descent Policy Optimization" (Tomar et al., 2020).

3. Mathematical definition

After Stage 1 and trajectory expansion, Stage 2 receives $\mathrm{Enc}_{env}$5 intention anchors. For each anchor $\mathrm{Enc}_{env}$6, the expansion algorithm produces a local subset

$\mathrm{Enc}_{env}$7

For each candidate $\mathrm{Enc}_{env}$8, the local decoder outputs a refined trajectory, a distance-to-expert score, supervised metric predictions, and metric-specific RL logits: $\mathrm{Enc}_{env}$9

MDPO acts on $cond$0 through a per-metric softmax over the local set: $cond$1 For each metric $cond$2, simulator-based rewards $cond$3 are retrieved and standardized within the same local set: $cond$4

The metric-decoupled set reward is

$cond$5

where $cond$6 ranges over the RL-optimized metrics NC, DAC, DDC, TLC, EP, TTC, LK, and HC. The overall RL objective is

$cond$7

The MDPO loss is therefore

$cond$8

This objective makes MDPO a per-metric soft policy gradient over a discrete candidate set. For each metric, gradients increase the logits of trajectories with high normalized reward while retaining a temperature-smoothed ranking among nearby candidates. The paper explicitly contrasts this design with PPO and GRPO: MDPO does not implement explicit KL or clipping, and instead relies on a per-metric softmax over local candidates, per-set standardization, and aggregation through $cond$9 (Yao et al., 4 Apr 2026).

4. Architectural integration and training procedure

The local decoder $T$0 uses a shared transformer backbone that processes both environment context $T$1 and candidate trajectory embeddings $T$2. On top of this shared backbone, HAD attaches multiple metric heads as small MLPs: a distance head, supervised metric heads, and RL metric heads. The RL metric heads share the same backbone but have separate output parameters for each metric, which the paper describes as enabling decoupled representations for metric-specific preference learning (Yao et al., 4 Apr 2026).

The batch-level training workflow is structured as follows. First, Stage 1 samples or diffuses global anchors with $T$3 and selects the top $T$4 anchors by distance-based score; there is no RL in this stage. Second, Structure-Preserved Trajectory Expansion generates local candidates around each selected anchor through polar perturbations. Third, the local decoder refines the candidates and outputs $T$5 together with distance, supervised metric, and RL logits. Fourth, reward retrieval maps each refined trajectory to the nearest trajectory in an 8192-trajectory vocabulary and reads cached simulator scores as RL rewards. Fifth, MDPO computes per-set standardized rewards, per-metric softmax probabilities, and the objective $T$6. Sixth, the RL loss backpropagates through the metric-specific logits and the shared decoder.

The RL loss is combined with supervised objectives in the local-stage loss,

$T$7

Several implementation choices are presented as essential for stability. Reward computation is offline and based on nearest-neighbor retrieval rather than online simulation. Rewards are normalized per metric and per local set. RL is restricted to Stage 2, because the global candidate space is described as large and unstructured. At inference, higher-level coefficients combine distance, PDMS-like supervised scores, and RL scores; the reported combination uses $T$8, $T$9, and $\tau = \{(x_t, y_t)\}_{t=1}^T.$0, keeping RL influence modest but meaningful.

5. Empirical behavior and comparative position

The central empirical evidence for MDPO comes from ablations that compare decoupled and coupled reward formulations on NAVSIM v2 (Yao et al., 4 Apr 2026).

Setting	EPDMS	Notes
Dist. only	62.6	Very poor; EC 3.9
PDMS as single scalar reward	87.8	Coupled reward RL
Dist. + PDMS	86.9	Still scalar/coupled
Decoupled Metrics (MDPO)	88.6	Best; decoupled metric heads

The decoupled setting is reported to use 51 metric heads and yields strong values across the fundamental metrics: NC 98.2, DAC 97.3, DDC 99.2, TLC 99.8, EP 87.4, TTC 97.5, LK 95.2, HC 98.3, and EC 86.2. Relative to the scalar-reward baselines, MDPO improves EPDMS by $\tau = \{(x_t, y_t)\}_{t=1}^T.$1 over single PDMS reward RL and by $\tau = \{(x_t, y_t)\}_{t=1}^T.$2 over the best coupled Dist.+PDMS setting. Relative to distance-only imitation, the gain is approximately $\tau = \{(x_t, y_t)\}_{t=1}^T.$3.

At the benchmark level, HAD with MDPO reaches EPDMS 88.6 on NAVSIM v2 for the C+L setting and 88.5 for the camera-only setting, exceeding DiffusionDriveV2 at 85.5, DiffRefiner at 86.2, EvaDrive at 86.3, and selection-based DriveSuprim and HydraMDP++ at 83.1 and 81.4. On HUGSIM closed-loop evaluation, HAD-L reaches Route Completion 47.5 and HD-Score 30.8, exceeding ZTRS at 42.6 and 28.9, with particularly strong results in Extreme scenarios.

The placement of RL also receives an ablation. Adding RL metrics in the global stage reduces EPDMS from 88.6 to 87.2. The paper interprets this as evidence that RL signals are too noisy in the large, unstructured global space, whereas the compact local sets created by hierarchical diffusion and structure-preserved expansion give MDPO a suitable operating regime.

The method is also situated relative to related research programs. Compared with single-reward driving RL, MDPO retains a vector-valued reward until the last aggregation through $\tau = \{(x_t, y_t)\}_{t=1}^T.$4. Compared with GDPO, which is cited as using multiple rewards with normalization, MDPO is described as conceptually similar in decoupling reward dimensions and normalizing per reward, but specialized to ranking a small discrete set of trajectories in autonomous driving rather than generic RL. Compared with reward decomposition or multi-task learning, MDPO extends shared-backbone multi-head structure by attaching RL training to each head rather than using only supervised regression.

6. Limitations, scope, and terminological ambiguity

The paper identifies several limitations. RL was ineffective or harmful in the global stage, which indicates that MDPO currently does not handle very large, unstructured action spaces well. Metric weights such as $\tau = \{(x_t, y_t)\}_{t=1}^T.$5, $\tau = \{(x_t, y_t)\}_{t=1}^T.$6, and the higher-level $\tau = \{(x_t, y_t)\}_{t=1}^T.$7 coefficients are hand-tuned rather than learned. Reward is approximated by nearest-neighbor retrieval from an 8192-trajectory vocabulary, introducing approximation error. The method is tightly bound to NAVSIM-style metrics, so extension to richer or human-feedback-based rewards would require new metrics and new heads. In addition, the Transfuser backbone is described as fragile on NavHard’s noisy synthetic images, suggesting sensitivity to observation quality (Yao et al., 4 Apr 2026).

Beyond these implementation limits, the acronym itself is ambiguous on arXiv. "Mirror Descent Policy Optimization" uses MDPO for a trust-region RL algorithm derived from mirror descent (Tomar et al., 2020). "MDPO: Overcoming the Training-Inference Divide of Masked Diffusion LLMs" uses MDPO for Masked Diffusion Policy Optimization in diffusion LLMs (He et al., 18 Aug 2025). "Model-Based Decentralized Policy Optimization" uses MDPO for a decentralized MARL method (Luo et al., 2023). Metric-Decoupled Policy Optimization in HAD is therefore a distinct usage: here, “metric-decoupled” refers to decoupling across driving metrics such as NC, DAC, EP, TTC, LK, and HC, not to mirror-descent geometry, masked diffusion trajectories for language, or decentralized model-based multi-agent control.

The broader conceptual takeaway stated by the HAD paper is that MDPO is useful when actions are selected from a structured set of candidates and multiple conflicting metrics must be optimized together. In that setting, separating reward into metric-specific components, normalizing them within local candidate sets, and training metric-specific preference heads provides a controlled mechanism for balancing safety, efficiency, and comfort. This suggests a broader design pattern, but in the present formulation MDPO is specifically a local-stage, multi-objective RL optimizer embedded inside hierarchical diffusion for end-to-end autonomous driving.