HAD: Combining Hierarchical Diffusion with Metric-Decoupled RL for End-to-End Driving

Abstract: End-to-end planning has emerged as a dominant paradigm for autonomous driving, where recent models often adopt a scoring-selection framework to choose trajectories from a large set of candidates, with diffusion-based decoding showing strong promise. However, directly selecting from the entire candidate space remains difficult to optimize, and Gaussian perturbations used in diffusion often introduce unrealistic trajectories that complicate the denoising process. In addition, for training these models, reinforcement learning (RL) has shown promise, but existing end-to-end RL approaches typically rely on a single coupled reward without structured signals, limiting optimization effectiveness. To address these challenges, we propose HAD, an end-to-end planning framework with a Hierarchical Diffusion Policy that decomposes planning into a coarse-to-fine process. To improve trajectory generation, we introduce Structure-Preserved Trajectory Expansion, which produces realistic candidates while maintaining kinematic structure. For policy learning, we develop Metric-Decoupled Policy Optimization (MDPO) to enable structured RL optimization across multiple driving objectives. Extensive experiments show that HAD achieves new state-of-the-art performance on both NAVSIM and HUGSIM, outperforming prior arts by a huge margin: +2.3 EPDMS on NAVSIM and +4.9 Route Completion on HUGSIM.

• The paper introduces a hierarchical planning pipeline that combines diffusion-based intention establishment with local trajectory refinement to produce kinematically plausible routes.
• It employs polar-coordinate perturbations with metric-decoupled RL that separately optimizes driving objectives, reducing reward interference and improving sample efficiency.
• An offline reward retrieval system minimizes simulation overhead, enabling rapid training while experimental results demonstrate state-of-the-art performance on open- and closed-loop benchmarks.

Hierarchical Diffusion Policies and Metric-Decoupled Reinforcement Learning for End-to-End Autonomous Driving

Introduction

This work presents HAD, an integrated hierarchical diffusion and RL-based approach for end-to-end planning in autonomous driving. The paper addresses critical shortcomings in prior end-to-end frameworks related to optimization complexity over the driving action space and the limitations of reward shaping in RL-based planners. Specifically, existing scoring-selection and diffusion-based models search over overly large candidate spaces with weak priors and utilize Gaussian perturbations that introduce non-kinematic, implausible trajectories. Reinforcement learning approaches are typically handicapped by reliance on a single, coupled reward, leading to coarse optimization and susceptibility to reward hacking, while online reward computation via simulation incurs substantial training overhead. The authors introduce a hierarchical coarse-to-fine planning pipeline, a structure-preserving trajectory expansion scheme leveraging polar coordinate perturbations, and a metric-decoupled RL policy optimization (MDPO), coupled with a highly efficient reward retrieval system.

Figure 1: Prior methods optimize in a large, coupled space and require online simulation for reward computation; HAD narrows search via hierarchical policy and exploits offline precomputed decoupled rewards.

Methodology

Hierarchical Diffusion Policy

HAD decomposes planning into two denoising stages: high-level driving intention establishment and low-level local trajectory refinement.

The first stage employs a sparse set of predefined trajectory anchors. Diffusion-based denoising (Transformer-driven) identifies plausible intention candidates via selection based on confidence that the anchor is in the same driving sub-region as expert demonstrations.

Figure 2: HAD decomposes planning into intention establishment via denoising over anchors, trajectory expansion, and final local refinement. MDPO provides structured RL supervision.

Local region exploration around selected anchors is achieved through Structure-Preserved Trajectory Expansion. Contrary to classic Gaussian noise or Cartesian perturbative expansion, HAD applies radial and angular perturbations in polar coordinates, allowing the generation of candidates that preserve trajectory geometry and kinematic structure. The resulting candidate set is refined in the second diffusion stage, with multiple MLP heads predicting a suite of metric scores for each candidate, including supervised imitation, safety constraints, and RL-driven quantities.

Figure 4: Random noise destroys kinematic structure, XY expansion limits local exploration, polar expansion in HAD comprehensively covers the local search region.

Metric-Decoupled Policy Optimization (MDPO)

Most prior RL-based planning aggregates driving objectives into a single reward. HAD’s MDPO explicitly decouples optimization such that each critical driving rule (e.g. collision avoidance, drivable area compliance, lane keeping, TTC, comfort, traffic light compliance) is mapped to an independent reward head. Each candidate trajectory evaluation predicts logits for every metric. Candidate selection is then driven by normalized, metric-wise reward signals, and global reward computation conducts weighted aggregation only after per-metric normalization, reducing cross-metric interference and reward gaming.

Offline Reward Retrieval

To support RL training without the heavy computation costs of online trajectory simulation, HAD introduces an offline reward retrieval scheme. Large, dense trajectory vocabularies are precomputed with associated safety-metric scores. During training, reward labels for any candidate trajectory are retrieved via nearest-neighbor lookup against this vocabulary. This strategy nearly eliminates simulation overhead during training and enables rapid, scalable RL on large datasets.

Figure 5: Simulation-based reward computation requires online expensive evaluation; HAD uses offline precomputed tables and fast nearest-neighbor lookups.

Experimental Results

Quantitative evaluation is presented on NAVSIM v1/v2 (open-loop) and HUGSIM (closed-loop) benchmarks. HAD achieves state-of-the-art performance: 90.2 PDMS (v1) and 88.6 EPDMS (v2) on NAVSIM, with gains up to +2.3 points over prior SOTA (2604.03581). On HUGSIM, HAD-L (image-only) reaches 47.5 route completion and 30.8 HD-Score, outperforming the best previous method by +4.9 in route completion.

The paper includes strong ablation studies. It demonstrates that

• Trajectory expansion in the polar domain outperforms random and Cartesian expansion by nearly +3.7 EPDMS.
• MDPO outperforms single-head RL by +0.8 EPDMS and improves sample efficiency.
• The offline reward retrieval system decreases per-trajectory reward latency from 0.24s to 0.004s, yielding a 5x speedup in total training.

Visualization results show superior intention alignment and safety in both complex urban and adversarially challenging scenarios. HAD trajectories (blue) match expert demonstrations (green) more closely than competitive baselines.

Figure 3: On NAVSIM examples, HAD predictions (blue) maintain strong safety margin and track lane centerlines robustly compared to precedent methods.

Figure 8: On HUGSIM, HAD produces smooth, context-aware interactive trajectories in dynamic and adversarial agent deployments.

Discussion and Implications

HAD’s approach highlights several theoretical and practical implications:

• Hierarchical policies reduce the effective search space at each stage, enabling tractable optimization even in multimodal, long-horizon planning problems, and facilitate more reliable denoising in trajectory generation.
• Polar coordinate trajectory perturbations preserve physically plausible kinematic constraints absent in naive noise models, demonstrating the necessity of structure-aware candidate generation in safety-critical domains.
• Decoupled multi-metric RL explicitly regularizes policy learning, preventing reward hacking and yielding granular control over tradeoffs between driving objectives—foundational for real-world deployment where safety, comfort, and regulatory compliance must all be harmonized.
• Offline reward retrieval mitigates training bottlenecks and could become standard when simulator-based rewards are expensive yet diverse reference sets are available.

Practically, HAD’s 30 FPS inference rate (A100 GPU) and competitive model capacity (63M parameters) ensure real-time deployability. The modular design (image-only vs. multi-modal sensor fusion) also facilitates ablation of perception system requirements.

Future lines of research include: integrating semantic priors for even more robust intention modeling, extending metric decoupling for interactive/collaborative driving settings, and adapting the framework for world models or vision-language-based closed-loop policies.

Conclusion

HAD introduces a scalable, sample-efficient framework unifying hierarchical diffusion policy planning, structure-preserved kinematic expansion, and decoupled policy optimization for end-to-end driving. By combining physically plausible candidate generation with multi-objective RL and rapid offline reward computation, HAD delivers SOTA empirical results on open- and closed-loop driving tasks, and sets a new baseline for future research in unified planning under real-world constraints (2604.03581).