Hydra-MDP: End-to-End Planning for Autonomous Driving

• Hydra-MDP is an end-to-end multimodal planning paradigm that unifies trajectory prediction with multi-target closed-loop evaluation in autonomous driving.
• The framework employs a teacher-student model, distilling guidance from both human and rule-based teachers via differentiable multi-head decoders.
• Empirical results demonstrate improved generalization and state-of-the-art closed-loop metrics, achieving 1st place in the Navsim challenge.

Hydra-MDP is an end-to-end multimodal planning paradigm for autonomous driving that employs multiple teachers in a teacher-student model, combining knowledge distillation from human and rule-based teachers within a single differentiable framework (Li et al., 2024). The method is designed to unify multimodal planning, in which multiple candidate futures are represented explicitly, with multi-target learning over closed-loop metrics such as No-fault Collision, Drivable-Area Compliance, Time-to-Collision, Comfort, and Ego-Progress. Its central claim is that rule-based planning knowledge can be absorbed directly by the student network from raw sensor observations, rather than being injected only through non-differentiable post-processing. In the reported Navsim challenge results, this formulation achieved $1^{st}$ place and showed improved generalization across diverse driving environments and conditions (Li et al., 2024).

1. Problem setting and conceptual motivation

Hydra-MDP is motivated by a persistent mismatch between open-loop imitation quality and closed-loop driving performance. End-to-end neural planners trained by imitation learning can mimic logged human trajectories, yet often fail under closed-loop evaluation, including collision, lane-departure, and inefficient-driving behaviors (Li et al., 2024). Prior multimodal planners address trajectory uncertainty by predicting multiple candidate trajectories, but typically rely on non-differentiable, rule-based cost functions on imperfect perception outputs to select among them. In the Hydra-MDP formulation, that separation is treated as a limitation because it breaks gradient flow and discards intermediate information that could otherwise shape representation learning.

The framework therefore unifies two ideas. The first is multimodal planning over a fixed vocabulary of trajectory candidates. The second is multi-target learning over closed-loop metrics that are normally computed by a simulator or a rule-based evaluation stack. The resulting system is not only a trajectory predictor; it is also a metric predictor over the same candidate set, trained so that the perception backbone, transformer blocks, and decoder heads jointly internalize the consequences of each candidate trajectory under simulation (Li et al., 2024).

A common misunderstanding is to view Hydra-MDP as a conventional multimodal planner with a separate rule-based selector appended at inference time. The defining distinction is that the rule-based knowledge is distilled during training into differentiable heads attached to the student. This suggests that the framework is intended less as a post hoc reranking mechanism than as an integrated planning-and-evaluation model.

2. Multi-teacher distillation formulation

Hydra-MDP uses two types of teachers: a human teacher and a rule-based teacher (Li et al., 2024). The human teacher accesses ground-truth perception $\hat P$ and the logged expert trajectory $\hat T$, and produces imitation targets over a fixed vocabulary of $k$ trajectories. The target distribution is defined by a distance-based softmax,

$$y_i = \frac{\exp\bigl(-\|\hat T - T_i\|^2\bigr)} {\sum_{j=1}^k \exp\bigl(-\|\hat T - T_j\|^2\bigr)}.$$

Rather than supervising a single regressed path, this formulation supervises a distribution over discrete candidate trajectories.

The rule-based teacher evaluates each candidate $T_i$ in an offline simulator with perfect perception $\hat P$, producing normalized binary or real-valued scores $\{\hat{\mathcal S}_i^m\}_{m\in M}$ for a set of metrics $m$. In the original Hydra-MDP exposition, the distilled Navsim sub-metrics are No-fault Collision (NC), Drivable-Area Compliance (DAC), Time-to-Collision (TTC), Comfort (C), and Ego-Progress (EP), all normalized to $[0,1]$ (Li et al., 2024). The student, by contrast, sees only raw sensor observations $\hat P$0, specifically camera and LiDAR, and must learn both the imitation target and the metric predictions end-to-end.

This multi-teacher organization is central to the term “Hydra-Distillation.” Human supervision supplies expert behavioral priors, while simulator-derived rule-based supervision injects closed-loop safety and efficiency criteria. The paper also notes that DDC, Drivable-Lane-Deviation, was omitted in the original Hydra-MDP setup due to an implementation issue (Li et al., 2024). That omission is relevant because it delineates the precise scope of the original distilled target set.

3. Student architecture, trajectory vocabulary, and hydra heads

The Hydra-MDP student is organized around a perception backbone, a discrete planning vocabulary, and a multi-head decoder (Li et al., 2024). In the original system, images are cropped front and side views at resolution $\hat P$1, or up to $\hat P$2 when scaling, and encoded by ResNet-34, ViT-L, or V2-99 depending on the experiment. LiDAR uses 4 consecutive sweeps projected into BEV and encoded by ResNet-34. Late fusion through transformer cross-attention yields environmental tokens $\hat P$3.

Planning is performed over a pre-computed vocabulary $\hat P$4 obtained by clustering 700K expert trajectories, where each trajectory consists of 40 $\hat P$5 waypoints. Each vocabulary center is embedded by an MLP, augmented with ego-state $\hat P$6, and processed by a transformer encoder,

$\hat P$7

The resulting queries are then fused with the environmental tokens by a transformer decoder,

$\hat P$8

From each latent vector in $\hat P$9, the model spawns multiple prediction heads. The imitation head is a linear layer with softmax producing $\hat T$0. For each metric $\hat T$1, a binary-classification head outputs $\hat T$2, interpreted as the probability that trajectory $\hat T$3 satisfies metric $\hat T$4 (Li et al., 2024). The hydra metaphor refers to this shared trunk with multiple specialized heads scoring the same vocabulary element under different criteria.

A descriptive rendering of the original framework has three parts: a perception backbone that fuses LiDAR BEV and multi-view images into $\hat T$5; planning queries, given by embedded K-means centers, that attend to $\hat T$6 to produce $\hat T$7; and a set of heads consisting of the imitation head plus NC, DAC, TTC, C, and EP heads, followed by final trajectory selection by an arg-min over weighted log-scores (Li et al., 2024).

4. Optimization objectives and differentiable integration of rule-based knowledge

The human-teacher signal is optimized through a cross-entropy imitation loss,

$\hat T$8

The rule-based teacher signal is optimized through a binary cross-entropy loss per metric,

$\hat T$9

The total training loss is

$k$0

with $k$1 set to $k$2 in most experiments (Li et al., 2024).

The paper also presents an optional temperature-scaled distillation form for soft logits,

$k$3

with

$k$4

However, the exposition explicitly states that in the Hydra-MDP paper the simpler BCE form suffices (Li et al., 2024).

The key methodological point is that each metric head is a differentiable neural module. As a result, gradients from $k$5 propagate backward through the transformer and into the perception backbone, so the system avoids non-differentiable post-processing. The rule-based teacher thereby “tells” the student how each trajectory will fare under closed-loop simulation, and the student internalizes these cost signals directly. This is the mechanism by which rule-based planning knowledge is integrated into end-to-end learning.

5. Inference procedure, evaluation metrics, and empirical behavior

At inference time, Hydra-MDP produces $k$6 and $k$7 for every candidate trajectory. The original system then forms a single cost per candidate through a log-sum weighted combination of imitation and metric scores and selects

$k$8

The weights $k$9 are tuned by grid search, typically with

$$y_i = \frac{\exp\bigl(-\|\hat T - T_i\|^2\bigr)} {\sum_{j=1}^k \exp\bigl(-\|\hat T - T_j\|^2\bigr)}.$$0

(Li et al., 2024).

The principal closed-loop evaluation metrics distilled in the original work are NC, DAC, TTC, C, and EP, all normalized to $$y_i = \frac{\exp\bigl(-\|\hat T - T_i\|^2\bigr)} {\sum_{j=1}^k \exp\bigl(-\|\hat T - T_j\|^2\bigr)}.$$1. The reported Navtest results are summarized below.

Method	NC	DAC	PDM-score
Vadv2-8192	97.2	89.1	80.9
Hydra-MDP-8192	97.9	91.7	83.0
Hydra-MDP-8192-W	98.1	96.1	85.7
Hydra-MDP-8192-W-EP	98.3	96.0	86.5

The same result set reports EP, TTC, and C as follows: Vadv2-8192 gives EP $$y_i = \frac{\exp\bigl(-\|\hat T - T_i\|^2\bigr)} {\sum_{j=1}^k \exp\bigl(-\|\hat T - T_j\|^2\bigr)}.$$2, TTC $$y_i = \frac{\exp\bigl(-\|\hat T - T_i\|^2\bigr)} {\sum_{j=1}^k \exp\bigl(-\|\hat T - T_j\|^2\bigr)}.$$3, C $$y_i = \frac{\exp\bigl(-\|\hat T - T_i\|^2\bigr)} {\sum_{j=1}^k \exp\bigl(-\|\hat T - T_j\|^2\bigr)}.$$4; Hydra-MDP-8192 gives EP $$y_i = \frac{\exp\bigl(-\|\hat T - T_i\|^2\bigr)} {\sum_{j=1}^k \exp\bigl(-\|\hat T - T_j\|^2\bigr)}.$$5, TTC $$y_i = \frac{\exp\bigl(-\|\hat T - T_i\|^2\bigr)} {\sum_{j=1}^k \exp\bigl(-\|\hat T - T_j\|^2\bigr)}.$$6, C $$y_i = \frac{\exp\bigl(-\|\hat T - T_i\|^2\bigr)} {\sum_{j=1}^k \exp\bigl(-\|\hat T - T_j\|^2\bigr)}.$$7; Hydra-MDP-8192-W gives EP $$y_i = \frac{\exp\bigl(-\|\hat T - T_i\|^2\bigr)} {\sum_{j=1}^k \exp\bigl(-\|\hat T - T_j\|^2\bigr)}.$$8, TTC $$y_i = \frac{\exp\bigl(-\|\hat T - T_i\|^2\bigr)} {\sum_{j=1}^k \exp\bigl(-\|\hat T - T_j\|^2\bigr)}.$$9, C $T_i$0; and Hydra-MDP-8192-W-EP gives EP $T_i$1, TTC $T_i$2, C $T_i$3 (Li et al., 2024). In this table, vocabulary size $T_i$4 gives the best base performance, the version without distillation scores $T_i$5, adding imitation plus hydra-distill reaches $T_i$6, adding weighted inference reaches $T_i$7, and finally including the EP head yields $T_i$8, which is reported as $T_i$9 place.

The ablations in the original exposition indicate that removing metric heads sharply degrades closed-loop collision and lane-compliance behavior; distilling the monolithic PDM score instead of per-metric targets worsens learning due to its irregular distribution; and weighted combination at inference is consistently better than unweighted (Li et al., 2024). Larger image backbones, specifically ViT-L and V2-99, yield a further $\hat P$0–$\hat P$1 points in final PDM score, up to $\hat P$2. Model ensembles, described as Mixture of Encoders plus sub-score ensembling, give a final Navtest PDM score of approximately $\hat P$3 (Li et al., 2024).

The implementation details reported for the original system are: 8 $\hat P$4 NVIDIA A100, batch size $\hat P$5, $\hat P$6 epochs, AdamW with learning rate $\hat P$7, weight decay $\hat P$8, default image encoder ResNet-34 at $\hat P$9, LiDAR encoder based on 4-frame BEV with ResNet-34, planning vocabulary $\{\hat{\mathcal S}_i^m\}_{m\in M}$0 or $\{\hat{\mathcal S}_i^m\}_{m\in M}$1, $\{\hat{\mathcal S}_i^m\}_{m\in M}$2, and no test-time augmentations (Li et al., 2024).

6. Hydra-MDP++ and the expansion of expert-guided distillation

Hydra-MDP++ extends the Hydra-MDP family by retaining the teacher-student knowledge distillation framework with a multi-head decoder while adding expanded evaluation metrics and a modified image-processing pipeline (Li et al., 17 Mar 2025). The ++ version processes raw images directly, without relying on privileged perception signals, and uses a lightweight ResNet-34 or a VoVNet-99 (“V2-99”) image encoder initialized from ImageNet pretraining. Its inputs are two consecutive frames, current plus one historical, from three stitched front-view cameras at $\{\hat{\mathcal S}_i^m\}_{m\in M}$3, together with current ego-status $\{\hat{\mathcal S}_i^m\}_{m\in M}$4, including speed, acceleration, and high-level navigation command.

A distinctive architectural addition is “Temporal Squeeze-and-Excitation” fusion. Historical features $\{\hat{\mathcal S}_i^m\}_{m\in M}$5 and current features $\{\hat{\mathcal S}_i^m\}_{m\in M}$6 are concatenated along the temporal dimension and passed through

$\{\hat{\mathcal S}_i^m\}_{m\in M}$7

The TemporalSE block performs channel-wise attention across the two-frame stack, followed by a $\{\hat{\mathcal S}_i^m\}_{m\in M}$8 convolution back to a fixed token dimension, with gradients on $\{\hat{\mathcal S}_i^m\}_{m\in M}$9 detached to speed convergence (Li et al., 17 Mar 2025).

Hydra-MDP++ also enlarges the rule-based teacher set. In addition to the NAVSIM-derived metrics $m$0, it adds Traffic-Light compliance (TL), Driving-Direction compliance (DDC), Lane-Keeping ability (LK), and Extended-Comfort (EC). TL is assigned by simulating $m$1 s ahead and checking stop-line crossing during a red phase. DDC is defined through projected segment motion onto the local lane centerline direction and a threshold $m$2. LK is defined through lateral deviation to nearby lanes with the same threshold $m$3. EC is defined by root-mean-square discrepancy between the student’s previous-step and current-step $m$4 s trajectories over acceleration, jerk, yaw-rate, and yaw-acceleration, with thresholds $m$5, $m$6, $m$7, and $m$8 (Li et al., 17 Mar 2025).

The extended score, EPDMS, is defined as

$m$9

The specified thresholds are $[0,1]$0 m, $[0,1]$1 m/s$[0,1]$2, $[0,1]$3 m/s$[0,1]$4, $[0,1]$5 rad/s, and $[0,1]$6 rad/s$[0,1]$7 (Li et al., 17 Mar 2025).

On Navtest, Hydra-MDP++ reports the following quantitative results.

Variant	PDMS	EPDMS	Latency
Hydra-MDP++ (ResNet-34)	86.6%	80.6%	206.2 ms
Hydra-MDP++ (V2-99)	91.0%	84.1%	271.0 ms

The corresponding original-metric breakdowns are NC $[0,1]$8, DAC $[0,1]$9, EP $\hat P$00, TTC $\hat P$01, C $\hat P$02 for the ResNet-34 variant, and NC $\hat P$03, DAC $\hat P$04, EP $\hat P$05, TTC $\hat P$06, C $\hat P$07 for the V2-99 variant. For extended metrics, the ResNet-34 variant gives TL $\hat P$08, DDC $\hat P$09, LK $\hat P$10, and EC $\hat P$11; the V2-99 variant gives TL $\hat P$12, DDC $\hat P$13, LK $\hat P$14, and EC $\hat P$15 (Li et al., 17 Mar 2025). The paper further states that the model runs at roughly $\hat P$16–$\hat P$17 Hz end-to-end.

Relative to the original Hydra-MDP, Hydra-MDP++ is described as preserving the trajectory vocabulary, transformer decoder, and multi-head distillation pattern while adding four new rule-based teachers, an expanded Extended-PDM score, and a grid-searched inference weighting scheme $\hat P$18 (Li et al., 17 Mar 2025). This suggests a family-level trajectory in which Hydra-MDP establishes the differentiable multi-teacher formulation and Hydra-MDP++ broadens its expert supervision to address unsafe behaviors not captured by the original NAVSIM-derived teachers.