---
title: 'Hydra-MDP: End-to-End Planning for Autonomous Driving'
url: https://www.emergentmind.com/topics/hydra-mdp
type: topic
---

# Hydra-MDP: End-to-End Planning for Autonomous Driving

Hydra-MDP is an end-to-end multimodal planning paradigm for autonomous driving that employs multiple teachers in a teacher-student model, combining knowledge distillation from human and rule-based teachers within a single differentiable framework [2406.06978]. The method is designed to unify multimodal planning, in which multiple candidate futures are represented explicitly, with multi-target learning over closed-loop metrics such as No-fault Collision, Drivable-Area Compliance, Time-to-Collision, Comfort, and Ego-Progress. Its central claim is that rule-based planning knowledge can be absorbed directly by the student network from raw sensor observations, rather than being injected only through non-differentiable post-processing. In the reported Navsim challenge results, this formulation achieved \(1^{st}\) place and showed improved generalization across diverse driving environments and conditions [2406.06978].

## 1. Problem setting and conceptual motivation

Hydra-MDP is motivated by a persistent mismatch between open-loop imitation quality and closed-loop driving performance. End-to-end neural planners trained by imitation learning can mimic logged human trajectories, yet often fail under closed-loop evaluation, including collision, lane-departure, and inefficient-driving behaviors [2406.06978]. Prior multimodal planners address trajectory uncertainty by predicting multiple candidate trajectories, but typically rely on non-differentiable, rule-based cost functions on imperfect perception outputs to select among them. In the Hydra-MDP formulation, that separation is treated as a limitation because it breaks gradient flow and discards intermediate information that could otherwise shape representation learning.

The framework therefore unifies two ideas. The first is multimodal planning over a fixed vocabulary of trajectory candidates. The second is multi-target learning over closed-loop metrics that are normally computed by a simulator or a rule-based evaluation stack. The resulting system is not only a trajectory predictor; it is also a metric predictor over the same candidate set, trained so that the perception backbone, transformer blocks, and decoder heads jointly internalize the consequences of each candidate trajectory under simulation [2406.06978].

A common misunderstanding is to view Hydra-MDP as a conventional multimodal planner with a separate rule-based selector appended at inference time. The defining distinction is that the rule-based knowledge is distilled during training into differentiable heads attached to the student. This suggests that the framework is intended less as a post hoc reranking mechanism than as an integrated planning-and-evaluation model.

## 2. Multi-teacher distillation formulation

Hydra-MDP uses two types of teachers: a human teacher and a rule-based teacher [2406.06978]. The human teacher accesses ground-truth perception \(\hat P\) and the logged expert trajectory \(\hat T\), and produces imitation targets over a fixed vocabulary of \(k\) trajectories. The target distribution is defined by a distance-based softmax,
\[
y_i = \frac{\exp\bigl(-\|\hat T - T_i\|^2\bigr)}
{\sum_{j=1}^k \exp\bigl(-\|\hat T - T_j\|^2\bigr)}.
\]
Rather than supervising a single regressed path, this formulation supervises a distribution over discrete candidate trajectories.

The rule-based teacher evaluates each candidate \(T_i\) in an offline simulator with perfect perception \(\hat P\), producing normalized binary or real-valued scores \(\{\hat{\mathcal S}_i^m\}_{m\in M}\) for a set of metrics \(m\). In the original Hydra-MDP exposition, the distilled Navsim sub-metrics are No-fault Collision (NC), Drivable-Area Compliance (DAC), Time-to-Collision (TTC), Comfort (C), and Ego-Progress (EP), all normalized to \([0,1]\) [2406.06978]. The student, by contrast, sees only raw sensor observations \(O\), specifically camera and LiDAR, and must learn both the imitation target and the metric predictions end-to-end.

This multi-teacher organization is central to the term “Hydra-Distillation.” Human supervision supplies expert behavioral priors, while simulator-derived rule-based supervision injects closed-loop safety and efficiency criteria. The paper also notes that DDC, Drivable-Lane-Deviation, was omitted in the original Hydra-MDP setup due to an implementation issue [2406.06978]. That omission is relevant because it delineates the precise scope of the original distilled target set.

## 3. Student architecture, trajectory vocabulary, and hydra heads

The Hydra-MDP student is organized around a perception backbone, a discrete planning vocabulary, and a multi-head decoder [2406.06978]. In the original system, images are cropped front and side views at resolution \(256\times 1024\), or up to \(512\times 2048\) when scaling, and encoded by ResNet-34, ViT-L, or V2-99 depending on the experiment. LiDAR uses 4 consecutive sweeps projected into BEV and encoded by ResNet-34. Late fusion through transformer cross-attention yields environmental tokens \(F_{env}\in\mathbb R^{N\times d}\).

Planning is performed over a pre-computed vocabulary \(\{T_i\}_{i=1}^k\) obtained by clustering 700K expert trajectories, where each trajectory consists of 40 \((x,y,\theta)\) waypoints. Each vocabulary center is embedded by an MLP, augmented with ego-state \(E\), and processed by a transformer encoder,
\[
V'_k = \mathrm{TransformerEnc}\bigl(\mathrm{MLP}(\mathcal V_k)\bigr) + E.
\]
The resulting queries are then fused with the environmental tokens by a transformer decoder,
\[
V''_k = \mathrm{TransformerDec}(Q=V'_k,\; KV=F_{env}).
\]

From each latent vector in \(V''_k\), the model spawns multiple prediction heads. The imitation head is a linear layer with softmax producing \(\mathcal S_i^{\mathrm{im}} = P(\text{choose }T_i)\). For each metric \(m\in M\), a binary-classification head outputs \(\mathcal S_i^m\), interpreted as the probability that trajectory \(T_i\) satisfies metric \(m\) [2406.06978]. The hydra metaphor refers to this shared trunk with multiple specialized heads scoring the same vocabulary element under different criteria.

A descriptive rendering of the original framework has three parts: a perception backbone that fuses LiDAR BEV and multi-view images into \(F_{env}\); planning queries, given by embedded K-means centers, that attend to \(F_{env}\) to produce \(V''_k\); and a set of heads consisting of the imitation head plus NC, DAC, TTC, C, and EP heads, followed by final trajectory selection by an arg-min over weighted log-scores [2406.06978].

## 4. Optimization objectives and differentiable integration of rule-based knowledge

The human-teacher signal is optimized through a cross-entropy imitation loss,
\[
\mathcal L_{\mathrm{im}}
= -\sum_{i=1}^k y_i \log \mathcal S_i^{\mathrm{im}}.
\]
The rule-based teacher signal is optimized through a binary cross-entropy loss per metric,
\[
\mathcal L_{\mathrm{kd}}^m
= -\sum_{i=1}^k
\Bigl[
\hat{\mathcal S}_i^m \log \mathcal S_i^m
+ (1-\hat{\mathcal S}_i^m)\log(1-\mathcal S_i^m)
\Bigr].
\]
The total training loss is
\[
\mathcal L
= \mathcal L_{\mathrm{im}}
+ \lambda \sum_{m\in M} \mathcal L_{\mathrm{kd}}^m,
\]
with \(\lambda\) set to \(1\) in most experiments [2406.06978].

The paper also presents an optional temperature-scaled distillation form for soft logits,
\[
p^t_{m,i} = \frac{\exp(z^t_{m,i}/\tau)}{\sum_j \exp(z^t_{m,j}/\tau)},
\qquad
p^s_{m,i} = \frac{\exp(z^s_{m,i}/\tau)}{\sum_j \exp(z^s_{m,j}/\tau)},
\]
with
\[
\mathcal L_{\mathrm{distill}}^m = -\sum_i p^t_{m,i}\log p^s_{m,i}.
\]
However, the exposition explicitly states that in the Hydra-MDP paper the simpler BCE form suffices [2406.06978].

The key methodological point is that each metric head is a differentiable neural module. As a result, gradients from \(\mathcal L_{\mathrm{kd}}^m\) propagate backward through the transformer and into the perception backbone, so the system avoids non-differentiable post-processing. The rule-based teacher thereby “tells” the student how each trajectory will fare under closed-loop simulation, and the student internalizes these cost signals directly. This is the mechanism by which rule-based planning knowledge is integrated into end-to-end learning.

## 5. Inference procedure, evaluation metrics, and empirical behavior

At inference time, Hydra-MDP produces \(\{\mathcal S_i^{\mathrm{im}}\}\) and \(\{\mathcal S_i^m\}\) for every candidate trajectory. The original system then forms a single cost per candidate through a log-sum weighted combination of imitation and metric scores and selects
\[
T^* = \arg\min_i \tilde f(T_i, O).
\]
The weights \(\{w_1,w_2,w_3,w_4\}\) are tuned by grid search, typically with
\[
w_1\in[0.01,0.1],\qquad
w_2,w_3\in[0.1,1],\qquad
w_4\in[1,10]
\]
[2406.06978].

The principal closed-loop evaluation metrics distilled in the original work are NC, DAC, TTC, C, and EP, all normalized to \([0,1]\). The reported Navtest results are summarized below.

| Method | NC | DAC | PDM-score |
|---|---:|---:|---:|
| Vadv2-8192 | 97.2 | 89.1 | 80.9 |
| Hydra-MDP-8192 | 97.9 | 91.7 | 83.0 |
| Hydra-MDP-8192-W | 98.1 | 96.1 | 85.7 |
| Hydra-MDP-8192-W-EP | 98.3 | 96.0 | 86.5 |

The same result set reports EP, TTC, and C as follows: Vadv2-8192 gives EP \(76.0\), TTC \(91.6\), C \(100\); Hydra-MDP-8192 gives EP \(77.6\), TTC \(92.9\), C \(100\); Hydra-MDP-8192-W gives EP \(77.8\), TTC \(93.9\), C \(100\); and Hydra-MDP-8192-W-EP gives EP \(78.7\), TTC \(94.6\), C \(100\) [2406.06978]. In this table, vocabulary size \(k=8192\) gives the best base performance, the version without distillation scores \(80.9\), adding imitation plus hydra-distill reaches \(83.0\), adding weighted inference reaches \(85.7\), and finally including the EP head yields \(86.5\), which is reported as \(1^{st}\) place.

The ablations in the original exposition indicate that removing metric heads sharply degrades closed-loop collision and lane-compliance behavior; distilling the monolithic PDM score instead of per-metric targets worsens learning due to its irregular distribution; and weighted combination at inference is consistently better than unweighted [2406.06978]. Larger image backbones, specifically ViT-L and V2-99, yield a further \(+4\)–\(5\) points in final PDM score, up to \(91.0\). Model ensembles, described as Mixture of Encoders plus sub-score ensembling, give a final Navtest PDM score of approximately \(91.0\) [2406.06978].

The implementation details reported for the original system are: 8 \(\times\) NVIDIA A100, batch size \(256\), \(20\) epochs, AdamW with learning rate \(1\times 10^{-4}\), weight decay \(0\), default image encoder ResNet-34 at \(256\times 1024\), LiDAR encoder based on 4-frame BEV with ResNet-34, planning vocabulary \(k=4096\) or \(8192\), \(\lambda_{\mathrm{kd}}=1\), and no test-time augmentations [2406.06978].

## 6. Hydra-MDP++ and the expansion of expert-guided distillation

Hydra-MDP++ extends the Hydra-MDP family by retaining the teacher-student knowledge distillation framework with a multi-head decoder while adding expanded evaluation metrics and a modified image-processing pipeline [2503.12820]. The ++ version processes raw images directly, without relying on privileged perception signals, and uses a lightweight ResNet-34 or a VoVNet-99 (“V2-99”) image encoder initialized from ImageNet pretraining. Its inputs are two consecutive frames, current plus one historical, from three stitched front-view cameras at \(256\times 1024\), together with current ego-status \(E\), including speed, acceleration, and high-level navigation command.

A distinctive architectural addition is “Temporal Squeeze-and-Excitation” fusion. Historical features \(F_{\mathrm{img}}^{pre}\) and current features \(F_{\mathrm{img}}^{cur}\) are concatenated along the temporal dimension and passed through
\[
F_{env} = \mathrm{Conv}\bigl(\mathrm{TemporalSE}([F_{\mathrm{img}}^{pre};F_{\mathrm{img}}^{cur}])\bigr).
\]
The TemporalSE block performs channel-wise attention across the two-frame stack, followed by a \(1\times 1\) convolution back to a fixed token dimension, with gradients on \(F_{\mathrm{img}}^{pre}\) detached to speed convergence [2503.12820].

Hydra-MDP++ also enlarges the rule-based teacher set. In addition to the NAVSIM-derived metrics \(\{NC, DAC, EP, TTC, C\}\), it adds Traffic-Light compliance (TL), Driving-Direction compliance (DDC), Lane-Keeping ability (LK), and Extended-Comfort (EC). TL is assigned by simulating \(4\) s ahead and checking stop-line crossing during a red phase. DDC is defined through projected segment motion onto the local lane centerline direction and a threshold \(\tau_D\). LK is defined through lateral deviation to nearby lanes with the same threshold \(\tau_D\). EC is defined by root-mean-square discrepancy between the student’s previous-step and current-step \(4\) s trajectories over acceleration, jerk, yaw-rate, and yaw-acceleration, with thresholds \(\tau_A\), \(\tau_J\), \(\tau_Y^R\), and \(\tau_Y^A\) [2503.12820].

The extended score, EPDMS, is defined as
\[
\mathrm{EPDMS}
=
\Bigl(\prod_{m\in\{NC,DAC,DDC,TL\}} S^m\Bigr)
\times
\frac{5S^{EP}+5S^{TTC}+2S^C+5S^{LK}+5S^{EC}}{22}.
\]
The specified thresholds are \(\tau_D=0.5\) m, \(\tau_A=0.7\) m/s\(^2\), \(\tau_J=0.5\) m/s\(^3\), \(\tau_Y^R=0.1\) rad/s, and \(\tau_Y^A=0.1\) rad/s\(^2\) [2503.12820].

On Navtest, Hydra-MDP++ reports the following quantitative results.

| Variant | PDMS | EPDMS | Latency |
|---|---:|---:|---:|
| Hydra-MDP++ (ResNet-34) | 86.6% | 80.6% | 206.2 ms |
| Hydra-MDP++ (V2-99) | 91.0% | 84.1% | 271.0 ms |

The corresponding original-metric breakdowns are NC \(97.6\%\), DAC \(96.0\%\), EP \(80.4\%\), TTC \(93.1\%\), C \(100\%\) for the ResNet-34 variant, and NC \(98.6\%\), DAC \(98.6\%\), EP \(85.7\%\), TTC \(95.1\%\), C \(100\%\) for the V2-99 variant. For extended metrics, the ResNet-34 variant gives TL \(100\%\), DDC \(98.9\%\), LK \(67.2\%\), and EC \(97.7\%\); the V2-99 variant gives TL \(100\%\), DDC \(99.1\%\), LK \(70.1\%\), and EC \(96.8\%\) [2503.12820]. The paper further states that the model runs at roughly \(4\)–\(5\) Hz end-to-end.

Relative to the original Hydra-MDP, Hydra-MDP++ is described as preserving the trajectory vocabulary, transformer decoder, and multi-head distillation pattern while adding four new rule-based teachers, an expanded Extended-PDM score, and a grid-searched inference weighting scheme \(\{k_{im},k_m,k_w\}\) [2503.12820]. This suggests a family-level trajectory in which Hydra-MDP establishes the differentiable multi-teacher formulation and Hydra-MDP++ broadens its expert supervision to address unsafe behaviors not captured by the original NAVSIM-derived teachers.

Source: https://www.emergentmind.com/topics/hydra-mdp