---
title: 'DeMo++: Unified Motion Estimation'
url: https://www.emergentmind.com/topics/demo
type: topic
---

# DeMo++: Unified Motion Estimation

DeMo++ is a unified motion estimation framework for autonomous driving that targets motion forecasting of surrounding agents, motion planning for the ego vehicle, and end-to-end planning from raw sensors. Its defining idea is to decouple trajectory modeling into two complementary components: **holistic motion intentions**, which represent diverse future directions, and **fine spatiotemporal states**, which represent how a trajectory evolves over time. DeMo++ couples these components through a hybrid Attention–Mamba architecture, augments them with cross-scene trajectory interaction and state anchor-based refinement, and reports state-of-the-art or top-tier results on Argoverse 2, nuScenes, nuPlan, and NAVSIM [2507.17342].

## 1. Origins, scope, and conceptual position

DeMo++ was introduced as a successor to **DeMo: Decoupling Motion Forecasting into Directional Intentions and Dynamic States**, which had already argued that the prevalent one-query-one-trajectory paradigm is too coarse for representing future trajectory evolution [2410.05982]. In that earlier formulation, a single query was expected to encode both multimodal intention and detailed temporal evolution. DeMo++ preserves the same basic decoupling principle, but broadens it from motion forecasting to a unified framework spanning forecasting, planning, and end-to-end planning, and adds two explicit extensions: **cross-scene trajectory interaction** and **state anchor-based refinement** [2507.17342].

The framework is motivated by a structural criticism of query-based motion prediction. In many DETR-like decoders, each learned query corresponds to one candidate future trajectory. That arrangement can represent diverse intentions, but it does not explicitly model intermediate dynamic states across future timesteps. DeMo++ therefore separates the representation of a future trajectory into a **mode dimension** and a **state dimension**. This separation is intended to let the model specialize one set of latent variables for multimodal direction selection and another set for temporally consistent state evolution.

The paper treats motion forecasting and planning as closely related motion estimation problems. In forecasting, the target is the future trajectory of traffic agents of interest. In planning, the target is the ego trajectory conditioned on scene context. In end-to-end planning, structured scene inputs are replaced by raw camera and LiDAR observations, but the model still infers scene context and predicts feasible future motion. DeMo++ is presented as a common representation and decoding framework across all three regimes.

## 2. Decoupled trajectory representation

The core representation consists of **mode queries** and **state queries**. Mode queries encode global motion alternatives, such as distinct directional intentions. State queries encode temporally ordered future-state representations. Their combination yields a hybrid spatiotemporal representation of each candidate trajectory [2507.17342].

For structured-input forecasting, the paper defines the vectorized HD map as
\[
{\rm M} \in \mathbb{R}^{N_{\rm m} \times L \times C_{\rm m}},
\]
agent histories as
\[
{\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},
\]
and target future trajectories as
\[
A_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm m} \times 2}.
\]

The state queries are
\[
Q_{\rm s} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm s} \times C},
\]
where \(T_{\rm s}\) is the number of latent future state steps. These queries are initialized from explicit future timestamps:
\[
Q_{\rm s} = {\rm MLP}([t_1, t_2, \cdots, t_{T_{\rm s}}]).
\]
This gives the state queries an explicit temporal semantics rather than treating them as generic learned slots.

The mode queries are
\[
Q_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times K \times C},
\]
where \(K\) is the number of trajectory modes. Each mode query corresponds to one candidate trajectory hypothesis. After separate processing, the two sets are coupled into
\[
Q_{\rm h} \in \mathbb{R}^{N_{\rm aoi} \times K \times T_{\rm s} \times C}.
\]
The paper states that this coupling is performed by simply adding \(Q_{\rm m}\) and \(Q_{\rm s}\), with broadcasting across the mode and time axes.

This decomposition yields a clear semantic split. The mode queries are responsible for multimodal diversity; the state queries are responsible for dynamic fidelity and temporal consistency. The hybrid queries then represent candidate trajectories as mode-conditioned state sequences rather than as one-shot trajectory tokens. The paper’s ablation analysis supports this reading: state-query outputs are stronger on single-trajectory fidelity, mode-query outputs are stronger on multimodal diversity, and final coupled outputs combine both advantages.

## 3. Hybrid Attention–Mamba architecture

For structured inputs, DeMo++ first encodes scene context and then processes state and mode queries with dedicated modules before coupling them. The map encoder is
\[
F_{\rm m} = {\rm PointNet}({\rm M}),
\]
the agent-history encoder is
\[
F_{\rm a} = {\rm UniMamba}({\rm A}),
\]
and fused scene context is
\[
F_{\rm s} = {\rm Transformer}({\rm Concat}(F_{\rm a},\, F_{\rm m})).
\]
PointNet is used for vectorized map polylines, UniMamba for causal historical motion encoding, and Transformer attention for global intra-scene interaction [2507.17342].

The **State Consistency Module** processes state queries as
\[
\begin{split}
Q_{\rm s} &= {\rm MLP}([t_{1}, t_{2}, \cdots, t_{T_{\rm s}}]), \\
Q_{\rm s} &= {\rm MHA}({\rm Q} = Q_{\rm s}, {\rm K} = F_{\rm s}, {\rm V} = F_{\rm s}), \\
Q_{\rm s} &= {\rm BiMamba}(Q_{\rm s}).
\end{split}
\]
The sequence is therefore: timestamp-based initialization, cross-attention to scene context, and bidirectional Mamba over the temporal axis. An auxiliary decoder maps processed state queries to a single future trajectory, with supervision meant to force the state branch to encode temporally coherent motion.

The **Mode Localization Module** processes mode queries as
\[
\begin{split}
Q_{\rm m} &= {\rm MHA}({\rm Q} = Q_{\rm m}, {\rm K} = F_{\rm s}, {\rm V} = F_{\rm s}), \\
Q_{\rm m} &= {\rm MHA}({\rm Q} = Q_{\rm m}, {\rm K} = Q_{\rm m}, {\rm V} = Q_{\rm m}).
\end{split}
\]
Cross-attention localizes mode queries in the scene, and self-attention among mode queries encourages diversity and coordination across candidate intentions.

After coupling,
\[
Q_{\rm h} = Q_{\rm m} + Q_{\rm s},
\]
the **Hybrid Coupling Module** applies
\[
\begin{split}
Q_{\rm h} &= {\rm MHA}({\rm Q} = Q_{\rm h}, {\rm K} = F_{\rm s}, {\rm V} = F_{\rm s}), \\
Q_{\rm h} &= {\rm HybridMHA}({\rm Q} = Q_{\rm h}, {\rm K} = Q_{\rm h}, {\rm V} = Q_{\rm h}), \\
Q_{\rm h} &= {\rm ModeMHA}({\rm Q} = Q_{\rm h}, {\rm K} = Q_{\rm h}, {\rm V} = Q_{\rm h}), \\
Q_{\rm h} &= {\rm BiMamba}(Q_{\rm h}).
\end{split}
\]
The paper attributes distinct roles to these operators: scene cross-attention injects context again; HybridMHA reasons jointly across time and modes; ModeMHA refines interactions among modes; BiMamba enforces temporal sequence structure over the coupled representation.

The architectural division of labor is explicit. Attention is used where global scene aggregation and nonlocal interaction dominate. Mamba is used where ordered state-sequence modeling is central. In the encoder, UniMamba is used because historical trajectories are causal. In the decoder, BiMamba is used because future latent states are jointly optimized and benefit from bidirectional sequence modeling. Ablations on Argoverse 2 validation show that **Bi-Mamba** gives the best state-sequence modeling result among the tested variants, with \(minFDE_6 = 1.268\), \(minADE_6 = 0.674\), and \(MR_6 = 0.152\).

## 4. Cross-scene interaction and state-anchor refinement

DeMo++ extends the original DeMo formulation with two mechanisms intended to improve continuity and precision: **cross-scene intention interaction** and **state anchor-based refinement** [2507.17342].

Cross-scene interaction is motivated by continuous driving. Standard benchmarks typically treat scenes independently, but deployed systems repeatedly forecast and plan over adjacent temporal windows. DeMo++ therefore reorganizes snapshot scenes into sequential **sub-scenes** using a sliding-window construction. The paper states that **Argoverse 2** is reorganized into **three** continuous and evenly spaced sub-scenes, each with **3 s history** and **6 s future**, while **nuPlan** is reorganized into **two** continuous and evenly spaced sub-scenes, each with **1.5 s history** and **8 s future**.

For mode-query interaction across scenes, current and historical mode trajectories are first decoded and then aligned into a common frame. Historical trajectories are projected into the current frame as
\[
Y'_{\rm m} = \mathcal{R}\cdot(Y'_{\rm m} - {y'}_{\rm m}^{\rm ori})^{\rm T},
\]
where \(\mathcal{R}\) is the rotation from the historical frame to the current frame, and \({y'}_{\rm m}^{\rm ori}\) is the historical waypoint corresponding to the current time step. Current mode queries are then updated via
\[
Q_{\rm m} = {\rm Transformer}(Q_{\rm m} + {\rm TE}(Y_{\rm m}), \; Q'_{\rm m} + {\rm TE}(Y'_{\rm m})),
\]
with \({\rm TE}(\cdot)\) an MLP embedding of flattened trajectories. The intended effect is temporally coherent intention modeling across adjacent scenes. The paper states that state queries are updated with historical features in a similar process, although it does not provide an explicit second formula.

The second extension is **state anchor-based refinement**. After proposal generation, each trajectory mode is refined independently using predicted state locations as anchors. The refinement uses distance-aware cross-attention between state queries and scene context, masking distant context elements so that each future state attends only to nearby relevant map and agent features. The exact attention formula, masking threshold, and weight function are not specified in the paper. For refined probability estimation, the model uses the endpoint state query of each trajectory mode and applies an MLP to predict refined probabilities.

Both extensions improve validation metrics. On Argoverse 2 validation, adding cross-scene intention interaction to the full DeMo configuration changes \(minFDE_1\) from \(3.917\) to \(3.839\), \(minADE_1\) from \(1.609\) to \(1.550\), \(minFDE_6\) from \(1.268\) to \(1.204\), \(minADE_6\) from \(0.674\) to \(0.637\), \(MR_6\) from \(0.152\) to \(0.139\), and \(b\text{-}minFDE_6\) from \(1.918\) to \(1.832\). Adding refinement instead changes the same metrics to \(3.856\), \(1.568\), \(1.230\), \(0.644\), \(0.148\), and \(1.856\). Using both together gives the best ablation result.

## 5. Training objectives, task adaptations, and implementation

For structured-input forecasting and planning, DeMo++ is trained end-to-end with a per-sub-scene objective
\[
\mathcal{L}_{\rm sub} = \mathcal{L}_{\rm prop} + \mathcal{L}_{\rm ref} + \mathcal{L}_{\rm ts} + \mathcal{L}_{\rm m}.
\]
The paper states equal weighting among losses. Trajectory regression uses **Smooth-L1**, probability classification uses **cross-entropy**, and a **winner-take-all** strategy is used so that only the best prediction with minimal average error to ground truth is optimized [2507.17342].

The state auxiliary loss is
\[
\mathcal{L}_{\rm ts} = {\rm SmoothL1}(Y_{\rm ts}, Y_{\rm gt}),
\]
and the mode auxiliary loss is
\[
\begin{split}
Y_{\rm best},\, P_{\rm best} &= {\rm SelectBest}(Y_{\rm m}, Y_{\rm gt}), \\
\mathcal{L}_{\rm m} &= {\rm SmoothL1}(Y_{\rm best}, Y_{\rm gt}) + {\rm CE}(P_{\rm m}, P_{\rm best}).
\end{split}
\]
For cross-scene training with \(N_{\rm sub}\) sub-scenes, the total loss is the sum over sub-scenes:
\[
\mathcal{L} = \mathcal{L}_{sub}^{1}+...+\mathcal{L}_{sub}^{N_{\rm sub}}.
\]

The framework is adapted across three task families. In **motion forecasting**, it predicts multi-mode trajectories for agents of interest from vectorized maps and agent histories. In **motion planning**, the same decoder structure predicts candidate ego trajectories with scores. In **end-to-end planning**, the paper introduces **DeMo-E2E++**, which uses raw multi-view camera images \(\mathcal{I}\), LiDAR observations \(\mathcal{P}\), and ego status. These are fused into BEV features
\[
F_{\rm bev} \in \mathbb{R}^{H \times W \times C},
\]
with extracted agent features
\[
F_{\rm agent} \in \mathbb{R}^{N_{\rm agent} \times C},
\]
and ego features
\[
F_{\rm ego} \in \mathbb{R}^{1 \times C}.
\]
The decoder still uses \(Q_{\rm m}\) and \(Q_{\rm s}\), but cross-attends them to BEV, agent, and ego features, and adds **deformable attention** to adaptively sample BEV features after coupling. Cross-scene intention interaction is not used in DeMo-E2E++ because NAVSIM lacks sequential sensor information.

The end-to-end model predicts auxiliary tasks—BEV segmentation, surrounding-agent detection, and the final multi-mode plan—with total loss
\[
\mathcal{L} = \mathcal{L}_{\rm bev} + \mathcal{L}_{\rm agent} + \mathcal{L}_{\rm ts} + \mathcal{L}_{\rm m} + \mathcal{L}_{\rm final}.
\]

The reported training settings are task-specific. For forecasting, the paper uses **60 epochs**, **AdamW**, **batch size 16 per GPU**, **learning rate \(3 \times 10^{-3}\)**, **weight decay \(1 \times 10^{-2}\)**, **dropout 0.2**, cosine scheduling, and **10-epoch warmup**. For planning, it uses **25 epochs**, **warmup 3 epochs**, **weight decay \(1 \times 10^{-4}\)**, and otherwise the same settings. For NAVSIM end-to-end planning, it trains on **navtrain for 100 epochs** with **batch size 16**, **AdamW**, **learning rate \(1 \times 10^{-4}\)**, **weight decay \(1 \times 10^{-4}\)**, a **ResNet-34** image backbone, and **20 planning modes**. All experiments use **8 NVIDIA GeForce RTX 3090 GPUs**.

## 6. Benchmarks, empirical performance, and limitations

DeMo++ is evaluated on **Argoverse 2**, **nuScenes**, **nuPlan**, and **NAVSIM** [2507.17342]. The reported top-line results are summarized below.

| Benchmark | Variant | Selected reported results |
|---|---|---|
| Argoverse 2 test | DeMo++ | \(minFDE_1=3.70\), \(minADE_1=1.50\), \(minFDE_6=1.12\), \(minADE_6=0.61\), \(MR_6=0.12\), \(b\text{-}minFDE_6=1.74\) |
| nuScenes test | DeMo++ | \(minFDE_1=6.33\), \(minADE_5=1.18\), \(minADE_{10}=0.87\), \(MR_5=0.40\), \(MR_{10}=0.33\) |
| nuPlan Test 14 Hard | DeMo++ | \(OLS=0.88\), \(NR\text{-}CLS=0.76\), \(R\text{-}CLS=0.69\) |
| NAVSIM navtest | DeMo-E2E++ | \(NC=98.4\), \(DAC=97.9\), \(TTC=95.1\), \(Comfort=100\), \(EP=84.2\), \(PDM\ Score=89.9\) |

On **Argoverse 2**, DeMo++ is best on \(minFDE_1\), \(minFDE_6\), \(MR_6\), and \(b\text{-}minFDE_6\), and tied best on \(minADE_6\). Relative to DeMo, it improves \(minFDE_1\) from \(3.74\) to \(3.70\), \(minFDE_6\) from \(1.17\) to \(1.12\), \(MR_6\) from \(0.13\) to \(0.12\), and \(b\text{-}minFDE_6\) from \(1.84\) to \(1.74\). On **nuScenes**, it improves all reported metrics over DeMo, moving from \(6.60\) to \(6.33\) on \(minFDE_1\), from \(1.22\) to \(1.18\) on \(minADE_5\), from \(0.89\) to \(0.87\) on \(minADE_{10}\), from \(0.43\) to \(0.40\) on \(MR_5\), and from \(0.34\) to \(0.33\) on \(MR_{10}\). On **nuPlan**, it achieves the best open-loop score and matches the best reactive closed-loop score. On **NAVSIM**, DeMo-E2E++ reports the strongest PDM Score among the listed methods. The paper defines PDM Score as
\[
\begin{aligned}
S_{\rm PDM} = &\ S_{\rm NC} \times S_{\rm DAC} \times \\
&\ \left( \frac{5 \times S_{\rm EP} + 5 \times S_{\rm TTC} + 2 \times S_{\rm CF}}{12} \right).
\end{aligned}
\]

The ablation studies provide the main interpretive evidence. Simply adding state queries without proper decoupling and training support degrades performance: in the Argoverse 2 validation ablation, ID-2 is worse than the plain baseline. Decoupling plus auxiliary losses helps only modestly, whereas the aggregation modules are critical. Full DeMo corresponds to a substantial jump, and the added DeMo++ mechanisms—cross-scene interaction and refinement—each produce further gains, with the best result obtained when both are present. This suggests that the paper’s improvements are not attributable to query proliferation alone; they depend on the semantic split, the dedicated modules, and the added continuity and refinement mechanisms.

The paper also reports that increasing the number of state queries improves performance: **60** state queries gives the best reported Argoverse 2 validation result among the tested values \(10, 20, 30, 60\). A plausible implication is that fine-grained temporal semantics are useful, but they come with a computational cost. The paper explicitly notes this as a limitation: the decoupled query paradigm can make models heavier when predicting long trajectories, and efficiency is not yet fully optimized. It suggests **sparse-state modeling** as a future direction.

The reported failure cases fall into two categories. One concerns **subjective behaviors**, such as a vehicle unexpectedly turning into an alley; the authors suggest that signals of driver intent, such as turn indicators, may be necessary. The other concerns **complex intersections with multiple route options**, where failures are attributed to incomplete understanding of complex map topology and data imbalance. These limitations indicate that the decoupled representation improves multimodal diversity and temporal consistency, but does not eliminate ambiguity arising from missing intent signals or difficult topology.

A recurring source of confusion is nomenclature. DeMo++ in this sense is the autonomous-driving framework introduced in 2025 [2507.17342]. It is preceded by the motion-forecasting method DeMo [2410.05982], but it is unrelated to **DeMo: Decoupled Momentum Optimization**, which is a distributed training optimizer [2411.19870], and to **FlexDeMo**, which adapts that optimizer family to hybrid-sharded FSDP training [2502.06728]. The shared acronym does not imply a shared technical lineage across those domains.

Source: https://www.emergentmind.com/topics/demo