DeMo++: Unified Motion Estimation

• DeMo++ is a unified framework that decouples trajectory modeling into holistic motion intentions and fine spatiotemporal states.
• It leverages a hybrid Attention–Mamba architecture with cross-scene interaction and state-anchor refinement to achieve state-of-the-art performance.
• The approach spans motion forecasting, ego planning, and end-to-end sensor-based planning, enhancing multimodal diversity and temporal consistency.

DeMo++ is a unified motion estimation framework for autonomous driving that targets motion forecasting of surrounding agents, motion planning for the ego vehicle, and end-to-end planning from raw sensors. Its defining idea is to decouple trajectory modeling into two complementary components: holistic motion intentions, which represent diverse future directions, and fine spatiotemporal states, which represent how a trajectory evolves over time. DeMo++ couples these components through a hybrid Attention–Mamba architecture, augments them with cross-scene trajectory interaction and state anchor-based refinement, and reports state-of-the-art or top-tier results on Argoverse 2, nuScenes, nuPlan, and NAVSIM (Zhang et al., 23 Jul 2025).

1. Origins, scope, and conceptual position

DeMo++ was introduced as a successor to DeMo: Decoupling Motion Forecasting into Directional Intentions and Dynamic States, which had already argued that the prevalent one-query-one-trajectory paradigm is too coarse for representing future trajectory evolution (Zhang et al., 2024). In that earlier formulation, a single query was expected to encode both multimodal intention and detailed temporal evolution. DeMo++ preserves the same basic decoupling principle, but broadens it from motion forecasting to a unified framework spanning forecasting, planning, and end-to-end planning, and adds two explicit extensions: cross-scene trajectory interaction and state anchor-based refinement (Zhang et al., 23 Jul 2025).

The framework is motivated by a structural criticism of query-based motion prediction. In many DETR-like decoders, each learned query corresponds to one candidate future trajectory. That arrangement can represent diverse intentions, but it does not explicitly model intermediate dynamic states across future timesteps. DeMo++ therefore separates the representation of a future trajectory into a mode dimension and a state dimension. This separation is intended to let the model specialize one set of latent variables for multimodal direction selection and another set for temporally consistent state evolution.

The paper treats motion forecasting and planning as closely related motion estimation problems. In forecasting, the target is the future trajectory of traffic agents of interest. In planning, the target is the ego trajectory conditioned on scene context. In end-to-end planning, structured scene inputs are replaced by raw camera and LiDAR observations, but the model still infers scene context and predicts feasible future motion. DeMo++ is presented as a common representation and decoding framework across all three regimes.

2. Decoupled trajectory representation

The core representation consists of mode queries and state queries. Mode queries encode global motion alternatives, such as distinct directional intentions. State queries encode temporally ordered future-state representations. Their combination yields a hybrid spatiotemporal representation of each candidate trajectory (Zhang et al., 23 Jul 2025).

For structured-input forecasting, the paper defines the vectorized HD map as

$${\rm M} \in \mathbb{R}^{N_{\rm m} \times L \times C_{\rm m}},$$

agent histories as

$${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$

and target future trajectories as

$$A_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm m} \times 2}.$$

The state queries are

$$Q_{\rm s} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm s} \times C},$$

where $T_{\rm s}$ is the number of latent future state steps. These queries are initialized from explicit future timestamps: $$Q_{\rm s} = {\rm MLP}([t_1, t_2, \cdots, t_{T_{\rm s}}]).$$ This gives the state queries an explicit temporal semantics rather than treating them as generic learned slots.

The mode queries are

$$Q_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times K \times C},$$

where $K$ is the number of trajectory modes. Each mode query corresponds to one candidate trajectory hypothesis. After separate processing, the two sets are coupled into

$$Q_{\rm h} \in \mathbb{R}^{N_{\rm aoi} \times K \times T_{\rm s} \times C}.$$

The paper states that this coupling is performed by simply adding $Q_{\rm m}$ and $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$0, with broadcasting across the mode and time axes.

This decomposition yields a clear semantic split. The mode queries are responsible for multimodal diversity; the state queries are responsible for dynamic fidelity and temporal consistency. The hybrid queries then represent candidate trajectories as mode-conditioned state sequences rather than as one-shot trajectory tokens. The paper’s ablation analysis supports this reading: state-query outputs are stronger on single-trajectory fidelity, mode-query outputs are stronger on multimodal diversity, and final coupled outputs combine both advantages.

3. Hybrid Attention–Mamba architecture

For structured inputs, DeMo++ first encodes scene context and then processes state and mode queries with dedicated modules before coupling them. The map encoder is

$${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$1

the agent-history encoder is

$${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$2

and fused scene context is

$${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$3

PointNet is used for vectorized map polylines, UniMamba for causal historical motion encoding, and Transformer attention for global intra-scene interaction (Zhang et al., 23 Jul 2025).

The State Consistency Module processes state queries as

$${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$4

The sequence is therefore: timestamp-based initialization, cross-attention to scene context, and bidirectional Mamba over the temporal axis. An auxiliary decoder maps processed state queries to a single future trajectory, with supervision meant to force the state branch to encode temporally coherent motion.

The Mode Localization Module processes mode queries as

$${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$5

Cross-attention localizes mode queries in the scene, and self-attention among mode queries encourages diversity and coordination across candidate intentions.

After coupling,

$${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$6

the Hybrid Coupling Module applies

$${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$7

The paper attributes distinct roles to these operators: scene cross-attention injects context again; HybridMHA reasons jointly across time and modes; ModeMHA refines interactions among modes; BiMamba enforces temporal sequence structure over the coupled representation.

The architectural division of labor is explicit. Attention is used where global scene aggregation and nonlocal interaction dominate. Mamba is used where ordered state-sequence modeling is central. In the encoder, UniMamba is used because historical trajectories are causal. In the decoder, BiMamba is used because future latent states are jointly optimized and benefit from bidirectional sequence modeling. Ablations on Argoverse 2 validation show that Bi-Mamba gives the best state-sequence modeling result among the tested variants, with $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$8, $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$9, and $$A_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm m} \times 2}.$$0.

4. Cross-scene interaction and state-anchor refinement

DeMo++ extends the original DeMo formulation with two mechanisms intended to improve continuity and precision: cross-scene intention interaction and state anchor-based refinement (Zhang et al., 23 Jul 2025).

Cross-scene interaction is motivated by continuous driving. Standard benchmarks typically treat scenes independently, but deployed systems repeatedly forecast and plan over adjacent temporal windows. DeMo++ therefore reorganizes snapshot scenes into sequential sub-scenes using a sliding-window construction. The paper states that Argoverse 2 is reorganized into three continuous and evenly spaced sub-scenes, each with 3 s history and 6 s future, while nuPlan is reorganized into two continuous and evenly spaced sub-scenes, each with 1.5 s history and 8 s future.

For mode-query interaction across scenes, current and historical mode trajectories are first decoded and then aligned into a common frame. Historical trajectories are projected into the current frame as

$$A_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm m} \times 2}.$$1

where $$A_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm m} \times 2}.$$2 is the rotation from the historical frame to the current frame, and $$A_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm m} \times 2}.$$3 is the historical waypoint corresponding to the current time step. Current mode queries are then updated via

$$A_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm m} \times 2}.$$4

with $$A_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm m} \times 2}.$$5 an MLP embedding of flattened trajectories. The intended effect is temporally coherent intention modeling across adjacent scenes. The paper states that state queries are updated with historical features in a similar process, although it does not provide an explicit second formula.

The second extension is state anchor-based refinement. After proposal generation, each trajectory mode is refined independently using predicted state locations as anchors. The refinement uses distance-aware cross-attention between state queries and scene context, masking distant context elements so that each future state attends only to nearby relevant map and agent features. The exact attention formula, masking threshold, and weight function are not specified in the paper. For refined probability estimation, the model uses the endpoint state query of each trajectory mode and applies an MLP to predict refined probabilities.

Both extensions improve validation metrics. On Argoverse 2 validation, adding cross-scene intention interaction to the full DeMo configuration changes $$A_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm m} \times 2}.$$6 from $$A_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm m} \times 2}.$$7 to $$A_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm m} \times 2}.$$8, $$A_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm m} \times 2}.$$9 from $$Q_{\rm s} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm s} \times C},$$0 to $$Q_{\rm s} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm s} \times C},$$1, $$Q_{\rm s} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm s} \times C},$$2 from $$Q_{\rm s} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm s} \times C},$$3 to $$Q_{\rm s} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm s} \times C},$$4, $$Q_{\rm s} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm s} \times C},$$5 from $$Q_{\rm s} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm s} \times C},$$6 to $$Q_{\rm s} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm s} \times C},$$7, $$Q_{\rm s} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm s} \times C},$$8 from $$Q_{\rm s} \in \mathbb{R}^{N_{\rm aoi} \times T_{\rm s} \times C},$$9 to $T_{\rm s}$0, and $T_{\rm s}$1 from $T_{\rm s}$2 to $T_{\rm s}$3. Adding refinement instead changes the same metrics to $T_{\rm s}$4, $T_{\rm s}$5, $T_{\rm s}$6, $T_{\rm s}$7, $T_{\rm s}$8, and $T_{\rm s}$9. Using both together gives the best ablation result.

5. Training objectives, task adaptations, and implementation

For structured-input forecasting and planning, DeMo++ is trained end-to-end with a per-sub-scene objective

$$Q_{\rm s} = {\rm MLP}([t_1, t_2, \cdots, t_{T_{\rm s}}]).$$0

The paper states equal weighting among losses. Trajectory regression uses Smooth-L1, probability classification uses cross-entropy, and a winner-take-all strategy is used so that only the best prediction with minimal average error to ground truth is optimized (Zhang et al., 23 Jul 2025).

The state auxiliary loss is

$$Q_{\rm s} = {\rm MLP}([t_1, t_2, \cdots, t_{T_{\rm s}}]).$$1

and the mode auxiliary loss is

$$Q_{\rm s} = {\rm MLP}([t_1, t_2, \cdots, t_{T_{\rm s}}]).$$2

For cross-scene training with $$Q_{\rm s} = {\rm MLP}([t_1, t_2, \cdots, t_{T_{\rm s}}]).$$3 sub-scenes, the total loss is the sum over sub-scenes: $$Q_{\rm s} = {\rm MLP}([t_1, t_2, \cdots, t_{T_{\rm s}}]).$$4

The framework is adapted across three task families. In motion forecasting, it predicts multi-mode trajectories for agents of interest from vectorized maps and agent histories. In motion planning, the same decoder structure predicts candidate ego trajectories with scores. In end-to-end planning, the paper introduces DeMo-E2E++, which uses raw multi-view camera images $$Q_{\rm s} = {\rm MLP}([t_1, t_2, \cdots, t_{T_{\rm s}}]).$$5, LiDAR observations $$Q_{\rm s} = {\rm MLP}([t_1, t_2, \cdots, t_{T_{\rm s}}]).$$6, and ego status. These are fused into BEV features

$$Q_{\rm s} = {\rm MLP}([t_1, t_2, \cdots, t_{T_{\rm s}}]).$$7

with extracted agent features

$$Q_{\rm s} = {\rm MLP}([t_1, t_2, \cdots, t_{T_{\rm s}}]).$$8

and ego features

$$Q_{\rm s} = {\rm MLP}([t_1, t_2, \cdots, t_{T_{\rm s}}]).$$9

The decoder still uses $$Q_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times K \times C},$$0 and $$Q_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times K \times C},$$1, but cross-attends them to BEV, agent, and ego features, and adds deformable attention to adaptively sample BEV features after coupling. Cross-scene intention interaction is not used in DeMo-E2E++ because NAVSIM lacks sequential sensor information.

The end-to-end model predicts auxiliary tasks—BEV segmentation, surrounding-agent detection, and the final multi-mode plan—with total loss

$$Q_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times K \times C},$$2

The reported training settings are task-specific. For forecasting, the paper uses 60 epochs, AdamW, batch size 16 per GPU, learning rate $$Q_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times K \times C},$$3, weight decay $$Q_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times K \times C},$$4, dropout 0.2, cosine scheduling, and 10-epoch warmup. For planning, it uses 25 epochs, warmup 3 epochs, weight decay $$Q_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times K \times C},$$5, and otherwise the same settings. For NAVSIM end-to-end planning, it trains on navtrain for 100 epochs with batch size 16, AdamW, learning rate $$Q_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times K \times C},$$6, weight decay $$Q_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times K \times C},$$7, a ResNet-34 image backbone, and 20 planning modes. All experiments use 8 NVIDIA GeForce RTX 3090 GPUs.

6. Benchmarks, empirical performance, and limitations

DeMo++ is evaluated on Argoverse 2, nuScenes, nuPlan, and NAVSIM (Zhang et al., 23 Jul 2025). The reported top-line results are summarized below.

Benchmark	Variant	Selected reported results
Argoverse 2 test	DeMo++	$$Q_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times K \times C},$$8, $$Q_{\rm m} \in \mathbb{R}^{N_{\rm aoi} \times K \times C},$$9, $K$0, $K$1, $K$2, $K$3
nuScenes test	DeMo++	$K$4, $K$5, $K$6, $K$7, $K$8
nuPlan Test 14 Hard	DeMo++	$K$9, $$Q_{\rm h} \in \mathbb{R}^{N_{\rm aoi} \times K \times T_{\rm s} \times C}.$$0, $$Q_{\rm h} \in \mathbb{R}^{N_{\rm aoi} \times K \times T_{\rm s} \times C}.$$1
NAVSIM navtest	DeMo-E2E++	$$Q_{\rm h} \in \mathbb{R}^{N_{\rm aoi} \times K \times T_{\rm s} \times C}.$$2, $$Q_{\rm h} \in \mathbb{R}^{N_{\rm aoi} \times K \times T_{\rm s} \times C}.$$3, $$Q_{\rm h} \in \mathbb{R}^{N_{\rm aoi} \times K \times T_{\rm s} \times C}.$$4, $$Q_{\rm h} \in \mathbb{R}^{N_{\rm aoi} \times K \times T_{\rm s} \times C}.$$5, $$Q_{\rm h} \in \mathbb{R}^{N_{\rm aoi} \times K \times T_{\rm s} \times C}.$$6, $$Q_{\rm h} \in \mathbb{R}^{N_{\rm aoi} \times K \times T_{\rm s} \times C}.$$7

On Argoverse 2, DeMo++ is best on $$Q_{\rm h} \in \mathbb{R}^{N_{\rm aoi} \times K \times T_{\rm s} \times C}.$$8, $$Q_{\rm h} \in \mathbb{R}^{N_{\rm aoi} \times K \times T_{\rm s} \times C}.$$9, $Q_{\rm m}$0, and $Q_{\rm m}$1, and tied best on $Q_{\rm m}$2. Relative to DeMo, it improves $Q_{\rm m}$3 from $Q_{\rm m}$4 to $Q_{\rm m}$5, $Q_{\rm m}$6 from $Q_{\rm m}$7 to $Q_{\rm m}$8, $Q_{\rm m}$9 from $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$00 to $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$01, and $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$02 from $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$03 to $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$04. On nuScenes, it improves all reported metrics over DeMo, moving from $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$05 to $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$06 on $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$07, from $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$08 to $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$09 on $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$10, from $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$11 to $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$12 on $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$13, from $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$14 to $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$15 on $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$16, and from $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$17 to $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$18 on $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$19. On nuPlan, it achieves the best open-loop score and matches the best reactive closed-loop score. On NAVSIM, DeMo-E2E++ reports the strongest PDM Score among the listed methods. The paper defines PDM Score as

$${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$20

The ablation studies provide the main interpretive evidence. Simply adding state queries without proper decoupling and training support degrades performance: in the Argoverse 2 validation ablation, ID-2 is worse than the plain baseline. Decoupling plus auxiliary losses helps only modestly, whereas the aggregation modules are critical. Full DeMo corresponds to a substantial jump, and the added DeMo++ mechanisms—cross-scene interaction and refinement—each produce further gains, with the best result obtained when both are present. This suggests that the paper’s improvements are not attributable to query proliferation alone; they depend on the semantic split, the dedicated modules, and the added continuity and refinement mechanisms.

The paper also reports that increasing the number of state queries improves performance: 60 state queries gives the best reported Argoverse 2 validation result among the tested values $${\rm A} \in \mathbb{R}^{N_{\rm a} \times T_{\rm h} \times C_{\rm a}},$$21. A plausible implication is that fine-grained temporal semantics are useful, but they come with a computational cost. The paper explicitly notes this as a limitation: the decoupled query paradigm can make models heavier when predicting long trajectories, and efficiency is not yet fully optimized. It suggests sparse-state modeling as a future direction.

The reported failure cases fall into two categories. One concerns subjective behaviors, such as a vehicle unexpectedly turning into an alley; the authors suggest that signals of driver intent, such as turn indicators, may be necessary. The other concerns complex intersections with multiple route options, where failures are attributed to incomplete understanding of complex map topology and data imbalance. These limitations indicate that the decoupled representation improves multimodal diversity and temporal consistency, but does not eliminate ambiguity arising from missing intent signals or difficult topology.

A recurring source of confusion is nomenclature. DeMo++ in this sense is the autonomous-driving framework introduced in 2025 (Zhang et al., 23 Jul 2025). It is preceded by the motion-forecasting method DeMo (Zhang et al., 2024), but it is unrelated to DeMo: Decoupled Momentum Optimization, which is a distributed training optimizer (Peng et al., 2024), and to FlexDeMo, which adapts that optimizer family to hybrid-sharded FSDP training (From et al., 10 Feb 2025). The shared acronym does not imply a shared technical lineage across those domains.