MMTransformer: Multimodal Motion Prediction

• MMTransformer is a transformer-based architecture designed for multimodal motion prediction in autonomous driving, leveraging stacked decoder modules with proposal-specific embeddings.
• It integrates heterogeneous scene context—motion history, HD map structure, and social interactions—to generate multiple plausible future trajectories with competitive accuracy metrics.
• The model employs a region-based training strategy (RTS) to enforce diversity among proposals, mitigating mode collapse and enhancing trajectory forecasting reliability.

mmTransformer is a fully transformer-based architecture for multimodal motion prediction in autonomous driving. Introduced in "Multimodal Motion Prediction with Stacked Transformers," it addresses the need to predict multiple plausible future trajectories for nearby vehicles by modeling multimodality at the feature level through stacked transformer decoders queried by fixed independent proposals, and by enforcing diversity through a region-based training strategy (RTS). The model ingests heterogeneous scene context—motion history, HD map structure, and social interactions—and outputs $K$ trajectory proposals with confidences; on Argoverse it reports state-of-the-art miss rate with competitive minADE and minFDE (Liu et al., 2021).

1. Problem setting and conceptual basis

Motion forecasting for autonomous driving is intrinsically multimodal. Intersections, merges, and lane changes admit several plausible futures, and single-mode predictors tend to average these futures, producing unsafe or unrealistic trajectories. The paper situates prior methods in two broad families. Latent-variable or probabilistic approaches can suffer optimization instability and latent feature collapse toward frequent modes. Proposal-based approaches narrow the search space, but rely heavily on hand-crafted anchors or candidate generation and do not guarantee diverse outputs when training data provides only one ground-truth future per scene (Liu et al., 2021).

mmTransformer is designed to address both limitations simultaneously. Its central claim is that multimodality should be represented not only at the output level, where multiple trajectories are produced, but also at the feature level, where proposal-specific latent states are maintained throughout context aggregation. This is implemented through stacked transformer modules whose decoder queries are learned proposal embeddings. A complementary external mechanism then maps those proposal features into trajectories and confidences.

This design suggests a two-level interpretation of modality. Internally, each proposal is intended to carry disentangled modality-specific features via proposal-specific positional encodings and divergent attention patterns. Externally, the same proposals are decoded into distinct trajectories and scores. RTS then acts as the explicit diversity-inducing mechanism that resolves the conflict between a single observed future and inherently multimodal output space.

2. Scene representation and stacked transformer backbone

The model operates in an agent-centric coordinate frame. Each scene is normalized by translating to the target agent’s last observed position and rotating so that its heading aligns with the $+y$ axis. Map features are cropped to a local $65\,\mathrm{m} \times 65\,\mathrm{m}$ region centered on the target agent. The observed past trajectories are represented as

$$H = \{h_i \in \mathbb{R}^{T_{\mathrm{obs}}\times 2} : 1 \le i \le N_{\mathrm{vehicle}}\},$$

with $T_{\mathrm{obs}} = 2\,\mathrm{s}$ in Argoverse. Lane geometry is encoded by vectorized centerline-based polylines, processed by a subgraph module in the style of VectorNet; both trajectories and lane centerlines are treated as token sequences with learned positional encodings (Liu et al., 2021).

The backbone contains three sequential encoder–decoder transformer modules. All transformer modules use $d_{\mathrm{model}} = 128$, the proposal feature dimension is $n = 128$, the motion extractor and map aggregator each use 2 encoder and 2 decoder layers, and the social constructor decoder uses 4 layers.

Module	Encoder input	Decoder queries / role
Motion extractor	Past positions for each observed vehicle	$K$ learned proposal embeddings refined by motion history
Map aggregator	Vectorized HD-map polyline embeddings	Motion-refined proposals updated with map topology and semantics
Social constructor	Vehicle-level features aggregated from all vehicles	Target-agent proposals updated via social context

The information flow is hierarchical: motion $\rightarrow$ map $\rightarrow$ social. First, each observed vehicle’s motion history is encoded. Second, the refined proposal features attend to local map structure. Third, proposals from all vehicles are summarized through an MLP into vehicle-level descriptors, and only the target agent’s proposals are further updated against this social memory. The paper reports that this order best aligns feature composition.

Within each transformer block, attention uses the standard formulation

$+y$0

$+y$1

$+y$2

The specialized constructions differ by module: in the motion extractor, proposal embeddings query encoded motion history; in the map aggregator, motion-refined proposals query encoded map tokens; in the social constructor, target-agent proposals query encoded vehicle-level social features. Learned positional encodings are added to keys and queries in all attention layers. Crucially, learned proposal positional encodings are added before each decoder layer to preserve proposal identity and enforce orthogonality, with the stated goal of preventing attention-induced collapse across proposals (Liu et al., 2021).

3. Proposal parameterization and region-based training

mmTransformer uses $+y$3 fixed proposals, initialized randomly as embeddings and learned end-to-end. These proposals are permutation-invariant in the decoder, but proposal-specific positional encodings and separate output heads are used to encourage specialization. The proposal feature decoder has two branches. For each proposal feature $+y$4, the trajectory generator predicts

$+y$5

using a 3-layer MLP, and the trajectory selector predicts a confidence

$+y$6

using an MLP with transformer-style FFN structure (Liu et al., 2021).

RTS is the mechanism that explicitly induces multimodality. After normalization into the agent-centric frame, the output plane is partitioned into $+y$7 disjoint regions, either by constrained K-means over normalized ground-truth endpoints or by manual fan-shaped sectors. The $+y$8 proposals are equally divided into $+y$9 groups, so that each region is assigned $65\,\mathrm{m} \times 65\,\mathrm{m}$0 proposals. During training, only proposals assigned to the ground-truth region contribute to the loss; proposals assigned to other regions are not penalized. The stated purpose is to preserve modality-specific features in regions not occupied by the observed future.

The loss has three components. For proposals in the ground-truth region, the regression term is a Huber loss over per-step coordinate offsets,

$65\,\mathrm{m} \times 65\,\mathrm{m}$1

Confidence learning uses a KL divergence between a predicted distribution over proposals,

$65\,\mathrm{m} \times 65\,\mathrm{m}$2

and a target distribution based on endpoint distance

$65\,\mathrm{m} \times 65\,\mathrm{m}$3

$65\,\mathrm{m} \times 65\,\mathrm{m}$4

with

$65\,\mathrm{m} \times 65\,\mathrm{m}$5

An auxiliary region-classification cross-entropy loss is added to stabilize learning at large $65\,\mathrm{m} \times 65\,\mathrm{m}$6. The total objective uses uncertainty-based multi-task weighting:

$65\,\mathrm{m} \times 65\,\mathrm{m}$7

Intermediate supervision is attached after each decoder layer of the social constructor in order to accelerate training (Liu et al., 2021).

4. Inference procedure and evaluation methodology

At inference time, the model normalizes the scene, encodes motion history, map polylines, and social context, decodes the $65\,\mathrm{m} \times 65\,\mathrm{m}$8 proposal embeddings through the stacked transformers, and produces $65\,\mathrm{m} \times 65\,\mathrm{m}$9 trajectories and $$H = \{h_i \in \mathbb{R}^{T_{\mathrm{obs}}\times 2} : 1 \le i \le N_{\mathrm{vehicle}}\},$$0 confidences. A diversity-preserving, NMS-like post-processing stage filters near-duplicate trajectories using endpoint Euclidean distance thresholds and selects the top-$$H = \{h_i \in \mathbb{R}^{T_{\mathrm{obs}}\times 2} : 1 \le i \le N_{\mathrm{vehicle}}\},$$1 outputs; Argoverse reports $$H = \{h_i \in \mathbb{R}^{T_{\mathrm{obs}}\times 2} : 1 \le i \le N_{\mathrm{vehicle}}\},$$2 (Liu et al., 2021).

Evaluation uses minADE@K, minFDE@K, and miss rate. Given ground-truth trajectory $$H = \{h_i \in \mathbb{R}^{T_{\mathrm{obs}}\times 2} : 1 \le i \le N_{\mathrm{vehicle}}\},$$3 and predictions $$H = \{h_i \in \mathbb{R}^{T_{\mathrm{obs}}\times 2} : 1 \le i \le N_{\mathrm{vehicle}}\},$$4,

$$H = \{h_i \in \mathbb{R}^{T_{\mathrm{obs}}\times 2} : 1 \le i \le N_{\mathrm{vehicle}}\},$$5

$$H = \{h_i \in \mathbb{R}^{T_{\mathrm{obs}}\times 2} : 1 \le i \le N_{\mathrm{vehicle}}\},$$6

Miss Rate is the fraction of cases where all $$H = \{h_i \in \mathbb{R}^{T_{\mathrm{obs}}\times 2} : 1 \le i \le N_{\mathrm{vehicle}}\},$$7 predictions miss the benchmark’s final-displacement threshold.

On the Argoverse test set, the reported leaderboard results as of 2020-11-16 are as follows (Liu et al., 2021):

Method	minADE	minFDE / MR
mmTransformer (6 proposals)	0.8435	1.3383 / 15.42%
mmTransformer + RTS (36 proposals, post-processed to 6 outputs)	0.8704	1.3688 / 13.00%
TNT	0.9358	1.5384 / 13.28%
LaneGCN	0.8679	1.3640 / 16.34%
WIMF	0.8995	1.4220 / 16.69%
LA	0.9436	1.5486 / 21.79%

The most notable empirical point is the trade-off between diversity and point accuracy. With RTS and 36 proposals, mmTransformer attains the best reported miss rate, ranked first on 2020-11-16, while minADE and minFDE become slightly worse than the 6-proposal configuration. The paper attributes this to the interaction between large proposal sets and endpoint-NMS: more proposals improve coverage, but post-processing may reduce the number of selected trajectories within the ground-truth region.

5. Ablations, training configuration, and limitations

Validation ablations separate the effects of context modules and RTS. Using only the motion extractor with 6 proposals yields $$H = \{h_i \in \mathbb{R}^{T_{\mathrm{obs}}\times 2} : 1 \le i \le N_{\mathrm{vehicle}}\},$$8 for minADE / minFDE / MR. Adding the map aggregator yields $$H = \{h_i \in \mathbb{R}^{T_{\mathrm{obs}}\times 2} : 1 \le i \le N_{\mathrm{vehicle}}\},$$9, and adding the social constructor yields $T_{\mathrm{obs}} = 2\,\mathrm{s}$0. Increasing the proposal count to 36 without RTS degrades to $T_{\mathrm{obs}} = 2\,\mathrm{s}$1, whereas 36 proposals with RTS improves to $T_{\mathrm{obs}} = 2\,\mathrm{s}$2. The interpretation given in the paper is that large proposal sets collapse without RTS, while RTS induces specialized regional proposals and lowers miss rate (Liu et al., 2021).

The region partition itself is not uniquely determined. With $T_{\mathrm{obs}} = 2\,\mathrm{s}$3 regions and $T_{\mathrm{obs}} = 2\,\mathrm{s}$4 proposals per region, constrained K-means gives minADE $T_{\mathrm{obs}} = 2\,\mathrm{s}$5, minFDE $T_{\mathrm{obs}} = 2\,\mathrm{s}$6, MR $T_{\mathrm{obs}} = 2\,\mathrm{s}$7, while a manual fan partition gives minADE $T_{\mathrm{obs}} = 2\,\mathrm{s}$8, minFDE $T_{\mathrm{obs}} = 2\,\mathrm{s}$9, MR $d_{\mathrm{model}} = 128$0. Reported miss-rate sweeps over the number of regions and proposals show that overly skewed $d_{\mathrm{model}} = 128$1 ratios hurt performance. In the listed configurations, $d_{\mathrm{model}} = 128$2 is best.

Training uses the Argoverse motion forecasting benchmark with 340k 5s sequences, split into 205,942 train, 39,472 validation, and 78,143 test sequences. The input is 2 s of history and the output horizon is 3 s. Optimization uses AdamW with initial learning rate $d_{\mathrm{model}} = 128$3, weight decay $d_{\mathrm{model}} = 128$4, gradient max norm $d_{\mathrm{model}} = 128$5, and Xavier initialization. Data augmentation consists of horizontal flips and randomly masking the first ten time steps of trajectories. Batch size, epoch count, runtime, and memory are not explicitly reported.

The paper identifies several limitations. Proposals are fixed in number and structure, so performance is sensitive to $d_{\mathrm{model}} = 128$6 and to the granularity of RTS regions. Manual or K-means partitions may misclassify endpoints near region boundaries. Performance depends on HD map quality and may vary across cities or datasets. Proposed future directions include learned proposal generation, dynamic anchors, end-to-end region learning, and differentiable diversity regularizers. A plausible implication is that mmTransformer’s gains are partly architectural and partly supervisory: stacked attention composes richer scene context, but the decisive diversity behavior comes from how training resolves single-ground-truth supervision over a multimodal output space.

6. Terminology and related usage of the name

The name “MMTransformer” is not unique across the later literature. The data include “MTrans” for automatic 3D annotation and object detection from LiDAR scans and images (Liu et al., 2022), “MTrans” for accelerated multi-modal MR imaging (Feng et al., 2021), and “MMTransformer” as the “Motion-Mimic Transformer” backbone inside the GBC framework for whole-body humanoid imitation (Yao et al., 13 Aug 2025). These systems share the general idea of transformer-based multimodal fusion, but they address different modalities, objectives, and architectural constraints.

Within this broader naming landscape, mmTransformer most specifically denotes the autonomous-driving model of stacked transformer decoders and region-based proposal training introduced in 2021 (Liu et al., 2021). This suggests that the term functions less as a single canonical architecture than as a recurring label for transformer systems that treat multimodality as a first-class design objective.