Trajectron++: Multi-Agent Trajectory Forecasting

• The paper introduces Trajectron++ with a dynamics-aware, graph-structured methodology that robustly forecasts multi-agent trajectories using multimodal latent variables and conditioning on external plans.
• The model employs LSTM-based node and edge encoders along with CNN map modules and additive attention to effectively capture agent interactions and environmental semantics.
• Empirical evaluations on datasets like ETH/UCY and nuScenes demonstrate significant improvements in displacement error and collision rates compared to previous state-of-the-art models.

Trajectron++ is a modular, graph-structured recurrent model tailored for multi-agent trajectory forecasting in dynamic, heterogeneous environments. The model is designed for integration with robotic planning and control systems, prominently targeting applications such as autonomous driving where it accommodates diverse agent types, environmental semantic information, and explicit agent dynamics. Its architectural innovations enforce dynamically-feasible predictions, account for multimodal futures, and allow conditioning on external plans (e.g., those of an ego-agent), establishing a new standard for interpretable, scenario-adaptive, and robust motion forecasting (Salzmann et al., 2020).

1. Graph-Structured Probabilistic Forecasting

Trajectron++ encodes the scene as a directed, time-varying spatiotemporal graph $G=(V,E)$ where each node represents an agent (e.g., car, pedestrian) with associated semantic class $S_i$, and directed edges correspond to social interactions initiated within agent-class-specific perceptual radii. Each agent maintains a D-dimensional state vector $\mathbf{s}_i^{(t)}$, with state histories over $H$ steps aggregated as $\mathbf{x}_i$ (Salzmann et al., 2020).

• Node History Encoder: A 32-dimensional LSTM processes each agent's history.
• Edge Encoders: For each semantic edge type, aggregated neighbor encodings are passed through an 8-dimensional shared LSTM.
• Attention Aggregator: Additive attention over per-edge-type outputs yields an “influence” vector, capturing high-level agent interactions.
• The resulting feature vector concatenates the node’s historical, interaction, and (optionally) environmental/contextual features.

The predictive task is to learn the conditional distribution $$p\left(\mathbf{y} \mid \mathbf{x},\{M_i^{(t)}\},\mathbf{y}_R\right)$$, where $\{M_i^{(t)}\}$ are agent-centric map crops and $\mathbf{y}_R$ is an optional robot plan.

2. Dynamics-Aware Multimodal Trajectory Generation

Core to Trajectron++ is dynamics-aware prediction: the model outputs control signals (e.g., acceleration, steering rates) rather than direct future positions, ensuring dynamic feasibility under known motion models.

• Decoder: A 128-dimensional GRU outputs a Gaussian distribution over control inputs at each timestep.
• Propagation: Controls are propagated via agent-specific dynamics models (single integrator for pedestrians, dynamically-extended unicycle for vehicles, with linearized integration per step).

A discrete latent variable $z \in \{1,...,K\}$ (typically $K=25$) captures high-level behavior modes, enabling multimodal forecasting: $$p(\mathbf{y}\mid\mathbf{x}) = \sum_{z=1}^K p_\theta(z\mid\mathbf{x}) p_\psi(\mathbf{y}\mid\mathbf{x},z)$$ A recognition network encodes the ground-truth future, supporting variational learning via a CVAE framework, specifically adopting the InfoVAE loss variant to control regularization and encourage informative latent representations.

3. Integration of Heterogeneous Data Sources

Trajectron++ is architected for seamless incorporation of scene semantics and further high-dimensional data:

• Map Encoder: Local semantic map $M_i^{(t)}$, rasterized and rotated into the agent's frame, is embedded via a 4-layer CNN (filter sizes {5,5,5,3}, strides {2,2,1,1}) followed by a 32-dim fully connected layer, delivering spatial context (e.g., road geometry, obstacles).
• Additional Modalities: Further sensors (e.g., LIDAR, RGB images) can be processed by modality-specific encoders, concatenated at the agent feature level.

The model optionally conditions on external plans such as an ego-agent’s (robot) intended trajectory, encoding these via a bidirectional LSTM whose final hidden state is appended to every agent’s representation, enabling response-aware forecasting.

4. Evaluation, Benchmarks, and Empirical Performance

Experiments span diverse trajectory forecasting datasets:

• ETH/UCY: Multi-pedestrian, 2.5 Hz, up to 1,536 agents.
• nuScenes: Urban driving scenes with 23 classes, dense semantic maps, 2 Hz over 20 seconds.

Metrics include average and final displacement error (ADE/FDE), best-of-$N$ ADE/FDE, kernel-density-estimated NLL (KDE NLL), and collision rate.

Model Variant	Dataset	ADE / FDE	KDE NLL	Collision Rate
Trajectron++ (Full)	ETH/UCY	Up to 60% lower than SOTA	NLL: –1.14 vs +1.79	↓ from 4.6% to 1.0%
Trajectron++ (Full, 3s FDE)	nuScenes	1.14 m (vs 1.23 m)	—	—

Trajectron++ outperforms deterministic and generative baselines (e.g., S-GAN, SoPhie, MATF, CAR-Net), with especially strong gains in both accuracy and uncertainty calibration (Salzmann et al., 2020). Dynamics-aware prediction and map-encoding are individually responsible for large ablation gains (e.g., NLL drop from +1.80 to –1.31; collision reduction from 4.6% to 1.0%).

5. Robustness, Sensitivity, and Security Considerations

Recent work has systematically evaluated Trajectron++’s vulnerability to input perturbations (Gibson et al., 2024):

• Dominant Sensitivity: The most recent agent state inputs (position and velocity) present a large attack surface; median ADE percent increases under small perturbations can exceed 10,000%.
• Image-Map and Graph Sensitivity: FGSM-style attacks on the image-based map input can increase ADE by 20–85%; perturbations to graph node/edge features produce up to 50% error increases.
• Downstream Planning Impact: Attacks on the prediction component can cause substantial failures in downstream planning—e.g., undetectable image perturbations may cause the vehicle to stop abruptly from highway speeds.

Proposed mitigations include adversarial training, denoising of semantic maps, and sanity-checking of velocity inputs. This highlights the need for architectural and pipeline-level hardening before closed-loop deployment in safety-critical systems (Gibson et al., 2024).

6. Extensions and Variants

Several modifications of the original Trajectron++ have explored different axes of robustness, tractability, and cognitive plausibility:

• Smooth-Trajectron++ introduces a temporal smoothness regularizer on the attention weights, inspired by cognitive models of human attention. The loss augments the CVAE objective with a total-variation penalty on changes in social attention over time, discouraging abrupt “jumps” in focus between different edge-types. Empirically, this reduces attention “jitter,” improves ADE/FDE on nuScenes (best at $\beta=0.1$), and yields more stable social reasoning, especially in dense scenes (Westerhout et al., 2023).
• Trajectron++ Without Ground-Truth replaces dataset-provided labels with pseudo-ground-truth generated by a SLAM-based sequential estimator pipeline (DynoSAM + EKF). This results in smoother, more stable training signals, enables learning in label-scarce regimes, and yields lower ADE/FDE and absolute consistency errors than even ground-truth- or EKF-supervised variants—suggesting the practical value of independent, physically consistent pose estimation for real-world deployment (Kliniewski et al., 13 Feb 2025).

7. Limitations and Future Directions

Trajectron++’s principal limitations include reliance on linearized dynamics integration for non-holonomic agents, fixed-size map crops for spatial context, and discrete latent spaces of limited cardinality for multimodality. Scaling to highly complex, real-time, and ultra-heterogeneous settings may necessitate richer dynamic models, continuous/hierarchical latent distributions, adaptive attention for non-grid semantic contexts, and computational pruning or distillation. Ongoing research addresses these dimensions, examining both architectural restrictions and system-level vulnerabilities (Salzmann et al., 2020, Gibson et al., 2024).