---
title: Multimodal Dynamic Trajectory Integration (MDTI)
url: https://www.emergentmind.com/topics/multimodal-dynamic-trajectory-integration-mdti
type: topic
---

# Multimodal Dynamic Trajectory Integration (MDTI)

Multimodal Dynamic Trajectory Integration (MDTI) denotes a family of trajectory-centric modeling paradigms in which heterogeneous signals are combined with time-varying motion evidence to estimate, represent, or generate future motion as a distribution over plausible trajectories rather than as a single path. In the current literature, the term is used across autonomous driving, vessel forecasting, travel time estimation, multimodal tracking, and navigation-conditioned generation, but it does not refer to one universally fixed architecture. Instead, it names a recurrent design pattern: static context and dynamic state are jointly encoded, fused through a learned integration mechanism, and decoded into multimodal trajectory hypotheses, probabilities, or continuous motion representations [2407.05811][2511.14265][2510.05840][2506.19341].

## 1. Terminological scope and conceptual boundaries

MDTI is best understood as a broad methodological label rather than a single standardized model class. In autonomous-driving trajectory prediction, it usually refers to the joint use of map priors, agent histories, sensor-derived kinematics, and interaction cues to model $p(Y \mid X)$ or an equivalent multimodal predictive distribution [2407.05811][2209.00062]. In maritime forecasting, the same term is used for architectures that combine dynamic interaction graphs, sustained route-level intentions, and transient maneuver latents to capture multiple socially or rule-consistent futures [2503.08065][2511.14265]. In trajectory representation learning for travel time estimation, MDTI instead denotes multimodal fusion of GPS sequences, grid trajectories, and road-network constraints, with dynamic handling of variable-length trajectories [2510.05840]. A related survey extends the idea further, describing MDTI as an end-to-end coupling of dynamic object tracking and trajectory prediction under multimodal sensing and contextual reasoning [2506.19341].

A central ambiguity concerns the word “multimodal.” In many MDTI papers it refers to heterogeneous inputs such as RGB, thermal, LiDAR, HD maps, IMU, GPS, road graphs, or language-conditioned context [2407.05811][2603.09287][2606.31654]. In other work, however, “multimodal” refers primarily to one-to-many future behavior under a fixed context, even when the inputs are not sensor-multimodal. DsMCL makes this distinction explicit: its multimodality is about the output distribution over plausible vehicle behaviors, not about multimodal sensors [2003.08638]. This distinction matters because some MDTI systems fuse sensing modalities, some organize multiple future modes, and some do both simultaneously.

The adjective “dynamic” is similarly non-uniform. It can denote temporal evolution of kinematic state, time-varying interaction graphs, state-space temporal propagation, or trajectory-length-adaptive information density. Across usages, the common denominator is that integration is not static feature concatenation alone; it is conditioned on temporal state, evolving context, or both [2503.08065][2603.01284][2510.05840].

## 2. Modalities and state representations

The most common MDTI input partition separates static context from dynamic motion state. In MapsTP, the future ego trajectory is written as $Y=\{y_t\}_{t=1}^T$ with $y_t=(x_t,y_t)\in\mathbb{R}^2$, and the input $X$ consists of a rasterized HD map image $I_{HD}$ together with IMU-derived speed, acceleration, and yaw rate. The modeling target is $p(Y\mid X)$, with the HD map providing lane, boundary, and intersection structure, and the IMU state grounding prediction in current motion dynamics [2407.05811]. Class-aware attention for multimodal prediction generalizes this decomposition to target and neighboring agents by explicitly defining
\[
X=(S_0,S_1,\ldots,S_N,M,O_0,O_1,\ldots,O_N),
\]
where $S_i$ stores past dynamics, $M$ is a rasterized semantic map, and $O_i=(C,l,w)$ encodes class and physical dimensions [2209.00062].

Other MDTI systems broaden the notion of modality well beyond map-plus-kinematics. LiMTR augments agent history and HD maps with local LiDAR crops centered on each tracked road user, encoding point sets over 11 frames and fusing the resulting feature into both local self-attention and motion decoding; the reported motivation is to expose fine-grained cues such as posture or gaze that are absent from coarse kinematic history alone [2410.15819]. DriVerse converts a future 3D trajectory into two distinct conditioning channels: textual prompt tokens based on a 12-sector trend vocabulary and 2D spatial motion prior maps constructed from projected anchors and fading trails [2504.18576]. DynFly, in UAV vision-language navigation, uses multimodal context $h_t$ from a vision-language front-end and a local 3D target $g_t$, then bridges them to continuous UAV motion through B-spline control points and dynamic-aware supervision [2606.31654].

In representation-learning settings, MDTI treats trajectory views themselves as modalities. MVTraj defines GPS, route, and grid/POI views linked through GPS-based map matching and grid assignment, using assignment matrices to bridge views during self-supervised learning [2410.13196]. The MDTI framework for travel time estimation similarly integrates GPS sequences, grid trajectories, and road-network constraints, emphasizing that the modalities differ not only semantically but also in trajectory length and information density [2510.05840].

A further generalization appears in tracking-oriented MDTI. MDTrack assigns dedicated experts to RGB, infrared, event, and depth streams, while maintaining separate temporal state-space models for RGB and the auxiliary modality; DM$^3$T operates on RGB and thermal features and treats fusion as an iterative cross-modal harmonization process [2603.09287][2511.22896]. The survey perspective in dynamic object tracking interprets such constructions as part of a broader MDTI loop in which tracking stabilizes prediction and prediction, in turn, assists tracking under occlusion and ambiguity [2506.19341].

## 3. Integration architectures and fusion mechanisms

MDTI papers do not converge on one fusion operator. Instead, several recurring integration patterns appear.

A first pattern is early fusion by concatenation. MapsTP uses ResNet-50 conv5 features pooled into $f_{map}$ and concatenates them with the agent state vector $s$ to obtain
\[
z=\phi([f_{map}, s]),
\]
which is then consumed by a Spatio-Temporal Probabilistic Network (STPN) that outputs trajectory hypotheses and probabilities [2407.05811]. This is architecturally simple and emphasizes feasibility constraints from lane topology together with current motion state.

A second pattern is attention-based interaction fusion. Class-aware attention combines raster map features, target history, neighbor histories, and physical metadata using distance-weighted and area-weighted attention. The fused context is
\[
z_l=\text{Concat}\{h_M,h_{S_0},h_{att},O\},
\]
where $h_{att}$ concatenates outputs of distance and area attention modules [2209.00062]. In MVTraj and the TTE-oriented MDTI framework, cross-modal interaction is organized as hierarchical attention across route, grid, and GPS-aligned views, with road-to-grid or pairwise inter-modal streams followed by a shared transformer-style global context module [2410.13196][2510.05840].

A third pattern is graph-based integration of dynamic interactions. STGDPM constructs a time-varying graph $G_t=(V_t,E_t)$ from AIS trajectories, defines distance-thresholded adjacency matrices $A_t$, normalizes them via
\[
\hat{A}=I-D^{-1/2}AD^{-1/2},
\]
and injects the resulting graph context into every denoising step of a conditional diffusion model [2503.08065]. GIMTP uses a dynamic adjacency that aggregates neighborhood structure, distance, and a potential-risk term derived from relative kinematics, then applies a bidirectional Diffusion Graph Convolutional Network to encode historical and future-guided graph embeddings before intention-specific fusion [2309.01981]. These approaches make “dynamic” literal: the fusion topology changes over time.

A fourth pattern is hierarchical intention integration. DI-MTP decomposes future behavior into sustained and transient intentions. Sustained intention trees are built from clustered historical AIS trajectory shapes, cross-attention selects top-$k$ prototype branches with probabilities, and a destination-aware CVAE samples transient maneuver variation within each branch. A non-local attention block then enforces scene-level consistency across interacting vessels [2511.14265]. This yields an explicitly interpretable mixture-of-intent architecture rather than an undifferentiated latent sampler.

A fifth pattern uses state-space or diffusion-inspired iterative harmonization. FoSS splits motion into a frequency-domain branch and a time-domain dynamic selective state-space model, aligns them with cross-attention, and generates multiple futures through learnable queries and a weighted fusion head [2603.01284]. MDTrack decouples temporal propagation for RGB and the auxiliary modality with separate SSMs, exchanging information only through controlled cross-attention [2603.09287]. DM$^3$T formulates RGB–thermal fusion as iterative cross-modal diffusion fusion, followed by a diffusion refiner and hierarchical tracker [2511.22896]. These systems replace monolithic fusion with progressive harmonization or decoupled temporal state updates.

A final variant appears in generative and planning settings. DriVerse aligns future trajectory control with a frozen 2D video generator by representing the same trajectory simultaneously as text tokens and 2D motion priors, then adding a latent motion alignment term for dynamic pixels [2504.18576]. DynFly conditions a Spline-DiT generator on visual-language context, a local target, and generation time, then decodes continuous UAV trajectories in B-spline space [2606.31654]. In both cases, MDTI is not only predictive but also control-facing: integration is designed to bridge abstract intent and executable motion.

## 4. Probabilistic formulations, output spaces, and learning objectives

MDTI systems differ sharply in how they represent uncertainty. One common formulation is a discrete set of trajectory hypotheses with categorical probabilities. MapsTP outputs $\{(P_i,\text{prob}(P_i))\}_{i=1}^N$ and selects the maximum-probability path via
\[
\hat{Y}=\arg\max_{P_i}\text{prob}(P_i),
\]
while still evaluating the top-$K$ set with $\text{MinADE}_k$, $\text{MinFDE}_k$, and $\text{MissRate}_{k,d}$ [2407.05811]. Class-aware attention likewise uses $K$ parallel decoders and a softmax score head over modes [2209.00062].

A second class uses explicit hierarchical or latent probabilistic decompositions. DI-MTP defines
\[
p(Y\mid X)=\sum_{m=1}^{M}\pi_m(X)\,p(Y\mid X,z_m),
\]
with sustained intention branches furnishing $\pi_m(X)$ and CVAE latents controlling within-branch variation [2511.14265]. DsMCL formalizes a dual-level mixture over intention modes and motion modes, assigning the regression gradient only to a winner chosen by endpoint and ADE criteria; this is a multimodal output construction even though the inputs need not be sensor-multimodal [2003.08638].

A third class uses diffusion or continuous stochastic-process formalisms. STGDPM learns
\[
p_\theta(Y\mid X,M)
\]
through a DDPM-style reverse process conditioned on history and graph context, with the simplified denoising objective
\[
\mathcal{L}_{simple}=\mathbb{E}_{y_0,k,\epsilon}\left[\left\|\epsilon-\epsilon_\theta(\sqrt{\bar{\alpha}_k}y_0+\sqrt{1-\bar{\alpha}_k}\epsilon,k,X,A)\right\|_2^2\right]
\]
[2503.08065]. Kernel Trajectory Maps go in a different direction and model the future as a mixture of continuous stochastic processes over basis-function weights, enabling continuous-time querying of sampled trajectories at arbitrary time steps [1907.05127].

Training objectives are correspondingly heterogeneous. MapsTP states directly that training minimizes $\text{MinADE}_k$ on the training set [2407.05811]. Class-aware attention reports that ReCoAt loss caused mode collapse and that MTP loss yielded better diversity and accuracy, though explicit equations are not given [2209.00062]. DI-MTP uses
\[
L_{DI\text{-}MTP}=\lambda_1 L_{clf}+\lambda_2 L_{cvae}+\lambda_3 L_{reg},
\]
with $\lambda_1=\lambda_2=\lambda_3=1$ in experiments [2511.14265]. The travel-time MDTI framework uses self-supervised contrastive alignment and masked language modeling during pretraining, then fine-tunes for scalar travel-time regression [2510.05840]. DynFly combines flow matching with dynamic-aware position, velocity, acceleration, heading, and target-alignment losses [2606.31654].

Evaluation metrics also reflect domain-specific interpretations of trajectory quality. Road-forecasting work centers on ADE, FDE, minADE, minFDE, miss rate, off-road rate, or RFS-like planning scores [2209.00062][2511.22181]. Maritime work uses ADE/FDE under best-of-$N$ sampling and, in DI-MTP, geodesic-distance versions of ADE/FDE [2503.08065][2511.14265]. Representation-learning variants target MAE, RMSE, and MAPE for travel-time estimation rather than geometric displacement of futures [2510.05840].

## 5. Empirical realizations across domains

The empirical literature shows that MDTI is not confined to one application domain. It appears in road forecasting, vessel prediction, travel time estimation, tracking, UAV navigation, and trajectory-conditioned generation.

| Work | Domain | Representative outcome |
|---|---|---|
| MapsTP [2407.05811] | Ego-trajectory prediction on nuScenes | MinADE\_5 = 1.30, MinFDE\_10 = 2.19, MissRate\_{10,2} = 0.42 |
| Class-Aware Attention [2209.00062] | Multimodal road-user prediction | minADE\_5 = 1.67, minFDE\_5 = 3.36, Off-Road Rate = 0.07, over 300 FPS |
| STGDPM [2503.08065] | Maritime vessel forecasting | AVG ADE = 0.094, FDE = 0.143 across TJP and CFW |
| DI-MTP [2511.14265] | Explainable vessel MTP | Zhoushan ADE/FDE = 64.91 / 102.33; Hainan = 42.87 / 69.07 |
| FoSS [2603.01284] | Autonomous-driving motion prediction | AV2 b-minFDE\_6 = 1.69, minADE\_6 = 0.61, MR\_6 = 0.11 |
| DynFly [2606.31654] | UAV vision-language navigation | Test Unseen Full: +4.69 NDTW, +2.40 SDTW, +2.14 SR, +4.87 OSR, −4.51 m NE |
| MDTI for TTE [2510.05840] | Travel time estimation | Porto MAE/MAPE = 1.419 / 0.152; Xi’an = 1.169 / 0.170; Chengdu = 1.143 / 0.183 |

In road prediction, the recurring empirical pattern is that integrating scene structure with dynamic state improves both displacement error and physically meaningful metrics. MapsTP attributes its gains to the combination of HD map priors and IMU-derived motion state, while class-aware attention reports improvements from distance-weighted and area-weighted neighbor attention and maintains real-time throughput above 300 FPS [2407.05811][2209.00062]. FoSS extends the road-forecasting branch of MDTI by combining temporal and spectral views of motion; its reported Argoverse 2 results and reduced parameter count suggest that efficient multimodal integration need not rely exclusively on quadratic-cost attention [2603.01284].

In maritime forecasting, MDTI has developed in two distinct directions. STGDPM uses spatio-temporal graphs and diffusion to represent interaction-driven uncertainty, with best performance reported at interaction threshold $\tau = 50$ hm [2503.08065]. DI-MTP instead emphasizes interpretability, binding each predicted future to a sustained intention prototype, a transient latent sample, and a decoded destination [2511.14265]. The coexistence of these two formulations illustrates that MDTI can be either graph-diffusive or intention-hierarchical without losing its core emphasis on multimodal, dynamic integration.

Beyond forecasting, the term now covers motion-interface modules in planning and generation. DynFly applies MDTI to bridge visual-language reasoning and executable UAV motion through B-spline control points, flow matching, and dynamic-aware supervision [2606.31654]. DriVerse uses trajectory tokenization and spatial motion priors to condition long-horizon driving-video generation, and reports that replacing multimodal trajectory prompting with direct 3D-to-2D projection degrades GAE from 1.68 m to 3.25 m on the Waymo ablation [2504.18576]. In trajectory representation learning for travel time estimation, MDTI’s gains come not from predicting multiple future paths but from integrating heterogeneous trajectory views and dynamic length handling [2510.05840].

## 6. Misconceptions, limitations, and open research directions

A first misconception is that MDTI always implies richer sensor fusion. The literature does not support that simplification. Some systems indeed fuse sensors or heterogeneous contextual channels, but others use the term mainly for multimodal future behavior, multi-view trajectory representations, or tracker–predictor couplings [2003.08638][2510.05840][2506.19341]. A second misconception is that any additional modality automatically improves prediction. MTR-VP offers a counterexample: in its ablation, blank images produced 3 s ADE 1.4238 and 5 s ADE 3.3509, nearly identical to the full-image model’s 1.4232 and 3.3485, indicating that simple transformer fusion of global image features and kinematics was not sufficient to make vision materially useful in that setting [2511.22181].

Several limitations recur across MDTI papers. Dependence on map quality and alignment remains a basic fragility in map-centric models such as MapsTP [2407.05811]. Graph-based and diffusion-based interaction models improve multimodality but introduce graph-construction or iterative-denoising overhead; STGDPM explicitly notes dynamic-graph cost and scaling pressure as vessel counts rise [2503.08065]. Intention-hierarchical systems improve explainability but can inherit prototype bias and may miss rare behaviors or new traffic patterns if the intention library is not representative [2511.14265]. Travel-time MDTI depends on reliable map matching and POI or road-network construction, so errors in preprocessing propagate into the fused representation [2510.05840].

Another persistent gap concerns calibration and uncertainty quality. Several road-forecasting systems evaluate minADE/minFDE but do not report NLL-, CRPS-, or calibration-oriented analyses. MapsTP explicitly notes the absence of explicit calibration studies, and the broader survey on DOT–TP integration identifies uncertainty propagation and decision-theoretic integration as open problems [2407.05811][2506.19341]. This suggests that many current MDTI systems are multimodal in the sense of producing multiple futures, yet still incomplete as uncertainty-aware decision components.

Open directions are correspondingly consistent across domains. The road and maritime papers repeatedly propose adding richer sensor streams, stronger social-interaction modeling, planning-aware costs, and risk-aware trajectory selection beyond pure argmax over mode probability [2407.05811][2511.14265]. Tracking-oriented MDTI suggests more explicit modality-aware reliability estimation and better handling of missing or asynchronous modalities [2603.09287][2511.22896]. Representation-learning variants point toward broader context integration and standardized evaluation across cities and domains [2410.13196][2510.05840]. Taken together, these works suggest that MDTI is evolving from a loose fusion label into a more general research program: integrating heterogeneous context, temporal state, and multimodal uncertainty in a form that remains computationally tractable, physically consistent, and useful for downstream decision-making.

Source: https://www.emergentmind.com/topics/multimodal-dynamic-trajectory-integration-mdti