Multimodal Dynamic Trajectory Integration (MDTI)

• MDTI is a method that integrates heterogeneous signals and dynamic states to generate a distribution over plausible future trajectories.
• It employs various fusion techniques such as concatenation, attention, and graph-based integration to enhance prediction accuracy and interpretability.
• MDTI is applied in domains like autonomous driving, vessel forecasting, travel time estimation, and UAV navigation, demonstrating its broad versatility.

Multimodal Dynamic Trajectory Integration (MDTI) denotes a family of trajectory-centric modeling paradigms in which heterogeneous signals are combined with time-varying motion evidence to estimate, represent, or generate future motion as a distribution over plausible trajectories rather than as a single path. In the current literature, the term is used across autonomous driving, vessel forecasting, travel time estimation, multimodal tracking, and navigation-conditioned generation, but it does not refer to one universally fixed architecture. Instead, it names a recurrent design pattern: static context and dynamic state are jointly encoded, fused through a learned integration mechanism, and decoded into multimodal trajectory hypotheses, probabilities, or continuous motion representations (Sharma et al., 2024, Zhang et al., 18 Nov 2025, Liu et al., 7 Oct 2025, Dong et al., 24 Jun 2025).

1. Terminological scope and conceptual boundaries

MDTI is best understood as a broad methodological label rather than a single standardized model class. In autonomous-driving trajectory prediction, it usually refers to the joint use of map priors, agent histories, sensor-derived kinematics, and interaction cues to model $p(Y \mid X)$ or an equivalent multimodal predictive distribution (Sharma et al., 2024, Pathiraja et al., 2022). In maritime forecasting, the same term is used for architectures that combine dynamic interaction graphs, sustained route-level intentions, and transient maneuver latents to capture multiple socially or rule-consistent futures (Wenzhe et al., 11 Mar 2025, Zhang et al., 18 Nov 2025). In trajectory representation learning for travel time estimation, MDTI instead denotes multimodal fusion of GPS sequences, grid trajectories, and road-network constraints, with dynamic handling of variable-length trajectories (Liu et al., 7 Oct 2025). A related survey extends the idea further, describing MDTI as an end-to-end coupling of dynamic object tracking and trajectory prediction under multimodal sensing and contextual reasoning (Dong et al., 24 Jun 2025).

A central ambiguity concerns the word “multimodal.” In many MDTI papers it refers to heterogeneous inputs such as RGB, thermal, LiDAR, HD maps, IMU, GPS, road graphs, or language-conditioned context (Sharma et al., 2024, Wang et al., 10 Mar 2026, Jiang et al., 30 Jun 2026). In other work, however, “multimodal” refers primarily to one-to-many future behavior under a fixed context, even when the inputs are not sensor-multimodal. DsMCL makes this distinction explicit: its multimodality is about the output distribution over plausible vehicle behaviors, not about multimodal sensors (Wang et al., 2020). This distinction matters because some MDTI systems fuse sensing modalities, some organize multiple future modes, and some do both simultaneously.

The adjective “dynamic” is similarly non-uniform. It can denote temporal evolution of kinematic state, time-varying interaction graphs, state-space temporal propagation, or trajectory-length-adaptive information density. Across usages, the common denominator is that integration is not static feature concatenation alone; it is conditioned on temporal state, evolving context, or both (Wenzhe et al., 11 Mar 2025, Huang et al., 1 Mar 2026, Liu et al., 7 Oct 2025).

2. Modalities and state representations

The most common MDTI input partition separates static context from dynamic motion state. In MapsTP, the future ego trajectory is written as $Y=\{y_t\}_{t=1}^T$ with $y_t=(x_t,y_t)\in\mathbb{R}^2$, and the input $X$ consists of a rasterized HD map image $I_{HD}$ together with IMU-derived speed, acceleration, and yaw rate. The modeling target is $p(Y\mid X)$, with the HD map providing lane, boundary, and intersection structure, and the IMU state grounding prediction in current motion dynamics (Sharma et al., 2024). Class-aware attention for multimodal prediction generalizes this decomposition to target and neighboring agents by explicitly defining

$X=(S_0,S_1,\ldots,S_N,M,O_0,O_1,\ldots,O_N),$

where $S_i$ stores past dynamics, $M$ is a rasterized semantic map, and $O_i=(C,l,w)$ encodes class and physical dimensions (Pathiraja et al., 2022).

Other MDTI systems broaden the notion of modality well beyond map-plus-kinematics. LiMTR augments agent history and HD maps with local LiDAR crops centered on each tracked road user, encoding point sets over 11 frames and fusing the resulting feature into both local self-attention and motion decoding; the reported motivation is to expose fine-grained cues such as posture or gaze that are absent from coarse kinematic history alone (Oerlemans et al., 2024). DriVerse converts a future 3D trajectory into two distinct conditioning channels: textual prompt tokens based on a 12-sector trend vocabulary and 2D spatial motion prior maps constructed from projected anchors and fading trails (Li et al., 22 Apr 2025). DynFly, in UAV vision-language navigation, uses multimodal context $Y=\{y_t\}_{t=1}^T$0 from a vision-language front-end and a local 3D target $Y=\{y_t\}_{t=1}^T$1, then bridges them to continuous UAV motion through B-spline control points and dynamic-aware supervision (Jiang et al., 30 Jun 2026).

In representation-learning settings, MDTI treats trajectory views themselves as modalities. MVTraj defines GPS, route, and grid/POI views linked through GPS-based map matching and grid assignment, using assignment matrices to bridge views during self-supervised learning (Qian et al., 2024). The MDTI framework for travel time estimation similarly integrates GPS sequences, grid trajectories, and road-network constraints, emphasizing that the modalities differ not only semantically but also in trajectory length and information density (Liu et al., 7 Oct 2025).

A further generalization appears in tracking-oriented MDTI. MDTrack assigns dedicated experts to RGB, infrared, event, and depth streams, while maintaining separate temporal state-space models for RGB and the auxiliary modality; DM$Y=\{y_t\}_{t=1}^T$2T operates on RGB and thermal features and treats fusion as an iterative cross-modal harmonization process (Wang et al., 10 Mar 2026, Li et al., 28 Nov 2025). The survey perspective in dynamic object tracking interprets such constructions as part of a broader MDTI loop in which tracking stabilizes prediction and prediction, in turn, assists tracking under occlusion and ambiguity (Dong et al., 24 Jun 2025).

3. Integration architectures and fusion mechanisms

MDTI papers do not converge on one fusion operator. Instead, several recurring integration patterns appear.

A first pattern is early fusion by concatenation. MapsTP uses ResNet-50 conv5 features pooled into $Y=\{y_t\}_{t=1}^T$3 and concatenates them with the agent state vector $Y=\{y_t\}_{t=1}^T$4 to obtain

$Y=\{y_t\}_{t=1}^T$5

which is then consumed by a Spatio-Temporal Probabilistic Network (STPN) that outputs trajectory hypotheses and probabilities (Sharma et al., 2024). This is architecturally simple and emphasizes feasibility constraints from lane topology together with current motion state.

A second pattern is attention-based interaction fusion. Class-aware attention combines raster map features, target history, neighbor histories, and physical metadata using distance-weighted and area-weighted attention. The fused context is

$Y=\{y_t\}_{t=1}^T$6

where $Y=\{y_t\}_{t=1}^T$7 concatenates outputs of distance and area attention modules (Pathiraja et al., 2022). In MVTraj and the TTE-oriented MDTI framework, cross-modal interaction is organized as hierarchical attention across route, grid, and GPS-aligned views, with road-to-grid or pairwise inter-modal streams followed by a shared transformer-style global context module (Qian et al., 2024, Liu et al., 7 Oct 2025).

A third pattern is graph-based integration of dynamic interactions. STGDPM constructs a time-varying graph $Y=\{y_t\}_{t=1}^T$8 from AIS trajectories, defines distance-thresholded adjacency matrices $Y=\{y_t\}_{t=1}^T$9, normalizes them via

$y_t=(x_t,y_t)\in\mathbb{R}^2$0

and injects the resulting graph context into every denoising step of a conditional diffusion model (Wenzhe et al., 11 Mar 2025). GIMTP uses a dynamic adjacency that aggregates neighborhood structure, distance, and a potential-risk term derived from relative kinematics, then applies a bidirectional Diffusion Graph Convolutional Network to encode historical and future-guided graph embeddings before intention-specific fusion (Wu et al., 2023). These approaches make “dynamic” literal: the fusion topology changes over time.

A fourth pattern is hierarchical intention integration. DI-MTP decomposes future behavior into sustained and transient intentions. Sustained intention trees are built from clustered historical AIS trajectory shapes, cross-attention selects top-$y_t=(x_t,y_t)\in\mathbb{R}^2$1 prototype branches with probabilities, and a destination-aware CVAE samples transient maneuver variation within each branch. A non-local attention block then enforces scene-level consistency across interacting vessels (Zhang et al., 18 Nov 2025). This yields an explicitly interpretable mixture-of-intent architecture rather than an undifferentiated latent sampler.

A fifth pattern uses state-space or diffusion-inspired iterative harmonization. FoSS splits motion into a frequency-domain branch and a time-domain dynamic selective state-space model, aligns them with cross-attention, and generates multiple futures through learnable queries and a weighted fusion head (Huang et al., 1 Mar 2026). MDTrack decouples temporal propagation for RGB and the auxiliary modality with separate SSMs, exchanging information only through controlled cross-attention (Wang et al., 10 Mar 2026). DM$y_t=(x_t,y_t)\in\mathbb{R}^2$2T formulates RGB–thermal fusion as iterative cross-modal diffusion fusion, followed by a diffusion refiner and hierarchical tracker (Li et al., 28 Nov 2025). These systems replace monolithic fusion with progressive harmonization or decoupled temporal state updates.

A final variant appears in generative and planning settings. DriVerse aligns future trajectory control with a frozen 2D video generator by representing the same trajectory simultaneously as text tokens and 2D motion priors, then adding a latent motion alignment term for dynamic pixels (Li et al., 22 Apr 2025). DynFly conditions a Spline-DiT generator on visual-language context, a local target, and generation time, then decodes continuous UAV trajectories in B-spline space (Jiang et al., 30 Jun 2026). In both cases, MDTI is not only predictive but also control-facing: integration is designed to bridge abstract intent and executable motion.

4. Probabilistic formulations, output spaces, and learning objectives

MDTI systems differ sharply in how they represent uncertainty. One common formulation is a discrete set of trajectory hypotheses with categorical probabilities. MapsTP outputs $y_t=(x_t,y_t)\in\mathbb{R}^2$3 and selects the maximum-probability path via

$y_t=(x_t,y_t)\in\mathbb{R}^2$4

while still evaluating the top-$y_t=(x_t,y_t)\in\mathbb{R}^2$5 set with $y_t=(x_t,y_t)\in\mathbb{R}^2$6, $y_t=(x_t,y_t)\in\mathbb{R}^2$7, and $y_t=(x_t,y_t)\in\mathbb{R}^2$8 (Sharma et al., 2024). Class-aware attention likewise uses $y_t=(x_t,y_t)\in\mathbb{R}^2$9 parallel decoders and a softmax score head over modes (Pathiraja et al., 2022).

A second class uses explicit hierarchical or latent probabilistic decompositions. DI-MTP defines

$X$0

with sustained intention branches furnishing $X$1 and CVAE latents controlling within-branch variation (Zhang et al., 18 Nov 2025). DsMCL formalizes a dual-level mixture over intention modes and motion modes, assigning the regression gradient only to a winner chosen by endpoint and ADE criteria; this is a multimodal output construction even though the inputs need not be sensor-multimodal (Wang et al., 2020).

A third class uses diffusion or continuous stochastic-process formalisms. STGDPM learns

$X$2

through a DDPM-style reverse process conditioned on history and graph context, with the simplified denoising objective

$X$3

(Wenzhe et al., 11 Mar 2025). Kernel Trajectory Maps go in a different direction and model the future as a mixture of continuous stochastic processes over basis-function weights, enabling continuous-time querying of sampled trajectories at arbitrary time steps (Zhi et al., 2019).

Training objectives are correspondingly heterogeneous. MapsTP states directly that training minimizes $X$4 on the training set (Sharma et al., 2024). Class-aware attention reports that ReCoAt loss caused mode collapse and that MTP loss yielded better diversity and accuracy, though explicit equations are not given (Pathiraja et al., 2022). DI-MTP uses

$X$5

with $X$6 in experiments (Zhang et al., 18 Nov 2025). The travel-time MDTI framework uses self-supervised contrastive alignment and masked language modeling during pretraining, then fine-tunes for scalar travel-time regression (Liu et al., 7 Oct 2025). DynFly combines flow matching with dynamic-aware position, velocity, acceleration, heading, and target-alignment losses (Jiang et al., 30 Jun 2026).

Evaluation metrics also reflect domain-specific interpretations of trajectory quality. Road-forecasting work centers on ADE, FDE, minADE, minFDE, miss rate, off-road rate, or RFS-like planning scores (Pathiraja et al., 2022, Keskar et al., 27 Nov 2025). Maritime work uses ADE/FDE under best-of-$X$7 sampling and, in DI-MTP, geodesic-distance versions of ADE/FDE (Wenzhe et al., 11 Mar 2025, Zhang et al., 18 Nov 2025). Representation-learning variants target MAE, RMSE, and MAPE for travel-time estimation rather than geometric displacement of futures (Liu et al., 7 Oct 2025).

5. Empirical realizations across domains

The empirical literature shows that MDTI is not confined to one application domain. It appears in road forecasting, vessel prediction, travel time estimation, tracking, UAV navigation, and trajectory-conditioned generation.

Work	Domain	Representative outcome
MapsTP (Sharma et al., 2024)	Ego-trajectory prediction on nuScenes	MinADE_5 = 1.30, MinFDE_10 = 2.19, MissRate_{10,2} = 0.42
Class-Aware Attention (Pathiraja et al., 2022)	Multimodal road-user prediction	minADE_5 = 1.67, minFDE_5 = 3.36, Off-Road Rate = 0.07, over 300 FPS
STGDPM (Wenzhe et al., 11 Mar 2025)	Maritime vessel forecasting	AVG ADE = 0.094, FDE = 0.143 across TJP and CFW
DI-MTP (Zhang et al., 18 Nov 2025)	Explainable vessel MTP	Zhoushan ADE/FDE = 64.91 / 102.33; Hainan = 42.87 / 69.07
FoSS (Huang et al., 1 Mar 2026)	Autonomous-driving motion prediction	AV2 b-minFDE_6 = 1.69, minADE_6 = 0.61, MR_6 = 0.11
DynFly (Jiang et al., 30 Jun 2026)	UAV vision-language navigation	Test Unseen Full: +4.69 NDTW, +2.40 SDTW, +2.14 SR, +4.87 OSR, −4.51 m NE
MDTI for TTE (Liu et al., 7 Oct 2025)	Travel time estimation	Porto MAE/MAPE = 1.419 / 0.152; Xi’an = 1.169 / 0.170; Chengdu = 1.143 / 0.183

In road prediction, the recurring empirical pattern is that integrating scene structure with dynamic state improves both displacement error and physically meaningful metrics. MapsTP attributes its gains to the combination of HD map priors and IMU-derived motion state, while class-aware attention reports improvements from distance-weighted and area-weighted neighbor attention and maintains real-time throughput above 300 FPS (Sharma et al., 2024, Pathiraja et al., 2022). FoSS extends the road-forecasting branch of MDTI by combining temporal and spectral views of motion; its reported Argoverse 2 results and reduced parameter count suggest that efficient multimodal integration need not rely exclusively on quadratic-cost attention (Huang et al., 1 Mar 2026).

In maritime forecasting, MDTI has developed in two distinct directions. STGDPM uses spatio-temporal graphs and diffusion to represent interaction-driven uncertainty, with best performance reported at interaction threshold $X$8 hm (Wenzhe et al., 11 Mar 2025). DI-MTP instead emphasizes interpretability, binding each predicted future to a sustained intention prototype, a transient latent sample, and a decoded destination (Zhang et al., 18 Nov 2025). The coexistence of these two formulations illustrates that MDTI can be either graph-diffusive or intention-hierarchical without losing its core emphasis on multimodal, dynamic integration.

Beyond forecasting, the term now covers motion-interface modules in planning and generation. DynFly applies MDTI to bridge visual-language reasoning and executable UAV motion through B-spline control points, flow matching, and dynamic-aware supervision (Jiang et al., 30 Jun 2026). DriVerse uses trajectory tokenization and spatial motion priors to condition long-horizon driving-video generation, and reports that replacing multimodal trajectory prompting with direct 3D-to-2D projection degrades GAE from 1.68 m to 3.25 m on the Waymo ablation (Li et al., 22 Apr 2025). In trajectory representation learning for travel time estimation, MDTI’s gains come not from predicting multiple future paths but from integrating heterogeneous trajectory views and dynamic length handling (Liu et al., 7 Oct 2025).

6. Misconceptions, limitations, and open research directions

A first misconception is that MDTI always implies richer sensor fusion. The literature does not support that simplification. Some systems indeed fuse sensors or heterogeneous contextual channels, but others use the term mainly for multimodal future behavior, multi-view trajectory representations, or tracker–predictor couplings (Wang et al., 2020, Liu et al., 7 Oct 2025, Dong et al., 24 Jun 2025). A second misconception is that any additional modality automatically improves prediction. MTR-VP offers a counterexample: in its ablation, blank images produced 3 s ADE 1.4238 and 5 s ADE 3.3509, nearly identical to the full-image model’s 1.4232 and 3.3485, indicating that simple transformer fusion of global image features and kinematics was not sufficient to make vision materially useful in that setting (Keskar et al., 27 Nov 2025).

Several limitations recur across MDTI papers. Dependence on map quality and alignment remains a basic fragility in map-centric models such as MapsTP (Sharma et al., 2024). Graph-based and diffusion-based interaction models improve multimodality but introduce graph-construction or iterative-denoising overhead; STGDPM explicitly notes dynamic-graph cost and scaling pressure as vessel counts rise (Wenzhe et al., 11 Mar 2025). Intention-hierarchical systems improve explainability but can inherit prototype bias and may miss rare behaviors or new traffic patterns if the intention library is not representative (Zhang et al., 18 Nov 2025). Travel-time MDTI depends on reliable map matching and POI or road-network construction, so errors in preprocessing propagate into the fused representation (Liu et al., 7 Oct 2025).

Another persistent gap concerns calibration and uncertainty quality. Several road-forecasting systems evaluate minADE/minFDE but do not report NLL-, CRPS-, or calibration-oriented analyses. MapsTP explicitly notes the absence of explicit calibration studies, and the broader survey on DOT–TP integration identifies uncertainty propagation and decision-theoretic integration as open problems (Sharma et al., 2024, Dong et al., 24 Jun 2025). This suggests that many current MDTI systems are multimodal in the sense of producing multiple futures, yet still incomplete as uncertainty-aware decision components.

Open directions are correspondingly consistent across domains. The road and maritime papers repeatedly propose adding richer sensor streams, stronger social-interaction modeling, planning-aware costs, and risk-aware trajectory selection beyond pure argmax over mode probability (Sharma et al., 2024, Zhang et al., 18 Nov 2025). Tracking-oriented MDTI suggests more explicit modality-aware reliability estimation and better handling of missing or asynchronous modalities (Wang et al., 10 Mar 2026, Li et al., 28 Nov 2025). Representation-learning variants point toward broader context integration and standardized evaluation across cities and domains (Qian et al., 2024, Liu et al., 7 Oct 2025). Taken together, these works suggest that MDTI is evolving from a loose fusion label into a more general research program: integrating heterogeneous context, temporal state, and multimodal uncertainty in a form that remains computationally tractable, physically consistent, and useful for downstream decision-making.