Multi-Level Trajectory Fusion Module

• Multi-Level Trajectory Fusion Module is a strategy that combines trajectory-derived signals with complementary representations at multiple scales and modalities.
• It employs techniques such as competitive gating, cross-modal attention, and late fusion to preserve both local details and global trends.
• Empirical results show significant performance gains in tasks like map extraction, object detection, and tracking by optimizing fusion at various levels.

In the cited literature, a multi-level trajectory fusion module denotes a computational component that fuses trajectory-derived information with complementary representations at multiple resolutions, modalities, or abstraction levels. The immediate task varies widely—road-mask extraction from aerial imagery and GPS trajectories, multi-frame camera-radar 3D detection, driving-behavior explanation, trajectory representation learning, destination prediction, multi-object tracking, and hybrid prediction with rule-based planners—but the recurring principle is stable: modality-specific or level-specific encoders first preserve heterogeneous structure, and fusion is then performed over aligned feature maps, temporal windows, candidate modes, or global embeddings by gates, attention, graph aggregation, clustering, or belief-driven sampling (Wu et al., 2020, Li et al., 31 Oct 2025, Veer et al., 2023).

1. Terminological scope and recurrent structure

Across the cited usage, “level” can refer to feature scale, temporal granularity, semantic abstraction, trajectory mode, or camera/view index. In DeepDualMapper, the relevant levels are the five decoder stages of two auxiliary U-Nets, where aerial-image features and trajectory features are fused at every scale (Wu et al., 2020). In M^3Detection, the levels are explicitly named global-level inter-object aggregation, local-level inter-grid aggregation, and trajectory-level multi-frame spatiotemporal reasoning (Li et al., 31 Oct 2025). In BEEF, the levels are decision-level trajectory outputs and mid-level perceptual features (Ben-Younes et al., 2020). In DouFu and MVTraj, the levels are movement, route, global semantic, GPS, road-network, and grid/POI views (Wang et al., 2022, Qian et al., 2024). In FusionTrack and Fusion-Poly, the levels extend to frame-level, short-term temporal, cross-view, and global identity fusion (Li et al., 24 May 2025, Wu et al., 9 Mar 2026).

Representative system	Levels fused	Mechanism
DeepDualMapper	decoder levels of aerial-image and trajectory features	adapters + complementary-aware gates
M^3Detection	global, local, trajectory levels	GOA + LGA + MSTR
BEEF	trajectory-level decision and mid-level perception	BLOCK bilinear fusion
DouFu	movement, route, global semantic	attention fusion + embedding fusion
WTTFNet	trajectory-based destination probabilities and weather-time context	Gate Multimodal Unit
MVTraj	route, GPS-on-road, grid, GPS-on-grid	inter-modal interaction attention + global context attention

This range indicates that the term is not tied to a single architecture. Instead, it names a family of modules that reconcile complementary but non-identical trajectory signals. A plausible implication is that “multi-level” is best understood as a design constraint: fusion is deferred until each source has been encoded at the level where it is most informative, and then repeated or staged so that local detail and global trend are both preserved.

2. Canonical operators and mathematical formulations

One common operator is competitive gating. DeepDualMapper projects image and trajectory features into a shared space with 1×1 adapters and then enforces complementary softmax gates at each decoder level. Its fused feature is

$$A_f^{(i)} = G_I^{(i)} \odot A_I^{(i)} + G_T^{(i)} \odot A_T^{(i)},$$

with the complementary constraint $G_I^{(i)}(p) + G_T^{(i)}(p)=1$ for every pixel $p$ (Wu et al., 2020). This makes the selector explicitly decide which modality to trust where and at which scale.

A second recurring operator is gated multimodal interpolation over compact embeddings. WTTFNet projects trajectory-based destination probabilities and a weather-time embedding into a shared dimension, computes a vector gate,

$$\gamma_n = \Psi_{\text{sgm}}\big(W_3 [P_{\text{pre}}(W_n)^\top, e_{\text{WT},n}^\top]^\top\big),$$

and fuses the branches as

$$f_{\text{fuse}} = \gamma_n \odot h_v + (1-\gamma_n)\odot h_e.$$

This is a per-dimension mixture between trajectory and context modalities rather than simple concatenation (Wu et al., 2024).

A third operator is cross-modal attention. MVTraj constructs query, key, and value tensors from concatenated trajectory-level and segment-level tokens for each modality, then computes

$$\boldsymbol{O}_a^b = \text{softmax}\left(\frac{Q_a K_b^\top}{\sqrt{d_k}}\right)V_b,$$

aggregating route, GPS, and grid views before a shared Transformer performs global context attention (Qian et al., 2024). MTFT uses an analogous two-stage strategy in the temporal domain: the Multi-scale Attention Head first produces multi-scale motion representations $R_m$, and the Continuity Representation-guided Multi-scale Fusion module then uses continuity-derived queries $Q_c$ to fuse them as

$$e^{\text{temp}} = \text{softmax}\left(\frac{Q_c K_m^\top}{d_K}\right)V_m,$$

thereby combining detailed motion information with overall continuity (Liu et al., 2024).

A fourth operator is late fusion over trajectory hypotheses or model outputs. ParallelNet generates independent multi-modal trajectory sets from four MobileNet v2 branches and then applies a Set Transformer over the hypothesis set (Wu et al., 2022). Multi-Predictor Fusion instead mixes whole trajectory sets from a learning-based predictor and a rule-based predictor according to an online belief distribution $b_t=[b_t^l,b_t^r]$, producing an explicit mixture at the trajectory level rather than the feature level (Veer et al., 2023).

3. Representative architectural families

A first family is spatially aligned raster fusion. DeepDualMapper rasterizes GPS trajectories onto the same 2D grid as the aerial image, applies Gated Fusion Modules at all five decoder stages, and refines the fused hierarchy with a Densely Supervised Refinement Decoder in a coarse-to-fine manner (Wu et al., 2020). This formulation is explicitly pixel-wise and benefits from exact alignment between the trajectory image and the RGB crop.

A second family is object-centric spatiotemporal fusion for perception and tracking. M^3Detection stores intermediate BEV features in a memory bank, uses tracked reference trajectories and candidate proposals, applies global-level inter-object aggregation on global BEV crops, local-level inter-grid aggregation with cross-level deformable attention on expanded local patches, and trajectory-level multi-frame attention over per-track feature sequences (Li et al., 31 Oct 2025). CoPAD begins even earlier, using Hungarian matching and Kalman filtering to fuse vehicle and infrastructure trajectories, then applies Past Time Attention, mode attention, and an anchor-oriented decoder (Wu et al., 19 Sep 2025). Fusion-Poly performs frequency-aware cascade matching at synchronized and asynchronous timestamps, then fuses motion state and existence state through differential Kalman updates and Noisy-OR confidence updates (Wu et al., 9 Mar 2026). FutrTrack combines a temporal smoother over detector outputs with a camera-LiDAR fusion transformer that propagates identity by query reuse rather than an explicit motion model (Teye et al., 22 Oct 2025). FusionTrack extends the same multi-level logic to arbitrary multi-view environments with a Tracklet Memory Pool, a ReID Transformer, an Object Update Module with cross-frame and cross-view attention, and view-guided hierarchical clustering (Li et al., 24 May 2025).

A third family targets explanation, representation learning, and semantic enrichment. BEEF fuses a flattened predicted trajectory $G_I^{(i)}(p) + G_T^{(i)}(p)=1$0 with a pooled mid-level perceptual vector $G_I^{(i)}(p) + G_T^{(i)}(p)=1$1 via BLOCK bilinear fusion,

$G_I^{(i)}(p) + G_T^{(i)}(p)=1$2

to produce cause labels or explanation features (Ben-Younes et al., 2020). DouFu encodes movement and route sequences with Transformers, performs attention fusion between them, reduces the fused sequence to a trajectory embedding, and then merges it with a global semantic embedding derived from functional zones and trip statistics (Wang et al., 2022). MVTraj uses GPS trajectories as a bridge between route and grid views, aligns representations by contrastive learning, and then fuses route, $G_I^{(i)}(p) + G_T^{(i)}(p)=1$3, grid, and $G_I^{(i)}(p) + G_T^{(i)}(p)=1$4 modalities through hierarchical cross-modal interaction (Qian et al., 2024). LLM-MLFFN adds a different semantic layer: numerical trajectory descriptors are converted into natural-language descriptions by GPT‑4o, encoded by RoBERTa, and fused with multi-scale convolutional and spatio-temporal attention features from the original trajectory signals (Li et al., 3 Mar 2026).

A fourth family focuses on contextual modulation of trajectory intent. WTTFNet augments a PoPPL destination classifier with weather-time embeddings and a Gate Multimodal Unit, then routes the fused destination estimate to destination-adapted trajectory predictors (Wu et al., 2024). TrajFusionNet predicts future pedestrian trajectory and vehicle speed, encodes the concatenated past and predicted sequence with a Sequence Attention Module, renders predicted boxes onto scene images for a Visual Attention Module, and fuses the two embeddings for crossing-intention prediction (Landry et al., 27 Aug 2025). This suggests that trajectory fusion modules often operate not only on observed motion but also on predicted motion used as a prior.

4. Supervision, objectives, and optimization regimes

The supervision strategy strongly shapes what “fusion” means. DeepDualMapper is unusually explicit: a shared prediction module is applied to $G_I^{(i)}(p) + G_T^{(i)}(p)=1$5, $G_I^{(i)}(p) + G_T^{(i)}(p)=1$6, $G_I^{(i)}(p) + G_T^{(i)}(p)=1$7, and $G_I^{(i)}(p) + G_T^{(i)}(p)=1$8 at five levels, yielding $G_I^{(i)}(p) + G_T^{(i)}(p)=1$9 supervised outputs. The total objective is a weighted sum of pixel-wise cross-entropy losses, with the refined branch weighted more heavily than modality-specific and fused branches (Wu et al., 2020). This directly supervises both the gates and the refinement decoder.

Other systems supervise only the final task outputs while leaving the fusion pathway implicit. M^3Detection trains the full second-stage module end-to-end with

$p$0

and explicitly states that there are no auxiliary trajectory or motion prediction losses for GOA, LGA, or MSTR (Li et al., 31 Oct 2025). FutrTrack likewise relies on Hungarian-style matching with regression, classification, and GIoU terms, so identity-consistent fusion is learned indirectly through the tracker’s query dynamics (Teye et al., 22 Oct 2025).

In explanation and representation learning, fusion is often jointly regularized with modality-specific objectives. BEEF minimizes $p$1 on HDD or $p$2 on BDD-X, so the explanation head remains tied to the underlying driving model (Ben-Younes et al., 2020). DouFu uses

$p$3

which keeps movement-only, route-only, and fused embeddings discriminative at the same time (Wang et al., 2022). MVTraj combines cross-view contrastive alignment with masked language modeling,

$p$4

thereby aligning trajectory-level representations while forcing fused token sequences to be predictive (Qian et al., 2024).

A distinct regime appears in WTTFNet and MPF. WTTFNet jointly supervises the preliminary and fused destination classifiers with focal loss,

$p$5

which stabilizes the intermediate trajectory branch before GMU fusion (Wu et al., 2024). MPF does not learn a fusion network in the same sense; it updates a belief distribution online by generalized Bayesian updating and IMM-style mixing, then samples trajectories from the component predictors according to that belief (Veer et al., 2023).

5. Empirical behavior and reported gains

The empirical record in the cited papers supports the practical value of multi-level fusion, although the gains are task-specific. DeepDualMapper reports the highest results among all baselines on three city-scale datasets: Porto reaches IoU $p$6 and F1 $p$7, Shanghai reaches IoU $p$8 and F1 $p$9, and Singapore reaches IoU $$\gamma_n = \Psi_{\text{sgm}}\big(W_3 [P_{\text{pre}}(W_n)^\top, e_{\text{WT},n}^\top]^\top\big),$$0 and F1 $$\gamma_n = \Psi_{\text{sgm}}\big(W_3 [P_{\text{pre}}(W_n)^\top, e_{\text{WT},n}^\top]^\top\big),$$1; it also remains strongest among fusion methods under an information loss attack in which $$\gamma_n = \Psi_{\text{sgm}}\big(W_3 [P_{\text{pre}}(W_n)^\top, e_{\text{WT},n}^\top]^\top\big),$$2 of the image area and $$\gamma_n = \Psi_{\text{sgm}}\big(W_3 [P_{\text{pre}}(W_n)^\top, e_{\text{WT},n}^\top]^\top\big),$$3 of the trajectory area are blanked out (Wu et al., 2020).

In multi-frame 3D detection, M^3Detection raises SFGFusion on VoD from $$\gamma_n = \Psi_{\text{sgm}}\big(W_3 [P_{\text{pre}}(W_n)^\top, e_{\text{WT},n}^\top]^\top\big),$$4 to $$\gamma_n = \Psi_{\text{sgm}}\big(W_3 [P_{\text{pre}}(W_n)^\top, e_{\text{WT},n}^\top]^\top\big),$$5 and HGSFusion from $$\gamma_n = \Psi_{\text{sgm}}\big(W_3 [P_{\text{pre}}(W_n)^\top, e_{\text{WT},n}^\top]^\top\big),$$6 to $$\gamma_n = \Psi_{\text{sgm}}\big(W_3 [P_{\text{pre}}(W_n)^\top, e_{\text{WT},n}^\top]^\top\big),$$7 on Entire Annotated Area mAP; on TJ4DRadSet it raises SFGFusion from $$\gamma_n = \Psi_{\text{sgm}}\big(W_3 [P_{\text{pre}}(W_n)^\top, e_{\text{WT},n}^\top]^\top\big),$$8 to $$\gamma_n = \Psi_{\text{sgm}}\big(W_3 [P_{\text{pre}}(W_n)^\top, e_{\text{WT},n}^\top]^\top\big),$$9 and HGSFusion from $$f_{\text{fuse}} = \gamma_n \odot h_v + (1-\gamma_n)\odot h_e.$$0 to $$f_{\text{fuse}} = \gamma_n \odot h_v + (1-\gamma_n)\odot h_e.$$1 on 3D mAP (Li et al., 31 Oct 2025). In cooperative V2X prediction, CoPAD improves over its single-vehicle version from minADE $$f_{\text{fuse}} = \gamma_n \odot h_v + (1-\gamma_n)\odot h_e.$$2, minFDE $$f_{\text{fuse}} = \gamma_n \odot h_v + (1-\gamma_n)\odot h_e.$$3, MR $$f_{\text{fuse}} = \gamma_n \odot h_v + (1-\gamma_n)\odot h_e.$$4 to minADE $$f_{\text{fuse}} = \gamma_n \odot h_v + (1-\gamma_n)\odot h_e.$$5, minFDE $$f_{\text{fuse}} = \gamma_n \odot h_v + (1-\gamma_n)\odot h_e.$$6, MR $$f_{\text{fuse}} = \gamma_n \odot h_v + (1-\gamma_n)\odot h_e.$$7 (Wu et al., 19 Sep 2025). In asynchronous camera-LiDAR 3D MOT, Fusion-Poly reaches $$f_{\text{fuse}} = \gamma_n \odot h_v + (1-\gamma_n)\odot h_e.$$8 AMOTA on the nuScenes test set (Wu et al., 9 Mar 2026). In camera-LiDAR fusion tracking, FutrTrack achieves an aMOTA of $$f_{\text{fuse}} = \gamma_n \odot h_v + (1-\gamma_n)\odot h_e.$$9 on the nuScenes test set (Teye et al., 22 Oct 2025).

Contextual and semantic fusion modules show comparable gains in their own regimes. WTTFNet reports a $$\boldsymbol{O}_a^b = \text{softmax}\left(\frac{Q_a K_b^\top}{\sqrt{d_k}}\right)V_b,$$0 increase in classification accuracy and $$\boldsymbol{O}_a^b = \text{softmax}\left(\frac{Q_a K_b^\top}{\sqrt{d_k}}\right)V_b,$$1 and $$\boldsymbol{O}_a^b = \text{softmax}\left(\frac{Q_a K_b^\top}{\sqrt{d_k}}\right)V_b,$$2 reductions of average and final displacement error over the original PoPPL baseline on the Osaka ATC dataset (Wu et al., 2024). MTFT reports a comprehensive performance improvement of more than $$\boldsymbol{O}_a^b = \text{softmax}\left(\frac{Q_a K_b^\top}{\sqrt{d_k}}\right)V_b,$$3 on the HighD dataset for incomplete vehicle trajectory prediction (Liu et al., 2024). LLM-MLFFN reports classification accuracy of over $$\boldsymbol{O}_a^b = \text{softmax}\left(\frac{Q_a K_b^\top}{\sqrt{d_k}}\right)V_b,$$4 on the Waymo open trajectory dataset (Li et al., 3 Mar 2026). TrajFusionNet reports the lowest total inference time among the compared methods, with $$\boldsymbol{O}_a^b = \text{softmax}\left(\frac{Q_a K_b^\top}{\sqrt{d_k}}\right)V_b,$$5 ms for the full model and $$\boldsymbol{O}_a^b = \text{softmax}\left(\frac{Q_a K_b^\top}{\sqrt{d_k}}\right)V_b,$$6 ms for TrajFusionNet-Small, while also achieving state-of-the-art results across the three most commonly used datasets for pedestrian crossing intention prediction (Landry et al., 27 Aug 2025). MPF, finally, is notable because it improves consistency rather than a single metric: on nuPlan-mini it attains the best minADE $$\boldsymbol{O}_a^b = \text{softmax}\left(\frac{Q_a K_b^\top}{\sqrt{d_k}}\right)V_b,$$7 and minFDE $$\boldsymbol{O}_a^b = \text{softmax}\left(\frac{Q_a K_b^\top}{\sqrt{d_k}}\right)V_b,$$8 and the lowest Mean Difference from Best, $$\boldsymbol{O}_a^b = \text{softmax}\left(\frac{Q_a K_b^\top}{\sqrt{d_k}}\right)V_b,$$9 (Veer et al., 2023).

These results do not imply that a single fusion mechanism dominates. Rather, they indicate that multi-level fusion is effective when the chosen levels reflect the structure of the task: pixel scale for map extraction, BEV object scale for 3D detection, view and identity scale for tracking, or semantic-description scale for behavior classification.

6. Limitations, controversies, and extension paths

The cited papers also identify recurring limitations. DeepDualMapper notes trajectory sparsity, localization error, and the fact that rasterization discards temporal order and speed/direction information; it also uses only point counts per pixel (Wu et al., 2020). BEEF depends on expensive cause labels or textual justifications and does not study robustness across domains or RL-based drivers (Ben-Younes et al., 2020). MVTraj assumes accurate GPS-map alignment, treats road topology and POI semantics as static, and incurs higher model size and training or inference cost than single-view baselines (Qian et al., 2024). LLM-MLFFN uses only speed, acceleration, and jerk, and GPT‑4o is used only for generating descriptions rather than as an online or jointly optimized component (Li et al., 3 Mar 2026). The multi-modal UAV fusion framework predicts per-frame UAV position without an explicit temporal model inside the fusion network; temporal smoothing is delegated to outlier correction and sliding-window averaging (Gao et al., 26 Jan 2026). Fusion-Poly shows that asynchronous observations can degrade performance unless frequency-aware matching and confidence-calibrated updates are added (Wu et al., 9 Mar 2026).

The extension paths are correspondingly consistent. DeepDualMapper explicitly lists attention or transformer-based fusion, temporal modeling of trajectories, uncertainty and confidence weighting, graph-based refinement, and richer trajectory channels such as density, mean speed, direction histogram, and temporal usage (Wu et al., 2020). M^3Detection motivates trajectory-based multi-level fusion directly on intermediate BEV features as an alternative to re-extracting point-cloud features, and its design suggests broader reuse in other camera-radar systems (Li et al., 31 Oct 2025). MVTraj points toward additional modalities, dynamic graphs, and richer pretext tasks such as temporal order prediction and missing-modality prediction (Qian et al., 2024). LLM-MLFFN suggests adding camera, LiDAR, radar, V2X, or graph-based social interaction signals (Li et al., 3 Mar 2026).

A broader inference is therefore warranted. The most durable formulation of a multi-level trajectory fusion module is not a specific layer type but a compositional strategy: preserve modality-specific representations long enough to exploit their inductive biases, derive fusion signals that encode reliability or continuity, and apply fusion repeatedly at the level where each ambiguity is actually resolved—across pixels, time steps, objects, modes, views, or predictor families.