---
title: Multi-Level Trajectory Fusion Module
url: https://www.emergentmind.com/topics/multi-level-trajectory-fusion-module
type: topic
---

# Multi-Level Trajectory Fusion Module

In the cited literature, a multi-level trajectory fusion module denotes a computational component that fuses trajectory-derived information with complementary representations at multiple resolutions, modalities, or abstraction levels. The immediate task varies widely—road-mask extraction from aerial imagery and GPS trajectories, multi-frame camera-radar 3D detection, driving-behavior explanation, trajectory representation learning, destination prediction, multi-object tracking, and hybrid prediction with rule-based planners—but the recurring principle is stable: modality-specific or level-specific encoders first preserve heterogeneous structure, and fusion is then performed over aligned feature maps, temporal windows, candidate modes, or global embeddings by gates, attention, graph aggregation, clustering, or belief-driven sampling [2002.06832][2510.27166][2307.01408].

## 1. Terminological scope and recurrent structure

Across the cited usage, “level” can refer to feature scale, temporal granularity, semantic abstraction, trajectory mode, or camera/view index. In DeepDualMapper, the relevant levels are the five decoder stages of two auxiliary U-Nets, where aerial-image features and trajectory features are fused at every scale [2002.06832]. In M^3Detection, the levels are explicitly named global-level inter-object aggregation, local-level inter-grid aggregation, and trajectory-level multi-frame spatiotemporal reasoning [2510.27166]. In BEEF, the levels are decision-level trajectory outputs and mid-level perceptual features [2012.04983]. In DouFu and MVTraj, the levels are movement, route, global semantic, GPS, road-network, and grid/POI views [2205.08356][2410.13196]. In FusionTrack and Fusion-Poly, the levels extend to frame-level, short-term temporal, cross-view, and global identity fusion [2505.18727][2603.08199].

| Representative system | Levels fused | Mechanism |
|---|---|---|
| DeepDualMapper | decoder levels of aerial-image and trajectory features | adapters + complementary-aware gates |
| M^3Detection | global, local, trajectory levels | GOA + LGA + MSTR |
| BEEF | trajectory-level decision and mid-level perception | BLOCK bilinear fusion |
| DouFu | movement, route, global semantic | attention fusion + embedding fusion |
| WTTFNet | trajectory-based destination probabilities and weather-time context | Gate Multimodal Unit |
| MVTraj | route, GPS-on-road, grid, GPS-on-grid | inter-modal interaction attention + global context attention |

This range indicates that the term is not tied to a single architecture. Instead, it names a family of modules that reconcile complementary but non-identical trajectory signals. A plausible implication is that “multi-level” is best understood as a design constraint: fusion is deferred until each source has been encoded at the level where it is most informative, and then repeated or staged so that local detail and global trend are both preserved.

## 2. Canonical operators and mathematical formulations

One common operator is competitive gating. DeepDualMapper projects image and trajectory features into a shared space with 1×1 adapters and then enforces complementary softmax gates at each decoder level. Its fused feature is
\[
A_f^{(i)} = G_I^{(i)} \odot A_I^{(i)} + G_T^{(i)} \odot A_T^{(i)},
\]
with the complementary constraint \(G_I^{(i)}(p) + G_T^{(i)}(p)=1\) for every pixel \(p\) [2002.06832]. This makes the selector explicitly decide which modality to trust where and at which scale.

A second recurring operator is gated multimodal interpolation over compact embeddings. WTTFNet projects trajectory-based destination probabilities and a weather-time embedding into a shared dimension, computes a vector gate,
\[
\gamma_n = \Psi_{\text{sgm}}\big(W_3 [P_{\text{pre}}(W_n)^\top, e_{\text{WT},n}^\top]^\top\big),
\]
and fuses the branches as
\[
f_{\text{fuse}} = \gamma_n \odot h_v + (1-\gamma_n)\odot h_e.
\]
This is a per-dimension mixture between trajectory and context modalities rather than simple concatenation [2405.18945].

A third operator is cross-modal attention. MVTraj constructs query, key, and value tensors from concatenated trajectory-level and segment-level tokens for each modality, then computes
\[
\boldsymbol{O}_a^b = \text{softmax}\left(\frac{Q_a K_b^\top}{\sqrt{d_k}}\right)V_b,
\]
aggregating route, GPS, and grid views before a shared Transformer performs global context attention [2410.13196]. MTFT uses an analogous two-stage strategy in the temporal domain: the Multi-scale Attention Head first produces multi-scale motion representations \(R_m\), and the Continuity Representation-guided Multi-scale Fusion module then uses continuity-derived queries \(Q_c\) to fuse them as
\[
e^{\text{temp}} = \text{softmax}\left(\frac{Q_c K_m^\top}{d_K}\right)V_m,
\]
thereby combining detailed motion information with overall continuity [2409.00904].

A fourth operator is late fusion over trajectory hypotheses or model outputs. ParallelNet generates independent multi-modal trajectory sets from four MobileNet v2 branches and then applies a Set Transformer over the hypothesis set [2212.10203]. Multi-Predictor Fusion instead mixes whole trajectory sets from a learning-based predictor and a rule-based predictor according to an online belief distribution \(b_t=[b_t^l,b_t^r]\), producing an explicit mixture at the trajectory level rather than the feature level [2307.01408].

## 3. Representative architectural families

A first family is spatially aligned raster fusion. DeepDualMapper rasterizes GPS trajectories onto the same 2D grid as the aerial image, applies Gated Fusion Modules at all five decoder stages, and refines the fused hierarchy with a Densely Supervised Refinement Decoder in a coarse-to-fine manner [2002.06832]. This formulation is explicitly pixel-wise and benefits from exact alignment between the trajectory image and the RGB crop.

A second family is object-centric spatiotemporal fusion for perception and tracking. M^3Detection stores intermediate BEV features in a memory bank, uses tracked reference trajectories and candidate proposals, applies global-level inter-object aggregation on global BEV crops, local-level inter-grid aggregation with cross-level deformable attention on expanded local patches, and trajectory-level multi-frame attention over per-track feature sequences [2510.27166]. CoPAD begins even earlier, using Hungarian matching and Kalman filtering to fuse vehicle and infrastructure trajectories, then applies Past Time Attention, mode attention, and an anchor-oriented decoder [2509.15984]. Fusion-Poly performs frequency-aware cascade matching at synchronized and asynchronous timestamps, then fuses motion state and existence state through differential Kalman updates and Noisy-OR confidence updates [2603.08199]. FutrTrack combines a temporal smoother over detector outputs with a camera-LiDAR fusion transformer that propagates identity by query reuse rather than an explicit motion model [2510.19981]. FusionTrack extends the same multi-level logic to arbitrary multi-view environments with a Tracklet Memory Pool, a ReID Transformer, an Object Update Module with cross-frame and cross-view attention, and view-guided hierarchical clustering [2505.18727].

A third family targets explanation, representation learning, and semantic enrichment. BEEF fuses a flattened predicted trajectory \(m_t\) with a pooled mid-level perceptual vector \(v_t^L\) via BLOCK bilinear fusion,
\[
\hat c_t = W_c\left(D \times_1 (W_m m_t) \times_2 (W_v v_t^L)\right),
\]
to produce cause labels or explanation features [2012.04983]. DouFu encodes movement and route sequences with Transformers, performs attention fusion between them, reduces the fused sequence to a trajectory embedding, and then merges it with a global semantic embedding derived from functional zones and trip statistics [2205.08356]. MVTraj uses GPS trajectories as a bridge between route and grid views, aligns representations by contrastive learning, and then fuses route, \(\mathcal{T}^{p|r}\), grid, and \(\mathcal{T}^{p|g}\) modalities through hierarchical cross-modal interaction [2410.13196]. LLM-MLFFN adds a different semantic layer: numerical trajectory descriptors are converted into natural-language descriptions by GPT‑4o, encoded by RoBERTa, and fused with multi-scale convolutional and spatio-temporal attention features from the original trajectory signals [2603.02528].

A fourth family focuses on contextual modulation of trajectory intent. WTTFNet augments a PoPPL destination classifier with weather-time embeddings and a Gate Multimodal Unit, then routes the fused destination estimate to destination-adapted trajectory predictors [2405.18945]. TrajFusionNet predicts future pedestrian trajectory and vehicle speed, encodes the concatenated past and predicted sequence with a Sequence Attention Module, renders predicted boxes onto scene images for a Visual Attention Module, and fuses the two embeddings for crossing-intention prediction [2508.19866]. This suggests that trajectory fusion modules often operate not only on observed motion but also on predicted motion used as a prior.

## 4. Supervision, objectives, and optimization regimes

The supervision strategy strongly shapes what “fusion” means. DeepDualMapper is unusually explicit: a shared prediction module is applied to \(A_I^{(i)}\), \(A_T^{(i)}\), \(A_f^{(i)}\), and \(A_r^{(i)}\) at five levels, yielding \(5\times4=20\) supervised outputs. The total objective is a weighted sum of pixel-wise cross-entropy losses, with the refined branch weighted more heavily than modality-specific and fused branches [2002.06832]. This directly supervises both the gates and the refinement decoder.

Other systems supervise only the final task outputs while leaving the fusion pathway implicit. M^3Detection trains the full second-stage module end-to-end with
\[
L = \beta_1 L_{loc} + \beta_2 L_{cls} + \beta_3 L_{dir},
\]
and explicitly states that there are no auxiliary trajectory or motion prediction losses for GOA, LGA, or MSTR [2510.27166]. FutrTrack likewise relies on Hungarian-style matching with regression, classification, and GIoU terms, so identity-consistent fusion is learned indirectly through the tracker’s query dynamics [2510.19981].

In explanation and representation learning, fusion is often jointly regularized with modality-specific objectives. BEEF minimizes \(L_{\text{total}} = L_{\text{drive}} + L_{\text{explain}}\) on HDD or \(L_{\text{total}} = L_{\text{drive}} + L_{\text{NLG}}\) on BDD-X, so the explanation head remains tied to the underlying driving model [2012.04983]. DouFu uses
\[
\mathcal{L} = \mathcal{L}_{\text{fusion}} + \alpha \mathcal{L}_{\text{move}} + \beta \mathcal{L}_{\text{route}},
\]
which keeps movement-only, route-only, and fused embeddings discriminative at the same time [2205.08356]. MVTraj combines cross-view contrastive alignment with masked language modeling,
\[
\mathcal{L} = w_1 \mathcal{L}_{\text{align}}^{\text{multi}} + w_2 \mathcal{L}_{\text{MLM}}^{\text{multi}},
\]
thereby aligning trajectory-level representations while forcing fused token sequences to be predictive [2410.13196].

A distinct regime appears in WTTFNet and MPF. WTTFNet jointly supervises the preliminary and fused destination classifiers with focal loss,
\[
L_T = (1-A_p)L_{\text{focal}}(w,P_F(w)) + A_p L_{\text{focal}}(w,P_{\text{pre}}(w)),
\]
which stabilizes the intermediate trajectory branch before GMU fusion [2405.18945]. MPF does not learn a fusion network in the same sense; it updates a belief distribution online by generalized Bayesian updating and IMM-style mixing, then samples trajectories from the component predictors according to that belief [2307.01408].

## 5. Empirical behavior and reported gains

The empirical record in the cited papers supports the practical value of multi-level fusion, although the gains are task-specific. DeepDualMapper reports the highest results among all baselines on three city-scale datasets: Porto reaches IoU \(0.717\) and F1 \(0.835\), Shanghai reaches IoU \(0.634\) and F1 \(0.776\), and Singapore reaches IoU \(0.620\) and F1 \(0.765\); it also remains strongest among fusion methods under an information loss attack in which \(25\%\) of the image area and \(25\%\) of the trajectory area are blanked out [2002.06832].

In multi-frame 3D detection, M^3Detection raises SFGFusion on VoD from \(55.76\) to \(62.70\) and HGSFusion from \(58.96\) to \(66.81\) on Entire Annotated Area mAP; on TJ4DRadSet it raises SFGFusion from \(35.57\) to \(43.25\) and HGSFusion from \(37.21\) to \(44.73\) on 3D mAP [2510.27166]. In cooperative V2X prediction, CoPAD improves over its single-vehicle version from minADE \(1.42\), minFDE \(2.30\), MR \(0.33\) to minADE \(1.24\), minFDE \(2.00\), MR \(0.29\) [2509.15984]. In asynchronous camera-LiDAR 3D MOT, Fusion-Poly reaches \(76.5\%\) AMOTA on the nuScenes test set [2603.08199]. In camera-LiDAR fusion tracking, FutrTrack achieves an aMOTA of \(74.7\) on the nuScenes test set [2510.19981].

Contextual and semantic fusion modules show comparable gains in their own regimes. WTTFNet reports a \(23.67\%\) increase in classification accuracy and \(9.16\%\) and \(7.07\%\) reductions of average and final displacement error over the original PoPPL baseline on the Osaka ATC dataset [2405.18945]. MTFT reports a comprehensive performance improvement of more than \(39\%\) on the HighD dataset for incomplete vehicle trajectory prediction [2409.00904]. LLM-MLFFN reports classification accuracy of over \(94\%\) on the Waymo open trajectory dataset [2603.02528]. TrajFusionNet reports the lowest total inference time among the compared methods, with \(12.09\) ms for the full model and \(4.63\) ms for TrajFusionNet-Small, while also achieving state-of-the-art results across the three most commonly used datasets for pedestrian crossing intention prediction [2508.19866]. MPF, finally, is notable because it improves consistency rather than a single metric: on nuPlan-mini it attains the best minADE \(0.54\) and minFDE \(0.81\) and the lowest Mean Difference from Best, \(4.3\%\) [2307.01408].

These results do not imply that a single fusion mechanism dominates. Rather, they indicate that multi-level fusion is effective when the chosen levels reflect the structure of the task: pixel scale for map extraction, BEV object scale for 3D detection, view and identity scale for tracking, or semantic-description scale for behavior classification.

## 6. Limitations, controversies, and extension paths

The cited papers also identify recurring limitations. DeepDualMapper notes trajectory sparsity, localization error, and the fact that rasterization discards temporal order and speed/direction information; it also uses only point counts per pixel [2002.06832]. BEEF depends on expensive cause labels or textual justifications and does not study robustness across domains or RL-based drivers [2012.04983]. MVTraj assumes accurate GPS-map alignment, treats road topology and POI semantics as static, and incurs higher model size and training or inference cost than single-view baselines [2410.13196]. LLM-MLFFN uses only speed, acceleration, and jerk, and GPT‑4o is used only for generating descriptions rather than as an online or jointly optimized component [2603.02528]. The multi-modal UAV fusion framework predicts per-frame UAV position without an explicit temporal model inside the fusion network; temporal smoothing is delegated to outlier correction and sliding-window averaging [2602.00107]. Fusion-Poly shows that asynchronous observations can degrade performance unless frequency-aware matching and confidence-calibrated updates are added [2603.08199].

The extension paths are correspondingly consistent. DeepDualMapper explicitly lists attention or transformer-based fusion, temporal modeling of trajectories, uncertainty and confidence weighting, graph-based refinement, and richer trajectory channels such as density, mean speed, direction histogram, and temporal usage [2002.06832]. M^3Detection motivates trajectory-based multi-level fusion directly on intermediate BEV features as an alternative to re-extracting point-cloud features, and its design suggests broader reuse in other camera-radar systems [2510.27166]. MVTraj points toward additional modalities, dynamic graphs, and richer pretext tasks such as temporal order prediction and missing-modality prediction [2410.13196]. LLM-MLFFN suggests adding camera, LiDAR, radar, V2X, or graph-based social interaction signals [2603.02528].

A broader inference is therefore warranted. The most durable formulation of a multi-level trajectory fusion module is not a specific layer type but a compositional strategy: preserve modality-specific representations long enough to exploit their inductive biases, derive fusion signals that encode reliability or continuity, and apply fusion repeatedly at the level where each ambiguity is actually resolved—across pixels, time steps, objects, modes, views, or predictor families.

Source: https://www.emergentmind.com/topics/multi-level-trajectory-fusion-module