TrajFusion: Fusion of Trajectories

• TrajFusion is a design principle that fuses trajectory-structured data with complementary context to support diverse applications.
• In sports, methods like TrajSV use trajectory-enhanced transformers to combine clip embeddings with visual features, achieving significant improvements in retrieval performance.
• In prediction and estimation tasks, trajectory fusion integrates spatio-temporal and reasoning data, enabling more accurate pedestrian intention prediction, pose estimation, and mathematical solution refinement.

TrajFusion denotes a family of trajectory fusion formulations rather than a single standardized architecture. In the cited literature, the term is used for several distinct but structurally related operations: fusing trajectory-derived and visual representations for sports videos, fusing sequential and visual representations for pedestrian crossing intention prediction, fusing spatial and temporal interactions in multi-agent prediction, fusing global spatio-temporal and local joint-trajectory streams for 3D human pose estimation, converting heterogeneous trajectory controls into unified conditioning for image-to-video generation, and fusing incorrect and correct reasoning trajectories during language-model fine-tuning (Wang et al., 15 Aug 2025, Landry et al., 27 Aug 2025, Deng et al., 4 Feb 2026). A common source of confusion is that “trajectory” does not always refer to geometric motion in physical space: in mathematical reasoning, it refers to reasoning trajectories, whereas in sports, autonomous driving, pose estimation, and controllable video generation it refers to spatio-temporal motion signals.

1. Scope and defining idea

Across the supplied sources, TrajFusion is best understood as the explicit combination of trajectory-structured information with complementary context. In TrajSV, the core clip-level fusion is

$c = \text{Concat}(\mathbf{t}, \mathbf{y}),$

where $\mathbf{t}$ is a trajectory-based clip embedding and $\mathbf{y}$ is a visual representation extracted with X-CLIP’s video encoder; video-level fusion then uses attention blocks and a learned seed vector to produce a compact video representation (Wang et al., 15 Aug 2025). In TrajFusionNet, fusion is late and branchwise: a Sequence Attention Module learns from observed and predicted pedestrian trajectory and vehicle speed, while a Visual Attention Module learns from scene frames with trajectory overlays; the two embeddings are projected, concatenated, and classified (Landry et al., 27 Aug 2025). In the mathematical reasoning formulation, the fused object is a supervision sequence rather than a feature tensor:

$$T_{\text{fuse}} = \operatorname{concat}\big(\tau^{\text{err}_1}, r_1, \ldots, \tau^{\text{err}_m}, r_m, \tau^{\text{corr}}\big),$$

which interleaves selected incorrect trajectories, reflection prompts, and a correct trajectory (Deng et al., 4 Feb 2026).

Setting	Fused elements	Representative paper
Sports video representation	trajectory-based clip embedding + visual features; clip embeddings + attention-based video aggregation	TrajSV (Wang et al., 15 Aug 2025)
Pedestrian crossing intention prediction	sequence-level trajectory/speed priors + visual trajectory overlays	TrajFusionNet (Landry et al., 27 Aug 2025)
Multi-agent trajectory prediction	spatial interactions + temporal interactions on an integrated 3D graph	STF (Han et al., 2023)
3D human pose estimation	global spatio-temporal stream + self-trajectory + mutual-trajectory streams	Fusionformer (Yu et al., 2022)
Image-to-video generation	SegID/TrajID/color trajectory controls + diffusion tokens	FlexTraj (Zhang et al., 9 Oct 2025)
Mathematical reasoning	incorrect trajectories + reflections + correct trajectory	TrajFusion (Deng et al., 4 Feb 2026)

This suggests that TrajFusion is less a fixed module than a recurrent design principle: preserve trajectory structure, expose it in a modality-appropriate representation, and fuse it with context at a stage chosen to balance expressivity, efficiency, and robustness.

2. TrajFusion in sports video representation

TrajSV was introduced to address three unresolved issues in sports analytics: data unavailability, lack of an effective trajectory-based framework, and requirement for sufficient supervision labels. Its pipeline comprises data preprocessing, Clip Representation Network (CRNet), and Video Representation Network (VRNet). The preprocessing module segments broadcast videos, calibrates cameras, and performs multi-object tracking to extract player and ball trajectories in field coordinates. Frame-wise camera-type classification uses a 1D CNN (3 layers, kernel size 21 frames) on 512-d PCA-reduced ResNet features; morphological opening/closing (kernel size 3) refines binary predictions, and clips shorter than 0.4 s are removed. Camera calibration follows a TVCalib-style field registration using field markings with DeepLabV3-based instance segmentation, while multi-object tracking uses FairMOT with a DLA-34 backbone; ball emphasis uses a weight of 10 for small objects (bbox area < 500 pixels) (Wang et al., 15 Aug 2025).

CRNet realizes clip-level TrajFusion through a trajectory-enhanced Transformer. Each clip is divided into non-overlapping 1 s segments, with $m=16$ segments per clip. The field is discretized into a uniform grid with cell size 3 m, and for each segment a binary segment matrix marks traversed grid cells. Matrices are mapped to a token set via Jaccard dissimilarity with threshold 0.3. The input token is

$x_i = s_i + p_i,$

where $s_i$ is the segment embedding and $p_i$ is a learnable positional embedding. Multi-head self-attention is applied over $X = [x_1,\dots,x_m] \in \mathbb{R}^{m\times d_1}$, followed by a position-wise feed-forward network that yields the trajectory-based clip embedding $\mathbf{t} \in \mathbb{R}^{d_3}$. The visual stream uses X-CLIP pretrained on Kinetics-600 to obtain $\mathbf{t}$0, and the final clip embedding is

$\mathbf{t}$1

with $\mathbf{t}$2, $\mathbf{t}$3, and $\mathbf{t}$4.

VRNet performs video-level fusion and aggregation. For each video, it selects $\mathbf{t}$5 consecutive clips, with default $\mathbf{t}$6 and $\mathbf{t}$7 for longer SoccerNet videos. The fused clip embeddings are $\mathbf{t}$8. The encoder uses two MSB layers,

$\mathbf{t}$9

and the decoder aggregates with a learned seed vector,

$\mathbf{y}$0

The paper interprets this as attention-based weighting of informative clips. Optimization is unsupervised and uses a triple contrastive loss over an original video, an intra-clip variant, and an inter-clip variant. The loss is

$\mathbf{y}$1

with default $\mathbf{y}$2, $\mathbf{y}$3, and temperature $\mathbf{y}$4.

The reported results establish the operational value of this form of TrajFusion. In sports video retrieval, the paper reports a nearly 70% improvement overall; at $\mathbf{y}$5, YouTube retrieval reaches HR@1 = 0.475 and MRR = 0.680 versus ResNet(MLP) HR@1 = 0.231 and MRR = 0.443, SoccerNet reaches HR@1 = 0.250 and MRR = 0.551 versus X-CLIP(MLP) HR@1 = 0.141 and MRR = 0.332, and SportsMOT reaches HR@1 = 0.614 and MRR = 0.475 versus ResNet(MLP) HR@1 = 0.347 and MRR = 0.125. In action spotting on SoccerNet, Baidu-AS+TrajSV reaches Avg-mAP 73.7 and yields state-of-the-art results in 9 out of 17 action categories. In captioning, commentary spotting reaches mAP@30 = 53.07 versus 49.40 and mAP@60 = 66.64 versus 63.10; DVC reports $\mathbf{y}$6, $\mathbf{y}$7, $\mathbf{y}$8, $\mathbf{y}$9, and SDVC reports SODA_c = 7.90. Ablations show that replacing CRNet with BiLSTM or LSTM reduces HR@1 to 0.419 and 0.384, replacing VRNet with Mean or MLP reduces HR@1 to 0.239 and 0.335, and removing one term of the triple contrastive loss also degrades performance. The paper further notes a deployed search engine system that embeds all database videos with TrajSV, builds an HNSW ANN index, and reuses clip-level embeddings for retrieval, action spotting, and captioning.

3. TrajFusionNet for pedestrian crossing intention prediction

TrajFusionNet is a transformer-based model for pedestrian crossing intention prediction that combines future pedestrian trajectory and vehicle speed predictions as priors. The task is to estimate

$$T_{\text{fuse}} = \operatorname{concat}\big(\tau^{\text{err}_1}, r_1, \ldots, \tau^{\text{err}_m}, r_m, \tau^{\text{corr}}\big),$$0

for a pedestrian $$T_{\text{fuse}} = \operatorname{concat}\big(\tau^{\text{err}_1}, r_1, \ldots, \tau^{\text{err}_m}, r_m, \tau^{\text{corr}}\big),$$1, where $$T_{\text{fuse}} = \operatorname{concat}\big(\tau^{\text{err}_1}, r_1, \ldots, \tau^{\text{err}_m}, r_m, \tau^{\text{corr}}\big),$$2 indicates a future crossing. Following the Kotseruba et al. benchmark, the observation window is 0.53 s, corresponding to 16 frames, and the label is whether a crossing occurs within 1–2 s after $$T_{\text{fuse}} = \operatorname{concat}\big(\tau^{\text{err}_1}, r_1, \ldots, \tau^{\text{err}_m}, r_m, \tau^{\text{corr}}\big),$$3, corresponding to 30–60 frames. The modalities are a bounding-box sequence $$T_{\text{fuse}} = \operatorname{concat}\big(\tau^{\text{err}_1}, r_1, \ldots, \tau^{\text{err}_m}, r_m, \tau^{\text{corr}}\big),$$4, ego-vehicle speed sequence $$T_{\text{fuse}} = \operatorname{concat}\big(\tau^{\text{err}_1}, r_1, \ldots, \tau^{\text{err}_m}, r_m, \tau^{\text{corr}}\big),$$5, and scene images $$T_{\text{fuse}} = \operatorname{concat}\big(\tau^{\text{err}_1}, r_1, \ldots, \tau^{\text{err}_m}, r_m, \tau^{\text{corr}}\big),$$6; the output is a binary probability

$$T_{\text{fuse}} = \operatorname{concat}\big(\tau^{\text{err}_1}, r_1, \ldots, \tau^{\text{err}_m}, r_m, \tau^{\text{corr}}\big),$$7

The model uses a weighted cross-entropy loss for classification (Landry et al., 27 Aug 2025).

The architecture has two branches with late fusion. The Sequence Attention Module (SAM) uses a non-autoregressive encoder–decoder transformer to predict future pedestrian trajectory and vehicle speed, concatenates the predicted sequence with the observed sequence, appends a type identifier indicating “past” or “predicted” to each timestep, and then applies an encoder-only transformer to obtain a sequence representation. The observed sequence is $$T_{\text{fuse}} = \operatorname{concat}\big(\tau^{\text{err}_1}, r_1, \ldots, \tau^{\text{err}_m}, r_m, \tau^{\text{corr}}\big),$$8 with $$T_{\text{fuse}} = \operatorname{concat}\big(\tau^{\text{err}_1}, r_1, \ldots, \tau^{\text{err}_m}, r_m, \tau^{\text{corr}}\big),$$9 and $m=16$0, comprising four box coordinates and speed; the predicted sequence length is $m=16$1. The prediction transformer uses 8 encoder layers and 8 decoder layers, each with 4 heads, $m=16$2, and FFN dimension 512. The sequence encoder uses 6 encoder layers, 12 heads, $m=16$3, and FFN dimension 1024, followed by a linear projection to a 40-D feature $m=16$4.

The Visual Attention Module (VAM) overlays trajectories directly onto scene frames. The frame $m=16$5 receives 15 observed rectangles, while the frame $m=16$6 receives 60 predicted rectangles. Overlays are drawn only into two channels, blue and green in BGR, and the remaining channel preserves appearance; colors are taken from the ADE20k palette. Two VAN-B2 backbones process the two images, their outputs are concatenated, and the result is projected to a 40-D feature $m=16$7. Fusion is then

$m=16$8

followed by Dense(80) $m=16$9 GELU $x_i = s_i + p_i,$0 Dense(40) $x_i = s_i + p_i,$1 GELU $x_i = s_i + p_i,$2 Dense(2) $x_i = s_i + p_i,$3 softmax. The attention equations are the standard scaled dot-product and multi-head attention:

$x_i = s_i + p_i,$4

Training is stage-wise. The trajectory prediction encoder–decoder is pretrained with

$x_i = s_i + p_i,$5

with $x_i = s_i + p_i,$6 and $x_i = s_i + p_i,$7, using z-score normalized targets. The predictor is then frozen while the SAM encoder is trained for classification; each VAN is trained independently; and a final fusion stage freezes SAM and VAM while training only the projection heads and final classifier. On an RTX 3060, the reported inference times are 12.09 ms for TrajFusionNet and 4.63 ms for TrajFusionNet-Small, with model-only time equal to model-plus-data-preprocessing time because the design uses only three lightweight modalities: boxes, ego speed, and raw images.

The quantitative results place TrajFusionNet at the top tier of the standard benchmarks. On PIE, TrajFusionNet reports Acc 0.92, AUC 0.91, F1 0.86, P 0.85, and R 0.88; on JAADall, Acc 0.89, AUC 0.86, F1 0.72, P 0.64, and R 0.82; on JAADbeh, Acc 0.74, AUC 0.68, F1 0.82, P 0.73, and R 0.93. The full model achieves the best reported F1 on JAADbeh and tied-best or state-of-the-art Acc and F1 on PIE. Ablations show that removing vehicle speed causes notable degradation on PIE and JAADbeh, removing trajectory prediction from SAM causes clear drops on all three datasets, and replacing the two-VAN design with one VAN that combines past and future overlays degrades JAADall and JAADbeh. The reported failure cases include sudden jaywalking after static observed motion, ego turning that induces apparent lateral motion of a roadside pedestrian, and low illumination with atypical trajectories.

4. Space–time fusion in trajectory prediction and human pose estimation

Related literature operationalizes closely related trajectory fusion ideas even when “TrajFusion” is not the model name. STF describes its core idea as fusing trajectory information across space and time in a single integrated computation. It constructs an integrated 3D graph whose nodes are agent–time pairs, one node per agent $x_i = s_i + p_i,$8 at each time $x_i = s_i + p_i,$9 in the history window. Spatial edges connect agents at the same time when Euclidean distance is below $s_i$0, temporal edges connect the same agent across adjacent time steps, and attention additionally considers cross-time interaction edges to spatial neighbors at $s_i$1 and $s_i$2. Node features are embedded by a two-layer MLP,

$s_i$3

and two GAT layers perform simultaneous aggregation over spatial, temporal, and cross-time neighbors (Han et al., 2023).

The graph-attention mechanism is written as

$s_i$4

$s_i$5

$s_i$6

A parallel ST-GCN branch separately captures spatial and temporal information, and the outputs of the S-T Fusion and ST-GCN branches are concatenated before a GRU seq2seq decoder predicts future positions. On ApolloScape, STF reports WSADE 1.384 and WSFDE 1.707; WSFDE is the best among reported methods, and at 3 s the weighted RMSE is 1.707 versus GRIP++ 4.026. The description attributes the gains especially to long-horizon prediction, where simultaneous space–time fusion mitigates compounding error.

Fusionformer provides a second related formulation, now for 3D human pose estimation from 2D joint sequences. The method consists of a Global Information Interaction Module, a Local Information Interaction Module, a fusion head, and a pose refinement network. The Local Information Interaction Module contains a Self-Trajectory Encoder for per-joint temporal modeling and a Cross-/Mutual-Trajectory Encoder for inter-joint temporal synergy. Given $s_i$7, trajectories are reorganized into $s_i$8, each joint trajectory is embedded and processed with temporal self-attention, and the set of trajectory tokens is then used to model deep correlation between different trajectories. In parallel, the spatio-temporal module applies a spatial encoder across joints and a temporal encoder across frames. The paper describes the overall goal as fusing “global spatio-temporal features and local joint trajectory features” through a linear network in a parallel manner (Yu et al., 2022).

The reported empirical gains are concrete. On Human3.6M with $s_i$9 frames and CPN 2D inputs, Fusionformer reports 48.7 mm under Protocol #1 and 37.9 mm under Protocol #2; the abstract states improvements of 2.4% MPJPE and 4.3% P-MPJPE over PoseFormer. On MPI-INF-3DHP, the method reports PCK 97.9, AUC 70.0, and MPJPE 28.2. Ablations show that the full combination of STE, CTE, and pose refinement yields the best Human3.6M MPJPE, and that increasing the receptive field from $p_i$0 to $p_i$1 steadily improves performance. Taken together, STF and Fusionformer suggest that trajectory fusion is effective both when the primitive objects are interacting traffic agents and when they are articulated human joints.

5. Unified trajectory control for image-to-video generation

FlexTraj is not named TrajFusion, but the supplied description presents it as a foundation for a trajectory fusion/control system for image-to-video generation. Its central representation is a unified point-based motion representation

$p_i$2

where $p_i$3 are 3D point coordinates, $p_i$4 is a segmentation ID, $p_i$5 is a temporally consistent trajectory ID, and $p_i$6 is an optional color vector. This representation supports dense control, spatially sparse control, temporally sparse control, and dynamic point sets. Two conditioning videos are rendered from $p_i$7: an ID-coded video $p_i$8, with SegID encoded in the red channel and TrajID encoded in green-blue channels, and a color-cue video $p_i$9 (Zhang et al., 9 Oct 2025).

The conditioning mechanism departs from token concatenation and ControlNet-style injection in favor of efficient sequence-concatenation. A pretrained video VAE encodes the condition videos as

$X = [x_1,\dots,x_m] \in \mathbb{R}^{m\times d_1}$0

and structure and appearance are fused with a zero-initialized projection,

$X = [x_1,\dots,x_m] \in \mathbb{R}^{m\times d_1}$1

If $X = [x_1,\dots,x_m] \in \mathbb{R}^{m\times d_1}$2 denotes noisy latent tokens of the target video and $X = [x_1,\dots,x_m] \in \mathbb{R}^{m\times d_1}$3 denotes text tokens, the diffusion transformer input is

$X = [x_1,\dots,x_m] \in \mathbb{R}^{m\times d_1}$4

With an attention mask $X = [x_1,\dots,x_m] \in \mathbb{R}^{m\times d_1}$5, condition tokens do not attend to noise or text tokens, but noise and text tokens may attend to condition tokens. The backbone is CogVideoX-5B I2V, the base DiT is frozen, and LoRA is applied on Q/K/V maps for condition tokens only. The paper reports that KV caching can reuse the condition keys and values across diffusion steps.

Training uses a standard diffusion objective

$X = [x_1,\dots,x_m] \in \mathbb{R}^{m\times d_1}$6

together with a four-stage annealing curriculum: complete dense aligned for 1,200 steps, dense aligned with dropped appearance for 2,400 steps, sparse aligned for 14,000 steps, and unaligned for 4,000 steps, with learning rate $X = [x_1,\dots,x_m] \in \mathbb{R}^{m\times d_1}$7 for aligned stages and $X = [x_1,\dots,x_m] \in \mathbb{R}^{m\times d_1}$8 for the final unaligned stage. The reported applications include dense motion cloning, sparse point dragging, temporally sparse interpolation, camera redirection, flexible action control, and mesh animations.

The empirical results are reported on DAVIS and FlexBench. On DAVIS dense control, the method reports FVD 532.4, Consistency 0.979, and TrajErr 0.017; on spatially sparse control, FVD 710.4, Consistency 0.980, and TrajErr 0.025; on temporally sparse control, FVD 837.0, Consistency 0.983, and TrajErr 0.031; and on unaligned control, FVD 622.3, Consistency 0.976, and TrajSIM 0.908. The ablation described in the supplied text states that replacing sequence-concatenation with ControlNet-style injection severely degrades trajectory accuracy, with TrajErr up to 0.131 versus 0.024–0.016 for FlexTraj in aligned tasks. The stated limitations include sensitivity to tracking and segmentation quality, difficulties under severe occlusion or topology changes, trade-offs under extremely dense constraints, and the memory and time costs of long sequences and high resolution.

6. Trajectory fusion as supervision construction in mathematical reasoning

The mathematical reasoning paper extends the meaning of TrajFusion from geometric trajectories to reasoning trajectories. Its starting point is rejection sampling fine-tuning (RFT), which samples $X = [x_1,\dots,x_m] \in \mathbb{R}^{m\times d_1}$9 candidate solutions for a problem and retains only those with correct final answers. The paper argues that this binary filtering discards teacher-generated errors that encode diagnostic information about computational slips, missing assumptions, misapplied formulas, and plausible but invalid argument paths. TrajFusion therefore reframes rejection sampling as a structured supervision construction process: selected incorrect trajectories are interleaved with brief reflection prompts and a correct trajectory, and the number of inserted incorrect segments is adaptively controlled by teacher error frequency and diversity (Deng et al., 4 Feb 2026).

Formally, for a problem $\mathbf{t} \in \mathbb{R}^{d_3}$0, the teacher generates

$\mathbf{t} \in \mathbb{R}^{d_3}$1

A verifier partitions them into correct and incorrect subsets $\mathbf{t} \in \mathbb{R}^{d_3}$2 and $\mathbf{t} \in \mathbb{R}^{d_3}$3. Incorrect trajectories are grouped by extracted final answers, and the shortest trajectory from each group is selected as a representative. The fused sequence is

$\mathbf{t} \in \mathbb{R}^{d_3}$4

Adaptive length control is defined by

$\mathbf{t} \in \mathbb{R}^{d_3}$5

$\mathbf{t} \in \mathbb{R}^{d_3}$6

When $\mathbf{t} \in \mathbb{R}^{d_3}$7, TrajFusion reduces to vanilla RFT. The training objective remains standard language modeling,

$\mathbf{t} \in \mathbb{R}^{d_3}$8

so the method requires no changes to the architecture or training objective.

The pipeline samples $\mathbf{t} \in \mathbb{R}^{d_3}$9 completions per problem with temperature 0.7 and top-p 0.95, filters malformed outputs, verifies final answers with an automatic math verifier, computes $\mathbf{t}$00, and either constructs a standard RFT sample or interleaves up to $\mathbf{t}$01 shortest representatives from the most frequent incorrect-answer groups. Reflection prompts are deliberately brief, such as “Wait… that doesn’t seem right. Let’s try again.” or “That doesn’t add up. Let’s give it another go.” Queries without any correct trajectory are excluded from training.

The empirical results show consistent gains over RFT. For LLaMA3-8B with 15k examples, TrajFusion reports 34.0 AVG versus RFT 27.9, with MATH 36.6 versus 29.5, GSM8K 79.2 versus 72.3, and TheoremQA 20.0 versus 14.9. For LLaMA3-8B with 100k examples, the averages are 41.3 versus 38.8. For DeepSeekMath-7B with 15k examples, the averages are 48.8 versus 45.8; with 100k, 52.3 versus 50.2. In long-form scaling with Qwen2.5-Math-7B-base, the 4K setting reports AIME24 Pass@1 23.3 versus 16.7, AIME25 16.7 versus 10.0, and MATH-500 82.6 versus 79.4; the 32K setting reports AIME24 Pass@1 43.3 versus 40.0 and Avg@16 40.8 versus 36.7, AIME25 Pass@1 30.0 versus 26.7 and Avg@16 29.6 versus 27.9, and MATH-500 90.4 versus 88.4. Ablations indicate a sweet spot around $\mathbf{t}$02, and HighDiv selection outperforms LowDiv by +1.67% on GSM8K and +2.50% on MATH at approximately equal average number of fused incorrect trajectories.

A common misconception would be to read this method as directly related to motion analysis; the supplied evidence instead shows a broader abstraction. Here, a “trajectory” is the ordered reasoning path of a sampled solution, and “fusion” means explicitly pairing failure-to-correction transitions within a single supervised sequence. The paper’s safety argument is correspondingly linguistic rather than geometric: harmful error reinforcement is mitigated by always ending with a verified correct trajectory, marking preceding content as likely wrong through reflection prompts, excluding problems without any correct teacher solution, and selecting diverse but concise incorrect representatives.