PatchTraj: Dynamic Multi-Scale Trajectory Prediction

Updated 7 July 2026

The paper introduces PatchTraj, a dynamic patch-based framework that unifies time-domain and frequency-domain representations for pedestrian trajectory prediction.
It employs multi-scale patch partitioning with adaptive embedding, MoE routing, and hierarchical FPN fusion to capture local motion details and long-range dependencies.
Empirical results on benchmarks like ETH-UCY, SDD, and JRDB demonstrate significant improvements in ADE and FDE metrics, validating its effectiveness.

Searching arXiv for the exact "PatchTraj" paper and closely related entries to ground the article. PatchTraj is a dynamic patch-based trajectory prediction framework for pedestrian trajectory forecasting that unifies time-domain and frequency-domain representations within a Transformer encoder-decoder architecture. In the formulation of "PatchTraj: Dynamic Patch Representation Learning for Time-Frequency Trajectory Prediction" (Liu et al., 25 Jul 2025), the method addresses two limitations attributed to earlier point-based and grid-based approaches: insufficient modeling of human motion dynamics across local and long-range scales, and limited interaction between time-domain and frequency-domain representations. Its central mechanism is multi-scale patch partitioning of trajectory sequences, followed by adaptive embedding, hierarchical aggregation, cross-modal attention, and autoregressive decoding for future trajectory prediction (Liu et al., 25 Jul 2025).

1. Definition and scope

PatchTraj is defined in the literature as a framework for pedestrian trajectory prediction: given an observed trajectory over a history window, the goal is to predict the future trajectory over a target horizon (Liu et al., 25 Jul 2025). For pedestrian $i$ , the observed trajectory is written as

$\mathbf{X}^{i}=[\mathbf{x}^{i}_{1},\mathbf{x}^{i}_{2},...,\mathbf{x}^{i}_{T_{obs}] \in \mathbb{R}^{T_{obs} \times d},$

and the future trajectory as

$\mathbf{Y}^{i}=[\mathbf{y}^{i}_{T_{obs}+1},...,\mathbf{y}^{i}_{T_{pred}] \in \mathbb{R}^{T_{pred} \times d}.$

The paper frames the prediction problem as modeling

$p(\mathbf{Y}^{i} \mid \mathbf{X}^{i}, \mathbf{X}^{-i}, \mathbf{C}),$

where $\mathbf{X}^{-i}$ denotes other pedestrians’ trajectories and $\mathbf{C}$ optional context such as scene semantics (Liu et al., 25 Jul 2025).

The method’s defining representational move is to replace point tokens or spatial grids with multi-scale, non-overlapping temporal patches. In the paper’s terminology, a patch is a contiguous trajectory segment, and “dynamic patch representation learning” refers concretely to using a predefined set of patch sizes $\mathcal S$ rather than a single fixed segmentation scale (Liu et al., 25 Jul 2025). This allows the model to represent fine-grained motions with small patches and broader trends with larger patches.

A terminological caveat is important. The exact model name PatchTraj is used in the pedestrian-trajectory paper (Liu et al., 25 Jul 2025). Other works in adjacent areas use patch-based trajectory ideas without that official name. For example, "A Prefixed Patch Time Series Transformer for Two-Point Boundary Value Problems in Three-Body Problems" (Hatakeyama et al., 2 Apr 2025) does not define or officially name its method “PatchTraj,” and instead uses “Prefixed Patch Time Series Transformer” or “Prefixed PatchTST.” This suggests that PatchTraj should be understood primarily as the specific trajectory-prediction framework introduced in (Liu et al., 25 Jul 2025), rather than as a generic label for all patch-based trajectory models.

2. Motivating deficiencies in prior trajectory representations

The motivation for PatchTraj is organized around two representational critiques. First, point-based methods are said to atomize motion into isolated observations, which makes it harder to encode motion semantics that span multiple points, such as short turns, pauses, or stepping rhythm (Liu et al., 25 Jul 2025). Second, grid-based methods are said to introduce artificial quantization, which can distort precise geometry and weaken smooth motion modeling (Liu et al., 25 Jul 2025).

The paper argues that trajectory prediction requires simultaneous modeling of three structures: local motion details, long-range spatiotemporal dependencies, and time-frequency interactions (Liu et al., 25 Jul 2025). Local details include short-term continuity, small direction changes, and speed variation. Long-range structure includes sustained trends and global path development. The frequency-domain claim is that trajectories also contain periodic and trend-like regularities that may be more compactly expressed in spectral form.

This framing differentiates PatchTraj from architectures that only concatenate multiple modalities late in the pipeline. The paper explicitly argues that simple concatenation of time and frequency embeddings gives only inconsistent gains, and that the problem concerns not only fusion but also the underlying tokenization of the trajectory (Liu et al., 25 Jul 2025). A plausible implication is that PatchTraj treats representation granularity itself as a principal modeling variable, rather than merely an implementation detail.

3. Representation learning: time branch, frequency branch, and dynamic patching

PatchTraj uses a two-branch representation consisting of a raw time-sequence branch and a frequency-component branch (Liu et al., 25 Jul 2025). In implementation, each 2D trajectory point is represented as a 6-dimensional vector combining absolute position, relative displacement, and velocity, though the exact ordering is not specified (Liu et al., 25 Jul 2025).

For the frequency branch, the observed trajectory is padded by repeating the final observed timestep $T_{pred}$ times, yielding total length

$T = T_{obs}+T_{pred}.$

A Type-II DCT is then applied along the temporal axis, after which only the first $l$ low-frequency coefficients are retained: $\mathbf{X}^{i}=[\mathbf{x}^{i}_{1},\mathbf{x}^{i}_{2},...,\mathbf{x}^{i}_{T_{obs}] \in \mathbb{R}^{T_{obs} \times d},$0 The intended motivation is that low frequencies preserve global motion trends and filter high-frequency noise (Liu et al., 25 Jul 2025).

Dynamic patch partitioning is defined through a set of patch sizes

$\mathbf{X}^{i}=[\mathbf{x}^{i}_{1},\mathbf{x}^{i}_{2},...,\mathbf{x}^{i}_{T_{obs}] \in \mathbb{R}^{T_{obs} \times d},$1

For each $\mathbf{X}^{i}=[\mathbf{x}^{i}_{1},\mathbf{x}^{i}_{2},...,\mathbf{x}^{i}_{T_{obs}] \in \mathbb{R}^{T_{obs} \times d},$2, the sequence is partitioned into $\mathbf{X}^{i}=[\mathbf{x}^{i}_{1},\mathbf{x}^{i}_{2},...,\mathbf{x}^{i}_{T_{obs}] \in \mathbb{R}^{T_{obs} \times d},$3 non-overlapping patches (Liu et al., 25 Jul 2025). The paper gives dataset-specific scale sets:

ETH-UCY / SDD: $\mathbf{X}^{i}=[\mathbf{x}^{i}_{1},\mathbf{x}^{i}_{2},...,\mathbf{x}^{i}_{T_{obs}] \in \mathbb{R}^{T_{obs} \times d},$4
NBA: $\mathbf{X}^{i}=[\mathbf{x}^{i}_{1},\mathbf{x}^{i}_{2},...,\mathbf{x}^{i}_{T_{obs}] \in \mathbb{R}^{T_{obs} \times d},$5
JRDB: $\mathbf{X}^{i}=[\mathbf{x}^{i}_{1},\mathbf{x}^{i}_{2},...,\mathbf{x}^{i}_{T_{obs}] \in \mathbb{R}^{T_{obs} \times d},$6 (Liu et al., 25 Jul 2025)

Although the text repeatedly uses terms such as “adaptive” and “dynamic,” the described implementation uses a predefined multi-scale patch set rather than a learned per-sample segmentation rule (Liu et al., 25 Jul 2025). This distinction matters: PatchTraj is dynamic in its use of multiple scales, not in the sense of free-form learned boundary discovery.

4. Architecture and computational pipeline

The end-to-end PatchTraj pipeline proceeds through input representation, dual-branch patchification, adaptive embedding, hierarchical aggregation, cross-domain enhancement, Transformer encoding, and autoregressive decoding (Liu et al., 25 Jul 2025).

At the embedding stage, each patch scale is processed by a Mixture-of-Experts (MoE) module. The gating network is given as

$\mathbf{X}^{i}=[\mathbf{x}^{i}_{1},\mathbf{x}^{i}_{2},...,\mathbf{x}^{i}_{T_{obs}] \in \mathbb{R}^{T_{obs} \times d},$7

For scale $\mathbf{X}^{i}=[\mathbf{x}^{i}_{1},\mathbf{x}^{i}_{2},...,\mathbf{x}^{i}_{T_{obs}] \in \mathbb{R}^{T_{obs} \times d},$8, the routed embedding is

$\mathbf{X}^{i}=[\mathbf{x}^{i}_{1},\mathbf{x}^{i}_{2},...,\mathbf{x}^{i}_{T_{obs}] \in \mathbb{R}^{T_{obs} \times d},$9

which means that top-$\mathbf{Y}^{i}=[\mathbf{y}^{i}_{T_{obs}+1},...,\mathbf{y}^{i}_{T_{pred}] \in \mathbb{R}^{T_{pred} \times d}.$0 experts are activated for each patch scale (Liu et al., 25 Jul 2025). The exact number of experts and top-$\mathbf{Y}^{i}=[\mathbf{y}^{i}_{T_{obs}+1},...,\mathbf{y}^{i}_{T_{pred}] \in \mathbb{R}^{T_{pred} \times d}.$1 are not reported in the provided text.

After scale-specific embeddings are produced, PatchTraj aggregates them through an FPN-like hierarchical fusion. Expert outputs for each scale are averaged, projected with Conv1D, and fused with top-down upsampling: $\mathbf{Y}^{i}=[\mathbf{y}^{i}_{T_{obs}+1},...,\mathbf{y}^{i}_{T_{pred}] \in \mathbb{R}^{T_{pred} \times d}.$2

$\mathbf{Y}^{i}=[\mathbf{y}^{i}_{T_{obs}+1},...,\mathbf{y}^{i}_{T_{pred}] \in \mathbb{R}^{T_{pred} \times d}.$3

$\mathbf{Y}^{i}=[\mathbf{y}^{i}_{T_{obs}+1},...,\mathbf{y}^{i}_{T_{pred}] \in \mathbb{R}^{T_{pred} \times d}.$4

The intended effect is to inject coarse-scale semantics into fine-scale representations (Liu et al., 25 Jul 2025).

The two branches then interact by cross-domain enhancement (CDE) through cross-attention. Time-domain features query frequency-domain features: $\mathbf{Y}^{i}=[\mathbf{y}^{i}_{T_{obs}+1},...,\mathbf{y}^{i}_{T_{pred}] \in \mathbb{R}^{T_{pred} \times d}.$5 followed by residual updates

$\mathbf{Y}^{i}=[\mathbf{y}^{i}_{T_{obs}+1},...,\mathbf{y}^{i}_{T_{pred}] \in \mathbb{R}^{T_{pred} \times d}.$6

The fused representation is then concatenated as

$\mathbf{Y}^{i}=[\mathbf{y}^{i}_{T_{obs}+1},...,\mathbf{y}^{i}_{T_{pred}] \in \mathbb{R}^{T_{pred} \times d}.$7

This design is meant to support complementary exchange between temporal and spectral cues rather than simple side-by-side concatenation (Liu et al., 25 Jul 2025).

The prediction module is a Transformer encoder-decoder. The encoder is described as a vanilla Transformer encoder operating on fused patch embeddings. The decoder is autoregressive, begins from a learnable prediction token

$\mathbf{Y}^{i}=[\mathbf{y}^{i}_{T_{obs}+1},...,\mathbf{y}^{i}_{T_{pred}] \in \mathbb{R}^{T_{pred} \times d}.$8

and produces predicted coordinates through an MLP head: $\mathbf{Y}^{i}=[\mathbf{y}^{i}_{T_{obs}+1},...,\mathbf{y}^{i}_{T_{pred}] \in \mathbb{R}^{T_{pred} \times d}.$9 Here $p(\mathbf{Y}^{i} \mid \mathbf{X}^{i}, \mathbf{X}^{-i}, \mathbf{C}),$ 0 is the number of future hypotheses (Liu et al., 25 Jul 2025).

5. Objective function, training configuration, and datasets

PatchTraj uses a combined marginal and joint objective: $p(\mathbf{Y}^{i} \mid \mathbf{X}^{i}, \mathbf{X}^{-i}, \mathbf{C}),$ 1

$p(\mathbf{Y}^{i} \mid \mathbf{X}^{i}, \mathbf{X}^{-i}, \mathbf{C}),$ 2

$p(\mathbf{Y}^{i} \mid \mathbf{X}^{i}, \mathbf{X}^{-i}, \mathbf{C}),$ 3

with implementation values

$p(\mathbf{Y}^{i} \mid \mathbf{X}^{i}, \mathbf{X}^{-i}, \mathbf{C}),$ 4

The stated interpretation is that the marginal loss selects the best hypothesis per agent, whereas the joint loss selects a single hypothesis index that best explains all agents jointly (Liu et al., 25 Jul 2025).

The principal training settings reported for PatchTraj are:

hidden dimension $p(\mathbf{Y}^{i} \mid \mathbf{X}^{i}, \mathbf{X}^{-i}, \mathbf{C}),$ 5
Transformer layers $p(\mathbf{Y}^{i} \mid \mathbf{X}^{i}, \mathbf{X}^{-i}, \mathbf{C}),$ 6
attention heads $p(\mathbf{Y}^{i} \mid \mathbf{X}^{i}, \mathbf{X}^{-i}, \mathbf{C}),$ 7
batch size $p(\mathbf{Y}^{i} \mid \mathbf{X}^{i}, \mathbf{X}^{-i}, \mathbf{C}),$ 8
learning rate $p(\mathbf{Y}^{i} \mid \mathbf{X}^{i}, \mathbf{X}^{-i}, \mathbf{C}),$ 9, halved every 10 epochs
training epochs $\mathbf{X}^{-i}$ 0
optimizer AdamW
framework PyTorch
hardware: single RTX 4090 (Liu et al., 25 Jul 2025)

The method is evaluated on four benchmarks with task-specific observation and prediction horizons:

ETH-UCY: observe 8 frames (3.2s), predict 12 frames (4.8s)
SDD: observe 8 frames (3.2s), predict 12 frames (4.8s)
NBA: observe 10 frames (2.0s), predict 20 frames (4.0s)
JRDB: observe 9 frames (3.6s), predict 12 frames (4.8s) (Liu et al., 25 Jul 2025)

The evaluation metrics are ADE, FDE, and their multimodal counterparts $\mathbf{X}^{-i}$ 1 and $\mathbf{X}^{-i}$ 2 (Liu et al., 25 Jul 2025).

6. Empirical performance and ablation evidence

PatchTraj reports state-of-the-art or near-state-of-the-art performance on ETH-UCY, SDD, NBA, and JRDB, with particularly strong gains on JRDB, SDD, and ETH-UCY average (Liu et al., 25 Jul 2025).

On JRDB deterministic evaluation with $\mathbf{X}^{-i}$ 3, PatchTraj reports 0.20 / 0.40 for ADE/FDE, compared with 0.26 / 0.48 for NMRF. The paper states that this corresponds to 23.1% ADE improvement and 16.7% FDE improvement over NMRF (Liu et al., 25 Jul 2025).

On JRDB multimodal evaluation with $\mathbf{X}^{-i}$ 4, PatchTraj reports:

1.2s: 0.02 / 0.03
2.4s: 0.05 / 0.08
3.6s: 0.08 / 0.13
4.8s: 0.11 / 0.19 which outperforms NMRF’s 0.15 / 0.23 at 4.8s (Liu et al., 25 Jul 2025).

On SDD with $\mathbf{X}^{-i}$ 5, PatchTraj reports 6.58 / 11.14, improving over V $\mathbf{X}^{-i}$ 6Net’s 7.12 / 11.39 and yielding an 8.6% ADE improvement over NMRF (Liu et al., 25 Jul 2025). On ETH-UCY, the per-subset results average to 0.17 / 0.31, compared with NMRF’s 0.19 / 0.32, which the paper summarizes as 10.5% ADE reduction and 3.1% FDE reduction (Liu et al., 25 Jul 2025). On NBA, PatchTraj reports 0.68 / 0.94, which is the best ADE among the compared methods but not the best FDE; the paper attributes this to the absence of explicit intention-aware modeling (Liu et al., 25 Jul 2025).

The ablation study is particularly central to the method’s interpretation. Starting from a vanilla Transformer with time branch only, the paper reports the following progression:

Configuration	NBA	SDD	ETH-UCY
Vanilla Transformer + time branch only	1.02 / 1.43	7.83 / 14.30	0.32 / 0.55
Add frequency branch	0.96 / 1.38	7.60 / 13.94	0.30 / 0.52
Add DPM	0.82 / 1.10	7.15 / 12.87	0.25 / 0.49
Add MoE embedding	0.77 / 1.08	6.89 / 11.71	0.23 / 0.44
Add FPN hierarchical fusion	0.71 / 1.00	6.68 / 11.64	0.20 / 0.38
Add CDE cross-domain enhancement	0.68 / 0.94	6.58 / 11.14	0.17 / 0.31

These results support four empirical claims made in the paper: frequency information helps even before patching, dynamic patching is a major contributor, MoE plus FPN further improves multi-scale representations, and explicit cross-modal attention provides an additional gain over independent branch processing (Liu et al., 25 Jul 2025).

The paper also claims “state-of-the-art performance with high efficiency,” but it does not provide explicit runtime, FLOPs, parameter-count, or memory tables in the provided text (Liu et al., 25 Jul 2025). A plausible implication is that the efficiency claim should be read as qualitative rather than fully quantified within the reported material.

7. Interpretation, limitations, and relation to nearby patch-based work

PatchTraj’s main technical contribution is representational rather than purely architectural: it changes the unit of trajectory modeling from isolated points to multi-scale patches while pairing raw time structure with truncated DCT structure (Liu et al., 25 Jul 2025). This suggests that its gains arise from a combination of segment-level tokenization, scale-aware specialization, and explicit temporal-spectral interaction.

The paper identifies one explicit limitation in the NBA setting: the absence of explicit intention-aware modeling, which is used to explain why PatchTraj improves ADE but does not achieve the best FDE there (Liu et al., 25 Jul 2025). Other limitations are implied by the method description. Patch size choices and DCT truncation lengths are dataset-specific, which suggests retuning may be necessary when moving to new domains. The method also requires patch sizes to divide sequence length, which can constrain deployment unless padding or dataset-specific design is used (Liu et al., 25 Jul 2025). In addition, while the formulation includes $\mathbf{X}^{-i}$ 7 and $\mathbf{X}^{-i}$ 8, the provided text does not elaborate an explicit scene-semantic or social-interaction submodule, which may matter in robotics or autonomous-driving settings.

PatchTraj should also be distinguished from several thematically related patch-based systems. "PatchTrack: Multiple Object Tracking Using Frame Patches" (Chen et al., 2022) is a Transformer-based joint-detection-and-tracking system that uses current-frame patches around motion-predicted locations; it is trajectory-aware in an online tracking sense, but not a trajectory-forecasting framework of the type defined by PatchTraj. Likewise, the prefixed PatchTST model for CR3BP trajectory generation (Hatakeyama et al., 2 Apr 2025) is a patch-based Transformer for boundary-value trajectory design, but the paper explicitly states that the method is not officially named PatchTraj. These comparisons indicate that PatchTraj occupies a specific niche: multi-hypothesis pedestrian trajectory forecasting using dynamic multi-scale temporal patches and time-frequency fusion.

In summary, PatchTraj denotes a trajectory-prediction framework that represents motion as multi-scale trajectory patches, learns scale-specialized embeddings through MoE routing, aggregates them hierarchically, exchanges information between temporal and spectral branches by cross-attention, and autoregressively decodes future paths (Liu et al., 25 Jul 2025). Its reported empirical gains, especially on JRDB, SDD, and ETH-UCY, together with the staged ablation evidence, position it as a representative example of patch-based time-frequency modeling in contemporary trajectory forecasting research.