---
title: 'PatchTraj: Dynamic Multi-Scale Trajectory Prediction'
url: https://www.emergentmind.com/topics/patchtraj
type: topic
---

# PatchTraj: Dynamic Multi-Scale Trajectory Prediction

Searching arXiv for the exact "PatchTraj" paper and closely related entries to ground the article.
PatchTraj is a dynamic patch-based trajectory prediction framework for pedestrian trajectory forecasting that unifies time-domain and frequency-domain representations within a Transformer encoder-decoder architecture. In the formulation of "PatchTraj: Dynamic Patch Representation Learning for Time-Frequency Trajectory Prediction" [2507.19119], the method addresses two limitations attributed to earlier point-based and grid-based approaches: insufficient modeling of human motion dynamics across local and long-range scales, and limited interaction between time-domain and frequency-domain representations. Its central mechanism is multi-scale patch partitioning of trajectory sequences, followed by adaptive embedding, hierarchical aggregation, cross-modal attention, and autoregressive decoding for future trajectory prediction [2507.19119].

## 1. Definition and scope

PatchTraj is defined in the literature as a framework for **pedestrian trajectory prediction**: given an observed trajectory over a history window, the goal is to predict the future trajectory over a target horizon [2507.19119]. For pedestrian \(i\), the observed trajectory is written as
\[
\mathbf{X}^{i}=[\mathbf{x}^{i}_{1},\mathbf{x}^{i}_{2},...,\mathbf{x}^{i}_{T_{obs}] \in \mathbb{R}^{T_{obs} \times d},
\]
and the future trajectory as
\[
\mathbf{Y}^{i}=[\mathbf{y}^{i}_{T_{obs}+1},...,\mathbf{y}^{i}_{T_{pred}] \in \mathbb{R}^{T_{pred} \times d}.
\]
The paper frames the prediction problem as modeling
\[
p(\mathbf{Y}^{i} \mid \mathbf{X}^{i}, \mathbf{X}^{-i}, \mathbf{C}),
\]
where \(\mathbf{X}^{-i}\) denotes other pedestrians’ trajectories and \(\mathbf{C}\) optional context such as scene semantics [2507.19119].

The method’s defining representational move is to replace point tokens or spatial grids with **multi-scale, non-overlapping temporal patches**. In the paper’s terminology, a patch is a contiguous trajectory segment, and “dynamic patch representation learning” refers concretely to using a predefined set of patch sizes \(\mathcal S\) rather than a single fixed segmentation scale [2507.19119]. This allows the model to represent fine-grained motions with small patches and broader trends with larger patches.

A terminological caveat is important. The exact model name **PatchTraj** is used in the pedestrian-trajectory paper [2507.19119]. Other works in adjacent areas use patch-based trajectory ideas without that official name. For example, "A Prefixed Patch Time Series Transformer for Two-Point Boundary Value Problems in Three-Body Problems" [2504.01464] does not define or officially name its method “PatchTraj,” and instead uses “Prefixed Patch Time Series Transformer” or “Prefixed PatchTST.” This suggests that PatchTraj should be understood primarily as the specific trajectory-prediction framework introduced in [2507.19119], rather than as a generic label for all patch-based trajectory models.

## 2. Motivating deficiencies in prior trajectory representations

The motivation for PatchTraj is organized around two representational critiques. First, **point-based methods** are said to atomize motion into isolated observations, which makes it harder to encode motion semantics that span multiple points, such as short turns, pauses, or stepping rhythm [2507.19119]. Second, **grid-based methods** are said to introduce artificial quantization, which can distort precise geometry and weaken smooth motion modeling [2507.19119].

The paper argues that trajectory prediction requires simultaneous modeling of three structures: local motion details, long-range spatiotemporal dependencies, and time-frequency interactions [2507.19119]. Local details include short-term continuity, small direction changes, and speed variation. Long-range structure includes sustained trends and global path development. The frequency-domain claim is that trajectories also contain periodic and trend-like regularities that may be more compactly expressed in spectral form.

This framing differentiates PatchTraj from architectures that only concatenate multiple modalities late in the pipeline. The paper explicitly argues that simple concatenation of time and frequency embeddings gives only inconsistent gains, and that the problem concerns not only fusion but also the underlying tokenization of the trajectory [2507.19119]. A plausible implication is that PatchTraj treats representation granularity itself as a principal modeling variable, rather than merely an implementation detail.

## 3. Representation learning: time branch, frequency branch, and dynamic patching

PatchTraj uses a **two-branch representation** consisting of a raw time-sequence branch and a frequency-component branch [2507.19119]. In implementation, each 2D trajectory point is represented as a **6-dimensional vector** combining absolute position, relative displacement, and velocity, though the exact ordering is not specified [2507.19119].

For the frequency branch, the observed trajectory is padded by repeating the final observed timestep \(T_{pred}\) times, yielding total length
\[
T = T_{obs}+T_{pred}.
\]
A Type-II DCT is then applied along the temporal axis, after which only the first \(l\) low-frequency coefficients are retained:
\[
\mathbf{c}^{i}=\mathbf{c}^{i}_{1:l}\in \mathbb{R}^{l \times d}.
\]
The intended motivation is that low frequencies preserve global motion trends and filter high-frequency noise [2507.19119].

Dynamic patch partitioning is defined through a set of patch sizes
\[
\mathcal{S}=\{S_1,\ldots,S_M\}.
\]
For each \(S \in \mathcal S\), the sequence is partitioned into \(P=T/S\) non-overlapping patches [2507.19119]. The paper gives dataset-specific scale sets:
- ETH-UCY / SDD: \(\{2,4,8\}\)
- NBA: \(\{2,5,10\}\)
- JRDB: \(\{1,3,9\}\) [2507.19119]

Although the text repeatedly uses terms such as “adaptive” and “dynamic,” the described implementation uses a predefined multi-scale patch set rather than a learned per-sample segmentation rule [2507.19119]. This distinction matters: PatchTraj is dynamic in its use of multiple scales, not in the sense of free-form learned boundary discovery.

## 4. Architecture and computational pipeline

The end-to-end PatchTraj pipeline proceeds through input representation, dual-branch patchification, adaptive embedding, hierarchical aggregation, cross-domain enhancement, Transformer encoding, and autoregressive decoding [2507.19119].

At the embedding stage, each patch scale is processed by a **Mixture-of-Experts (MoE)** module. The gating network is given as
\[
\mathbf{G} = \text{Softmax}(\text{MLP}(\text{Flatten}(\mathbf{X}))).
\]
For scale \(m\), the routed embedding is
\[
\mathbf{U}_{m} = \sum\nolimits_{n=1}^N \mathbb{I}(n \in \text{TopK}(\mathbf{G}{:,n,m})) \cdot \mathbf{W}_n^m(\mathbf{X}_{P_m}),
\]
which means that top-\(k\) experts are activated for each patch scale [2507.19119]. The exact number of experts and top-\(k\) are not reported in the provided text.

After scale-specific embeddings are produced, PatchTraj aggregates them through an **FPN-like hierarchical fusion**. Expert outputs for each scale are averaged, projected with Conv1D, and fused with top-down upsampling:
\[
\mathbf{\bar{U}_{p_s} = \frac{1}{|\mathcal{E}_{p_s}|}\sum_{n\in\mathcal{E}_{p_s}\mathbf{U}_{p_s}^n,
\]
\[
\mathbf{F}_{p_i} = \text{Conv1D}(\mathbf{\bar{U}_{p_i}),
\]
\[
\mathbf{P}_{p_i}  = \begin{cases} \text{Conv1D}(\mathbf{F}_{p_i}), & \text{if } i=1, \\
\text{Conv1D}(\mathbf{F}_{p_i} + \text{Upsample}(\mathbf{P}_{p_{i-1})), & \text{otherwise}. \end{cases}
\]
The intended effect is to inject coarse-scale semantics into fine-scale representations [2507.19119].

The two branches then interact by **cross-domain enhancement (CDE)** through cross-attention. Time-domain features query frequency-domain features:
\[
\mathcal{\hat{F}_{t}=\text{Attention}(Q_t,K_f,V_f) = \text{softmax}\left(\frac{Q_t K_f^T}{\sqrt{d_k}\right)V_f,
\]
followed by residual updates
\[
\mathcal{F}_t' = \mathcal{F}_t + \mathcal{\hat{F}_{t}, \quad \mathcal{F}_f' = \mathcal{F}_f + \mathcal{\hat{F}_{f}.
\]
The fused representation is then concatenated as
\[
[\mathcal F_t'; \mathcal F_f'].
\]
This design is meant to support complementary exchange between temporal and spectral cues rather than simple side-by-side concatenation [2507.19119].

The prediction module is a Transformer encoder-decoder. The encoder is described as a vanilla Transformer encoder operating on fused patch embeddings. The decoder is autoregressive, begins from a learnable prediction token
\[
\mathcal{T} \in \mathbb{R}^{T_{pred} \times D},
\]
and produces predicted coordinates through an MLP head:
\[
\mathbf{\hat{Y} = \text{MLP}(\mathcal{T}^{(N)} ) \in \mathbb{R}^{K \times T_{pred} \times 2}.
\]
Here \(K\) is the number of future hypotheses [2507.19119].

## 5. Objective function, training configuration, and datasets

PatchTraj uses a combined marginal and joint objective:
\[
\mathcal{L}_{marginal} = \sum\nolimits_{n}^{N} \operatorname{min}_{k}^{K}\|\mathbf{Y}_{n}-\hat{\mathbf{Y}_{n}^{k}\|_{2},
\]
\[
\mathcal{L}_{joint} = \operatorname{min}_{k}^{K}\sum\nolimits_{n}^{N}\|\mathbf{Y}_{n}-\hat{\mathbf{Y}_{n}^{k}\|_{2},
\]
\[
\mathcal{L} = \lambda_{1}\mathcal{L}_{marginal} + \lambda_{2}\mathcal{L}_{joint},
\]
with implementation values
\[
\lambda_1 = 5.0,\qquad \lambda_2 = 1.0.
\]
The stated interpretation is that the marginal loss selects the best hypothesis per agent, whereas the joint loss selects a single hypothesis index that best explains all agents jointly [2507.19119].

The principal training settings reported for PatchTraj are:
- hidden dimension \(D=256\)
- Transformer layers \(L=4\)
- attention heads \(H=4\)
- batch size \(12\)
- learning rate \(1\times 10^{-3}\), halved every 10 epochs
- training epochs \(200\)
- optimizer AdamW
- framework PyTorch
- hardware: single RTX 4090 [2507.19119]

The method is evaluated on four benchmarks with task-specific observation and prediction horizons:
- **ETH-UCY**: observe 8 frames (3.2s), predict 12 frames (4.8s)
- **SDD**: observe 8 frames (3.2s), predict 12 frames (4.8s)
- **NBA**: observe 10 frames (2.0s), predict 20 frames (4.0s)
- **JRDB**: observe 9 frames (3.6s), predict 12 frames (4.8s) [2507.19119]

The evaluation metrics are **ADE**, **FDE**, and their multimodal counterparts \(\text{minADE}_K\) and \(\text{minFDE}_K\) [2507.19119].

## 6. Empirical performance and ablation evidence

PatchTraj reports state-of-the-art or near-state-of-the-art performance on ETH-UCY, SDD, NBA, and JRDB, with particularly strong gains on JRDB, SDD, and ETH-UCY average [2507.19119].

On **JRDB deterministic** evaluation with \(K=1\), PatchTraj reports **0.20 / 0.40** for ADE/FDE, compared with **0.26 / 0.48** for NMRF. The paper states that this corresponds to **23.1% ADE improvement** and **16.7% FDE improvement** over NMRF [2507.19119].

On **JRDB multimodal** evaluation with \(K=20\), PatchTraj reports:
- 1.2s: **0.02 / 0.03**
- 2.4s: **0.05 / 0.08**
- 3.6s: **0.08 / 0.13**
- 4.8s: **0.11 / 0.19**
which outperforms NMRF’s **0.15 / 0.23** at 4.8s [2507.19119].

On **SDD** with \(K=20\), PatchTraj reports **6.58 / 11.14**, improving over V\(^2\)Net’s **7.12 / 11.39** and yielding an **8.6% ADE improvement** over NMRF [2507.19119]. On **ETH-UCY**, the per-subset results average to **0.17 / 0.31**, compared with NMRF’s **0.19 / 0.32**, which the paper summarizes as **10.5% ADE reduction** and **3.1% FDE reduction** [2507.19119]. On **NBA**, PatchTraj reports **0.68 / 0.94**, which is the best ADE among the compared methods but not the best FDE; the paper attributes this to the absence of explicit intention-aware modeling [2507.19119].

The ablation study is particularly central to the method’s interpretation. Starting from a vanilla Transformer with time branch only, the paper reports the following progression:

| Configuration | NBA | SDD | ETH-UCY |
|---|---|---|---|
| Vanilla Transformer + time branch only | 1.02 / 1.43 | 7.83 / 14.30 | 0.32 / 0.55 |
| Add frequency branch | 0.96 / 1.38 | 7.60 / 13.94 | 0.30 / 0.52 |
| Add DPM | 0.82 / 1.10 | 7.15 / 12.87 | 0.25 / 0.49 |
| Add MoE embedding | 0.77 / 1.08 | 6.89 / 11.71 | 0.23 / 0.44 |
| Add FPN hierarchical fusion | 0.71 / 1.00 | 6.68 / 11.64 | 0.20 / 0.38 |
| Add CDE cross-domain enhancement | 0.68 / 0.94 | 6.58 / 11.14 | 0.17 / 0.31 |

These results support four empirical claims made in the paper: frequency information helps even before patching, dynamic patching is a major contributor, MoE plus FPN further improves multi-scale representations, and explicit cross-modal attention provides an additional gain over independent branch processing [2507.19119].

The paper also claims “state-of-the-art performance with high efficiency,” but it does **not** provide explicit runtime, FLOPs, parameter-count, or memory tables in the provided text [2507.19119]. A plausible implication is that the efficiency claim should be read as qualitative rather than fully quantified within the reported material.

## 7. Interpretation, limitations, and relation to nearby patch-based work

PatchTraj’s main technical contribution is representational rather than purely architectural: it changes the unit of trajectory modeling from isolated points to multi-scale patches while pairing raw time structure with truncated DCT structure [2507.19119]. This suggests that its gains arise from a combination of segment-level tokenization, scale-aware specialization, and explicit temporal-spectral interaction.

The paper identifies one explicit limitation in the NBA setting: **the absence of explicit intention-aware modeling**, which is used to explain why PatchTraj improves ADE but does not achieve the best FDE there [2507.19119]. Other limitations are implied by the method description. Patch size choices and DCT truncation lengths are dataset-specific, which suggests retuning may be necessary when moving to new domains. The method also requires patch sizes to divide sequence length, which can constrain deployment unless padding or dataset-specific design is used [2507.19119]. In addition, while the formulation includes \(\mathbf X^{-i}\) and \(\mathbf C\), the provided text does not elaborate an explicit scene-semantic or social-interaction submodule, which may matter in robotics or autonomous-driving settings.

PatchTraj should also be distinguished from several thematically related patch-based systems. "PatchTrack: Multiple Object Tracking Using Frame Patches" [2201.00080] is a Transformer-based joint-detection-and-tracking system that uses current-frame patches around motion-predicted locations; it is trajectory-aware in an online tracking sense, but not a trajectory-forecasting framework of the type defined by PatchTraj. Likewise, the prefixed PatchTST model for CR3BP trajectory generation [2504.01464] is a patch-based Transformer for boundary-value trajectory design, but the paper explicitly states that the method is not officially named PatchTraj. These comparisons indicate that PatchTraj occupies a specific niche: **multi-hypothesis pedestrian trajectory forecasting using dynamic multi-scale temporal patches and time-frequency fusion**.

In summary, PatchTraj denotes a trajectory-prediction framework that represents motion as multi-scale trajectory patches, learns scale-specialized embeddings through MoE routing, aggregates them hierarchically, exchanges information between temporal and spectral branches by cross-attention, and autoregressively decodes future paths [2507.19119]. Its reported empirical gains, especially on JRDB, SDD, and ETH-UCY, together with the staged ablation evidence, position it as a representative example of patch-based time-frequency modeling in contemporary trajectory forecasting research.

Source: https://www.emergentmind.com/topics/patchtraj