---
title: 'OmniTraj: Omni-Semantic & Temporal Framework'
url: https://www.emergentmind.com/topics/omnitraj
type: topic
---

# OmniTraj: Omni-Semantic & Temporal Framework

OmniTraj is a name used in recent trajectory-learning literature for distinct but conceptually related systems that pursue broader coverage across trajectory semantics or data domains. In one usage, OmniTraj denotes an omni-semantic trajectory retrieval framework that aligns raw trajectories, topology, road segments, and regions in a shared latent space to support large-scale, condition-based querying [2505.17437]. In another, it denotes a Transformer-based human trajectory prediction model pre-trained on heterogeneous data and explicitly conditioned on frame rate to support adaptive and zero-shot transfer across temporal setups [2507.23657]. A related line of work, UniTraj, provides unified infrastructure for scalable vehicle trajectory prediction across datasets and is explicitly positioned as groundwork for an “OmniTraj” objective centered on omni-modal, omni-domain coverage [2403.15098].

## 1. Terminological scope and conceptual framing

The term “OmniTraj” does not designate a single canonical method in the supplied literature. Instead, it appears in at least two 2025 formulations with different task definitions, data modalities, and evaluation protocols. The retrieval-oriented OmniTraj addresses spatio-temporal data mining and trajectory search, emphasizing generalized and flexible omni-semantic retrieval from any single modality or from combinations of modalities [2505.17437]. The prediction-oriented OmniTraj addresses human trajectory forecasting under temporal setup shift, emphasizing zero-shot transfer to unseen datasets with varying frame rates and observation or prediction horizons [2507.23657].

These two usages share a common design intuition: trajectory understanding benefits from explicit treatment of heterogeneity rather than collapsing all information into a single trajectory representation. In the retrieval case, heterogeneity is semantic, spanning raw coordinates, topology, road segments, and regions. In the prediction case, heterogeneity is temporal and multi-modal, spanning trajectories, pose, and bounding boxes, with explicit conditioning on FPS. A plausible implication is that “OmniTraj” functions less as a fixed architecture family than as a naming convention for trajectory systems that aim to remain effective across multiple representational regimes.

Within vehicle forecasting, UniTraj is not itself named OmniTraj, but it is explicitly described as providing “the critical infrastructure—data harmonization, standardized model IO, and unified metrics—to progress toward an OmniTraj goal” [2403.15098]. That positioning links the later OmniTraj terminology to an earlier unification program in trajectory modeling.

## 2. OmniTraj as omni-semantic trajectory retrieval

In the retrieval formulation, OmniTraj is a generalized and flexible trajectory retrieval framework that learns a shared, omni-semantic representation of mobility traces and supports accurate queries from any single modality or from combinations of modalities [2505.17437]. Its target problem is not forecasting but retrieval over large urban-scale trajectory corpora, where classical heuristic measures such as Dynamic Time Warping, Fréchet distance, and Hausdorff distance are computationally prohibitive and cannot answer condition-based queries such as “must pass road segment $s$” or “intersect region $R$” [2505.17437].

The framework models four complementary semantics. Raw trajectories are time-ordered sequences of spatial points, written as $T=\{(x_t,y_t,t)\}_{t=1}^{L}$ or $T=\{(x_t,y_t)\}_{t=1}^{L}$ when time is implicit. Topology is defined not as a graph neural representation but as the sparse sequence of critical points along $T$, written as $T^{(top)}=\{tp_1,tp_2,\dots,tp_k\}$ with $tp_i\in\mathbb{R}^2$ and $k\leq L$. Road semantics are map-matched segment IDs, $T^{(road)}=\{r_1,r_2,\dots,r_j\}$, and region semantics are spatial cells or polygons intersected by the trajectory, $T^{(reg)}=\{\mathcal{R}_1,\mathcal{R}_2,\dots,\mathcal{R}_m\}$ with $T\cap\mathcal{R}_i\neq\varnothing$ [2505.17437].

This decomposition is central to the claim of “omni-semantic supervision.” Omni-semantic retrieval means that retrieval is driven not only by overall geometric similarity but also by conditions over multiple semantics of movement. Omni-semantic supervision means that embeddings from the four modalities are simultaneously aligned in a shared latent space via contrastive learning, so any modality can supervise the others [2505.17437]. This directly addresses what the paper identifies as three core limitations of prior work: large-scale efficiency, condition-based queries, and single-modality rigidity.

The representation strategy is intentionally modular. Rather than operating directly on the road network graph $G=(V,E)$ with a GNN, OmniTraj encodes the map-matched segment sequence $T^{(road)}$ and the region sequence $T^{(reg)}$ [2505.17437]. This suggests that the method treats graph structure primarily through sequence semantics rather than explicit message passing on the urban network.

## 3. Retrieval architecture, alignment objective, and query processing

The retrieval architecture consists of four dedicated encoders followed by projection heads into a shared latent space [2505.17437]. The trajectory encoder uses a Transformer with patches: the raw trajectory is normalized and resampled to fixed length $L$ and split into non-overlapping patches of size $P$, each projected into a $d$-dimensional token. A learnable $[\mathrm{CLS}]$ token and positional embeddings are added, and the token sequence is processed by Transformer blocks to produce $z_{traj}\in\mathbb{R}^{d}$. Patchification reduces attention cost from $O(L^2)$ to $O((L/P)^2)$ [2505.17437].

The topology encoder also uses a Transformer, but with Rotary Positional Embedding (RoPE). Each critical point is projected to an embedding $e_i\in\mathbb{R}^{d}$, transformed by $\mathrm{RoPE}(e_i)=R(\theta_i)\cdot[e_i^{(1)};e_i^{(2)}]$, and then aggregated by Transformer blocks into $z_{top}\in\mathbb{R}^{d}$ [2505.17437]. The road encoder uses ID embeddings, RoPE, and augmentation, including reversing, discarding or truncating, local shuffling, and random replacement. The region encoder uses ID embeddings and a Transformer without explicit positional encoding, with augmentations including random shuffling and removal [2505.17437].

Each modality-specific embedding $z_{mod}$ is mapped into a shared latent space through a two-layer linear projection head, $h_{mod}=W_{mod}\cdot z_{mod}$, with $h_{mod}\in\mathbb{R}^{h}$ for $mod\in\{\text{traj},\text{top},\text{road},\text{region}\}$ [2505.17437]. Fusion for multi-modality queries can be performed either by concatenation and reprojection,
$$
z = W_{fuse}\cdot \mathrm{concat}(h_{raw},h_{top},h_{road},h_{region}),
$$
or by a weighted sum,
$$
z=\sum_i w_i h_i,
$$
with optional normalization before scoring [2505.17437].

Training uses contrastive learning with InfoNCE and cosine similarity,
$$
\mathrm{sim}(u,v)=\frac{u^\top v}{\|u\|\|v\|}.
$$
For a query embedding $h_q$ and positive $h_p$, the loss is
$$
\mathcal{L}_{\mathrm{InfoNCE}}=-\log\frac{\exp(\mathrm{sim}(h_q,h_p)/\tau)}{\sum_i \exp(\mathrm{sim}(h_q,h_i)/\tau)}.
$$
Bidirectional alignment is applied, and in practice losses are summed over modality pairs such as traj–top, traj–road, and traj–region [2505.17437]. The paper emphasizes simplicity and scalability: vanilla InfoNCE, in-batch negatives, and robustness-oriented augmentations.

Retrieval supports both single-modality and multi-modality queries. For a single modality $q_i$, candidates are ranked by cosine similarity in the shared space. For multi-modality queries, scoring can either sum weighted per-modality similarities or compare fused query and candidate embeddings [2505.17437]. Condition-based querying is implemented through a coarse-to-fine two-stage strategy: first filter candidates by coarse semantics such as region or road using set-membership predicates, then rank the filtered subset by finer topology similarities. Conditions can be composed by set operations, with AND via intersection constraints and OR via union constraints [2505.17437]. This is the mechanism that allows OmniTraj to move beyond nearest-neighbor retrieval by raw-shape similarity alone.

## 4. Retrieval data, evaluation, and empirical behavior

The retrieval OmniTraj is evaluated on taxi trajectories from Chengdu and Xi’an [2505.17437]. Each city uses approximately $1.1$M trajectories for training, $50$k for validation, and $20$k for testing. Preprocessing removes trajectories outside urban areas or with fewer than $20$ points, interpolates raw coordinates to fixed length $L=200$, partitions the city into a $16\times16$ grid with $256$ regions, and map-matches road segments. The data statistics reported are $7{,}597$ road-segment IDs for Chengdu and $6{,}018$ for Xi’an; average per-trajectory counts are topology $13.33$, road $21.08$, and region $9.52$ in Chengdu, and topology $12.02$, road $22.12$, and region $9.27$ in Xi’an [2505.17437].

The encoders use output dimension $d=256$, projection dimension $h=512$, $6$ Transformer blocks, and $8$ attention heads, with pooling via $[\mathrm{CLS}]$/BOS. Approximate parameters per encoder are reported as $\sim4.7$M. Training uses Adam with initial learning rate $2\times10^{-4}$; the implementation uses Python 3.9 and PyTorch 1.8, with training on NVIDIA A100 and evaluation on A6000 [2505.17437].

Evaluation uses Mean Rank (MR), Mean Reciprocal Rank (MRR), Hit Rate at $k$ (HR@$k$), and Containment Rate (CR@$k$) [2505.17437]. The reported baselines include heuristics such as DTW, EDR, Hausdorff, and Fréchet; learned similarity methods such as E2DTC, t2vec, TrjSR, and TrajCL; and condition-based CLIP-style baselines [2505.17437].

For similarity retrieval, OmniTraj reports state-of-the-art results across both cities. In Chengdu, HR@1 reaches $0.857$ with MRR $=0.909$, compared with TrajCL at HR@1 $=0.791$ and MRR $=0.846$. In Xi’an, HR@1 is $0.847$ with MRR $=0.903$, compared with TrajCL at HR@1 $=0.805$ and MRR $=0.832$ [2505.17437]. Fusing modalities improves accuracy over single-modality encoders, and topology is reported to contribute the finest granularity for ranking [2505.17437].

For condition-based retrieval, OmniTraj attains the highest reported CR@1 and CR@5 on both road and region conditions. In Chengdu, road CR@1 is $0.989$ and CR@5 is $0.670$, while region CR@1 is $0.989$ and CR@5 is $0.915$. In Xi’an, road CR@1 is $0.987$ and CR@5 is $0.676$, while region CR@1 is $0.994$ and CR@5 is $0.893$ [2505.17437]. The paper further reports that data augmentation substantially boosts CR@5, that replacing Transformer encoders with LSTM or CNN reduces accuracy, and that increasing encoder depth from $2$ to $6$ improves performance before saturation [2505.17437].

Scalability claims are also explicit. Processing time per $1{,}000$ queries scales from $0.334$ s at $20$k candidates to $0.391$ s at $200$k [2505.17437]. Two-stage retrieval is reported to preserve accuracy while scaling to $200$k candidates [2505.17437]. A plausible implication is that the method is designed to integrate naturally with ANN systems, consistent with the paper’s observation that the vector representation is compatible with standard approximate nearest neighbor indexes such as FAISS and HNSW.

## 5. OmniTraj as temporally adaptive human trajectory prediction

In the prediction formulation, OmniTraj is a compact, decoupled-interaction Transformer pre-trained on a unified, heterogeneous human-motion corpus and made temporally adaptive by explicit conditioning on frame rate [2507.23657]. The core problem is that most trajectory predictors hard-code a training-time temporal setup: a fixed frame rate and fixed observation and prediction horizons. When the test dataset uses a different FPS or horizon, the temporal prior becomes incorrect because a “one-step” at $10$ Hz and a “one-step” at $2$ Hz correspond to different physical durations and therefore different motion statistics [2507.23657].

The paper formalizes human trajectory prediction as forecasting future positions of a primary agent given observed trajectories of $N$ agents over $t=1,\dots,T_{obs}$. Positions are denoted $x_i^t\in\mathbb{R}^2$ or $\mathbb{R}^3$, sampled at interval $\Delta t$, with frame rate $f$ satisfying $\Delta t=1/f$ [2507.23657]. The paper distinguishes temporal generalization from dataset shift by constructing a controlled NBA-only protocol in which training uses two temporal setups and zero-shot testing uses a third unseen setup, thereby removing changes in scene, players, court, and viewpoint [2507.23657].

The architecture has three stated design pillars: explicit FPS-aware conditioning injected into input tokens, a decoupled interaction stack, and masked multi-modal pre-training on UniHuMotion++ [2507.23657]. Inputs can include trajectory, pose (2D or 3D), and bounding boxes (2D or 3D). For modality $c\in\{T,3dP,2dP,3dB,2dB\}$ and agent $i$, the observed tensor is $x_i^c\in\mathbb{R}^{T_{obs}\times e_c\times f_c}$ [2507.23657].

Each modality is embedded with its own MLP and temporal positional encoding. The temporal metadata is the frame rate $r$ in FPS, encoded via an MLP and added to every token:
$$
E_r=\mathrm{MLP}_r(r)\in\mathbb{R}^{D},\qquad e_{i,t}^c=\mathrm{MLP}_c(x_{i,t}^c,\cdot)+P_t+E_r.
$$
The paper reports that several conditioning schemes were tested in a controlled NBA study: no FPS encoder, FiLM-based conditioning, codebook embeddings, concatenation as an extra token, and MLP encoding with latent-space summation, with the last being best [2507.23657]. The stated interpretation is explicit: temporal metadata is not treated as a nuisance variable but as a signal that causally determines discretization of motion.

After token construction, the Cross-Modality Encoder (CME) uses shared-parameter Transformer layers over all modalities and agents to produce a motion-centric representation [2507.23657]. Historical social interactions are then processed by the Historical Interaction Encoder (HIE). Future reasoning is handled by the Predictive Interaction Decoder (PID), which introduces ego queries and context queries, yielding $Z_{ego}=D(H,Q_{ego}^{L})$ and $Z_{ctx}=D(H,Q_{ctx}^{L})$, followed by ego-centric cross-attention $Z_{ego}^{*}=CA(Z_{ego},Z_{ctx})$ [2507.23657]. This decoupled design is reported to separate encoding of historical social context from reasoning about predicted interactions.

Pre-training uses structured masking: $30\%$ modality masking, $50\%$ spatial masking on pose, and temporal masking that keeps the last two frames while masking $75\%$ of the others [2507.23657]. The primary agent’s representation is passed through an MLP head to predict future positions. Training uses a simple $L_2$ regression loss rather than a probabilistic head:
$$
\mathcal{L}=\sum_{t=T_{obs}+1}^{T_{pred}}\|\hat{y}_t-y_t\|_2^2.
$$
Evaluation reports ADE/FDE or MinADE$_K$/MinFDE$_K$ with $K=20$ [2507.23657].

## 6. Heterogeneous pre-training, empirical results, and relation to UniTraj

The prediction OmniTraj is pre-trained on UniHuMotion++, described as a unified human-motion data framework with $859$ hours from $12$ datasets and native support for heterogeneous temporal setups [2507.23657]. The listed datasets include NBA SportVU, JRDB-Pose, JTA, Human3.6M, AMASS, 3DPW, NuScenes, WOMD, Argoverse2, WorldPose, SDD, and Trajnet++. For zero-shot experiments, SDD and Trajnet++ are held out from pre-training [2507.23657]. Heterogeneity is handled by CME, FPS conditioning, and masking, and the paper explicitly states that no special re-sampling is required because horizons vary and are naturally handled by token length [2507.23657].

The architecture size is reported as approximately $7.5$M parameters. CME uses $6$ Transformer layers with $4$ heads; HIE uses $4$ layers with $4$ heads; the PID decoder uses $2$ layers with $4$ heads. Optimization uses Adam with learning rate $10^{-4}$, decayed by $\times0.1$ after $80\%$ of $30$ epochs, and pre-training runs on $6\times$ NVIDIA H100 $80$GB [2507.23657]. Inference speed is reported as $595\pm19$ predictions/s on JTA on a single H100 [2507.23657].

The most prominent result concerns cross-setup zero-shot transfer in NBA under an unseen $1$ FPS, $3\to3$ setup. Multi-Transmotion reports $1.68/2.15$ MinADE$_{20}$/MinFDE$_{20}$, TrajSDE reports $1.71/2.05$, OmniTraj without FPS encoder reports $1.87/2.49$, FiLM-based FPS reports $1.62/2.26$, codebook FPS reports $1.46/1.66$, and MLP FPS with latent-space summation reports $1.18/1.22$ [2507.23657]. The paper states that explicit FPS conditioning via a tiny MLP and latent summation is “both necessary and sufficient” to dramatically reduce errors under temporal setup shift [2507.23657].

On unseen datasets, the reported zero-shot gains are likewise large. On Trajnet++, ADE drops from $3.40$ for Multi-Transmotion to $1.57$ for OmniTraj trajectory-only and to $1.01$ for OmniTraj with multi-modal pre-training, corresponding to a $53.8\%$ gain and a $70.2\%$ gain respectively. On SDD, ADE drops from $3.58$ to $1.91$ and then to $0.93$, corresponding to a $46.6\%$ gain and a $74.0\%$ gain [2507.23657]. After fine-tuning, the method reports state-of-the-art or matching results on four datasets: NBA at $0.73/0.91$ MinADE$_{20}$/MinFDE$_{20}$, JTA at $0.90/1.81$ ADE/FDE with trajectory-only inference, WorldPose at $2.38/5.46$ ADE/FDE with trajectory-only inference, and ETH-UCY at $0.21/0.35$ average MinADE$_{20}$/MinFDE$_{20}$ [2507.23657].

Ablation studies attribute gains to both temporal conditioning and interaction design. On the unseen NBA setup, the chosen FPS-encoder reduces MinADE$_{20}$ by $36.9\%$ and MinFDE$_{20}$ by $51.0\%$ versus the no-FPS model [2507.23657]. Decoupled interaction modules improve from $0.91/1.33$ with no interaction modules to $0.74/0.94$ with HIE, decoder, and ego-centric cross-attention [2507.23657]. On NuScenes pedestrian prediction with only two observed frames, OmniTraj degrades from $0.66/1.37$ to $0.73/1.48$, while the cited specialized baseline degrades from $0.69/1.45$ to $1.08/2.03$ [2507.23657]. The paper also reports monotonic improvement with more pre-training data and identifies $75\%$ temporal masking as the best tested masking ratio, with MinADE$_{20}=0.9171$ at that ratio [2507.23657].

The broader “OmniTraj” agenda is clarified by comparison with UniTraj. UniTraj unifies datasets, models, and evaluation criteria for vehicle trajectory prediction, repurposes ScenarioNet as a common scenario description format, standardizes agent-centric vectorized inputs, and shows that multi-dataset training over $2{,}012$k trajectories and $1{,}337$ hours across $15$ cities improves cross-domain generalization and yields a new state-of-the-art nuScenes leaderboard result with minADE5 $=0.98$ for MTR-UniTraj [2403.15098]. The same paper states that extending toward OmniTraj would require adding pedestrians and cyclists, incorporating rasterized context and raw perception outputs, unifying 2D and 3D coordinates, and curating hard long-tail scenarios [2403.15098].

Taken together, the supplied literature gives “OmniTraj” a dual meaning. In retrieval, it denotes omni-semantic representation learning for flexible, condition-based search at city scale [2505.17437]. In prediction, it denotes temporally adaptive pre-training for zero-shot and fine-tuned human trajectory forecasting across heterogeneous temporal setups [2507.23657]. UniTraj, although differently named, provides the explicit infrastructural precursor for an omni-domain forecasting objective in autonomous-driving settings [2403.15098]. A plausible synthesis is that the term now marks a broader research direction: trajectory systems designed to remain effective when semantics, modalities, temporal discretizations, or source domains vary substantially.

Source: https://www.emergentmind.com/topics/omnitraj