OmniTraj: Omni-Semantic & Temporal Framework

• OmniTraj is a trajectory learning framework that unifies heterogeneous modalities by aligning raw trajectories, topology, road segments, and regions in a shared latent space.
• The framework employs Transformer-based architectures and contrastive learning to support both omni-semantic retrieval and temporally adaptive human trajectory prediction.
• Its design enhances scalability and flexibility, enabling condition-based queries and zero-shot transfer across diverse temporal setups and data domains.

OmniTraj is a name used in recent trajectory-learning literature for distinct but conceptually related systems that pursue broader coverage across trajectory semantics or data domains. In one usage, OmniTraj denotes an omni-semantic trajectory retrieval framework that aligns raw trajectories, topology, road segments, and regions in a shared latent space to support large-scale, condition-based querying (Zhu et al., 23 May 2025). In another, it denotes a Transformer-based human trajectory prediction model pre-trained on heterogeneous data and explicitly conditioned on frame rate to support adaptive and zero-shot transfer across temporal setups (Gao et al., 31 Jul 2025). A related line of work, UniTraj, provides unified infrastructure for scalable vehicle trajectory prediction across datasets and is explicitly positioned as groundwork for an “OmniTraj” objective centered on omni-modal, omni-domain coverage (Feng et al., 2024).

1. Terminological scope and conceptual framing

The term “OmniTraj” does not designate a single canonical method in the supplied literature. Instead, it appears in at least two 2025 formulations with different task definitions, data modalities, and evaluation protocols. The retrieval-oriented OmniTraj addresses spatio-temporal data mining and trajectory search, emphasizing generalized and flexible omni-semantic retrieval from any single modality or from combinations of modalities (Zhu et al., 23 May 2025). The prediction-oriented OmniTraj addresses human trajectory forecasting under temporal setup shift, emphasizing zero-shot transfer to unseen datasets with varying frame rates and observation or prediction horizons (Gao et al., 31 Jul 2025).

These two usages share a common design intuition: trajectory understanding benefits from explicit treatment of heterogeneity rather than collapsing all information into a single trajectory representation. In the retrieval case, heterogeneity is semantic, spanning raw coordinates, topology, road segments, and regions. In the prediction case, heterogeneity is temporal and multi-modal, spanning trajectories, pose, and bounding boxes, with explicit conditioning on FPS. A plausible implication is that “OmniTraj” functions less as a fixed architecture family than as a naming convention for trajectory systems that aim to remain effective across multiple representational regimes.

Within vehicle forecasting, UniTraj is not itself named OmniTraj, but it is explicitly described as providing “the critical infrastructure—data harmonization, standardized model IO, and unified metrics—to progress toward an OmniTraj goal” (Feng et al., 2024). That positioning links the later OmniTraj terminology to an earlier unification program in trajectory modeling.

2. OmniTraj as omni-semantic trajectory retrieval

In the retrieval formulation, OmniTraj is a generalized and flexible trajectory retrieval framework that learns a shared, omni-semantic representation of mobility traces and supports accurate queries from any single modality or from combinations of modalities (Zhu et al., 23 May 2025). Its target problem is not forecasting but retrieval over large urban-scale trajectory corpora, where classical heuristic measures such as Dynamic Time Warping, Fréchet distance, and Hausdorff distance are computationally prohibitive and cannot answer condition-based queries such as “must pass road segment $s$” or “intersect region $R$” (Zhu et al., 23 May 2025).

The framework models four complementary semantics. Raw trajectories are time-ordered sequences of spatial points, written as $T=\{(x_t,y_t,t)\}_{t=1}^{L}$ or $T=\{(x_t,y_t)\}_{t=1}^{L}$ when time is implicit. Topology is defined not as a graph neural representation but as the sparse sequence of critical points along $T$, written as $T^{(top)}=\{tp_1,tp_2,\dots,tp_k\}$ with $tp_i\in\mathbb{R}^2$ and $k\leq L$. Road semantics are map-matched segment IDs, $T^{(road)}=\{r_1,r_2,\dots,r_j\}$, and region semantics are spatial cells or polygons intersected by the trajectory, $$T^{(reg)}=\{\mathcal{R}_1,\mathcal{R}_2,\dots,\mathcal{R}_m\}$$ with $R$0 (Zhu et al., 23 May 2025).

This decomposition is central to the claim of “omni-semantic supervision.” Omni-semantic retrieval means that retrieval is driven not only by overall geometric similarity but also by conditions over multiple semantics of movement. Omni-semantic supervision means that embeddings from the four modalities are simultaneously aligned in a shared latent space via contrastive learning, so any modality can supervise the others (Zhu et al., 23 May 2025). This directly addresses what the paper identifies as three core limitations of prior work: large-scale efficiency, condition-based queries, and single-modality rigidity.

The representation strategy is intentionally modular. Rather than operating directly on the road network graph $R$1 with a GNN, OmniTraj encodes the map-matched segment sequence $R$2 and the region sequence $R$3 (Zhu et al., 23 May 2025). This suggests that the method treats graph structure primarily through sequence semantics rather than explicit message passing on the urban network.

3. Retrieval architecture, alignment objective, and query processing

The retrieval architecture consists of four dedicated encoders followed by projection heads into a shared latent space (Zhu et al., 23 May 2025). The trajectory encoder uses a Transformer with patches: the raw trajectory is normalized and resampled to fixed length $R$4 and split into non-overlapping patches of size $R$5, each projected into a $R$6-dimensional token. A learnable $R$7 token and positional embeddings are added, and the token sequence is processed by Transformer blocks to produce $R$8. Patchification reduces attention cost from $R$9 to $T=\{(x_t,y_t,t)\}_{t=1}^{L}$0 (Zhu et al., 23 May 2025).

The topology encoder also uses a Transformer, but with Rotary Positional Embedding (RoPE). Each critical point is projected to an embedding $T=\{(x_t,y_t,t)\}_{t=1}^{L}$1, transformed by $T=\{(x_t,y_t,t)\}_{t=1}^{L}$2, and then aggregated by Transformer blocks into $T=\{(x_t,y_t,t)\}_{t=1}^{L}$3 (Zhu et al., 23 May 2025). The road encoder uses ID embeddings, RoPE, and augmentation, including reversing, discarding or truncating, local shuffling, and random replacement. The region encoder uses ID embeddings and a Transformer without explicit positional encoding, with augmentations including random shuffling and removal (Zhu et al., 23 May 2025).

Each modality-specific embedding $T=\{(x_t,y_t,t)\}_{t=1}^{L}$4 is mapped into a shared latent space through a two-layer linear projection head, $T=\{(x_t,y_t,t)\}_{t=1}^{L}$5, with $T=\{(x_t,y_t,t)\}_{t=1}^{L}$6 for $T=\{(x_t,y_t,t)\}_{t=1}^{L}$7 (Zhu et al., 23 May 2025). Fusion for multi-modality queries can be performed either by concatenation and reprojection,

$T=\{(x_t,y_t,t)\}_{t=1}^{L}$8

or by a weighted sum,

$T=\{(x_t,y_t,t)\}_{t=1}^{L}$9

with optional normalization before scoring (Zhu et al., 23 May 2025).

Training uses contrastive learning with InfoNCE and cosine similarity,

$T=\{(x_t,y_t)\}_{t=1}^{L}$0

For a query embedding $T=\{(x_t,y_t)\}_{t=1}^{L}$1 and positive $T=\{(x_t,y_t)\}_{t=1}^{L}$2, the loss is

$T=\{(x_t,y_t)\}_{t=1}^{L}$3

Bidirectional alignment is applied, and in practice losses are summed over modality pairs such as traj–top, traj–road, and traj–region (Zhu et al., 23 May 2025). The paper emphasizes simplicity and scalability: vanilla InfoNCE, in-batch negatives, and robustness-oriented augmentations.

Retrieval supports both single-modality and multi-modality queries. For a single modality $T=\{(x_t,y_t)\}_{t=1}^{L}$4, candidates are ranked by cosine similarity in the shared space. For multi-modality queries, scoring can either sum weighted per-modality similarities or compare fused query and candidate embeddings (Zhu et al., 23 May 2025). Condition-based querying is implemented through a coarse-to-fine two-stage strategy: first filter candidates by coarse semantics such as region or road using set-membership predicates, then rank the filtered subset by finer topology similarities. Conditions can be composed by set operations, with AND via intersection constraints and OR via union constraints (Zhu et al., 23 May 2025). This is the mechanism that allows OmniTraj to move beyond nearest-neighbor retrieval by raw-shape similarity alone.

4. Retrieval data, evaluation, and empirical behavior

The retrieval OmniTraj is evaluated on taxi trajectories from Chengdu and Xi’an (Zhu et al., 23 May 2025). Each city uses approximately $T=\{(x_t,y_t)\}_{t=1}^{L}$5M trajectories for training, $T=\{(x_t,y_t)\}_{t=1}^{L}$6k for validation, and $T=\{(x_t,y_t)\}_{t=1}^{L}$7k for testing. Preprocessing removes trajectories outside urban areas or with fewer than $T=\{(x_t,y_t)\}_{t=1}^{L}$8 points, interpolates raw coordinates to fixed length $T=\{(x_t,y_t)\}_{t=1}^{L}$9, partitions the city into a $T$0 grid with $T$1 regions, and map-matches road segments. The data statistics reported are $T$2 road-segment IDs for Chengdu and $T$3 for Xi’an; average per-trajectory counts are topology $T$4, road $T$5, and region $T$6 in Chengdu, and topology $T$7, road $T$8, and region $T$9 in Xi’an (Zhu et al., 23 May 2025).

The encoders use output dimension $T^{(top)}=\{tp_1,tp_2,\dots,tp_k\}$0, projection dimension $T^{(top)}=\{tp_1,tp_2,\dots,tp_k\}$1, $T^{(top)}=\{tp_1,tp_2,\dots,tp_k\}$2 Transformer blocks, and $T^{(top)}=\{tp_1,tp_2,\dots,tp_k\}$3 attention heads, with pooling via $T^{(top)}=\{tp_1,tp_2,\dots,tp_k\}$4/BOS. Approximate parameters per encoder are reported as $T^{(top)}=\{tp_1,tp_2,\dots,tp_k\}$5M. Training uses Adam with initial learning rate $T^{(top)}=\{tp_1,tp_2,\dots,tp_k\}$6; the implementation uses Python 3.9 and PyTorch 1.8, with training on NVIDIA A100 and evaluation on A6000 (Zhu et al., 23 May 2025).

Evaluation uses Mean Rank (MR), Mean Reciprocal Rank (MRR), Hit Rate at $T^{(top)}=\{tp_1,tp_2,\dots,tp_k\}$7 (HR@$T^{(top)}=\{tp_1,tp_2,\dots,tp_k\}$8), and Containment Rate (CR@$T^{(top)}=\{tp_1,tp_2,\dots,tp_k\}$9) (Zhu et al., 23 May 2025). The reported baselines include heuristics such as DTW, EDR, Hausdorff, and Fréchet; learned similarity methods such as E2DTC, t2vec, TrjSR, and TrajCL; and condition-based CLIP-style baselines (Zhu et al., 23 May 2025).

For similarity retrieval, OmniTraj reports state-of-the-art results across both cities. In Chengdu, HR@1 reaches $tp_i\in\mathbb{R}^2$0 with MRR $tp_i\in\mathbb{R}^2$1, compared with TrajCL at HR@1 $tp_i\in\mathbb{R}^2$2 and MRR $tp_i\in\mathbb{R}^2$3. In Xi’an, HR@1 is $tp_i\in\mathbb{R}^2$4 with MRR $tp_i\in\mathbb{R}^2$5, compared with TrajCL at HR@1 $tp_i\in\mathbb{R}^2$6 and MRR $tp_i\in\mathbb{R}^2$7 (Zhu et al., 23 May 2025). Fusing modalities improves accuracy over single-modality encoders, and topology is reported to contribute the finest granularity for ranking (Zhu et al., 23 May 2025).

For condition-based retrieval, OmniTraj attains the highest reported CR@1 and CR@5 on both road and region conditions. In Chengdu, road CR@1 is $tp_i\in\mathbb{R}^2$8 and CR@5 is $tp_i\in\mathbb{R}^2$9, while region CR@1 is $k\leq L$0 and CR@5 is $k\leq L$1. In Xi’an, road CR@1 is $k\leq L$2 and CR@5 is $k\leq L$3, while region CR@1 is $k\leq L$4 and CR@5 is $k\leq L$5 (Zhu et al., 23 May 2025). The paper further reports that data augmentation substantially boosts CR@5, that replacing Transformer encoders with LSTM or CNN reduces accuracy, and that increasing encoder depth from $k\leq L$6 to $k\leq L$7 improves performance before saturation (Zhu et al., 23 May 2025).

Scalability claims are also explicit. Processing time per $k\leq L$8 queries scales from $k\leq L$9 s at $T^{(road)}=\{r_1,r_2,\dots,r_j\}$0k candidates to $T^{(road)}=\{r_1,r_2,\dots,r_j\}$1 s at $T^{(road)}=\{r_1,r_2,\dots,r_j\}$2k (Zhu et al., 23 May 2025). Two-stage retrieval is reported to preserve accuracy while scaling to $T^{(road)}=\{r_1,r_2,\dots,r_j\}$3k candidates (Zhu et al., 23 May 2025). A plausible implication is that the method is designed to integrate naturally with ANN systems, consistent with the paper’s observation that the vector representation is compatible with standard approximate nearest neighbor indexes such as FAISS and HNSW.

5. OmniTraj as temporally adaptive human trajectory prediction

In the prediction formulation, OmniTraj is a compact, decoupled-interaction Transformer pre-trained on a unified, heterogeneous human-motion corpus and made temporally adaptive by explicit conditioning on frame rate (Gao et al., 31 Jul 2025). The core problem is that most trajectory predictors hard-code a training-time temporal setup: a fixed frame rate and fixed observation and prediction horizons. When the test dataset uses a different FPS or horizon, the temporal prior becomes incorrect because a “one-step” at $T^{(road)}=\{r_1,r_2,\dots,r_j\}$4 Hz and a “one-step” at $T^{(road)}=\{r_1,r_2,\dots,r_j\}$5 Hz correspond to different physical durations and therefore different motion statistics (Gao et al., 31 Jul 2025).

The paper formalizes human trajectory prediction as forecasting future positions of a primary agent given observed trajectories of $T^{(road)}=\{r_1,r_2,\dots,r_j\}$6 agents over $T^{(road)}=\{r_1,r_2,\dots,r_j\}$7. Positions are denoted $T^{(road)}=\{r_1,r_2,\dots,r_j\}$8 or $T^{(road)}=\{r_1,r_2,\dots,r_j\}$9, sampled at interval $$T^{(reg)}=\{\mathcal{R}_1,\mathcal{R}_2,\dots,\mathcal{R}_m\}$$0, with frame rate $$T^{(reg)}=\{\mathcal{R}_1,\mathcal{R}_2,\dots,\mathcal{R}_m\}$$1 satisfying $$T^{(reg)}=\{\mathcal{R}_1,\mathcal{R}_2,\dots,\mathcal{R}_m\}$$2 (Gao et al., 31 Jul 2025). The paper distinguishes temporal generalization from dataset shift by constructing a controlled NBA-only protocol in which training uses two temporal setups and zero-shot testing uses a third unseen setup, thereby removing changes in scene, players, court, and viewpoint (Gao et al., 31 Jul 2025).

The architecture has three stated design pillars: explicit FPS-aware conditioning injected into input tokens, a decoupled interaction stack, and masked multi-modal pre-training on UniHuMotion++ (Gao et al., 31 Jul 2025). Inputs can include trajectory, pose (2D or 3D), and bounding boxes (2D or 3D). For modality $$T^{(reg)}=\{\mathcal{R}_1,\mathcal{R}_2,\dots,\mathcal{R}_m\}$$3 and agent $$T^{(reg)}=\{\mathcal{R}_1,\mathcal{R}_2,\dots,\mathcal{R}_m\}$$4, the observed tensor is $$T^{(reg)}=\{\mathcal{R}_1,\mathcal{R}_2,\dots,\mathcal{R}_m\}$$5 (Gao et al., 31 Jul 2025).

Each modality is embedded with its own MLP and temporal positional encoding. The temporal metadata is the frame rate $$T^{(reg)}=\{\mathcal{R}_1,\mathcal{R}_2,\dots,\mathcal{R}_m\}$$6 in FPS, encoded via an MLP and added to every token:

$$T^{(reg)}=\{\mathcal{R}_1,\mathcal{R}_2,\dots,\mathcal{R}_m\}$$7

The paper reports that several conditioning schemes were tested in a controlled NBA study: no FPS encoder, FiLM-based conditioning, codebook embeddings, concatenation as an extra token, and MLP encoding with latent-space summation, with the last being best (Gao et al., 31 Jul 2025). The stated interpretation is explicit: temporal metadata is not treated as a nuisance variable but as a signal that causally determines discretization of motion.

After token construction, the Cross-Modality Encoder (CME) uses shared-parameter Transformer layers over all modalities and agents to produce a motion-centric representation (Gao et al., 31 Jul 2025). Historical social interactions are then processed by the Historical Interaction Encoder (HIE). Future reasoning is handled by the Predictive Interaction Decoder (PID), which introduces ego queries and context queries, yielding $$T^{(reg)}=\{\mathcal{R}_1,\mathcal{R}_2,\dots,\mathcal{R}_m\}$$8 and $$T^{(reg)}=\{\mathcal{R}_1,\mathcal{R}_2,\dots,\mathcal{R}_m\}$$9, followed by ego-centric cross-attention $R$00 (Gao et al., 31 Jul 2025). This decoupled design is reported to separate encoding of historical social context from reasoning about predicted interactions.

Pre-training uses structured masking: $R$01 modality masking, $R$02 spatial masking on pose, and temporal masking that keeps the last two frames while masking $R$03 of the others (Gao et al., 31 Jul 2025). The primary agent’s representation is passed through an MLP head to predict future positions. Training uses a simple $R$04 regression loss rather than a probabilistic head:

$R$05

Evaluation reports ADE/FDE or MinADE$R$06/MinFDE$R$07 with $R$08 (Gao et al., 31 Jul 2025).

6. Heterogeneous pre-training, empirical results, and relation to UniTraj

The prediction OmniTraj is pre-trained on UniHuMotion++, described as a unified human-motion data framework with $R$09 hours from $R$10 datasets and native support for heterogeneous temporal setups (Gao et al., 31 Jul 2025). The listed datasets include NBA SportVU, JRDB-Pose, JTA, Human3.6M, AMASS, 3DPW, NuScenes, WOMD, Argoverse2, WorldPose, SDD, and Trajnet++. For zero-shot experiments, SDD and Trajnet++ are held out from pre-training (Gao et al., 31 Jul 2025). Heterogeneity is handled by CME, FPS conditioning, and masking, and the paper explicitly states that no special re-sampling is required because horizons vary and are naturally handled by token length (Gao et al., 31 Jul 2025).

The architecture size is reported as approximately $R$11M parameters. CME uses $R$12 Transformer layers with $R$13 heads; HIE uses $R$14 layers with $R$15 heads; the PID decoder uses $R$16 layers with $R$17 heads. Optimization uses Adam with learning rate $R$18, decayed by $R$19 after $R$20 of $R$21 epochs, and pre-training runs on $R$22 NVIDIA H100 $R$23GB (Gao et al., 31 Jul 2025). Inference speed is reported as $R$24 predictions/s on JTA on a single H100 (Gao et al., 31 Jul 2025).

The most prominent result concerns cross-setup zero-shot transfer in NBA under an unseen $R$25 FPS, $R$26 setup. Multi-Transmotion reports $R$27 MinADE$R$28/MinFDE$R$29, TrajSDE reports $R$30, OmniTraj without FPS encoder reports $R$31, FiLM-based FPS reports $R$32, codebook FPS reports $R$33, and MLP FPS with latent-space summation reports $R$34 (Gao et al., 31 Jul 2025). The paper states that explicit FPS conditioning via a tiny MLP and latent summation is “both necessary and sufficient” to dramatically reduce errors under temporal setup shift (Gao et al., 31 Jul 2025).

On unseen datasets, the reported zero-shot gains are likewise large. On Trajnet++, ADE drops from $R$35 for Multi-Transmotion to $R$36 for OmniTraj trajectory-only and to $R$37 for OmniTraj with multi-modal pre-training, corresponding to a $R$38 gain and a $R$39 gain respectively. On SDD, ADE drops from $R$40 to $R$41 and then to $R$42, corresponding to a $R$43 gain and a $R$44 gain (Gao et al., 31 Jul 2025). After fine-tuning, the method reports state-of-the-art or matching results on four datasets: NBA at $R$45 MinADE$R$46/MinFDE$R$47, JTA at $R$48 ADE/FDE with trajectory-only inference, WorldPose at $R$49 ADE/FDE with trajectory-only inference, and ETH-UCY at $R$50 average MinADE$R$51/MinFDE$R$52 (Gao et al., 31 Jul 2025).

Ablation studies attribute gains to both temporal conditioning and interaction design. On the unseen NBA setup, the chosen FPS-encoder reduces MinADE$R$53 by $R$54 and MinFDE$R$55 by $R$56 versus the no-FPS model (Gao et al., 31 Jul 2025). Decoupled interaction modules improve from $R$57 with no interaction modules to $R$58 with HIE, decoder, and ego-centric cross-attention (Gao et al., 31 Jul 2025). On NuScenes pedestrian prediction with only two observed frames, OmniTraj degrades from $R$59 to $R$60, while the cited specialized baseline degrades from $R$61 to $R$62 (Gao et al., 31 Jul 2025). The paper also reports monotonic improvement with more pre-training data and identifies $R$63 temporal masking as the best tested masking ratio, with MinADE$R$64 at that ratio (Gao et al., 31 Jul 2025).

The broader “OmniTraj” agenda is clarified by comparison with UniTraj. UniTraj unifies datasets, models, and evaluation criteria for vehicle trajectory prediction, repurposes ScenarioNet as a common scenario description format, standardizes agent-centric vectorized inputs, and shows that multi-dataset training over $R$65k trajectories and $R$66 hours across $R$67 cities improves cross-domain generalization and yields a new state-of-the-art nuScenes leaderboard result with minADE5 $R$68 for MTR-UniTraj (Feng et al., 2024). The same paper states that extending toward OmniTraj would require adding pedestrians and cyclists, incorporating rasterized context and raw perception outputs, unifying 2D and 3D coordinates, and curating hard long-tail scenarios (Feng et al., 2024).

Taken together, the supplied literature gives “OmniTraj” a dual meaning. In retrieval, it denotes omni-semantic representation learning for flexible, condition-based search at city scale (Zhu et al., 23 May 2025). In prediction, it denotes temporally adaptive pre-training for zero-shot and fine-tuned human trajectory forecasting across heterogeneous temporal setups (Gao et al., 31 Jul 2025). UniTraj, although differently named, provides the explicit infrastructural precursor for an omni-domain forecasting objective in autonomous-driving settings (Feng et al., 2024). A plausible synthesis is that the term now marks a broader research direction: trajectory systems designed to remain effective when semantics, modalities, temporal discretizations, or source domains vary substantially.