Trajectory Transformer Overview

• Trajectory Transformer is a class of transformer-based models that represent motion sequences by encoding historical states, scene context, constraints, and future trajectory structure.
• They are applied across domains like autonomous driving, robotics, GPS recovery, and spacecraft optimization using varied architectures such as encoder-decoder and causal decoders.
• These methods enhance forecasting and planning performance through techniques including multimodal regression, diffusion denoising, and constraint handling with both soft penalties and hard masks.

Searching arXiv for relevant papers on trajectory transformers across domains. Trajectory Transformer denotes a class of transformer-based models that represent trajectories, trajectory-conditioned decisions, or trajectory-related constraints as structured sequences and apply attention-based learning to forecasting, generation, recovery, planning, or value estimation. In the literature surveyed here, the term does not identify a single canonical architecture; it refers instead to a family of designs that range from autoregressive offline reinforcement-learning models and multi-modal autonomous-driving forecasters to diffusion-based GPS generators, road-network-aware recovery systems, constrained robot planners, and transformer-guided warm-start mechanisms for trajectory optimization (Lin et al., 2022, Postnikov et al., 2021, Zhang et al., 7 Oct 2025, Chen et al., 2022, Alanazi et al., 23 Oct 2025, Celestini et al., 2024). This breadth suggests that the unifying principle is not the output domain alone, but the use of transformer sequence modeling to encode motion history, scene context, constraints, and future trajectory structure within a common predictive framework.

1. Sequence modeling as the core abstraction

A foundational formulation treats offline reinforcement learning as sequence modeling over trajectories. In that setting, a trajectory is serialized as

$$\tau = (s_0,a_0,r_0,s_1,a_1,r_1,\dots,s_T,a_T,r_T),$$

augmented with a return-to-go token

$R_t=\sum_{t'=t}^{T}\gamma^{t'-t}r_{t'}.$

The model is then trained autoregressively to maximize the likelihood of trajectory tokens, and action selection at inference time is performed by beam search over imagined futures conditioned on current history and a desired target return (Lin et al., 2022).

This control-oriented usage is distinct from forecasting-only formulations. In autonomous driving, transformer-based trajectory predictors are typically multi-hypothesis regressors: they observe tracked-agent history and scene context, then output multiple plausible continuous future trajectories with confidence scores and, in some cases, an auxiliary uncertainty estimate. One representative system predicts up to $K=5$ future hypotheses over $T=25$ timesteps and ranks them with normalized confidence values, thereby treating motion prediction as a multimodal distribution-learning problem rather than a single-sequence continuation task (Postnikov et al., 2021).

Across domains, the same abstraction reappears in different operational roles. In trajectory generation for spacecraft, a causal transformer does not replace the optimal-control solver; instead it predicts a near-optimal warm start $(\hat X,\hat U)$ for a downstream optimization stage,

${X, U} = Opt \left(x(t_1), \hat X, \hat U \right),$

preserving the robustness and hard-constraint handling of optimization while using learned sequence priors to accelerate convergence (Celestini et al., 2024). In robot manipulation, the sequence model becomes a constrained decoder over lattice points, where the next token must satisfy adjacency and workspace legality at every step (Alanazi et al., 23 Oct 2025).

2. Trajectory representations and conditioning schemes

Trajectory-transformer methods differ most visibly in how they encode state, environment, and task context. Some operate directly on continuous positions; others use rasterized scenes, road-network subgraphs, multimodal prompt tokens, or discretized spatial lattices. The choice of representation determines what the attention mechanism can express and what kinds of inductive bias are injected before self-attention is applied.

Model family	Domain	Representation
Transformer-based motion prediction	Autonomous driving	Continuous future 2D coordinate sequences, often with rasterized scene context (Postnikov et al., 2021, Li et al., 2024)
ART-style warm-starting	Spacecraft	Interleaved state, control, scene, and performance descriptor tokens $\{x_i,P_i,S_i,u_i\}$ (Celestini et al., 2024)
Traj-Transformer / PathFormer / RNTrajRec	GPS, robot planning, trajectory recovery	GPS coordinate embeddings, 3-grid lattice paths, or point-local road-network subgraphs (Zhang et al., 7 Oct 2025, Alanazi et al., 23 Oct 2025, Chen et al., 2022)

In autonomous driving, one trajectory predictor rasterizes a multi-channel image containing dynamic-agent state and map context. The dynamic channels include position, velocity, acceleration, and orientation, while the map channels include lane direction, lane priority, speed limits, traffic-light association, road boundaries, crosswalks, and traffic-light states. The transformer then consumes a scene-level representation produced by a modified Vision Transformer encoder (Postnikov et al., 2021). A related formulation, the Context-Aware Transformer, combines past agent states with a semantic map rasterized as an RGB bird’s-eye-view image and encoded by MobileNet-V3 pretrained on ImageNet before fusion into transformer embeddings (Li et al., 2024).

In spacecraft trajectory generation, the representation is explicitly multimodal. The sequence interleaves state $x_i$, control $u_i$, scene descriptors $S_i$, and performance descriptors $R_t=\sum_{t'=t}^{T}\gamma^{t'-t}r_{t'}.$0, where the latter may include goal state $R_t=\sum_{t'=t}^{T}\gamma^{t'-t}r_{t'}.$1, time-to-go $R_t=\sum_{t'=t}^{T}\gamma^{t'-t}r_{t'}.$2, reward-to-go $R_t=\sum_{t'=t}^{T}\gamma^{t'-t}r_{t'}.$3, and constraint-violation budget $R_t=\sum_{t'=t}^{T}\gamma^{t'-t}r_{t'}.$4. This makes the trajectory token stream encode not only how the system moves, but also what the mission requires and how far the rollout is from satisfying that specification (Celestini et al., 2024).

For GPS trajectory generation, the representation itself becomes a major design variable. Traj-Transformer compares a location embedding, in which each point $R_t=\sum_{t'=t}^{T}\gamma^{t'-t}r_{t'}.$5 is embedded jointly, with a longitude-latitude embedding, in which longitude and latitude are embedded separately: $R_t=\sum_{t'=t}^{T}\gamma^{t'-t}r_{t'}.$6 A corresponding 2D positional encoding distinguishes coordinate type and trajectory order, allowing self-attention to model longitude–latitude interactions without fusing them prematurely (Zhang et al., 7 Oct 2025).

For robot-arm trajectory generation, PathFormer uses a discrete 3-grid representation organized as “where / what / when”: a spatial lattice over the reachable workspace, a task graph over subtask primitives, and a sequence grid encoding temporal order and causal structure. The generated path

$R_t=\sum_{t'=t}^{T}\gamma^{t'-t}r_{t'}.$7

is therefore conditioned simultaneously on geometry, task phase, and execution order (Alanazi et al., 23 Oct 2025).

For low-sample GPS recovery, RNTrajRec departs most sharply from plain tokenization. Each GPS point is interpreted together with a weighted subgraph of nearby road segments, so the token at each timestep is not just a point embedding but a point-plus-local-road-topology object. The subgraph weights are defined by

$R_t=\sum_{t'=t}^{T}\gamma^{t'-t}r_{t'}.$8

and pooled into a point representation used by the spatial-temporal transformer encoder (Chen et al., 2022).

3. Architectural patterns and transformer variants

The most common architectural template is the encoder-decoder transformer. In autonomous driving, this structure is used to encode agent and lane context and then decode future trajectory hypotheses. DyTTP adopts a HiVT-style encoder-decoder transformer in which self-attention captures agent-agent interactions, temporal dependencies, agent-lane interactions, and global context. Its central architectural modification is to replace LayerNorm with DynamicTanh,

$R_t=\sum_{t'=t}^{T}\gamma^{t'-t}r_{t'}.$9

yielding a normalization-free transformer backbone while preserving the overall HiVT structure (Zhu et al., 7 Apr 2025).

A second recurrent pattern is the causal decoder. PathFormer uses a causal Transformer decoder to autoregressively predict the next lattice waypoint,

$K=5$0

where $K=5$1 contains task-graph features and sequence information. The crucial innovation is not the decoder alone but its coupling to a legality mask that restricts logits to the one-step neighborhood $K=5$2, so every decoded move is lattice-adjacent and in-bounds by construction (Alanazi et al., 23 Oct 2025).

GPT-style causal transformers also appear in optimization-oriented settings. In spacecraft warm-starting, each modality first passes through a modality-specific encoder into a shared embedding space, after which a causal attention stack predicts future controls and states autoregressively from an initial prompt $K=5$3. The architecture supports variable-length trajectories and varying final times by training on subsequences of length $K=5$4 and rolling out with the last $K=5$5 tokens as context (Celestini et al., 2024).

Several methods hybridize attention with domain-specific structural modules. RNTrajRec alternates a TransformerEncoder over timesteps with a Graph Refinement Layer operating within each point’s local road-network subgraph: $K=5$6 This design makes temporal attention graph-aware rather than purely sequential (Chen et al., 2022). TrajGATFormer adopts a different hybridization strategy: a transformer encoder-decoder models temporal dependencies, while Graph Attention Networks capture worker-worker and worker-obstacle interactions in off-site construction environments (Alduais et al., 25 Oct 2025).

Diffusion-based trajectory transformers use the transformer as a denoiser rather than a direct autoregressive predictor. Traj-Transformer replaces the conventional UNet noise predictor with a transformer backbone and injects diffusion timestep and conditioning information via adaptive LayerNorm rather than cross-attention: $K=5$7 This yields a transformer-conditioned reverse diffusion model for GPS trajectories (Zhang et al., 7 Oct 2025).

4. Training objectives, inference procedures, and constraint handling

The learning objectives used by trajectory-transformer methods reflect the fact that “trajectory” can mean prediction target, latent rollout, denoising variable, or warm-start proposal. In uncertainty-aware autonomous-driving forecasting, one transformer-based system trains a mixture-of-Gaussians negative log-likelihood over $K=5$8 predicted trajectories,

$K=5$9

and supplements it with

$T=25$0

so that the scalar uncertainty head regresses the observed difficulty of the prediction instance (Postnikov et al., 2021).

CATF uses a multi-task objective combining a classification-style likelihood over multimodal futures with an auxiliary off-road loss that penalizes predicted waypoints falling into non-drivable grid cells: $T=25$1 The explicit purpose of $T=25$2 is feasibility regularization with respect to the road layout (Li et al., 2024).

DyTTP uses a standard multimodal forecasting decomposition into regression and classification,

$T=25$3

and adds snapshot ensembling through cosine annealing with warm restarts. Multiple checkpoints collected during a single training run are averaged at inference time, so diversity is introduced through saved local optima rather than training multiple independent models (Zhu et al., 7 Apr 2025).

In multi-task offline RL, SwitchTT augments the base trajectory-sequence formulation in two ways. First, dense feed-forward blocks are replaced by Switch/MoE layers, routing each token to one expert via a gating network. Second, scalar trajectory valuation is replaced by a categorical distribution over return atoms

$T=25$4

Planning then proceeds by a beam-search-like rollout: candidate actions are proposed, future states are imagined, and the distributional value model ranks the resulting partial trajectories (Lin et al., 2022).

In diffusion-based GPS generation, the training objective is the standard noise-prediction loss

$T=25$5

with DDIM-style sampling used to reduce reverse-process cost (Zhang et al., 7 Oct 2025). In spacecraft applications, by contrast, the transformer is not the final decision mechanism; it produces $T=25$6, which a conventional optimizer then refines into a constraint-satisfying solution (Celestini et al., 2024).

Constraint handling ranges from soft penalties to hard decoding restrictions. CATF’s off-road loss is a soft feasibility prior (Li et al., 2024). PathFormer’s neighborhood mask is a hard legality mechanism: $T=25$7 which guarantees Manhattan adjacency, workspace validity, and the absence of illegal jumps (Alanazi et al., 23 Oct 2025). RNTrajRec occupies an intermediate position: topology is injected into encoding and decoding through road-network subgraphs and masked road-segment prediction rather than through a purely unconstrained coordinate regression head (Chen et al., 2022).

5. Applications and reported empirical behavior

Autonomous driving remains the most densely populated application area. The uncertainty-aware transformer baseline for the Shifts Vehicle Motion Prediction Challenge ranked $T=25$8 on the 2021 leaderboard and was optimized for R-AUC cNLL rather than only ADE or FDE, emphasizing calibration under domain change (Postnikov et al., 2021). CATF reports state-of-the-art performance on the Lyft l5kit benchmark at the time of writing, with minADE$T=25$9 = 3.66, minADE$(\hat X,\hat U)$0 = 1.89, minFDE$(\hat X,\hat U)$1 = 3.34, and off-road rate $(\hat X,\hat U)$2; its linear-attention variant CATF$(\hat X,\hat U)$3 is about 1.6× faster inference and uses about 1.8× lower memory usage (Li et al., 2024). DyTTP, evaluated on Argoverse, reports minADE: 0.7845, minFDE: 1.1948, and MR: 0.1331, while also comparing average and standard deviation of inference time to assess real-time suitability and robustness (Zhu et al., 7 Apr 2025).

In offline reinforcement learning, SwitchTT reports an improvement of 10% over Trajectory Transformer across 10-task learning and up to 90% increase in offline model training speed on ten sparse-reward gym-mini-grid tasks. The paper attributes these gains to the combination of sparse conditional computation via switch experts and a distributional trajectory value estimator that is more informative than Monte Carlo value estimation in sparse-reward settings (Lin et al., 2022).

In mobility modeling, Traj-Transformer evaluates on the Chengdu and Xi’an taxi datasets, with 5,779,528 trajectories and 3,885,527 trajectories, respectively. The best reported configuration, L/lon-lat, achieves Chengdu density error: 0.0303, Chengdu pattern score: 0.828, Xi’an density error: 0.0295, and Xi’an pattern score: 0.830. The paper further reports that even the smallest transformer model, with 4.5M parameters, outperforms larger convolutional baselines (Zhang et al., 7 Oct 2025).

In trajectory recovery, RNTrajRec reports the strongest results across Shanghai-L, Chengdu, and Porto under both $(\hat X,\hat U)$4 and $(\hat X,\hat U)$5 downsampling. On Chengdu with $(\hat X,\hat U)$6, it reports F1 0.8272, Accuracy 0.6609, MAE 132.69, and RMSE 219.20; it also reports approximately 50.2 microseconds to recover a low-sample trajectory at inference time (Chen et al., 2022).

In robotic planning, PathFormer is trained on 53,755 trajectories with an 80% train / 20% validation split and reports 89.44% stepwise accuracy, 93.32% precision, 89.44% recall, 90.40% F1, and 99.99% valid path percent on offline lattice decoding. On an xArm Lite 6, it reports up to 97.5% reach success and 92.5% pick success in controlled tests, and 86.7% (52/60) successful executions across 60 language-specified pickup-place tasks in cluttered scenes (Alanazi et al., 23 Oct 2025).

In spacecraft autonomy, multimodal transformer warm-starting is evaluated on a 2D free-flyer testbed. The reported gains include up to 30% cost improvement and 80% reduction in infeasible cases relative to traditional approaches, with hardware experiments showing physically executable plans under obstacle and final-time variation (Celestini et al., 2024).

6. Conceptual boundaries, limitations, and bibliographic caveats

A central interpretive point is that “Trajectory Transformer” now names a methodological family rather than a fixed blueprint. In offline RL it denotes autoregressive modeling of state-action-return trajectories for planning (Lin et al., 2022). In autonomous driving it usually denotes a multimodal motion forecaster that maps scene context to several continuous future trajectories with confidences (Postnikov et al., 2021, Li et al., 2024, Zhu et al., 7 Apr 2025). In GPS generation it can denote a transformer denoiser inside a diffusion process (Zhang et al., 7 Oct 2025). In road-network recovery it becomes a spatio-graph-temporal encoder (Chen et al., 2022). In robot planning it refers to a constrained autoregressive decoder over legal workspace moves (Alanazi et al., 23 Oct 2025). This suggests that the essential commonality is the use of attention to model structured trajectory dependencies, while the surrounding inference stack remains domain-specific.

Several recurring limitations appear across the literature. Rasterized scene representations can lose explicit geometric structure relative to vectorized or graph-based alternatives (Postnikov et al., 2021, Li et al., 2024). Fixed covariance and a fixed number of predicted modes limit the probabilistic richness of some uncertainty-aware forecasters (Postnikov et al., 2021). Grid-cell off-road penalties are coarse and do not encode lane directionality, traffic rules, or interaction priorities (Li et al., 2024). Road-network-enhanced recovery methods require map infrastructure and map-matched supervision, and incur higher training cost and memory use because subgraphs must be sampled and stored (Chen et al., 2022). Raw GPS diffusion models report strong fidelity without explicit road constraints, which implies that geometric realism is learned statistically rather than guaranteed (Zhang et al., 7 Oct 2025). Optimization-assisted methods preserve hard constraints but remain dependent on the quality and cost profile of the downstream solver (Celestini et al., 2024).

A related misconception is that every arXiv item with a trajectory-transformer-like title contains a substantive method. The arXiv record “Accelerating Trajectory Generation for Quadrotors Using Transformers” (Tankasala et al., 2023) does not provide such content: it is only a generic l4dc2022 document template with placeholder sections and no scientific discussion of quadrotor trajectory generation, trajectory transformers, architecture, training setup, experiments, or results. It therefore cannot serve as technical evidence about quadrotor trajectory-transformer methods (Tankasala et al., 2023).

Taken together, the surveyed works indicate that the trajectory-transformer paradigm is most effective when attention is coupled to domain structure rather than used in isolation. That structure may be return-to-go conditioning, map semantics, road topology, diffusion-time conditioning, optimization prompts, graph interactions, or hard motion masks. The resulting systems are therefore best understood not as generic transformers applied to motion data, but as trajectory-centric transformer formulations whose success depends on how they encode feasibility, multimodality, uncertainty, and environment-specific constraints.