Decision Transformer Trajectory Planner

• Decision Transformer-based trajectory planning is a framework that models trajectories as ordered token sequences using states, actions, and return-to-go for conditional decision making.
• It employs a GPT-like autoregressive Transformer with causal masking and loss functions to predict next actions from historical context, ensuring long-horizon coherence.
• Variants like EDT, ADT, LPT, and DODT enhance performance by introducing elastic memory, learned prompts, latent plans, or imagined futures to overcome limitations such as trajectory stitching.

A Decision Transformer-based trajectory planner is a trajectory-generation or sequential-decision architecture that casts planning as conditional sequence modeling over trajectories, typically using tokens for states, actions, and a control variable such as return-to-go (RTG), a learned prompt, or a latent plan. In its canonical form, the planner is a GPT-like autoregressive Transformer trained on offline trajectories and queried at inference time with a desired outcome. Subsequent work broadens that template by treating prompts as high-level actions, inferring episode-level latent plans, adapting conditioning history at test time, or augmenting trajectory modeling with imagined futures. Across offline reinforcement learning, multi-vehicle coordination, drone navigation, and related trajectory-generation settings, the common objective is to obtain coherent long-horizon behavior by generating actions from sequence context rather than by explicit online search over full trajectories (Ma et al., 2023, Kong et al., 2024, Lee et al., 2024).

1. Canonical sequence-model formulation

The standard formulation treats a trajectory as an ordered token sequence. In the RTG-conditioned form used by Decision Transformer, the trajectory can be written as

$$\tau = \left( \widehat{R}_0, s_0, a_0, \widehat{R}_1, s_1, a_1, \dots, \widehat{R}_T, s_T, a_T\right),$$

with

$\widehat{R}_t = \sum_{i=t}^{T} r_i.$

Training is performed as autoregressive next-action prediction from a context window of past tokens, and inference supplies a desired return at the beginning of rollout and updates it by subtracting observed rewards (Ma et al., 2023).

In domain instantiations, this canonical structure is usually implemented by a GPT-like architecture with positional encoding, masked self-attention, and a feedforward network. The masking enforces causal conditioning, so the model attends only to past context. In a continuous-control example for unsignalized intersections, the model is trained on a context

$$\tau = (s_{t-K}, a_{t-K}, G_{t-K}, \cdots, s_t, a_t, G_t)$$

and predicts the next action with an MSE loss, while evaluation starts from $(s_0,\emptyset,\hat G_0)$ and repeatedly appends the new state, action, and updated RTG (Lee et al., 2024).

A more abstract interpretation views the architecture as a hierarchical policy. In that formulation,

$$\pi(a \mid s) = \int \pi^{h}(p \mid s)\,\pi^{l}(a \mid s,p)\,dp,$$

where a high-level policy chooses a prompt $p$ and a low-level policy outputs an action conditioned on the prompt. Under this view, vanilla Decision Transformer is recovered as a special case in which the prompt is the return-to-go and the prompt schedule is effectively fixed rather than learned (Ma et al., 2023).

2. Why vanilla Decision Transformer is a limited trajectory planner

A central criticism in the literature is that vanilla Decision Transformer is a prompt-conditioned behavior selector rather than a strong trajectory planner. The main failure mode is trajectory stitching: combining a useful prefix from one trajectory with a useful suffix from another even when that stitched trajectory does not explicitly appear in the dataset. The hierarchical analysis argues that DT “depends on recalling trajectories from the dataset rather than composing them,” and that this weakness is especially severe when offline data are suboptimal (Ma et al., 2023).

The stitching problem is illustrated by a two-trajectory example in which $a\to b\to c$ has poor outcome and $d\to b\to e$ has high return. A prompt at $b$ may induce the good continuation through $e$, but at $\widehat{R}_t = \sum_{i=t}^{T} r_i.$0 the model has not been trained to associate the prefix $\widehat{R}_t = \sum_{i=t}^{T} r_i.$1 with that improved continuation. In this sense, conditioning on the return observed along a single trajectory does not backpropagate future good outcomes through alternative prefixes. The hierarchical critique therefore concludes that DT is “not a true planner that backpropagates value information across trajectories” (Ma et al., 2023).

A closely related diagnosis appears in Elastic Decision Transformer. There, the argument is that a fixed prefix can dominate action choice even when a better continuation exists in the dataset. DT is therefore good at sequence reproduction but weak at sequence recombination. The proposed remedy is not to change the causal Transformer backbone but to change how much of the past is presented to it at inference time, so that bad prefixes can be forgotten and better continuations can be selected (Wu et al., 2023).

Latent-plan work makes a different criticism. When step-wise rewards are sparse or absent, direct RTG conditioning is described as unreliable because RTG is only a proxy and can be noisy or sparse. On that view, the real difficulty is temporal consistency over long horizons: the model must identify action sequences coherent with a desired outcome, assign credit across delayed consequences, and remain adaptable under environmental contingencies (Kong et al., 2024).

3. Architectural variants and planning mechanisms

Several extensions retain the Transformer trajectory generator but replace or modify the conditioning variable and the test-time planning procedure. Representative variants are summarized below.

Variant	Planning variable	Distinguishing mechanism
EDT	history length $\widehat{R}_t = \sum_{i=t}^{T} r_i.$2	searches truncated histories at test time and uses the one with the largest estimated return
ADT	prompt $\widehat{R}_t = \sum_{i=t}^{T} r_i.$3	learns a high-level prompt policy and a low-level transformer policy jointly
LPT	latent plan $\widehat{R}_t = \sum_{i=t}^{T} r_i.$4	infers a trajectory-level latent variable from expected return before execution
DODT	imagined trajectories	inserts Dreamer-generated forecasted trajectories into the Online DT replay buffer

EDT keeps the DT sequence-modeling backbone but introduces elastic memory. At test time it searches over candidate history lengths, estimates the achievable return for each truncated history with an expectile-regression objective, and then predicts the action from the history length with the highest estimated return. The selection problem is written as

$\widehat{R}_t = \sum_{i=t}^{T} r_i.$5

and the maximum-return estimator is approximated with a high expectile, using $\widehat{R}_t = \sum_{i=t}^{T} r_i.$6. To reduce inference cost, the search is performed with a step size $\widehat{R}_t = \sum_{i=t}^{T} r_i.$7 (Wu et al., 2023).

ADT replaces the fixed RTG prompt with a learned prompt generated by a high-level policy. In value-prompted ADT, the prompt is a scalar and the high-level policy approximates the in-sample optimal value using IQL. In goal-prompted ADT, the prompt is a subgoal and the high-level policy is trained with HIQL. In both cases, the low-level transformer is trained with an advantage-weighted regression objective rather than the original DT sequence-prediction loss. The practical recipe is to pretrain critics, relabel prompts or subgoals for each trajectory, and then train the transformer actor on concatenated $\widehat{R}_t = \sum_{i=t}^{T} r_i.$8 tokens (Ma et al., 2023).

LPT reframes offline planning as generative modeling over whole trajectories paired with final returns. It defines

$\widehat{R}_t = \sum_{i=t}^{T} r_i.$9

where the latent vector $$\tau = (s_{t-K}, a_{t-K}, G_{t-K}, \cdots, s_t, a_t, G_t)$$0 is a plan-like abstraction. The prior is an implicit generator,

$$\tau = (s_{t-K}, a_{t-K}, G_{t-K}, \cdots, s_t, a_t, G_t)$$1

the trajectory model is an autoregressive causal Transformer with finite context $$\tau = (s_{t-K}, a_{t-K}, G_{t-K}, \cdots, s_t, a_t, G_t)$$2, and the return model is a Gaussian whose mean is predicted from $$\tau = (s_{t-K}, a_{t-K}, G_{t-K}, \cdots, s_t, a_t, G_t)$$3. Training is maximum-likelihood estimation with posterior sampling by Langevin dynamics, and the paper describes the learning loop as resembling an EM-style latent-variable maximum-likelihood method. At test time, the method performs “planning as inference”: it infers a latent plan from an expected return using the return predictor only, then holds $$\tau = (s_{t-K}, a_{t-K}, G_{t-K}, \cdots, s_t, a_t, G_t)$$4 fixed throughout execution (Kong et al., 2024).

DODT changes the planning mechanism again by combining an Online Decision Transformer with Dreamer. Dreamer learns a latent world model, imagines trajectories $$\tau = (s_{t-K}, a_{t-K}, G_{t-K}, \cdots, s_t, a_t, G_t)$$5, and inserts those forecasted trajectories into the transformer replay buffer. The paper describes this as a parallel training architecture and a bidirectional enhancement loop: Dreamer improves trajectory quality, ODT improves decision quality using those trajectories, and the combined system is iteratively refined (Jiang et al., 2024).

4. Task-specific realizations

Decision Transformer-based trajectory planners have been instantiated in several planning domains with distinct state, action, and reward parameterizations.

For multi-vehicle coordination at unsignalized intersections, the problem is formulated as an offline sequential decision-making and trajectory prediction problem in a single-lane, 4-way unsignalized intersection with five vehicles. The joint state contains each vehicle’s position, velocity, heading angle, and destination; the action is continuous longitudinal acceleration $$\tau = (s_{t-K}, a_{t-K}, G_{t-K}, \cdots, s_t, a_t, G_t)$$6; and the reward combines a speed term with a collision penalty. The planner is trained on trajectories generated from reservation-based Autonomous Intersection Management and from an uncoordinated collision regime, so that collision-free and unsafe interaction patterns both appear in the offline distribution (Lee et al., 2024).

For drone planning in unknown environments, the task is formulated as a Markov Decision Process with a local 3D occupancy grid and a 12-dimensional local target observation. The Decision Transformer does not output raw controls; instead, it predicts a 9-dimensional action vector consisting of the higher-order coefficients of a short-horizon fifth-order polynomial trajectory segment. The distinctive design choice is to treat RTG as a temperature-like control parameter for the safety-efficiency trade-off. To reduce ambiguity in unknown environments, the method replaces full-episode RTG with a short-horizon $$\tau = (s_{t-K}, a_{t-K}, G_{t-K}, \cdots, s_t, a_t, G_t)$$7-step RTG,

$$\tau = (s_{t-K}, a_{t-K}, G_{t-K}, \cdots, s_t, a_t, G_t)$$8

and the reported best setting is $$\tau = (s_{t-K}, a_{t-K}, G_{t-K}, \cdots, s_t, a_t, G_t)$$9 (Ji et al., 29 Jul 2025).

For spacecraft trajectory generation, a related but non-canonical line uses a multimodal autoregressive transformer as a trajectory warm-start planner rather than as a direct closed-loop policy. The model predicts state-control trajectories conditioned on scene descriptors and performance descriptors, then passes the predicted trajectory to Sequential Convex Programming: $(s_0,\emptyset,\hat G_0)$0 The paper explicitly characterizes this method as “Decision Transformer-inspired” rather than a canonical DT, because it conditions not only on reward-to-go but also on obstacle geometry, goal state, time bounds, and constraint budgets, and because hard constraints are ultimately enforced by the downstream optimizer (Celestini et al., 2024).

5. Empirical performance and demonstrated planning behavior

On offline RL benchmarks, the principal empirical question is whether these planners can stitch trajectories, assign long-horizon credit, and outperform plain RTG prompting. ADT reports that V-ADT substantially beats DT and QLDT, especially on AntMaze and Kitchen, with average scores of 65.0 for V-ADT, 43.8 for DT, and 45.4 for QLDT. G-ADT reports an average score of 72.6, compared with 66.0 for Waypoint Transformer and 77.1 for HIQL. The ablations further show that removing the value prompt hurts performance, replacing the AWR-style low-level objective with DT’s original sequence-prediction loss reduces performance, and manual target-return tuning does not close the gap to learned prompting (Ma et al., 2023).

EDT reports especially strong gains on diverse offline datasets where stitching opportunities are abundant. On D4RL locomotion, Hopper medium-replay improves from $(s_0,\emptyset,\hat G_0)$1 for DT to $(s_0,\emptyset,\hat G_0)$2 for EDT, and Walker2d medium-replay improves from $(s_0,\emptyset,\hat G_0)$3 for DT to $(s_0,\emptyset,\hat G_0)$4 for EDT. In multi-task locomotion, the reported sums are $(s_0,\emptyset,\hat G_0)$5 for DT-1, $(s_0,\emptyset,\hat G_0)$6 for IQL-1, and $(s_0,\emptyset,\hat G_0)$7 for EDT-1. The paper also reports improved average human-normalized score over DT and IQL on 20 Atari games (Wu et al., 2023).

LPT is evaluated on offline Gym-Mujoco locomotion tasks, Maze2D navigation, Antmaze, and Connect Four. The paper reports that LPT is competitive with, and often better than, Decision Transformer and Q-learning-enhanced DT baselines, especially in the final-return setting where step-wise reward is removed. It reports improvements over DT on Gym-Mujoco final-return benchmarks, substantial gains over DT on Maze2D, improvements over DT and online DT-style baselines on Antmaze, and strongest-baseline-level performance with very low variance in Connect Four. The ablations also show that removing the expressive prior $(s_0,\emptyset,\hat G_0)$8 degrades performance across tasks, especially in stochastic ones (Kong et al., 2024).

DODT reports an overall total score of 646.19 compared with 605.02 for ODT. The paper highlights Half-Cheetah-medium, where ODT scores 42.16 and DODT scores 60.93, and antmaze-umaze-diverse, where ODT scores 56.00 and DODT scores 74.7. These results are presented as evidence that Dreamer’s anticipatory trajectories improve sample efficiency, reward maximization, and robustness relative to ODT alone (Jiang et al., 2024).

Application papers emphasize domain-specific performance. In unsignalized intersections, the 10% mixed dataset is selected because it yields the lowest collision rate among the mixed settings while maintaining similar average length and return; the reported values are return 1259.36, length 18.67, collision 0.08, compared with AIM’s return 1259.25, length 18.20, collision 0.0. Against AIM on 91 non-collision scenarios, the DT produces shorter episode lengths in 15 scenarios; in scenario #1 it reduces length by 4 s compared to AIM and is only 0.6 s away from the optimal solution; in scenario #90 it performs worse than AIM by 4.6 s. Under up to 2% velocity noise, average return decreases by 0.08%, average length increases by 1.5%, and collision rate increases by 6% (Lee et al., 2024).

In drone planning, low RTG produces safer trajectories and high RTG produces more efficient ones. On grid maps, the planner with RTG $(s_0,\emptyset,\hat G_0)$9 reports flight distance 33.72 m, average velocity 0.67 m/s, flight time 49.41 s, and safety 0.32 DTW error, whereas RTG $$\pi(a \mid s) = \int \pi^{h}(p \mid s)\,\pi^{l}(a \mid s,p)\,dp,$$0 reports flight distance 32.16 m, average velocity 0.92 m/s, flight time 37.32 s, and safety 0.47 DTW error. On random maps, RTG $$\pi(a \mid s) = \int \pi^{h}(p \mid s)\,\pi^{l}(a \mid s,p)\,dp,$$1 reports safety 1 obstacle within 0.5 m and RTG $$\pi(a \mid s) = \int \pi^{h}(p \mid s)\,\pi^{l}(a \mid s,p)\,dp,$$2 reports flight time 39.47 s with safety 2. The ablation study further states that without $$\pi(a \mid s) = \int \pi^{h}(p \mid s)\,\pi^{l}(a \mid s,p)\,dp,$$3-step RTG the planner is unstable and collisions occurred, while $$\pi(a \mid s) = \int \pi^{h}(p \mid s)\,\pi^{l}(a \mid s,p)\,dp,$$4 yields stable and consistent trajectories (Ji et al., 29 Jul 2025).

The DT-inspired spacecraft planner reports up to 30% cost improvement and 80% reduction in infeasible cases with respect to traditional approaches. For unseen scenarios and varying final times, the paper reports $$\pi(a \mid s) = \int \pi^{h}(p \mid s)\,\pi^{l}(a \mid s,p)\,dp,$$5 suboptimality improvement, $$\pi(a \mid s) = \int \pi^{h}(p \mid s)\,\pi^{l}(a \mid s,p)\,dp,$$6 iteration improvement, and $$\pi(a \mid s) = \int \pi^{h}(p \mid s)\,\pi^{l}(a \mid s,p)\,dp,$$7 infeasible cases for ART-Rs, and $$\pi(a \mid s) = \int \pi^{h}(p \mid s)\,\pi^{l}(a \mid s,p)\,dp,$$8 suboptimality improvement, $$\pi(a \mid s) = \int \pi^{h}(p \mid s)\,\pi^{l}(a \mid s,p)\,dp,$$9 iteration improvement, and $p$0 infeasible cases for ART-Rt. It also reports that using relative obstacle features is much better than absolute obstacle positions, since absolute obstacle information degrades performance by +77% average cost, +49% average iterations, and +290% average infeasibility rate (Celestini et al., 2024).

6. Misconceptions, limitations, and research directions

A recurring misconception is that any RTG-conditioned Transformer is already a strong planner. Several papers dispute that characterization. The hierarchical interpretation states that DT is a special hierarchical policy with a dummy prompt generator; EDT shows that fixed-memory conditioning can trap the model in bad prefixes; and LPT argues that temporal consistency in sparse-reward settings is better captured by a latent episode-level plan than by step-wise reward prompting (Ma et al., 2023, Wu et al., 2023, Kong et al., 2024). This suggests a distinction between prompt-conditioned imitation and planning mechanisms that explicitly support stitching, abstraction, or inference over future-consistent latent structure.

Safety and robustness remain domain-dependent. In unsignalized intersections, collision rate is not always zero, performance depends on the quality and diversity of offline data, and the paper explicitly notes the need for a safety filter to correct unsafe DT actions in future work (Lee et al., 2024). In drone planning, the safety-efficiency knob depends on a reward that intentionally ties higher reward to flying closer to obstacles, and the ablation study shows that RTG conditioning becomes unstable without the short-horizon $p$1-step modification (Ji et al., 29 Jul 2025). In DODT, the integrated model is heavier than ODT, requiring about 24.6 GB RAM and 12.3 hours versus 22.1 GB RAM and 10.5 hours for ODT, and the usefulness of imagined trajectories depends on world-model fidelity (Jiang et al., 2024). In the spacecraft setting, generalization depends strongly on dataset diversity and representational choice, and the method is not itself a safety guarantee because SCP, not the transformer, enforces hard constraints (Celestini et al., 2024).

The present literature therefore spans several meanings of “Decision Transformer-based trajectory planner.” At one end are canonical RTG-conditioned policies trained as next-action predictors; at another are hierarchical prompt-learning methods, elastic-memory selectors, latent-plan inference models, and planner-controller hybrids. A plausible implication is that the field is moving away from the narrow view of DT as a return-prompted sequence model and toward a broader family of Transformer planners in which the critical design choice is not only the backbone architecture but also the structure of the planning variable—RTG, learned prompt, latent plan, or imagined future trajectory.