Prompting Decision Transformer (PDT)

• Prompting Decision Transformer (PDT) is a prompt-conditioned extension of Decision Transformer that uses trajectory prompts as fixed task descriptors in multi-task offline reinforcement learning.
• It leverages both stochastic prompt sampling and optimized prompt-tuning methods (via bandit strategies or Gaussian noise) to significantly enhance few-shot policy generalization, as demonstrated on MuJoCo benchmarks.
• Hierarchical variants further extend PDT by dynamically generating or retrieving task-specific prompts, enabling robust performance in constrained control, zero-shot reach-avoid tasks, and clinical recommendation settings.

Searching arXiv for papers on Prompting Decision Transformer and closely related variants. Prompting Decision Transformer (PDT) denotes a prompt-conditioned extension of Decision Transformer in which a transformer policy is conditioned not only on the recent trajectory history but also on a task-specific prompt, typically a short sequence of demonstration tokens. In the multi-task offline RL setting, PDT seeks a generalized policy $\pi(\mathbf{s}, \psi) \to \mathbf{a}$, where the task context variable $\psi$ is realized as a trajectory prompt sampled from a task-specific dataset, and the prompt is maintained in context during rollouts (Rietz et al., 7 Feb 2025). In closely related few-shot formulations, the same idea appears as Prompt-DT: a trajectory prompt containing a few segments of state, action, and reward-to-go tuples is concatenated with the recent context, enabling adaptation to unseen tasks without parameter updates at test time (Xu et al., 2022). Across subsequent work, this prompting paradigm has been extended to hierarchical prompting, black-box prompt tuning, bandit-based prompt selection, language-model initialization, constrained control, zero-shot reach-avoid control, and goal-prompted clinical recommendation (Ma et al., 2023).

1. Core sequence-modeling formulation

The canonical PDT construction inherits the Decision Transformer view of RL as autoregressive sequence modeling, but it prepends a trajectory prompt $\rho$ to the rollout context. In the multi-task formulation, the prompt is assembled from $J$ prompt segments, each of length $H$:

$\rho = \big( \overbrace{\hat{r}_j^\star, \mathbf{s}_j^\star, \mathbf{a}_j^\star, ..., \hat{r}_{j+H}^\star, \mathbf{s}_{j+H}^\star, \mathbf{a}_{j+H}^\star}^{\Tilde{\tau}_1}, ..., \overbrace{\hat{r}_k^\star, \mathbf{s}_k^\star, \mathbf{a}_k^\star, ..., \hat{r}_{k+H}^\star, \mathbf{s}_{k+H}^\star, \mathbf{a}_{k+H}^\star}^{\Tilde{\tau}_J} \big),$

and the model input is

$$\mathbf{x} = (\rho) \odot (\hat{r}_{t-K}, \mathbf{s}_{t-K}, \mathbf{a}_{t-K}, ..., \hat{r}_{t}, \mathbf{s}_{t}, \mathbf{a}_{t}),$$

with $\odot$ denoting concatenation (Rietz et al., 7 Feb 2025). The prompt is fixed during inference for a target task and is prepended to the current trajectory context, so the content of the prompt acts as the task descriptor.

In the few-shot policy generalization formulation, the prompt is likewise a trajectory prefix,

$$\tau^{\star}_i = (\hat{r}^{\star}_1, s^{\star}_1, a^{\star}_1, \hat{r}^{\star}_2, s^{\star}_2, a^{\star}_2, ..., \hat{r}^{\star}_{K^{\star}}, s^{\star}_{K^{\star}}, a^{\star}_{K^{\star}}),$$

and the full input is

$$\tau^{\text{input}}_i = (\tau^{\star}_i,\, \tau_i),$$

where $\psi$0 is the recent interaction history (Xu et al., 2022). Prompt-DT changes the original DT minimally: the sequence input is lengthened to $\psi$1 tokens, prompt and history use identical embeddings, and training samples prompts stochastically from demonstration data.

This prompt construction differs from standard DT’s return-to-go conditioning in an important way. Rather than asking the model to infer the task from a scalar target, PDT conditions on state-action-reward fragments that encode task-specific transition and control structure. This suggests that the prompt is not merely a target value but an in-context behavioral specification.

2. Few-shot policy generalization and prompt semantics

The original few-shot prompt-based formulation was developed to study whether architecture inductive bias alone could yield rapid adaptation in offline meta-RL. The resulting Prompt-DT was evaluated on five MuJoCo meta-RL benchmarks—Cheetah-dir, Cheetah-vel, Ant-dir, Dial, and MetaWorld reach-v2—and was reported to be a strong few-shot learner without any extra finetuning on unseen target tasks (Xu et al., 2022).

A central empirical result is that Prompt-DT consistently outperforms Prompt-DT variants and strong meta offline RL baselines by a large margin with a trajectory prompt containing only a few timesteps (Xu et al., 2022). The paper further reports that prompt length is not the dominant factor in those environments: even 2-step prompts suffice to reach near-maximal reward, and increasing prompt length does not lead to significant improvement. On Cheetah-dir, for example, Prompt-DT achieves about 926 episode reward with as little as a 2-step prompt (Xu et al., 2022).

The prompt, however, is not a generic prefix that can be arbitrarily chosen. Prompt quality matters. Performance tracks prompt quality: expert prompts yield expert-like performance, whereas random prompts at test yield poor policy behavior, and training with mismatched prompt quality degrades generalization (Xu et al., 2022). The same paper reports that Prompt-DT and Prompt-MT-BC outperform non-prompted DT and finetune-based baselines on out-of-distribution Ant-dir tasks, indicating that trajectory prompts can support extrapolation when relevant demonstration fragments are available.

No parameter updates are performed at test time in Prompt-DT (Xu et al., 2022). This gives the method a distinctive adaptation profile relative to meta-RL algorithms such as MACAW: the few-shot signal is carried entirely through the prompt, not through online gradient updates. A plausible implication is that PDT shifts adaptation burden from optimizer design to prompt informativeness and prompt-representation quality.

3. Prompt optimization and prompt-tuning

A recurring limitation of PDT is that prompts are often sampled uniformly at random from demonstration datasets, even though prompt informativeness varies substantially. Both bandit-based prompt-tuning papers state this limitation explicitly: vanilla PDT samples prompt segments uniformly at random from per-task demonstration datasets or expert demonstrations, which can lead to non-informative, redundant, or suboptimal prompts and thereby degrade performance (Rietz et al., 7 Feb 2025).

One line of work addresses this with black-box prompt optimization. "Prompt-Tuning DT" treats the trajectory segment itself as the tunable object, perturbs a vectorized prompt with Gaussian noise,

$\psi$2

and uses a preference-ranking oracle with ZO-RankSGD to estimate a direction for prompt improvement (Hu et al., 2023). Only the prompt is tuned; the rest of the model is frozen. The paper reports that with only $\psi$3 of the parameters learned, Prompt-Tuning DT achieves comparable or even better performance than full-model fine-tuning in low-data scenarios (Hu et al., 2023). It also states that prompt-tuning is particularly advantageous in few-shot adaptation and is robust to prompt length, although it is sensitive to prompt initialization.

A second line formulates prompt selection as a contextual multi-armed bandit problem. In "Enhancing Pre-Trained Decision Transformers with Prompt-Tuning Bandits," a separate reward model $\psi$4 is learned for each prompt segment position, and the frozen PDT backbone is conditioned on prompts selected by $\psi$5-greedy or UCB exploration (Rietz et al., 7 Feb 2025). The paper reports consistent and significant improvements over vanilla PDT and other baselines. In MuJoCo Half Cheetah with $\psi$6, performance improves from $\psi$7 to $\psi$8, and in an OOD setting from $\psi$9 to $\rho$0 (Rietz et al., 7 Feb 2025). The companion paper emphasizes lower sample complexity than perturbation-based methods such as ZORankSGD and Gaussian hill climbing, and describes rapid convergence to optimal prompts within the first few rollouts in its experimental setting (Rietz et al., 10 Feb 2025).

These prompt-optimization results establish a broad distinction between prompt-conditioned control and prompt selection. In vanilla PDT, the prompt is a stochastic descriptor. In tuned PDT, prompt choice becomes an optimization problem in its own right.

4. Hierarchical prompting and adaptive guidance

Another major direction reinterprets prompt-conditioned DT through hierarchical RL. "Rethinking Decision Transformer via Hierarchical Reinforcement Learning" introduces Autotuned Decision Transformer (ADT), in which a high-level policy first proposes an ideal prompt for the current state and a low-level policy then generates an action conditioned on that prompt (Ma et al., 2023). Within this framework, DT emerges as a special case under particular choices of high-level and low-level policies, and those choices can fail because they do not reliably support trajectory stitching.

ADT studies two prompt types. In value-prompted V-ADT, the prompt is a scalar estimating the optimal attainable return from the current state. In goal-prompted G-ADT, the prompt is a sub-goal state to be reached a few steps ahead (Ma et al., 2023). The low-level policy is trained with advantage-weighted regression,

$\rho$1

and the paper reports that joint optimization of high-level and low-level policies is necessary for stitching ability. Its main aggregate scores are explicit: Table 1 reports V-ADT average score $\rho$2 versus DT $\rho$3, and Table 2 reports G-ADT average score $\rho$4 (Ma et al., 2023).

A different hierarchical design appears in "Hierarchical Prompt Decision Transformer: Improving Few-Shot Policy Generalization with Global and Adaptive Guidance" (Wang et al., 2024). HPDT introduces two layers of soft prompt tokens: a global token summarizing task-level information and adaptive tokens that provide timestep-specific guidance retrieved by KNN from a curated set of demonstration segments. The global token is computed by mean aggregation over projected demonstration tuples, while adaptive tokens are constructed from the top-$\rho$5 retrieved neighbors. HPDT fuses adaptive tokens by summation rather than concatenation and replaces lookup-table timestep embeddings with Time2Vec. Experiments across seven MuJoCo and MetaWorld tasks report that HPDT consistently outperforms all baseline methods, and Table 3 shows best or competitive results in 6 of 7 environments (Wang et al., 2024).

These hierarchical variants shift PDT from static prompt conditioning toward state-dependent prompt generation or retrieval. This suggests a broader interpretation of prompting in RL: prompts need not be fixed demonstration prefixes; they can also be learned high-level plans or dynamically retrieved local guidance.

5. Architectural extensions and domain-specific PDT variants

Subsequent work extends PDT beyond the original few-shot offline meta-RL setting. "Pre-trained LLMs Improve the Few-shot Prompt Ability of Decision Transformer" introduces LPDT, which initializes the prompt-conditioned DT backbone from DistilGPT2, adapts it with LoRA,

$\rho$6

and adds prompt regularization through either a classifier head or an InfoNCE objective (Yang et al., 2024). The paper reports that LM initialization significantly enhances performance on unseen tasks compared to baseline methods and that with only $\rho$7 of the data, LPDT variants maintain robust performance and outperform Prompt-DT on the MuJoCo tasks considered (Yang et al., 2024).

In continual learning, "P2DT: Mitigating Forgetting in task-incremental Learning with progressive prompt Decision Transformer" appends task-specific decision tokens during new-task training and separates general processing from task-specific expert processing (Wang et al., 2024). The paper states that adding a prompt token incurs only a tiny parameter overhead of approximately $\rho$8–$\rho$9 KB per prompt, versus approximately $J$0 MB for the full model, and reports much higher retained scores than DT in task-incremental D4RL MuJoCo experiments (Wang et al., 2024).

Prompt-conditioned DTs have also been specialized to constrained control and safe planning. "Attention-Enhanced Prompt Decision Transformers for UAV-Assisted Communications with AoI" introduces APDT, which adds an attention mechanism for varying numbers of terrestrial users, uses short trajectory demonstrations as prompts for rapid adaptation to new scenarios, and introduces reward-to-go and cost-to-go tokens to handle long-term UAV energy constraints (Lu et al., 28 May 2025). Simulations report twice faster convergence and an $J$1 reduction in average AoI compared to conventional DT, as well as $J$2 faster convergence and over $J$3 average AoI reduction versus conventional DRL baselines (Lu et al., 28 May 2025).

"Prompting Decision Transformers for Zero-Shot Reach-Avoid Policies" introduces RADT, a reward-free, goal-conditioned, avoid-region-conditioned DT in which the prompt explicitly includes goal and avoid-region tokens:

$J$4

RADT is trained with goal and avoid-region hindsight relabeling and uses an action loss plus an avoid-awareness loss (Li et al., 25 May 2025). Across 11 tasks, environments, and experimental settings, the paper reports zero-shot generalization to out-of-distribution avoid region sizes and counts, including a $J$5 improvement in normalized cost over the best retrained baseline in one zero-shot setting while maintaining high goal-reaching success (Li et al., 25 May 2025).

A clinically oriented goal-prompted variant appears in "Clinical Decision Transformer: Intended Treatment Recommendation through Goal Prompting" (Lee et al., 2023). CDT prepends a desired future clinical state as a goal prompt to treatment history and models medication sequences conditioned on that goal. On a diabetes dataset of 4788 patients, the paper reports intended treatment effects aligned with prompt ranges such as NormalA1c, LowerA1c, and HigherA1c, unlike behavior cloning (Lee et al., 2023). Although CDT is framed as goal prompting rather than trajectory prompting, it illustrates the broader portability of prompt-conditioned DTs to structured sequential decision domains.

6. Limitations, nomenclature, and theoretical context

Several limitations recur across the literature. Static prompt segments can be inadequate because they provide one fixed summary for an entire rollout and cannot deliver context-specific guidance at different timesteps (Wang et al., 2024). Uniform prompt sampling is another limitation because not all prompt segments are equally informative for distinguishing tasks, especially when demonstration data is sparse, noisy, or mixed-quality (Rietz et al., 7 Feb 2025). In bandit-based tuning, the independence assumption across prompt slots is explicitly described as high variance because each reward model attributes the total rollout return to its own segment while ignoring the contribution of other segments (Rietz et al., 10 Feb 2025).

There is also a methodological caution specific to RL prompting. "Prompt-Tuning Decision Transformer with Preference Ranking" states that prompt learning is widely used in NLP but has limited applicability to RL because RL prompts contain complex physical meaning and environment-specific information, and because directly extending NLP prompt-tuning mechanisms is challenging when prompts guide agent behavior through environmental modeling rather than missing-information completion (Hu et al., 2023). This caution helps explain why much of the PDT literature focuses on trajectory prompts, goal prompts, or learned soft tokens rather than textual instructions.

A separate line of critique concerns the role of the transformer backbone itself. "How Crucial is Transformer in Decision Transformer?" replaces the transformer in DT with an LSTM while keeping the sequence formulation unchanged, and reports that Decision LSTM outperforms DT on several continuous-control tasks, including pendulum stabilization and real Furuta pendulum swing-up (Siebenborn et al., 2022). The paper argues that the strength of DT in those settings may lie in the overall sequential modeling architecture rather than the transformer per se, and further reports that the query RTG value appears to have no influence on episode return in its continuous-control experiments (Siebenborn et al., 2022). This does not invalidate PDT, but it narrows what should be attributed specifically to transformer-based in-context conditioning.

Finally, the acronym “PDT” is not unique. In "Future-conditioned Unsupervised Pretraining for Decision Transformer," PDT refers to Pretrained Decision Transformer, a reward-free pretraining method that conditions on latent future trajectory information rather than demonstration prompts (Xie et al., 2023). This nomenclature overlap is important in the literature because Prompting Decision Transformer and Pretrained Decision Transformer address different problems.

From a theoretical standpoint, "Prompting a Pretrained Transformer Can Be a Universal Approximator" shows that prefix-tuning a pretrained transformer can universally approximate sequence-to-sequence functions, that a single attention head can approximate any continuous function on the hypersphere, and that a transformer with depth linear in sequence length can approximate any sequence-to-sequence function when the right prefixes are supplied (Petrov et al., 2024). This does not establish a practical guarantee for any specific PDT implementation, but it suggests that prompt conditioning can in principle arbitrarily modify the behavior of a fixed pretrained transformer. Within the PDT literature, a plausible implication is that prompt design and prompt selection are not merely auxiliary heuristics; they are a primary locus of expressivity and control.