Pretrained Decision Transformer Overview

• Pretrained Decision Transformer is a framework that integrates GPT-based pretraining with sequence modeling to enable offline RL and control across diverse applications.
• It employs techniques like low-rank adaptation (LoRA), prompt regularization, and future conditioning to transfer knowledge from unsupervised data to task-specific policies.
• Empirical results show improved performance in quantitative trading, robotics, and HVAC control by effectively leveraging pretrained models and tailored adaptation strategies.

Pretrained Decision Transformer (PDT) denotes a family of Decision Transformer formulations in which policy learning is coupled to some form of pretraining, most commonly transfer from GPT-style LLMs or unsupervised pretraining on offline trajectories, and then adapted to downstream offline reinforcement learning or control tasks. In the cited literature, the term covers both a specific future-conditioned unsupervised pretraining method and GPT-2-initialized Decision Transformers fine-tuned with Low-Rank Adaptation (LoRA); related variants extend the same paradigm with prompting, prompt regularization, contextual-bandit prompt selection, attention routing, and in-context control (Xie et al., 2023, Yun, 2024, Zhang et al., 2024, Yang et al., 2024, Zhao et al., 2024, Rietz et al., 7 Feb 2025, Berkes, 2024).

1. Terminological scope and lineage

The literature uses closely related names for several architectures that share a common decision-as-sequence-modeling premise. In all cases, the model operates on ordered control tokens rather than natural-language tokens, and the pretrained backbone is either reused directly or the decision model is itself pretrained before downstream adaptation.

Paper	Usage of the term	Core mechanism
"Future-conditioned Unsupervised Pretraining for Decision Transformer" (Xie et al., 2023)	PDT	future-conditioned unsupervised pretraining
"Pretrained LLM Adapted with LoRA as a Decision Transformer for Offline RL in Quantitative Trading" (Yun, 2024)	PDT	pretrained GPT-2 small + LoRA
"Decision Transformer as a Foundation Model for Partially Observable Continuous Control" (Zhang et al., 2024)	Pretrained Decision Transformer	GPT-2 initialization + LoRA
"Pre-trained LLMs Improve the Few-shot Prompt Ability of Decision Transformer" (Yang et al., 2024)	LPDT	DistilGPT2 initialization + prompt regularization

In the 2023 unsupervised-pretraining formulation, PDT is explicitly introduced as “a conceptually simple approach for unsupervised RL pretraining” that “leverages future trajectory information as a privileged context to predict actions during training” (Xie et al., 2023). In the 2024 quantitative-trading formulation, PDT is a Decision Transformer “initialized from a pretrained GPT-2 small model,” “augmented with small ‘adapter’ (LoRA) modules in every Transformer layer,” and “fine-tuned on expert trading trajectories by minimizing MSE on actions” (Yun, 2024). In partially observable control, the same family is presented as initializing DT from “the Generative Pre-trained Transformer (GPT) series” and then training it for control tasks using LoRA (Zhang et al., 2024).

This terminological spread matters because “pretrained” can refer either to language-model initialization or to reward-free pretraining on trajectories. A plausible implication is that PDT is best understood as a design space rather than a single fixed algorithm.

2. Sequence formulation and architectural pattern

Across these papers, PDT inherits the Decision Transformer view of offline RL as autoregressive sequence modeling. One concise statement is that Decision Transformer “casts RL as next-token prediction over sequences of triplets $(\hat R_t, s_t, a_t)$ where $\hat R_t = \sum_{t'=t}^T r_{t'}$ is the return-to-go” (Rietz et al., 7 Feb 2025). The quantitative-trading variant converts each expert trajectory into the interleaved format

$$\tau = (\hat R_0, s_0, a_0, \hat R_1, s_1, a_1, \dots, \hat R_T, s_T, a_T),$$

then applies a sliding window of length $K$ timesteps, yielding sequence length $3K$ (Yun, 2024). The partial-observability formulation uses the order

$$(\hat R_{t-K+1}, o_{t-K+1}, a_{t-K+1}, \dots, \hat R_t, o_t)$$

and predicts the action from the token corresponding to the last observation (Zhang et al., 2024).

Backbone choice varies by application. The trading and partially observable control papers start from GPT-2 small with “12 layers, 12 heads, hidden size 768” (Yun, 2024, Zhang et al., 2024). LPDT instead uses “DistilGPT2 (6 Transformer blocks, hidden size $d=768$, 12 self-attention heads, total ≈82M parameters)” (Yang et al., 2024). HVAC-DPT uses a smaller “GPT-2 style causal transformer with $L = 3$ layers, embedding dimension $d = 128$, $H = 8$ attention heads per layer” and is deployed without parameter updates, relying on in-context conditioning (Berkes, 2024).

A recurring architectural modification is replacement of discrete tokenization by continuous projections. LPDT uses “three separate ‘token’ types for Return-to-Go (scalar), State $\hat R_t = \sum_{t'=t}^T r_{t'}$0, Action $\hat R_t = \sum_{t'=t}^T r_{t'}$1,” each projected by a small linear layer into the shared hidden dimension $\hat R_t = \sum_{t'=t}^T r_{t'}$2, with “continuous floats for $\hat R_t = \sum_{t'=t}^T r_{t'}$3 directly projected; no subword BPE” (Yang et al., 2024). The trading PDT likewise replaces GPT-2’s discrete token embedding: each modality is embedded via a small residual MLP,

$\hat R_t = \sum_{t'=t}^T r_{t'}$4

followed by interleaving and normalization (Yun, 2024). In partial observability, the three modalities are mapped by three small MLPs $\hat R_t = \sum_{t'=t}^T r_{t'}$5, $\hat R_t = \sum_{t'=t}^T r_{t'}$6, and $\hat R_t = \sum_{t'=t}^T r_{t'}$7, followed by layer-norm and dropout (Zhang et al., 2024).

Prompted variants prepend task information. LPDT forms a prompt prefix of length $\hat R_t = \sum_{t'=t}^T r_{t'}$8,

$\hat R_t = \sum_{t'=t}^T r_{t'}$9

concatenates it with the training trajectory, and uses total sequence length $$\tau = (\hat R_0, s_0, a_0, \hat R_1, s_1, a_1, \dots, \hat R_T, s_T, a_T),$$0; in practice $$\tau = (\hat R_0, s_0, a_0, \hat R_1, s_1, a_1, \dots, \hat R_T, s_T, a_T),$$1 (Yang et al., 2024). The Prompting Decision Transformer uses a stochastic “trajectory prompt” $$\tau = (\hat R_0, s_0, a_0, \hat R_1, s_1, a_1, \dots, \hat R_T, s_T, a_T),$$2 composed of $$\tau = (\hat R_0, s_0, a_0, \hat R_1, s_1, a_1, \dots, \hat R_T, s_T, a_T),$$3 segments, kept fixed throughout the episode (Rietz et al., 7 Feb 2025).

3. Pretraining, objectives, and parameter-efficient adaptation

One branch of PDT centers on unsupervised future conditioning. The 2023 PDT defines a policy $$\tau = (\hat R_0, s_0, a_0, \hat R_1, s_1, a_1, \dots, \hat R_T, s_T, a_T),$$4, a future encoder $$\tau = (\hat R_0, s_0, a_0, \hat R_1, s_1, a_1, \dots, \hat R_T, s_T, a_T),$$5, a prior $$\tau = (\hat R_0, s_0, a_0, \hat R_1, s_1, a_1, \dots, \hat R_T, s_T, a_T),$$6, and a return predictor $$\tau = (\hat R_0, s_0, a_0, \hat R_1, s_1, a_1, \dots, \hat R_T, s_T, a_T),$$7 (Xie et al., 2023). Its pretraining loss combines behavior cloning and future regularization:

$$\tau = (\hat R_0, s_0, a_0, \hat R_1, s_1, a_1, \dots, \hat R_T, s_T, a_T),$$8

where $$\tau = (\hat R_0, s_0, a_0, \hat R_1, s_1, a_1, \dots, \hat R_T, s_T, a_T),$$9 includes KL terms for the latent future representation (Xie et al., 2023). During finetuning, the method adds a return-prediction term,

$K$0

so that future embeddings can be sampled according to predicted returns (Xie et al., 2023).

The more common 2024 formulations transfer pretrained language-model weights and adapt them with LoRA. In trading PDT, if $K$1 is any frozen GPT-2 weight matrix, the update is

$K$2

with only $K$3 and $K$4 learned; “in our experiments $K$5,” adding “only ≈900 k trainable parameters (∼0.7% of GPT-2 small) across all layers” (Yun, 2024). In partially observable control, LoRA is applied to “all attention Q,K,V,O and FFN W1,W2,” with rank $K$6, and all GPT weights $K$7 are frozen (Zhang et al., 2024). LPDT uses LoRA while freezing all pretrained Transformer weights except newly inserted linear layers, and updates only the LoRA matrices together with the prompt-encoder head (Yang et al., 2024).

The downstream objective is typically supervised action regression. The trading PDT minimizes

$K$8

with “no explicit RL loss or Q-value regression” (Yun, 2024). The partial-observability model uses mean-squared error between predicted action and demonstrated action (Zhang et al., 2024). LPDT gives both a negative log-likelihood Decision Transformer objective and an MSE replacement for continuous actions (Yang et al., 2024).

Prompt-specific regularization is a distinctive extension. LPDT augments $K$9 with a prompt-encoder loss $3K$0:

$3K$1

The regularizer is either supervised classification on task ID or unsupervised InfoNCE on prompt embeddings, with a 2-layer MLP prompt encoder of hidden size 128 and ReLU (Yang et al., 2024). The bandit-based extension to Prompting Decision Transformer freezes the pretrained backbone and adapts only prompt selection online; each prompt segment is treated as an arm in a contextual multi-armed bandit, and each slot maintains a parametric reward model $3K$2 trained by MSE on observed returns (Rietz et al., 7 Feb 2025).

4. Reported empirical behavior across domains

PDT-style methods have been evaluated in quantitative trading, MuJoCo control, Meta-World, partial-observability control, and HVAC control. The breadth of these applications reflects the sequence-modeling generality of the framework rather than a single benchmark convention.

Setting	Representative reported outcome	Source
Quantitative trading	“PDT often ranks in the top two for returns and Sharpe.”	(Yun, 2024)
Partially observable control	“Zero-shot DT scores 0.6–1.2 on new tasks” and “10-shot adaptation quickly reaches or exceeds expert performance (≈1.0+).”	(Zhang et al., 2024)
HVAC control	“Yearly reduction vs. Baseline: 45.6 %.”	(Berkes, 2024)

In quantitative trading, the environment comprises “the 29 DJIA constituent stocks via the FinRL framework, using daily OHLCV+technical indicators,” and five continuous-control RL agents—“A2C, PPO, SAC, TD3, DDPG”—generate expert deterministic trajectories (Yun, 2024). On the test period “07/2020–10/2021,” the paper reports that PDT “often ranks in the top two for returns and Sharpe”; under TD3 expert, PDT achieves “the highest return (46.62%) and Sharpe (2.14), matching or exceeding the expert itself,” and under SAC it attains “the best MDD (–8.54%) and Sharpe (1.69)” (Yun, 2024). The same study emphasizes the pretrained-vs-random contrast: “every expert saw a lift in return (e.g. +5.1 pp under DDPG: 47.98% vs 42.88%) and improved MDD/Sharpe” (Yun, 2024).

In few-shot prompt-based meta-RL, LPDT is evaluated on MuJoCo control meta-RL and Meta-World ML1. With full dataset and prompt length $3K$3, the reported results include Cheetah-dir: Prompt-DT $3K$4, Prompt-Tuning DT $3K$5, Prompt Diffuser $3K$6, LPDT (cls) $3K$7, LPDT (InfoNCE) $3K$8; Cheetah-vel: Prompt-DT $3K$9, LPDT (cls) $$(\hat R_{t-K+1}, o_{t-K+1}, a_{t-K+1}, \dots, \hat R_t, o_t)$$0; Ant-dir: Prompt Diffuser $$(\hat R_{t-K+1}, o_{t-K+1}, a_{t-K+1}, \dots, \hat R_t, o_t)$$1, LPDT (InfoNCE) $$(\hat R_{t-K+1}, o_{t-K+1}, a_{t-K+1}, \dots, \hat R_t, o_t)$$2; MW reach-v2: Prompt-DT $$(\hat R_{t-K+1}, o_{t-K+1}, a_{t-K+1}, \dots, \hat R_t, o_t)$$3, LPDT (InfoNCE) $$(\hat R_{t-K+1}, o_{t-K+1}, a_{t-K+1}, \dots, \hat R_t, o_t)$$4; MW pick-place-v2: LPDT (InfoNCE) $$(\hat R_{t-K+1}, o_{t-K+1}, a_{t-K+1}, \dots, \hat R_t, o_t)$$5 (Yang et al., 2024). These results are heterogeneous across tasks, but the same paper states that “initializing with a pre-trained LLM significantly enhances the performance of Prompt-DT on unseen tasks compared to baseline methods” (Yang et al., 2024).

In partially observable continuous control, the reported tasks span “five distinct control tasks, ranging from maneuvering aerospace systems to controlling partial differential equations,” and multi-task experiments involve “30 tasks during training, 9 in-distribution and 9 out-of-distribution test tasks” (Zhang et al., 2024). The reported single-task results say that on expert data “DT often exceeds 1.0, and beats PPO, CQL, BC,” while on medium data “DT achieves ≈0.5–0.6 vs. baselines around 0.0” (Zhang et al., 2024). The same paper reports that “Zero-shot DT scores 0.6–1.2 on new tasks” and “10-shot adaptation quickly reaches or exceeds expert performance (≈1.0+)” (Zhang et al., 2024).

HVAC-DPT shifts emphasis from weight updates to in-context adaptation. It trains on trajectories from “100 PPO agents” on the “ASHRAE-standard small office prototype (‘B_train,’ 5 zones, 511 m², Denver climate), each for 1 000 episodes of length $$(\hat R_{t-K+1}, o_{t-K+1}, a_{t-K+1}, \dots, \hat R_t, o_t)$$6,” then deploys on an unseen building without gradient updates (Berkes, 2024). On “B_Denver, 15 zones, 4 982 m²,” the reported totals are “Baseline: 56.75 MWh in January…74.38 MWh in December (≈687 MWh total),” “HVAC-DPT: 32.38 MWh in January…46.71 MWh in December (≈373 MWh total),” and “Yearly reduction vs. Baseline: 45.6 %”; HVAC-DPT is also “within 5.8 % of the Expert controller’s energy” (Berkes, 2024).

5. Mechanistic analysis, misconceptions, and failure modes

A recurrent misconception is that language-model initialization is uniformly beneficial across planning regimes. The Markov-head analysis directly argues against that simplification. In pretrained GPT-style transformers, an attention head $$(\hat R_{t-K+1}, o_{t-K+1}, a_{t-K+1}, \dots, \hat R_t, o_t)$$7 is called a Markov Head when the matrix

$$(\hat R_{t-K+1}, o_{t-K+1}, a_{t-K+1}, \dots, \hat R_t, o_t)$$8

satisfies the stated diagonal-dominance conditions, and such heads induce extreme attention on the last-input token (Zhao et al., 2024). The paper further states that “this extreme attention cannot be changed by re-training embedding layer or fine-tuning” and that pretrained Markov Heads remain “last-token focused” throughout RL fine-tuning (Zhao et al., 2024).

The empirical consequences are mixed rather than uniformly positive. On short-term MuJoCo tasks, GPT-DT improves over DT: Hopper-m $$(\hat R_{t-K+1}, o_{t-K+1}, a_{t-K+1}, \dots, \hat R_t, o_t)$$9, Walker2d-m $d=768$0, HalfCheetah-m $d=768$1, while GPT2-DTMA gives $d=768$2, $d=768$3, and $d=768$4 respectively (Zhao et al., 2024). On long-term PointMaze_large, however, DT is $d=768$5, GPT-DT degrades to $d=768$6, and GPT2-DTMA recovers to $d=768$7 (Zhao et al., 2024). The paper summarizes this by stating that GPT-DT “degrades in long-term tasks (e.g. +30 % longer episodes) because Markov Heads over-focus on the current state and ignore distant context,” whereas GPT2-DTMA “recovers or surpasses DT in long-term tasks by adaptively down-weighting Markov Heads and re-routing attention to non-Markov Heads” (Zhao et al., 2024).

Prompt construction is another failure mode. The bandit-based prompt-tuning work states that Prompting Decision Transformer samples prompt segments uniformly at random from the per-task demonstration pool, and that “uniform sampling often picks poor segments,” producing “suboptimal action predictions at inference time and degraded returns, especially when the prompt length $d=768$8 is small” (Rietz et al., 7 Feb 2025). Its reported gains are substantial: on Sparse 2D Point with $d=768$9, “PDT+no tuning $L = 3$0 vs $L = 3$1-greedy bandit $L = 3$2 (optimal=10)”; with $L = 3$3, $L = 3$4 vs $L = 3$5; with $L = 3$6, $L = 3$7 vs $L = 3$8 (Rietz et al., 7 Feb 2025).

LPDT reports a different limitation: few-shot prompting can fail to distinguish tasks when the pretraining dataset is limited. Its response is prompt regularization. The ablations state that “Removing prompt regularization hurts performance by 3–10%,” that “Text-based regularization (LaMo style) is inferior and extra cost,” and that “With only 10% of the data, LPDT still matches or outperforms full-data Prompt-DT” (Yang et al., 2024).

Application-specific limitations are also explicit. The trading PDT lists “only single-expert trajectories (future work: mixture-of-experts),” “no interpretability,” and evaluation on “only GPT-2 small and DJIA” (Yun, 2024). HVAC-DPT notes that “non-stationary environments” may degrade performance, that the approach assumes “consistent state representations,” and that “memory and compute scale with context length” (Berkes, 2024). The unsupervised PDT notes “higher compute/training time than vanilla DT/ODT,” “dataset-specific tuning of $L = 3$9,” and possible “future encoder collapse without careful regularization” (Xie et al., 2023).

6. Extensions and research directions

Several extensions define the current frontier of PDT-style research. One direction is task-aware prompting. LPDT combines pre-trained language-model initialization with prompt regularization and LoRA, while the prompt-tuning bandit freezes the Transformer backbone and adapts only prompt choice online (Yang et al., 2024, Rietz et al., 7 Feb 2025). This suggests a division between parameter-space adaptation and context-space adaptation.

Another direction is architectural control over inherited attention biases. GPT2-DTMA replaces standard multi-head attention by a Mixture-of-Attention module whose gating network

$d = 128$0

weights pretrained heads per token (Zhao et al., 2024). The paper’s stated “Best Practice” is: “When porting large-scale, cross-domain pretrained transformers into RL, inspect the QK-matrix spectra for Markov structure. If long-term tasks are envisaged, add a gating or routing layer (e.g. MoA) rather than blanket fine-tuning” (Zhao et al., 2024).

A third direction is broadening deployment regimes. The partial-observability work frames DT as a “foundational controller for general control applications,” reporting robustness on out-of-distribution perturbations and fast 10-shot recovery (Zhang et al., 2024). HVAC-DPT shows that a “small causal transformer, pretrained on diverse offline RL trajectories, can serve as a plug-and-play HVAC controller,” and explicitly proposes extensions such as “explicit return-to-go tokens,” “sparsified attention or memory-compressed transformers,” and pretraining on “a federated fleet of real buildings” (Berkes, 2024). The trading PDT proposes “Combine multiple experts’ data for richer offline RL,” “Align decision outputs with text explanations,” and “Benchmark on other markets (crypto, FX) and with larger pretrained backbones” (Yun, 2024).

The unsupervised-pretraining branch pushes in a different but complementary direction. Because PDT conditions on learned future embeddings rather than scalar returns during pretraining, it can learn from “reward-free and sub-optimal offline data” and later steer behavior by sampling futures according to predicted return (Xie et al., 2023). The paper’s proposed extensions include “richer latent priors (e.g. mixture models or VQ-VAE), diffusion-based policy generation, vision-based pretraining” (Xie et al., 2023).

Taken together, these works indicate that PDT is not a single settled architecture. It is a convergent family of methods in which Decision Transformer inherits either pretrained sequence priors or task-agnostic trajectory structure, and then adapts through LoRA, prompting, future conditioning, attention routing, or in-context inference according to the constraints of the downstream domain.