---
title: Pretrained Decision Transformer Overview
url: https://www.emergentmind.com/topics/pretrained-decision-transformer-pdt
type: topic
---

# Pretrained Decision Transformer Overview

Pretrained Decision Transformer (PDT) denotes a family of Decision Transformer formulations in which policy learning is coupled to some form of pretraining, most commonly transfer from GPT-style language models or unsupervised pretraining on offline trajectories, and then adapted to downstream offline reinforcement learning or control tasks. In the cited literature, the term covers both a specific future-conditioned unsupervised pretraining method and GPT-2-initialized Decision Transformers fine-tuned with Low-Rank Adaptation (LoRA); related variants extend the same paradigm with prompting, prompt regularization, contextual-bandit prompt selection, attention routing, and in-context control [2305.16683; 2411.17900; 2404.02407; 2408.01402; 2409.06985; 2502.04979; 2411.19746].

## 1. Terminological scope and lineage

The literature uses closely related names for several architectures that share a common decision-as-sequence-modeling premise. In all cases, the model operates on ordered control tokens rather than natural-language tokens, and the pretrained backbone is either reused directly or the decision model is itself pretrained before downstream adaptation.

| Paper | Usage of the term | Core mechanism |
|---|---|---|
| "Future-conditioned Unsupervised Pretraining for Decision Transformer" [2305.16683] | PDT | future-conditioned unsupervised pretraining |
| "Pretrained LLM Adapted with LoRA as a Decision Transformer for Offline RL in Quantitative Trading" [2411.17900] | PDT | pretrained GPT-2 small + LoRA |
| "Decision Transformer as a Foundation Model for Partially Observable Continuous Control" [2404.02407] | Pretrained Decision Transformer | GPT-2 initialization + LoRA |
| "Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformer" [2408.01402] | LPDT | DistilGPT2 initialization + prompt regularization |

In the 2023 unsupervised-pretraining formulation, PDT is explicitly introduced as “a conceptually simple approach for unsupervised RL pretraining” that “leverages future trajectory information as a privileged context to predict actions during training” [2305.16683]. In the 2024 quantitative-trading formulation, PDT is a Decision Transformer “initialized from a pretrained GPT-2 small model,” “augmented with small ‘adapter’ (LoRA) modules in every Transformer layer,” and “fine-tuned on expert trading trajectories by minimizing MSE on actions” [2411.17900]. In partially observable control, the same family is presented as initializing DT from “the Generative Pre-trained Transformer (GPT) series” and then training it for control tasks using LoRA [2404.02407].

This terminological spread matters because “pretrained” can refer either to language-model initialization or to reward-free pretraining on trajectories. A plausible implication is that PDT is best understood as a design space rather than a single fixed algorithm.

## 2. Sequence formulation and architectural pattern

Across these papers, PDT inherits the Decision Transformer view of offline RL as autoregressive sequence modeling. One concise statement is that Decision Transformer “casts RL as next-token prediction over sequences of triplets $(\hat R_t, s_t, a_t)$ where $\hat R_t = \sum_{t'=t}^T r_{t'}$ is the return-to-go” [2502.04979]. The quantitative-trading variant converts each expert trajectory into the interleaved format
$$
\tau = (\hat R_0, s_0, a_0, \hat R_1, s_1, a_1, \dots, \hat R_T, s_T, a_T),
$$
then applies a sliding window of length $K$ timesteps, yielding sequence length $3K$ [2411.17900]. The partial-observability formulation uses the order
$(\hat R_{t-K+1}, o_{t-K+1}, a_{t-K+1}, \dots, \hat R_t, o_t)$
and predicts the action from the token corresponding to the last observation [2404.02407].

Backbone choice varies by application. The trading and partially observable control papers start from GPT-2 small with “12 layers, 12 heads, hidden size 768” [2411.17900; 2404.02407]. LPDT instead uses “DistilGPT2 (6 Transformer blocks, hidden size $d=768$, 12 self-attention heads, total ≈82M parameters)” [2408.01402]. HVAC-DPT uses a smaller “GPT-2 style causal transformer with $L = 3$ layers, embedding dimension $d = 128$, $H = 8$ attention heads per layer” and is deployed without parameter updates, relying on in-context conditioning [2411.19746].

A recurring architectural modification is replacement of discrete tokenization by continuous projections. LPDT uses “three separate ‘token’ types for Return-to-Go (scalar), State $(\mathbb{R}^{d_s})$, Action $(\mathbb{R}^{d_a})$,” each projected by a small linear layer into the shared hidden dimension $d=768$, with “continuous floats for $(R,s,a)$ directly projected; no subword BPE” [2408.01402]. The trading PDT likewise replaces GPT-2’s discrete token embedding: each modality is embedded via a small residual MLP,
$$
\mathrm{Embed}(x) = x + \mathrm{MLP}(x),
$$
followed by interleaving and normalization [2411.17900]. In partial observability, the three modalities are mapped by three small MLPs $W_R$, $W_o$, and $W_a$, followed by layer-norm and dropout [2404.02407].

Prompted variants prepend task information. LPDT forms a prompt prefix of length $K^*$,
$$
\tau^*=[R^*_1,s^*_1,a^*_1,\dots,R^*_{K^*},s^*_{K^*},a^*_{K^*}],
$$
concatenates it with the training trajectory, and uses total sequence length $\ell=3(K^*+K)$; in practice $K^*=5,\;K=20\Rightarrow\ell=75$ [2408.01402]. The Prompting Decision Transformer uses a stochastic “trajectory prompt” $\rho$ composed of $J$ segments, kept fixed throughout the episode [2502.04979].

## 3. Pretraining, objectives, and parameter-efficient adaptation

One branch of PDT centers on unsupervised future conditioning. The 2023 PDT defines a policy $\pi_\theta(a \mid \tau_{1:t-1}, s_t, z)$, a future encoder $g_\theta(z \mid \tau_{future})$, a prior $p_\theta(z \mid s_t)$, and a return predictor $f_\theta(\hat R_t \mid z, s_t)$ [2305.16683]. Its pretraining loss combines behavior cloning and future regularization:
$$
L_{\mathrm{pre}} = L_{\mathrm{BC}} + L_{\mathrm{future}},
$$
where $L_{\mathrm{future}}$ includes KL terms for the latent future representation [2305.16683]. During finetuning, the method adds a return-prediction term,
$$
L_{\mathrm{fine}} = L_{\mathrm{pre}} + L_{\mathrm{return}},
$$
so that future embeddings can be sampled according to predicted returns [2305.16683].

The more common 2024 formulations transfer pretrained language-model weights and adapt them with LoRA. In trading PDT, if $W_0\in\mathbb{R}^{d\times k}$ is any frozen GPT-2 weight matrix, the update is
$$
W = W_0 + \Delta W,\quad \Delta W = BA,
$$
with only $A$ and $B$ learned; “in our experiments $r=16$,” adding “only ≈900 k trainable parameters (∼0.7% of GPT-2 small) across all layers” [2411.17900]. In partially observable control, LoRA is applied to “all attention Q,K,V,O and FFN W1,W2,” with rank $r=32$, and all GPT weights $W_0$ are frozen [2404.02407]. LPDT uses LoRA while freezing all pretrained Transformer weights except newly inserted linear layers, and updates only the LoRA matrices together with the prompt-encoder head [2408.01402].

The downstream objective is typically supervised action regression. The trading PDT minimizes
$$
L_{MSE} = \frac{1}{N\,T}\sum_{n=1}^N\sum_{t=1}^T\|a_t^{(n)} - \hat a_t^{(n)}\|_2^2,
$$
with “no explicit RL loss or Q-value regression” [2411.17900]. The partial-observability model uses mean-squared error between predicted action and demonstrated action [2404.02407]. LPDT gives both a negative log-likelihood Decision Transformer objective and an MSE replacement for continuous actions [2408.01402].

Prompt-specific regularization is a distinctive extension. LPDT augments $\mathcal{L}_{\rm DT}$ with a prompt-encoder loss $\mathcal{L}_\phi$:
$$
\mathcal{L}_{\rm total} = \mathcal{L}_{\rm DT} + \lambda\,\mathcal{L}_\phi,\quad \lambda\approx0.1.
$$
The regularizer is either supervised classification on task ID or unsupervised InfoNCE on prompt embeddings, with a 2-layer MLP prompt encoder of hidden size 128 and ReLU [2408.01402]. The bandit-based extension to Prompting Decision Transformer freezes the pretrained backbone and adapts only prompt selection online; each prompt segment is treated as an arm in a contextual multi-armed bandit, and each slot maintains a parametric reward model $\phi_j(\cdot)$ trained by MSE on observed returns [2502.04979].

## 4. Reported empirical behavior across domains

PDT-style methods have been evaluated in quantitative trading, MuJoCo control, Meta-World, partial-observability control, and HVAC control. The breadth of these applications reflects the sequence-modeling generality of the framework rather than a single benchmark convention.

| Setting | Representative reported outcome | Source |
|---|---|---|
| Quantitative trading | “PDT often ranks in the top two for returns and Sharpe.” | [2411.17900] |
| Partially observable control | “Zero-shot DT scores 0.6–1.2 on new tasks” and “10-shot adaptation quickly reaches or exceeds expert performance (≈1.0+).” | [2404.02407] |
| HVAC control | “Yearly reduction vs. Baseline: 45.6 %.” | [2411.19746] |

In quantitative trading, the environment comprises “the 29 DJIA constituent stocks via the FinRL framework, using daily OHLCV+technical indicators,” and five continuous-control RL agents—“A2C, PPO, SAC, TD3, DDPG”—generate expert deterministic trajectories [2411.17900]. On the test period “07/2020–10/2021,” the paper reports that PDT “often ranks in the top two for returns and Sharpe”; under TD3 expert, PDT achieves “the highest return (46.62%) and Sharpe (2.14), matching or exceeding the expert itself,” and under SAC it attains “the best MDD (–8.54%) and Sharpe (1.69)” [2411.17900]. The same study emphasizes the pretrained-vs-random contrast: “every expert saw a lift in return (e.g. +5.1 pp under DDPG: 47.98% vs 42.88%) and improved MDD/Sharpe” [2411.17900].

In few-shot prompt-based meta-RL, LPDT is evaluated on MuJoCo control meta-RL and Meta-World ML1. With full dataset and prompt length $K^*=5$, the reported results include Cheetah-dir: Prompt-DT $933.9\pm7.0$, Prompt-Tuning DT $941.5\pm3.2$, Prompt Diffuser $945.3\pm7.2$, LPDT (cls) **$947.8\pm1.5$**, LPDT (InfoNCE) **$951.7\pm4.1$**; Cheetah-vel: Prompt-DT $-34.7\pm2.8$, LPDT (cls) **$-31.6\pm2.7$**; Ant-dir: Prompt Diffuser $432.1\pm6.7$, LPDT (InfoNCE) **$412.5\pm21.0$**; MW reach-v2: Prompt-DT $692.3\pm9.3$, LPDT (InfoNCE) **$528.2\pm114.2$**; MW pick-place-v2: LPDT (InfoNCE) $3543.4\pm191.3$ [2408.01402]. These results are heterogeneous across tasks, but the same paper states that “initializing with a pre-trained language model significantly enhances the performance of Prompt-DT on unseen tasks compared to baseline methods” [2408.01402].

In partially observable continuous control, the reported tasks span “five distinct control tasks, ranging from maneuvering aerospace systems to controlling partial differential equations,” and multi-task experiments involve “30 tasks during training, 9 in-distribution and 9 out-of-distribution test tasks” [2404.02407]. The reported single-task results say that on expert data “DT often exceeds 1.0, and beats PPO, CQL, BC,” while on medium data “DT achieves ≈0.5–0.6 vs. baselines around 0.0” [2404.02407]. The same paper reports that “Zero-shot DT scores 0.6–1.2 on new tasks” and “10-shot adaptation quickly reaches or exceeds expert performance (≈1.0+)” [2404.02407].

HVAC-DPT shifts emphasis from weight updates to in-context adaptation. It trains on trajectories from “100 PPO agents” on the “ASHRAE-standard small office prototype (‘B_train,’ 5 zones, 511 m², Denver climate), each for 1 000 episodes of length $H=2 967$,” then deploys on an unseen building without gradient updates [2411.19746]. On “B_Denver, 15 zones, 4 982 m²,” the reported totals are “Baseline: 56.75 MWh in January…74.38 MWh in December (≈687 MWh total),” “HVAC-DPT: 32.38 MWh in January…46.71 MWh in December (≈373 MWh total),” and “Yearly reduction vs. Baseline: 45.6 %”; HVAC-DPT is also “within 5.8 % of the Expert controller’s energy” [2411.19746].

## 5. Mechanistic analysis, misconceptions, and failure modes

A recurrent misconception is that language-model initialization is uniformly beneficial across planning regimes. The Markov-head analysis directly argues against that simplification. In pretrained GPT-style transformers, an attention head $i$ is called a Markov Head when the matrix
$$
M_i = W_i^q (W_i^k)^T
$$
satisfies the stated diagonal-dominance conditions, and such heads induce extreme attention on the last-input token [2409.06985]. The paper further states that “this extreme attention cannot be changed by re-training embedding layer or fine-tuning” and that pretrained Markov Heads remain “last-token focused” throughout RL fine-tuning [2409.06985].

The empirical consequences are mixed rather than uniformly positive. On short-term MuJoCo tasks, GPT-DT improves over DT: Hopper-m $67.4 \rightarrow 77.9$, Walker2d-m $74.3 \rightarrow 77.1$, HalfCheetah-m $42.8 \rightarrow 42.6$, while GPT2-DTMA gives $77.4$, $79.9$, and $43.0$ respectively [2409.06985]. On long-term PointMaze_large, however, DT is $195.3$, GPT-DT degrades to $257.3$, and GPT2-DTMA recovers to $203.0$ [2409.06985]. The paper summarizes this by stating that GPT-DT “degrades in long-term tasks (e.g. +30 % longer episodes) because Markov Heads over-focus on the current state and ignore distant context,” whereas GPT2-DTMA “recovers or surpasses DT in long-term tasks by adaptively down-weighting Markov Heads and re-routing attention to non-Markov Heads” [2409.06985].

Prompt construction is another failure mode. The bandit-based prompt-tuning work states that Prompting Decision Transformer samples prompt segments uniformly at random from the per-task demonstration pool, and that “uniform sampling often picks poor segments,” producing “suboptimal action predictions at inference time and degraded returns, especially when the prompt length $J\times H$ is small” [2502.04979]. Its reported gains are substantial: on Sparse 2D Point with $J=1$, “PDT+no tuning $0.0\pm2.1$ vs $\epsilon$-greedy bandit $9.0\pm0.6$ (optimal=10)”; with $J=2$, $6.3\pm0.8$ vs $9.4\pm0.3$; with $J=4$, $8.3\pm0.6$ vs $9.6\pm0.2$ [2502.04979].

LPDT reports a different limitation: few-shot prompting can fail to distinguish tasks when the pretraining dataset is limited. Its response is prompt regularization. The ablations state that “Removing prompt regularization hurts performance by 3–10%,” that “Text-based regularization (LaMo style) is inferior and extra cost,” and that “With only 10% of the data, LPDT still matches or outperforms full-data Prompt-DT” [2408.01402].

Application-specific limitations are also explicit. The trading PDT lists “only single-expert trajectories (future work: mixture-of-experts),” “no interpretability,” and evaluation on “only GPT-2 small and DJIA” [2411.17900]. HVAC-DPT notes that “non-stationary environments” may degrade performance, that the approach assumes “consistent state representations,” and that “memory and compute scale with context length” [2411.19746]. The unsupervised PDT notes “higher compute/training time than vanilla DT/ODT,” “dataset-specific tuning of $\beta$,” and possible “future encoder collapse without careful regularization” [2305.16683].

## 6. Extensions and research directions

Several extensions define the current frontier of PDT-style research. One direction is task-aware prompting. LPDT combines pre-trained language-model initialization with prompt regularization and LoRA, while the prompt-tuning bandit freezes the Transformer backbone and adapts only prompt choice online [2408.01402; 2502.04979]. This suggests a division between parameter-space adaptation and context-space adaptation.

Another direction is architectural control over inherited attention biases. GPT2-DTMA replaces standard multi-head attention by a Mixture-of-Attention module whose gating network
$$
g_t = \mathrm{Softmax}(W_g x_t)
$$
weights pretrained heads per token [2409.06985]. The paper’s stated “Best Practice” is: “When porting large-scale, cross-domain pretrained transformers into RL, inspect the QK-matrix spectra for Markov structure. If long-term tasks are envisaged, add a gating or routing layer (e.g. MoA) rather than blanket fine-tuning” [2409.06985].

A third direction is broadening deployment regimes. The partial-observability work frames DT as a “foundational controller for general control applications,” reporting robustness on out-of-distribution perturbations and fast 10-shot recovery [2404.02407]. HVAC-DPT shows that a “small causal transformer, pretrained on diverse offline RL trajectories, can serve as a plug-and-play HVAC controller,” and explicitly proposes extensions such as “explicit return-to-go tokens,” “sparsified attention or memory-compressed transformers,” and pretraining on “a federated fleet of real buildings” [2411.19746]. The trading PDT proposes “Combine multiple experts’ data for richer offline RL,” “Align decision outputs with text explanations,” and “Benchmark on other markets (crypto, FX) and with larger pretrained backbones” [2411.17900].

The unsupervised-pretraining branch pushes in a different but complementary direction. Because PDT conditions on learned future embeddings rather than scalar returns during pretraining, it can learn from “reward-free and sub-optimal offline data” and later steer behavior by sampling futures according to predicted return [2305.16683]. The paper’s proposed extensions include “richer latent priors (e.g. mixture models or VQ-VAE), diffusion-based policy generation, vision-based pretraining” [2305.16683].

Taken together, these works indicate that PDT is not a single settled architecture. It is a convergent family of methods in which Decision Transformer inherits either pretrained sequence priors or task-agnostic trajectory structure, and then adapts through LoRA, prompting, future conditioning, attention routing, or in-context inference according to the constraints of the downstream domain.

Source: https://www.emergentmind.com/topics/pretrained-decision-transformer-pdt