Online Decision Transformers (ODT)

• Online Decision Transformers (ODTs) are reinforcement learning methods that recast policy learning as return-conditioned sequence modeling in an interactive online setting.
• They blend offline pretraining with online finetuning using autoregressive negative log-likelihood objectives and entropy regularization to drive exploration.
• Variants incorporate world models, memory modules, and pure RL gradients, resulting in improved sample efficiency and robustness across multiple benchmarks.

Searching arXiv for papers on Online Decision Transformers and related variants. Online Decision Transformers (ODTs) are reinforcement-learning methods that extend the Decision Transformer paradigm from static offline datasets to interactive learning, while preserving the view of policy learning as return-conditioned sequence modeling. In the narrow formulation introduced by Zheng et al., an ODT blends offline pretraining with online finetuning in a unified framework, uses a stochastic transformer policy rather than a deterministic decoder, trains with an autoregressive negative log-likelihood objective, and enforces exploration through sequence-level entropy regularization (Zheng et al., 2022). Later work broadened the ODT family to include variants with world-model augmentation, explicit action-outcome memory, robustness to frame dropping, and online finetuning with pure reinforcement-learning gradients rather than supervised sequence loss (Jiang et al., 2024, Aref et al., 19 Sep 2025, Yotsufuji et al., 18 Jun 2026, Luo et al., 1 Jan 2026).

1. Origins and conceptual scope

Decision Transformer (DT) recast offline reinforcement learning as sequence modeling over trajectories of returns-to-go, states, and actions. ODT preserves that trajectory-centric representation but moves into an online regime in which the agent interacts with the environment, collects new trajectories, and updates the transformer continually (Zheng et al., 2022). In the baseline formulation, the trajectory of length $K$ is tokenized as

$\tau = (g_1, s_1, a_1, \ldots, g_K, s_K, a_K),$

with

$g_t = \sum_{k=t}^K \gamma^{k-t} r_k.$

The policy remains a conditional sequence model, but the data distribution is no longer fixed; it becomes the combination of offline trajectories and online replay (Aref et al., 19 Sep 2025).

Two usages of the term coexist. In the strict sense, ODT denotes the value-free, entropy-regularized online DT of Zheng et al., which uses no explicit $V$- or $Q$-function and no TD bootstrapping (Zheng et al., 2022, Aref et al., 19 Sep 2025). In a broader usage, later work treats any online transformer policy that preserves DT-style trajectory conditioning as an ODT variant, even when the online phase incorporates actor-critic learning, model-based augmentation, or explicit value gradients (Jiang et al., 2024, Yan et al., 2024, Luo et al., 1 Jan 2026). A common misconception is therefore that ODTs are defined by being value-free. The narrower algorithm is value-free, but the broader research line now includes hybrids with critics and world models.

2. Sequence-modeling formulation

The baseline ODT models a stochastic policy as a Gaussian whose mean and covariance are produced by the transformer from state and return context: $$\pi_\theta(a_t \mid s_{-K,t}, g_{-K,t}) = \mathcal{N}\big(\mu_\theta(s_{-K,t}, g_{-K,t}), \Sigma_\theta(s_{-K,t}, g_{-K,t})\big),$$ with diagonal $\Sigma_\theta$ (Aref et al., 19 Sep 2025). The supervised core objective is negative log-likelihood over $K$-step contexts: $$\mathcal{J}(\theta) = \frac{1}{K}\,\mathbb{E}_{(a,s,g)\sim\mathcal{T}} \left[\sum_{k=1}^K -\log \pi_\theta(a_k \mid s_{-K,k}, g_{-K,k})\right].$$ This is the direct online analogue of offline DT training: the transformer predicts actions conditioned on trajectory history and return-to-go, rather than learning Bellman targets.

Exploration is introduced by a sequence-level entropy constraint. ODT imposes

$$\mathcal{H}_{\theta}^{\tau}[r(a \mid s,g)] = \frac{1}{K}\,\mathbb{E}_{(s,g)\sim\mathcal{T}} \left[\sum_{k=1}^{K}\mathcal{H}\left[\pi_\theta(a_k \mid s_{-K,k}, g_{-K,k})\right]\right] \ge \beta,$$

and optimizes the Lagrangian

$\tau = (g_1, s_1, a_1, \ldots, g_K, s_K, a_K),$0

This gives ODT its characteristic status as a value-free online RL algorithm in which exploration arises from entropy constraints and credit assignment is delegated to the transformer’s sequence modeling rather than Bellman recursion (Aref et al., 19 Sep 2025).

A related but more specific formulation appears in the original ODT paper, where the action model is likewise stochastic Gaussian and the online objective is a max-entropy sequence-modeling objective implemented with a dual temperature variable. The paper emphasizes that ODT keeps the same transformer and essentially the same objective in both offline and online phases, differing mainly in the fact that the data distribution becomes policy-dependent once online interaction begins (Zheng et al., 2022).

3. Online learning mechanics

The standard ODT loop is trajectory-based. The replay buffer stores full episodes rather than isolated transitions, and online finetuning repeatedly alternates rollout, buffer update, and transformer optimization (Zheng et al., 2022). In the formulation summarized by DODT, the online loop is: initialize $\tau = (g_1, s_1, a_1, \ldots, g_K, s_K, a_K),$1 with top $\tau = (g_1, s_1, a_1, \ldots, g_K, s_K, a_K),$2 trajectories from offline data; for rounds $\tau = (g_1, s_1, a_1, \ldots, g_K, s_K, a_K),$3, collect a new trajectory by executing $\tau = (g_1, s_1, a_1, \ldots, g_K, s_K, a_K),$4, update the replay buffer, and fine-tune $\tau = (g_1, s_1, a_1, \ldots, g_K, s_K, a_K),$5 for $\tau = (g_1, s_1, a_1, \ldots, g_K, s_K, a_K),$6 gradient steps on minibatches from $\tau = (g_1, s_1, a_1, \ldots, g_K, s_K, a_K),$7 (Jiang et al., 2024).

A central mechanism is hindsight return relabeling. During online interaction, actions are produced while conditioning on an exploration return $\tau = (g_1, s_1, a_1, \ldots, g_K, s_K, a_K),$8, but the stored sequence is relabeled with the actual achieved return-to-go

$\tau = (g_1, s_1, a_1, \ldots, g_K, s_K, a_K),$9

The original ODT paper presents this as a critical stabilizing device: it corrects discrepancies between specified target return and realized return during online exploration (Zheng et al., 2022). OnDeFog adopts the same principle and states explicitly that ODT replaces the target return specified during action decision with the actual return obtained (Yotsufuji et al., 18 Jun 2026).

The online phase therefore remains supervised in a precise sense: the targets are observed actions from newly collected trajectories, not value targets. This distinguishes the baseline ODT family from actor-critic methods. It also explains why replay-buffer composition matters. The original ODT uses a FIFO-style update that removes the oldest trajectory (Zheng et al., 2022), whereas DODT retains a high-quality buffer by discarding the lowest-reward trajectory instead, because Dreamer-generated or Dreamer-guided trajectories are intended to bias ODT toward richer and higher-return behavior (Jiang et al., 2024).

4. Variants and extensions

Several later variants modify either the memory mechanism, the source of online data, or the optimization objective.

DODT integrates Dreamer’s actor-critic world model with ODT in a parallel, bidirectionally enhanced system. Dreamer interacts with the environment, imagines trajectories in latent space, and contributes beneficial trajectories to ODT’s replay buffer; ODT’s behavior in turn changes the data distribution seen by Dreamer (Jiang et al., 2024). The paper reports that in online settings DODT improves over ODT on multiple benchmarks, including HalfCheetah-medium, HalfCheetah-medium-replay, and AntMaze-umaze-diverse.

EWA–VQ–ODT adds explicit per-action “mental accounts” to baseline ODT. Continuous actions are quantized into a small codebook, and each code $g_t = \sum_{k=t}^K \gamma^{k-t} r_k.$0 maintains an attraction

$g_t = \sum_{k=t}^K \gamma^{k-t} r_k.$1

These attractions are injected as bounded additive biases into attention logits for action-token columns, without changing the backbone transformer or the main objective (Aref et al., 19 Sep 2025). The paper reports large gains on walker2d-medium-replay-v2, with evaluation/return_mean_gm $g_t = \sum_{k=t}^K \gamma^{k-t} r_k.$2 for EWA–VQ–ODT versus $g_t = \sum_{k=t}^K \gamma^{k-t} r_k.$3 for ODT, and more modest but consistent gains on hopper-medium-v2.

A different line of work argues that the standard ODT online objective is itself the bottleneck. “Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers” shows that RTG values far from expected return hamper online finetuning and that simply adding TD3 gradients to the finetuning process of ODT effectively improves performance, especially if ODT is pretrained with low-reward offline data (Yan et al., 2024). The actor loss augments the DT imitation term with a critic term of the form $g_t = \sum_{k=t}^K \gamma^{k-t} r_k.$4, thereby importing local value gradients into the sequence model.

“Online Finetuning Decision Transformers with Pure RL Gradients” goes further and removes the supervised online objective entirely. It identifies hindsight return relabeling as fundamentally incompatible with importance-sampling-based RL algorithms such as GRPO, removes relabeling, and introduces pure-RL online finetuning with sub-trajectory optimization, sequence-level likelihood ratios, and active sampling of uncertain reset points (Luo et al., 1 Jan 2026). This line is important because it breaks with the assumption that ODTs must remain behavior-cloning-like during online learning.

OnDeFog addresses robustness rather than credit assignment. It extends ODT to environments with dropped observations and rewards by combining train-time frame dropping with a drop-span embedding that tells the model how many consecutive frames have been dropped. The resulting architecture significantly outperforms ODT at high drop rates and outperforms DeFog on medium-replay datasets containing much low-reward data (Yotsufuji et al., 18 Jun 2026).

5. Empirical regimes and deployments

The original ODT paper reports that ODT is competitive with the state-of-the-art in absolute performance on the D4RL benchmark but shows much more significant gains during the finetuning procedure (Zheng et al., 2022). On Gym and AntMaze tasks, the reported online gains are especially large relative to offline initialization, and the method is markedly more effective than training from scratch with SAC under the same online budget.

Subsequent work widened the empirical profile. DODT reports higher asymptotic returns and faster convergence than ODT in several MuJoCo and AntMaze environments, especially those with complex dynamics and long horizons (Jiang et al., 2024). EWA–VQ–ODT improves sample efficiency and average return over ODT, particularly in early training, while preserving the same backbone, optimizer, and training schedule (Aref et al., 19 Sep 2025). Pure-RL online finetuning with GRPO or PPO achieves new state-of-the-art performance across multiple benchmarks and is especially strong on low-quality offline datasets where supervised ODT variants struggle (Luo et al., 1 Jan 2026). TD3-augmented ODT likewise performs strongly on Adroit, Antmaze, MuJoCo, and Maze2D, especially when pretrained with low-reward data (Yan et al., 2024).

ODT ideas have also moved into production systems. In notification optimization at LinkedIn, a DT-based policy with circular buffer-based sequence processing was deployed for near-real-time inference in a high-throughput system, using rolling per-user context and multi-objective RTG conditioning. Compared to a multi-objective CQL-based agent, the DT-based approach achieved a $g_t = \sum_{k=t}^K \gamma^{k-t} r_k.$5 increase in sessions while reducing notification volume by $g_t = \sum_{k=t}^K \gamma^{k-t} r_k.$6 and keeping CTR changes statistically insignificant (Ocejo et al., 2 Sep 2025). This was not online parameter updating in the narrow ODT sense, but it was an online inference deployment of a return-conditioned transformer policy with continuously updated trajectory context.

Empirical evidence is not uniformly favorable. In VizDoom, a study comparing transformer-based RL approaches found that DTQN underperformed DQN-DRQN in online Team Deathmatch and offline DT underperformed PPO in the basic scenario (Batth et al., 24 Apr 2025). The paper does not implement ODT, but it explicitly argues that a naive ODT in action-heavy, partially observable FPS environments would likely underperform established baselines. This suggests that ODTs are not uniformly advantageous across domains, especially where explicit recurrence and tactical memory dominate.

6. Theory, misconceptions, and open directions

The strongest formal analysis currently available treats ODT through the lens of command-conditioned supervised learning. “On the Convergence and Stability of Upside-Down Reinforcement Learning, Goal-Conditioned Supervised Learning, and Online Decision Transformers” embeds ODT into an $g_t = \sum_{k=t}^K \gamma^{k-t} r_k.$7-regularized episodic UDRL recursion and proves that near-optimal behavior is achieved if the transition kernel is located in a sufficiently small neighborhood of a deterministic kernel (Štrupl et al., 8 Feb 2025). In deterministic environments, the regularized recursion converges to a class of near-optimal stochastic policies; in near-deterministic environments, command-conditioned policies, values, and the goal-reaching objective remain continuous with respect to the transition kernel, with explicit bounds.

A complementary expressivity result comes from “Continuous Latent Contexts Enable Efficient Online Learning in Transformers,” which gives explicit constructions of constant-depth transformers that implement weighted majority and tabular $g_t = \sum_{k=t}^K \gamma^{k-t} r_k.$8-learning by storing algorithmic state in a small number of latent context tokens (Anand et al., 11 May 2026). This is not an ODT paper in the narrow sense, but it shows that transformers can realize canonical online learning procedures with compact persistent state. A plausible implication is that future ODTs need not rely exclusively on return tokens and replay buffers; they may instead use explicit latent contexts to carry online-learned state.

Theoretical and empirical critiques also sharpen the boundary conditions of the paradigm. The offline DT study “When should we prefer Decision Transformers for Offline Reinforcement Learning?” finds that DT is substantially better than both CQL and BC in sparse-reward and low-quality data settings, but that CQL excels in situations characterized by the combination of high stochasticity and low data quality (Bhargava et al., 2023). This suggests that ODT-style methods should be strongest in long-horizon, low-stochasticity regimes and comparatively weaker when stochastic dynamics dominate. The critique in (Yan et al., 2024) identifies RTG far from expected return as a core failure mode for online DT finetuning, while (Luo et al., 1 Jan 2026) identifies hindsight return relabeling as a critical obstacle to pure-RL finetuning.

Several open directions follow directly from the current literature. One is whether online DT training should remain value-free or should systematically integrate critics, as in TD3+ODT or DODT. Another is whether explicit action-memory modules, latent contexts, or morphology-aware prompts can make ODTs more robust in settings with partial observability, embodiment, or long-horizon exploration (Aref et al., 19 Sep 2025, Ji et al., 2024). A third is whether the practical ODT recipe—offline pretraining, replay-buffer online data, and return conditioning—will remain central once pure-RL transformer finetuning matures. The recent trajectory of the field suggests that ODTs are evolving from a single algorithm into a broader class of online sequence-model policies whose common feature is not a single loss, but the use of autoregressive trajectory representations as the core interface between learning and control.