Reinforcement Learning Transformers

Updated 8 July 2026

Reinforcement Learning Transformers are methods that integrate transformer architectures to encode sequential data and enhance policy inference in RL.
They address challenges like partial observability and long-horizon dependencies through adaptive attention, gating, and memory mechanisms.
Empirical studies reveal improved online, offline, and meta-learning performance, despite challenges in computational cost and optimization stability.

Reinforcement learning transformers are reinforcement-learning methods in which transformer architectures are used either as stronger sequence encoders inside conventional RL algorithms or as the primary mechanism for modeling trajectories, histories, task context, rewards, or latent dynamics. Surveys of the area organize the field as representation learning, model learning, sequential decision-making, and generalist agents, or, equivalently, as representation learning, transition-function learning, reward-function learning, and policy learning; a closely related distinction separates architecture enhancement under standard RL objectives from sequence modeling of trajectories (Li et al., 2023, Agarwal et al., 2023, Luong et al., 26 May 2026).

1. Scope, taxonomies, and core mechanisms

Transformer-based RL is motivated by recurrent difficulties in RL: partial observability, long-horizon dependencies, delayed rewards, unstable training, poor credit assignment, sample inefficiency, and the need to process structured or heterogeneous inputs. The common computational primitive is scaled dot-product attention,

$\mathrm{Attention}(Q,K,V)=\operatorname{Softmax}\!\left(\frac{QK^\top}{\sqrt{d_q}}\right)V,$

which lets a model weight past states, actions, rewards, or entities according to relevance rather than compressing all history into a single recurrent state (Agarwal et al., 2023).

The surveys emphasize that transformers in RL are rarely used unchanged. They are adapted with causal masking, recurrence or memory, gating, sparse or local attention, relative positional schemes, latent world models, or task-conditioning prompts. In online RL, the transformer often acts as a history encoder for a policy or value function; in offline RL, it often becomes an autoregressive model over trajectories; in meta-RL and in-context RL, it is prompted with prior episodes or preference comparisons and acts without parameter updates at test time (Li et al., 2023, Luong et al., 26 May 2026).

Family	Transformer role	Representative examples
Representation and memory	History encoder for policy/value learning	GTrXL, Adaptive Transformer, Swin DQN
Trajectory sequence modeling	Autoregressive model over returns, states, and actions	Decision Transformer, Trajectory Transformer, STRL
In-context and meta-RL	Task-conditioned policy from episodes, preferences, or beliefs	HTrMRL, ICPRL, HT
World models	Latent dynamics model for imagination and planning	TransDreamer

A plausible implication is that “reinforcement learning transformers” denotes not one algorithmic family but a design space in which self-attention is inserted at different points of the RL pipeline: state encoding, temporal memory, reward inference, latent dynamics, or direct action generation.

2. Stabilized architectures for online and partially observable RL

Early work established that simply inserting a standard transformer into RL is often ineffective. “Stabilizing Transformers for Reinforcement Learning” showed that the canonical Transformer and Transformer-XL are difficult to optimize in RL and can perform near random, and proposed Gated Transformer-XL (GTrXL), which combines relative attention with memory, identity map reordering, gating, and identity-favoring gate initialization. On DMLab-30, average human-normalized performance was $117.6 \pm 0.3$ for GTrXL (GRU), compared with $99.3 \pm 1.0$ for the LSTM baseline and $5.0 \pm 0.2$ for canonical TrXL; the paper also reported stronger scaling than LSTMs on Numpad and human-level performance within 2B steps on Memory Maze in sensitivity experiments (Parisotto et al., 2019).

“Adaptive Transformers in RL” extended the stabilized transformer line by adding adaptive attention span to the Stable Transformer. On DMLab30 rooms_select_nonmatching_object, the reported final 100-episode average return increased from $7.17$ for the Stable Transformer to $8.84$ for the Adaptive Transformer, while parameter count decreased from $3.5$M to $3.3$M. The learned maximum attention spans at the best span penalty $0.025$ were 33, 2, and 2 for layers 1–3, supporting the paper’s claim that some layers can shrink to very short effective spans while preserving or improving performance (Kumar et al., 2020).

Not all partially observable control problems favor attention over recurrence. “Transformer Based Reinforcement Learning For Games” replaced the recurrent module in deep recurrent Q-learning with a transformer encoder on partially observable CartPole. The study trained DQN, DRQN, and DTQN for 5000 episodes over 10 independent runs and reported that DRQN performed best overall, DTQN was worst on average, and the maximum score reached by DRQN was 135 in one test case. This result directly contradicts any blanket claim that transformer memory is uniformly superior to GRU- or LSTM-based memory in small POMDPs (Upadhyay et al., 2019).

A distinct systems-oriented line addresses transformer latency rather than transformer accuracy alone. “Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation” introduced Actor-Learner Distillation (ALD), with a large-capacity transformer learner $\mathcal{M}_L$ and a small LSTM actor $117.6 \pm 0.3$ 0. In I-Maze, the LSTM actor processed 1053 steps/s whereas GTrXL achieved 70 steps/s, and learner idle time rose from 34% with LSTM actors to 84% with transformer actors. ALD recovered much of the transformer learner’s sample-efficiency advantage while maintaining the fast inference and wall-clock properties of the LSTM actor, indicating that transformer-based RL can be limited as much by distributed-systems constraints as by optimization alone (Parisotto et al., 2021).

For large-scale visual online RL, “Deep Reinforcement Learning with Swin Transformers” replaced the CNN backbone of Double DQN with a Swin Transformer hierarchy. On 49 Atari games in the Arcade Learning Environment, Swin DQN achieved significantly higher maximal evaluation scores than the baseline in 45 games and higher mean evaluation scores in 40 games, while remaining about 3 to 4 times slower than DQN in wall-clock time. This supports the narrower claim that localized and shifted self-attention can be computationally practical enough for large-scale online Atari training, even if it does not remove the cost gap relative to CNNs (Meng et al., 2022).

3. Trajectory sequence modeling and offline-to-online learning

A major shift in the field came from recasting RL as conditional sequence modeling. The surveys identify Decision Transformer as the canonical offline RL example and Trajectory Transformer as a model-based autoregressive variant. In this formulation, a trajectory is tokenized as

$117.6 \pm 0.3$ 1

and the transformer predicts actions conditioned on desired return-to-go and past context rather than performing explicit Bellman backups during training (Li et al., 2023).

“Energy-Efficient Deep Reinforcement Learning with Spiking Transformers” reformulated this sequence-modeling idea in spiking form. STRL is explicitly described as a spiking-neural-network rewrite of the Decision Transformer idea: each timestep is represented by $117.6 \pm 0.3$ 2, transformer blocks are replaced with multi-step Leaky Integrate-and-Fire dynamics, and training remains supervised action prediction on offline trajectories. On a synthetic procedural $117.6 \pm 0.3$ 3 maze dataset, the paper reported Decision Transformer test loss $117.6 \pm 0.3$ 4 and test accuracy $117.6 \pm 0.3$ 5, versus STRL test loss $117.6 \pm 0.3$ 6 and test accuracy $117.6 \pm 0.3$ 7; on D4RL maze2d-umaze-v1, validation accuracy approached $117.6 \pm 0.3$ 8 by epoch 20 (Uddin et al., 20 May 2025).

Offline-to-online adaptation exposes limitations of pure return conditioning. “Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers” analyzed Online Decision Transformer (ODT) and argued that when pretrained with low-reward offline data, a target return far above dataset support makes online finetuning difficult. The proposed remedy was to add TD3 gradients to the DT finetuning process, yielding TD3+ODT. Across Adroit, Antmaze, and MuJoCo, the method improved ODT substantially, especially on lower-quality offline data; in the MuJoCo table, average final reward was $117.6 \pm 0.3$ 9 for TD3+ODT and $99.3 \pm 1.0$ 0 for TD3, while ODT and PDT often failed to improve on random datasets (Yan et al., 2024).

A separate line shows that trajectory transformers can also be trained without policy gradients. “Utilizing Evolution Strategies to Train Transformers in Reinforcement Learning” used OpenAI-ES to train Decision Transformers in MuJoCo Humanoid locomotion and Atari Hero. The Decision Transformer had 825,098 parameters for Humanoid and 2,486,272 parameters for Atari, and the paper concluded that OpenAI-ES can train Decision Transformers successfully and that pretraining was not necessary, although it yielded useful observations about robustness and exploration (Lorenc, 23 Jan 2025).

4. Meta-reinforcement learning and in-context adaptation

Transformer-based meta-RL treats a transformer as a task-conditioned learning algorithm rather than a fixed policy. “Hierarchical Transformers are Efficient Meta-Reinforcement Learners” defined the goal as learning $99.3 \pm 1.0$ 1 so that $99.3 \pm 1.0$ 2 adapts rapidly to unseen tasks sampled from $99.3 \pm 1.0$ 3. HTrMRL stores the last $99.3 \pm 1.0$ 4 episodes, processes intra-episode transition sequences with $99.3 \pm 1.0$ 5, aggregates across episodes with $99.3 \pm 1.0$ 6, and conditions the policy on the resulting task representation. The architecture reduces attention complexity from $99.3 \pm 1.0$ 7 for a flat transformer to $99.3 \pm 1.0$ 8. On Meta-World, the paper reported that HTrMRL was the only method to exceed an average training success rate of $99.3 \pm 1.0$ 9 on ML10 and $5.0 \pm 0.2$ 0 on ML45, with roughly $5.0 \pm 0.2$ 1 higher final success than the baselines and out-of-distribution test performance of about $5.0 \pm 0.2$ 2 on ML10 and $5.0 \pm 0.2$ 3 on ML45 (Shala et al., 2024).

Reward-free in-context RL extends this paradigm by removing scalar rewards from both pretraining and deployment. “Learning in Context, Guided by Choice” introduced In-Context Preference-based Reinforcement Learning (ICPRL), in which both pretraining and deployment use only preference feedback. The paper studied Immediate Preference-based RL (I-PRL) with per-step preferences and Trajectory Preference-based RL (T-PRL) with trajectory-level comparisons, and proposed preference-native methods ICPO and ICRG that do not require reward supervision or optimal action labels. Experiments on dueling bandits, DarkRoom, navigation, and Meta-World Reach-v2 showed that ICPRL can achieve performance comparable to in-context RL methods trained with full reward supervision (Dong et al., 9 Feb 2026).

“Heuristic Transformer: Belief Augmented In-Context Reinforcement Learning” added a different source of task information: a VAE-derived belief over rewards. HT learns a low-dimensional latent $5.0 \pm 0.2$ 4 representing a posterior distribution over rewards, appends this belief to the in-context dataset and query state, and trains a GPT-style transformer policy in two stages. The paper reported that HT learns faster online than DPT and GFT in Darkroom, remains robust under 20% and 40% stochastic action noise, outperforms DPT in Miniworld, and that HT-SP is strongest among HT variants in MuJoCo (Dippel et al., 13 Nov 2025).

Mechanistic and theoretical work has begun to characterize what these in-context transformers compute. “From Memories to Maps” trained a GPT-2–style causal transformer on planning problems inspired by rodent behavior and argued that in-context RL is supported not by standard model-free or model-based planning, but by caching intermediate computations inside memory tokens and retrieving them at decision time; in gridworld, the model could infer shortcut paths through states it never saw in context (Fang et al., 24 Jun 2025). “Optimal Dynamic Regret by Transformers for Non-Stationary Reinforcement Learning” proved that transformers can achieve nearly optimal dynamic regret in non-stationary environments and, under sufficient data and approximation quality, match the rate

$5.0 \pm 0.2$ 5

while experiments in linear bandits showed performance matching or exceeding standard expert algorithms in low and moderate non-stationarity regimes (Chen et al., 22 Aug 2025).

5. World models and application domains

Model-based RL provides a different insertion point for transformers. “TransDreamer: Reinforcement Learning with Transformer World Models” replaced Dreamer’s recurrent state-space model with a Transformer State-Space Model (TSSM), in which the deterministic state is computed by a transformer over past latent states and actions, while the posterior is simplified to $5.0 \pm 0.2$ 6 for parallel training. On Hidden Order Discovery, TransDreamer outperformed Dreamer on long-memory tasks: in 2D 4-ball settings, episode reward reached around 7 for TransDreamer versus around 4 for Dreamer, with success rates 23% and 7%; in 3D settings, appendix success rates included 18% vs 10% on 4-ball dense and 11% vs 1% on 4-ball sparse for TransDreamer versus DreamerV2 (Chen et al., 2022).

Application-specific transformer-RL systems frequently use the transformer as a state encoder within a standard optimizer. “Enhancing IoT Intelligence: A Transformer-based Reinforcement Learning Methodology” combined a transformer encoder with PPO for smart-city IoT simulation. Continuous sensor variables were normalized by

$5.0 \pm 0.2$ 7

categorical variables were one-hot encoded, and the transformer generated the state representation $5.0 \pm 0.2$ 8 for PPO. The implemented configuration used embedding size 512, 8 attention heads, and 6 encoder layers, with a two-hidden-layer PPO network of 256 neurons per layer and training for 1000 episodes. The study reported higher total reward, faster task completion, lower response times, and more moderate latency growth than “Traditional RL Methods” and a “Baseline Transformer Model,” but also noted the absence of real-world deployment, public benchmark datasets, detailed statistical tables, or ablations (Rjoub et al., 2024).

The broadest domain survey in the provided literature is the communication-networks review. “Transformer-Enhanced Reinforcement Learning: Fundamentals and Applications in Communication Networks” frames the area as architecture enhancement and sequence modeling, and surveys applications in resource allocation, computation offloading, routing and trajectory control, network security, and semantic communication. It argues that transformer-enhanced RL is particularly suitable when the decision problem is sequential, long-horizon, partially observable, high-dimensional, multi-agent, or multi-modal, and highlights both transformer-augmented DQN/A2C/PPO pipelines and Decision Transformer-style sequence models (Luong et al., 26 May 2026). A wider survey similarly catalogues applications in robotics, medicine, language modeling, cloud computing, and combinatorial optimization, underscoring that transformer-based RL now spans both core algorithmic research and domain-specific deployment (Agarwal et al., 2023).

6. Limitations, controversies, and open directions

The literature is consistent that transformer-based RL is not simply a drop-in superiority claim over CNNs, GRUs, or LSTMs. The major recurring limitations are optimization instability in RL objectives, quadratic or otherwise high attention cost, actor-latency constraints, finite context windows, heavy data requirements, weak inductive bias in some low-data settings, and failure modes in stochastic or poorly covered offline datasets (Agarwal et al., 2023, Li et al., 2023, Luong et al., 26 May 2026).

Several empirical results sharpen these limitations. DTQN underperformed DRQN on partially observable CartPole, showing that attention alone did not outperform recurrence in a small low-dimensional temporal-control problem (Upadhyay et al., 2019). TransDreamer eventually matched Dreamer on DMC and Atari but tended to learn more slowly on short-memory tasks, supporting the more specific claim that transformers are especially useful when long-range dependency matters (Chen et al., 2022). ALD showed that even when transformers improve sample efficiency, rollout latency on CPU actors can negate those gains unless acting and learning are decoupled (Parisotto et al., 2021). For online finetuning of Decision Transformers, high desired return-to-go can become out-of-distribution when offline pretraining data are low-reward, and critic gradients may be needed to restore a local direction of improvement (Yan et al., 2024).

Open directions in the surveyed literature are correspondingly concrete. One line pursues more efficient attention and memory mechanisms, including adaptive span, compressive or sparse attention, and architectural designs that preserve long-horizon reasoning without prohibitive runtime (Kumar et al., 2020, Agarwal et al., 2023). Another line broadens the supervision signal from scalar rewards to preferences or learned beliefs, as in ICPRL and HT, suggesting that transformer-based RL need not be tied to explicit reward engineering (Dong et al., 9 Feb 2026, Dippel et al., 13 Nov 2025). A third line seeks stronger theory and mechanistic understanding, ranging from memory-token analyses of in-context RL to dynamic-regret guarantees in non-stationary environments (Fang et al., 24 Jun 2025, Chen et al., 22 Aug 2025). Taken together, these works suggest that the central research problem is no longer whether transformers can be used in RL, but under which training regimes, memory structures, and deployment constraints they are the appropriate computational substrate.