---
title: Offline Reinforcement Learning
url: https://www.emergentmind.com/topics/offline-rl
type: topic
---

# Offline Reinforcement Learning

Offline reinforcement learning (offline RL), also called batch RL, is the problem of learning a decision-making policy exclusively from a fixed dataset of previously collected interactions, without additional environment queries during training. A typical dataset consists of transitions $\mathcal D=\{(s_t,a_t,r_t,s_{t+1})\}$ generated by one or more behavior policies. Given an MDP with state space $\mathcal S$, action space $\mathcal A$, transition kernel $P$, reward function $r$, initial-state distribution, and discount factor $\gamma$, the objective is to obtain a policy maximizing expected discounted return while relying only on information represented in $\mathcal D$. Offline RL is motivated by settings in which interaction is expensive, dangerous, ethically constrained, latency-limited, or impossible, including healthcare, robotics, industrial control, recommendation, advertising, and safety-critical systems [2011.14379]. Unlike online RL, the learner cannot collect corrective transitions after changing its policy. The resulting problem is governed jointly by value estimation, dataset support, distribution shift, uncertainty, representation, and policy-selection issues.

## 1. Problem formulation and data assumptions

In the standard discounted formulation, an MDP is written as
$$
M=(\mathcal S,\mathcal A,P,\mu,r,\gamma),
$$
where $P(s'\mid s,a)$ is the transition kernel, $\mu$ is the initial-state distribution, $r(s,a)$ is the reward, and $\gamma\in(0,1)$ is the discount factor. The action-value and state-value functions are
$$
Q^\pi(s,a)=\mathbb E_{\pi,P}\left[\sum_{k=0}^{\infty}\gamma^k r(s_{t+k},a_{t+k})\mid s_t=s,a_t=a\right],
$$
and
$$
V^\pi(s)=\mathbb E_{\pi,P}\left[\sum_{k=0}^{\infty}\gamma^k r(s_{t+k},a_{t+k})\mid s_t=s\right].
$$

Offline RL receives a static dataset generated by an unknown behavior policy, conventionally denoted $\pi_\beta$. The learner may observe states, actions, rewards, next states, episode boundaries, and sometimes richer features, but it cannot add transitions to the dataset. In many applications, the dataset is heterogeneous: trajectories can originate from expert, mediocre, random, deterministic, stochastic, multimodal, or task-mixture behavior. The data may be unlabeled by task intent, as in undirected locomotion or manipulation logs [2010.13611].

Dataset quality is multidimensional. **State coverage** describes which states are visited, whereas **state-action coverage** describes which actions are observed at those states. High-return trajectories provide evidence for desirable behavior and routes into rewarding regions; diverse trajectories provide alternative actions, transitions, and fragments that can support policy improvement and stitching. A deterministic expert dataset can contain excellent trajectories but little evidence about alternatives, while random data can provide broad coverage but few successful long-horizon behaviors. Experiments with MiniGrid support the hypothesis that medium-quality, sufficiently stochastic data can provide a useful compromise between quality and coverage, although this is not a universal theorem [2011.14379].

The usual offline-RL objective is not necessarily the unconstrained optimum of the true MDP. If relevant states, actions, transitions, or high-return behaviors are absent from $\mathcal D$, the identifiable problem is often the best policy supported by the dataset. OPAL formalizes a related requirement through a representativeness parameter $\zeta$: if the optimal policy is not approximately represented in the behavior-policy mixture, primitive learning cannot reliably recover it [2010.13611]. In linear offline RL, analogous coverage conditions can be expressed in feature space rather than through pointwise state-action density ratios. FOGAS requires information primarily in the feature-occupancy direction associated with the target policy, rather than full covariance coverage of the entire feature space [2405.13755].

Offline RL also includes settings beyond ordinary reward-labeled transitions. Offline inverse reinforcement learning uses exploratory transitions together with a small set of expert demonstrations when the cost function is unknown [2106.05068]. Resource-constrained offline RL allows rich features during training but only limited observations at deployment [2110.03165]. Preference-based offline RL uses offline trajectories without numerical rewards and acquires comparisons over synthetic or logged trajectory pairs [2406.18450]. Offline multitask representation learning uses datasets from multiple source tasks to infer a shared low-dimensional transition representation [2403.11574].

## 2. Distribution shift, extrapolation, and bootstrapping error

The central difficulty is the mismatch between the dataset distribution and the state-action distribution induced by the learned policy:
$$
(s,a)\sim d^{\pi_\beta}\quad\text{during training},\qquad
(s,a)\sim d^\pi\quad\text{during execution}.
$$

A policy may select an action that is rare or absent at a particular state. The critic must then extrapolate its value from neighboring data or from function-approximation structure. In bootstrapping methods, the extrapolated estimate enters subsequent Bellman targets:
$$
Q(s,a)\leftarrow r+\gamma\mathbb E_{a'\sim\pi(\cdot\mid s')}[Q(s',a')].
$$
An erroneously high value at an unsupported action can attract policy optimization, after which the resulting estimate is propagated through further backups. This process is called extrapolation error, bootstrapping error accumulation, or the out-of-distribution action problem [2011.14379].

POPO analyzes this mechanism through an estimation gap between the true MDP and the empirical transition model induced by the dataset:
$$
\delta_{\mathrm{MDP}}(s)=V^\pi(s)-V^\pi_{\mathcal D}(s).
$$
The gap contains an immediate transition-distribution discrepancy and a recursive term propagating previously accumulated value error. The paper’s remedy is pessimistic value estimation combined with distribution-constrained action generation [2012.13682].

The Least Restriction framework identifies target-action selection inside Bellman backups as a critical intervention point. It proposes rejecting actions judged insufficiently supported by a discriminator trained on dataset state-action pairs, then perturbing proposed actions until they pass a support threshold. The intended condition is local support preservation rather than global behavior matching:
$$
\beta(a\mid s)\leq\epsilon\quad\Longrightarrow\quad \pi(a\mid s)=0.
$$
The discriminator test is only a proxy: the paper does not establish that discriminator acceptance provides a mathematically valid support guarantee [2107.01757].

Distribution shift can also concern states rather than only actions. Forward model-based imagination may begin from a dataset state and then enter unsupported regions where model error and value overestimation compound. ROMI addresses this by learning reverse dynamics and initiating rollouts from target states sampled directly from the dataset. Its structural conservatism comes from anchoring the endpoint of every reverse rollout at a real dataset state, not from an assumption that reverse models are intrinsically more accurate than forward models [2110.00188].

A further complication is that a single global closeness constraint can be inappropriate. In heteroskedastic datasets, the variability of $\pi_\beta(\cdot\mid s)$ changes across states. Narrow states may require strong support preservation, whereas broad states may contain multiple supported actions and benefit from aggressive reweighting toward a good mode. CQL (ReDS) therefore modifies CQL with a state-dependent reweighted distribution intended to approximate a support constraint rather than uniform behavior matching [2211.01052].

## 3. Conservative, behavior-constrained, and value-based methods

Offline RL methods differ primarily in where they impose conservatism: the policy, the value function, a learned model, an action generator, or an estimated occupancy distribution.

Behavior cloning (BC) treats offline RL as supervised action prediction. For discrete actions it uses cross-entropy, and for continuous actions the paper describes a mean-squared-error loss. BC avoids deliberate action extrapolation and is simple and stable, particularly when the dataset contains coherent near-optimal behavior. It generally cannot improve beyond the demonstrations, treats all logged actions as targets, and may suffer from compounding errors when deployment states differ from training states. Nevertheless, empirical studies show that BC can be a strong baseline, especially on medium- and high-quality continuous-control data [2011.14379].

Explicit policy-constraint methods attempt to keep $\pi$ close to $\pi_\beta$ under a divergence. Their limitations include the fact that $\pi_\beta$ is usually unknown, excessive conservatism can prevent improvement, and multimodal datasets make “the behavior policy” ambiguous. BCQ uses a generative action model to constrain policy actions, whereas BEAR uses distribution matching, including maximum mean discrepancy. The Least Restriction framework argues that exact behavior matching is stronger than necessary when broad support permits reward-directed improvement [2012.13682; 2107.01757].

CQL regularizes the critic to suppress high values for actions outside the dataset distribution. A representative discrete-action regularizer is
$$
\alpha\left(\mathbb E_{s}\left[\log\sum_a\exp Q_\theta(s,a)\right]
-\mathbb E_{(s,a)\sim\mathcal D}[Q_\theta(s,a)]\right).
$$
Its aim is to produce conservative value estimates and prevent policies from exploiting optimistic extrapolation. CQL can perform strongly on some discrete and medium-quality datasets, but its coefficient is dataset-dependent; excessive pessimism can suppress a minority optimal mode, whereas insufficient pessimism reintroduces distribution shift [2011.14379].

CRR can be viewed as value-filtered behavioral cloning. It weights logged actions according to estimated advantage, using either binary or exponential filters. This makes it suitable when datasets mix good and bad behavior and successful trajectories contain complementary fragments. In sparse-reward MiniGrid, CRR can stitch useful pieces from different trajectories and exceed the maximum complete-trajectory return observed in the dataset. Its limitations include dependence on critic accuracy and difficulty synthesizing improvements when the action distribution is narrow [2011.14379].

POPO combines a conditional VAE with a pessimistic distributional critic. The VAE generates behavior-like central actions, while a residual actor produces bounded perturbations. Multiple candidates are ranked using a lower-tail risk distortion of a quantile critic. Its main implementation uses Wang$(-0.75)$, 32 quantiles, ten candidate actions, and a residual coefficient $\xi=0.05$. The VAE addresses action distribution shift, while pessimistic valuation discourages exploitation of optimistic errors. The paper reports that the VAE supplies a particularly large contribution, with pessimistic valuation providing an additional robustness mechanism [2012.13682].

CQL (ReDS) modifies CQL by combining the learned policy $\pi$ with an auxiliary distribution $\rho$:
$$
\pi^{\mathrm{re}}(\cdot\mid s)
=\frac12\pi(\cdot\mid s)+\frac12\rho(\cdot\mid s).
$$
The auxiliary policy is trained with negative-advantage weighting so that it emphasizes poor actions that remain likely under the behavior policy. This permits CQL to penalize unsupported actions while allowing concentration on good actions within the behavior support. The method is designed for state-dependent action variability and is not a universal replacement for ordinary CQL [2211.01052].

## 4. Temporal abstraction, model-based augmentation, and structured behavior

Offline RL can be improved not only by changing the objective but also by changing the representation of behavior and transitions.

OPAL learns a continuous latent space of temporally extended primitives from unlabeled trajectory segments. Its encoder produces $q_\phi(z\mid\tau)$, its decoder is a latent-conditioned low-level policy $\pi_\theta(a\mid s,z)$, and its prior predicts plausible primitives from the initial state, $\rho_\omega(z\mid s_0)$. A latent $z$ is held fixed for $c$ environment steps. Reward-labeled segments are converted into high-level transitions
$$
(s_0,z,R^{(c)},s_c),
\qquad
R^{(c)}=\sum_{t=0}^{c-1}\gamma^t r_t,
$$
after which an offline RL algorithm such as CQL is applied in the latent action space. OPAL addresses distribution shift at the level of multi-step behavior and reduces the effective horizon from approximately $T$ decisions to $T/c$. The main experiments use $c=10$ and $\dim(\mathcal Z)=8$; the $c=1$ ablation returns close to standard CQL performance, supporting temporal abstraction as a major source of the reported gains [2010.13611].

ROMI provides a different form of model-based augmentation. It learns a probabilistic reverse dynamics model $p(s,r\mid s',a)$ and a reverse rollout policy conditioned on successor states. Starting from dataset target states, it generates plausible predecessor transitions and stores them in a model buffer combined with the original dataset. Reverse imagination is intended to interpolate between trajectories while avoiding the forward-model failure mode of terminating in unsupported, overvalued states. The method can be combined with BCQ or CQL. Its strongest gains occur in navigation domains with obstacles, narrow passages, and useful target states already represented in the dataset [2110.00188].

HUBL modifies Bellman targets by blending Monte Carlo trajectory returns with bootstrapped values. For a next-state heuristic $h'$ and blending coefficient $\lambda'$, it relabels each transition as
$$
\widetilde r=r+\gamma\lambda' h',
\qquad
\widetilde\gamma=\gamma(1-\lambda').
$$
High-return trajectories receive greater heuristic weight, while low-return trajectories retain more bootstrapping. HUBL reduces the effective continuation horizon but introduces bias when the heuristic differs from the optimal value. It is implemented by reward and discount relabeling and can be combined with ATAC, CQL, TD3+BC, and IQL. Across 27 D4RL and Meta-World datasets, the paper reports an average improvement of approximately 9% [2306.00321].

Sequence-modeling methods treat offline RL as conditional generation. Decision Transformer, Trajectory Transformer, Decision Diffuser, and related methods model trajectories or return-conditioned actions. Trifle argues that likelihood modeling alone is insufficient: stochastic behavior and dynamics make return-to-go a random variable, and evaluation requires conditional and marginal probabilistic queries. It uses a tractable probabilistic circuit together with an expressive autoregressive action prior to compute high-return action distributions, perform marginalization, incorporate constraints, and guide beam search. Trifle reports scores superior to Trajectory Transformer on all nine evaluated Gym-MuJoCo tasks, and it performs better in stochastic Taxi and action-constrained MuJoCo settings [2311.00094].

## 5. Representation, partial observability, and uncertainty adaptation

Representation determines whether offline RL can transfer information across trajectories. In a fully observed MDP, value iteration can stitch prefixes and suffixes that meet at the same state. Under partial observability, conditioning on raw observation histories may prevent such sharing: two histories can correspond to the same action-relevant situation while differing in irrelevant observations. The effective history space can grow exponentially with horizon, producing poor sample complexity [2310.20663].

The proposed remedy is a compact history representation approximating a bisimulation metric. For histories $\tau$ and $\tau'$, an on-policy metric has the form
$$
d^\pi(\tau,\tau')
=
|r^\pi(\tau)-r^\pi(\tau')|
+
W_1\!\left(P^\pi(\cdot\mid\tau),P^\pi(\cdot\mid\tau')\right).
$$
An encoder is trained so that embedding distances approximate reward and next-history distribution differences. If histories can be aggregated into a compact representation space $\mathcal Z$ with small bisimulation error, the raw-history sample-complexity dependence on $|\mathcal T|$ can be replaced by dependence on $|\mathcal Z|$, plus representation error [2310.20663].

APE-V addresses a different representation problem: uncertainty about which MDP generated the evaluation environment. Its Bayesian formulation treats offline RL as an implicit POMDP whose hidden state includes the environment identity. The optimal policy may depend on the current physical state and an evolving belief over environmental hypotheses:
$$
\pi(a_t\mid h_t)=\pi(a_t\mid s_t,b_t).
$$
APE-V approximates this structure using an ensemble of value functions, a belief vector, a Bellman-error-based surrogate likelihood, and a belief-conditioned actor. Unlike a static lower-confidence-bound policy, it can revise its hypothesis weights after observing evaluation transitions. In Locked Doors and Procgen Mazes, adaptation improves recovery from incorrect initial hypotheses; on D4RL, it is competitive with strong conservative methods and is most useful where multiple salient hypotheses remain [2207.02200].

Resource-constrained offline RL concerns a train–deployment observation mismatch. The offline dataset may contain rich features generated by expensive language, vision, sensor, or embedding systems, while the deployed policy can use only limited observations. A teacher trained with the rich state produces action targets for a student trained on deployment-available features. The continuous-control student objective combines critic improvement, behavior cloning toward the logged action, and transfer toward the teacher action:
$$
\lambda Q(\bar s,\pi_\phi(\bar s))
-\beta_1\|\pi_\phi(\bar s)-a\|_2^2
-\beta_2\|\pi_{\phi^T}(s)-\pi_\phi(\bar s)\|_2^2,
$$
with $\beta_1+\beta_2=1$. The method improves substantially over limited-feature TD3+BC on many medium and medium-replay datasets, but does not remove the irreducible teacher–student gap when omitted features are action-relevant [2110.03165].

Offline multitask representation learning exploits a different source of structure. In low-rank MDPs, multiple tasks share $\phi_h^*(s,a)$ while having task-specific transition measures $\mu_h^{(*,t)}$. MORL jointly estimates the shared representation and task-specific transition components from fixed source-task datasets. Its analysis gives average transition error of order $\widetilde O(1/\sqrt{nT})$ and supports downstream reward-free, offline, and online RL under partial feature coverage rather than full state-action concentrability [2403.11574].

## 6. Model learning, inverse rewards, preferences, and deployment

Model-based offline RL estimates dynamics or reward models from the dataset, then uses them for planning or data augmentation. Its principal risk is model error outside the data support. MOPO penalizes model uncertainty, while ROMI constrains imagination structurally through reverse rollouts. CAMERON extends this framework to offline inverse reinforcement learning, where the reward is unknown and only a small set of expert demonstrations is available.

CAMERON learns a transition model, estimates policy-dependent occupancy distributions using the Off-policy Idle GAN-based procedure, generates short model rollouts, and optimizes a discriminator-based cost with COMBO. Its purpose is to approximate the occupancy distributions required by adversarial IRL without current-policy trajectories from the real environment. The method outperforms ORIL and TGR and is competitive with COMBO on Ant, Half-Cheetah, and Hopper, although its density-ratio estimation, GAN optimization, and model rollouts introduce substantial computational and statistical demands [2106.05068].

Sim-OPRL addresses unknown rewards through offline preference elicitation. It learns a transition model and confidence set, constructs policies that are plausible optima under reward models consistent with existing preferences, and queries comparisons between pessimistically simulated rollouts selected for

Source: https://www.emergentmind.com/topics/offline-rl