Offline Reinforcement Learning

• Offline Reinforcement Learning (Offline RL) is the practice of learning to make decisions from a fixed dataset of previous interactions, essential in environments where interaction is costly or impossible.
• Offline RL methods include conservative, behavior-constrained, and value-based approaches, each with unique strategies to address distribution shift and extrapolation errors, tailored for different dataset qualities.
• Offline RL extends to multiple reinforcement learning subfields such as model-based augmentation, temporal abstraction, and transition representation under uncertainty.

Offline reinforcement learning (offline RL), also called batch RL, is the problem of learning a decision-making policy exclusively from a fixed dataset of previously collected interactions, without additional environment queries during training. A typical dataset consists of transitions $\mathcal D=\{(s_t,a_t,r_t,s_{t+1})\}$ generated by one or more behavior policies. Given an MDP with state space $\mathcal S$, action space $\mathcal A$, transition kernel $P$, reward function $r$, initial-state distribution, and discount factor $\gamma$, the objective is to obtain a policy maximizing expected discounted return while relying only on information represented in $\mathcal D$. Offline RL is motivated by settings in which interaction is expensive, dangerous, ethically constrained, latency-limited, or impossible, including healthcare, robotics, industrial control, recommendation, advertising, and safety-critical systems (Monier et al., 2020). Unlike online RL, the learner cannot collect corrective transitions after changing its policy. The resulting problem is governed jointly by value estimation, dataset support, distribution shift, uncertainty, representation, and policy-selection issues.

1. Problem formulation and data assumptions

In the standard discounted formulation, an MDP is written as

$M=(\mathcal S,\mathcal A,P,\mu,r,\gamma),$

where $P(s'\mid s,a)$ is the transition kernel, $\mu$ is the initial-state distribution, $\mathcal S$0 is the reward, and $\mathcal S$1 is the discount factor. The action-value and state-value functions are

$\mathcal S$2

and

$\mathcal S$3

Offline RL receives a static dataset generated by an unknown behavior policy, conventionally denoted $\mathcal S$4. The learner may observe states, actions, rewards, next states, episode boundaries, and sometimes richer features, but it cannot add transitions to the dataset. In many applications, the dataset is heterogeneous: trajectories can originate from expert, mediocre, random, deterministic, stochastic, multimodal, or task-mixture behavior. The data may be unlabeled by task intent, as in undirected locomotion or manipulation logs (Ajay et al., 2020).

Dataset quality is multidimensional. State coverage describes which states are visited, whereas state-action coverage describes which actions are observed at those states. High-return trajectories provide evidence for desirable behavior and routes into rewarding regions; diverse trajectories provide alternative actions, transitions, and fragments that can support policy improvement and stitching. A deterministic expert dataset can contain excellent trajectories but little evidence about alternatives, while random data can provide broad coverage but few successful long-horizon behaviors. Experiments with MiniGrid support the hypothesis that medium-quality, sufficiently stochastic data can provide a useful compromise between quality and coverage, although this is not a universal theorem (Monier et al., 2020).

The usual offline-RL objective is not necessarily the unconstrained optimum of the true MDP. If relevant states, actions, transitions, or high-return behaviors are absent from $\mathcal S$5, the identifiable problem is often the best policy supported by the dataset. OPAL formalizes a related requirement through a representativeness parameter $\mathcal S$6: if the optimal policy is not approximately represented in the behavior-policy mixture, primitive learning cannot reliably recover it (Ajay et al., 2020). In linear offline RL, analogous coverage conditions can be expressed in feature space rather than through pointwise state-action density ratios. FOGAS requires information primarily in the feature-occupancy direction associated with the target policy, rather than full covariance coverage of the entire feature space (Neu et al., 2024).

Offline RL also includes settings beyond ordinary reward-labeled transitions. Offline inverse reinforcement learning uses exploratory transitions together with a small set of expert demonstrations when the cost function is unknown (Jarboui et al., 2021). Resource-constrained offline RL allows rich features during training but only limited observations at deployment (Regatti et al., 2021). Preference-based offline RL uses offline trajectories without numerical rewards and acquires comparisons over synthetic or logged trajectory pairs (Pace et al., 2024). Offline multitask representation learning uses datasets from multiple source tasks to infer a shared low-dimensional transition representation (Ishfaq et al., 2024).

2. Distribution shift, extrapolation, and bootstrapping error

The central difficulty is the mismatch between the dataset distribution and the state-action distribution induced by the learned policy:

$\mathcal S$7

A policy may select an action that is rare or absent at a particular state. The critic must then extrapolate its value from neighboring data or from function-approximation structure. In bootstrapping methods, the extrapolated estimate enters subsequent Bellman targets:

$\mathcal S$8

An erroneously high value at an unsupported action can attract policy optimization, after which the resulting estimate is propagated through further backups. This process is called extrapolation error, bootstrapping error accumulation, or the out-of-distribution action problem (Monier et al., 2020).

POPO analyzes this mechanism through an estimation gap between the true MDP and the empirical transition model induced by the dataset:

$\mathcal S$9

The gap contains an immediate transition-distribution discrepancy and a recursive term propagating previously accumulated value error. The paper’s remedy is pessimistic value estimation combined with distribution-constrained action generation (He et al., 2020).

The Least Restriction framework identifies target-action selection inside Bellman backups as a critical intervention point. It proposes rejecting actions judged insufficiently supported by a discriminator trained on dataset state-action pairs, then perturbing proposed actions until they pass a support threshold. The intended condition is local support preservation rather than global behavior matching:

$\mathcal A$0

The discriminator test is only a proxy: the paper does not establish that discriminator acceptance provides a mathematically valid support guarantee (Su, 2021).

Distribution shift can also concern states rather than only actions. Forward model-based imagination may begin from a dataset state and then enter unsupported regions where model error and value overestimation compound. ROMI addresses this by learning reverse dynamics and initiating rollouts from target states sampled directly from the dataset. Its structural conservatism comes from anchoring the endpoint of every reverse rollout at a real dataset state, not from an assumption that reverse models are intrinsically more accurate than forward models (Wang et al., 2021).

A further complication is that a single global closeness constraint can be inappropriate. In heteroskedastic datasets, the variability of $\mathcal A$1 changes across states. Narrow states may require strong support preservation, whereas broad states may contain multiple supported actions and benefit from aggressive reweighting toward a good mode. CQL (ReDS) therefore modifies CQL with a state-dependent reweighted distribution intended to approximate a support constraint rather than uniform behavior matching (Singh et al., 2022).

3. Conservative, behavior-constrained, and value-based methods

Offline RL methods differ primarily in where they impose conservatism: the policy, the value function, a learned model, an action generator, or an estimated occupancy distribution.

Behavior cloning (BC) treats offline RL as supervised action prediction. For discrete actions it uses cross-entropy, and for continuous actions the paper describes a mean-squared-error loss. BC avoids deliberate action extrapolation and is simple and stable, particularly when the dataset contains coherent near-optimal behavior. It generally cannot improve beyond the demonstrations, treats all logged actions as targets, and may suffer from compounding errors when deployment states differ from training states. Nevertheless, empirical studies show that BC can be a strong baseline, especially on medium- and high-quality continuous-control data (Monier et al., 2020).

Explicit policy-constraint methods attempt to keep $\mathcal A$2 close to $\mathcal A$3 under a divergence. Their limitations include the fact that $\mathcal A$4 is usually unknown, excessive conservatism can prevent improvement, and multimodal datasets make “the behavior policy” ambiguous. BCQ uses a generative action model to constrain policy actions, whereas BEAR uses distribution matching, including maximum mean discrepancy. The Least Restriction framework argues that exact behavior matching is stronger than necessary when broad support permits reward-directed improvement (He et al., 2020, Su, 2021).

CQL regularizes the critic to suppress high values for actions outside the dataset distribution. A representative discrete-action regularizer is

$\mathcal A$5

Its aim is to produce conservative value estimates and prevent policies from exploiting optimistic extrapolation. CQL can perform strongly on some discrete and medium-quality datasets, but its coefficient is dataset-dependent; excessive pessimism can suppress a minority optimal mode, whereas insufficient pessimism reintroduces distribution shift (Monier et al., 2020).

CRR can be viewed as value-filtered behavioral cloning. It weights logged actions according to estimated advantage, using either binary or exponential filters. This makes it suitable when datasets mix good and bad behavior and successful trajectories contain complementary fragments. In sparse-reward MiniGrid, CRR can stitch useful pieces from different trajectories and exceed the maximum complete-trajectory return observed in the dataset. Its limitations include dependence on critic accuracy and difficulty synthesizing improvements when the action distribution is narrow (Monier et al., 2020).

POPO combines a conditional VAE with a pessimistic distributional critic. The VAE generates behavior-like central actions, while a residual actor produces bounded perturbations. Multiple candidates are ranked using a lower-tail risk distortion of a quantile critic. Its main implementation uses Wang$\mathcal A$6, 32 quantiles, ten candidate actions, and a residual coefficient $\mathcal A$7. The VAE addresses action distribution shift, while pessimistic valuation discourages exploitation of optimistic errors. The paper reports that the VAE supplies a particularly large contribution, with pessimistic valuation providing an additional robustness mechanism (He et al., 2020).

CQL (ReDS) modifies CQL by combining the learned policy $\mathcal A$8 with an auxiliary distribution $\mathcal A$9:

$P$0

The auxiliary policy is trained with negative-advantage weighting so that it emphasizes poor actions that remain likely under the behavior policy. This permits CQL to penalize unsupported actions while allowing concentration on good actions within the behavior support. The method is designed for state-dependent action variability and is not a universal replacement for ordinary CQL (Singh et al., 2022).

4. Temporal abstraction, model-based augmentation, and structured behavior

Offline RL can be improved not only by changing the objective but also by changing the representation of behavior and transitions.

OPAL learns a continuous latent space of temporally extended primitives from unlabeled trajectory segments. Its encoder produces $P$1, its decoder is a latent-conditioned low-level policy $P$2, and its prior predicts plausible primitives from the initial state, $P$3. A latent $P$4 is held fixed for $P$5 environment steps. Reward-labeled segments are converted into high-level transitions

$P$6

after which an offline RL algorithm such as CQL is applied in the latent action space. OPAL addresses distribution shift at the level of multi-step behavior and reduces the effective horizon from approximately $P$7 decisions to $P$8. The main experiments use $P$9 and $r$0; the $r$1 ablation returns close to standard CQL performance, supporting temporal abstraction as a major source of the reported gains (Ajay et al., 2020).

ROMI provides a different form of model-based augmentation. It learns a probabilistic reverse dynamics model $r$2 and a reverse rollout policy conditioned on successor states. Starting from dataset target states, it generates plausible predecessor transitions and stores them in a model buffer combined with the original dataset. Reverse imagination is intended to interpolate between trajectories while avoiding the forward-model failure mode of terminating in unsupported, overvalued states. The method can be combined with BCQ or CQL. Its strongest gains occur in navigation domains with obstacles, narrow passages, and useful target states already represented in the dataset (Wang et al., 2021).

HUBL modifies Bellman targets by blending Monte Carlo trajectory returns with bootstrapped values. For a next-state heuristic $r$3 and blending coefficient $r$4, it relabels each transition as

$r$5

High-return trajectories receive greater heuristic weight, while low-return trajectories retain more bootstrapping. HUBL reduces the effective continuation horizon but introduces bias when the heuristic differs from the optimal value. It is implemented by reward and discount relabeling and can be combined with ATAC, CQL, TD3+BC, and IQL. Across 27 D4RL and Meta-World datasets, the paper reports an average improvement of approximately 9% (Geng et al., 2023).

Sequence-modeling methods treat offline RL as conditional generation. Decision Transformer, Trajectory Transformer, Decision Diffuser, and related methods model trajectories or return-conditioned actions. Trifle argues that likelihood modeling alone is insufficient: stochastic behavior and dynamics make return-to-go a random variable, and evaluation requires conditional and marginal probabilistic queries. It uses a tractable probabilistic circuit together with an expressive autoregressive action prior to compute high-return action distributions, perform marginalization, incorporate constraints, and guide beam search. Trifle reports scores superior to Trajectory Transformer on all nine evaluated Gym-MuJoCo tasks, and it performs better in stochastic Taxi and action-constrained MuJoCo settings (Liu et al., 2023).

5. Representation, partial observability, and uncertainty adaptation

Representation determines whether offline RL can transfer information across trajectories. In a fully observed MDP, value iteration can stitch prefixes and suffixes that meet at the same state. Under partial observability, conditioning on raw observation histories may prevent such sharing: two histories can correspond to the same action-relevant situation while differing in irrelevant observations. The effective history space can grow exponentially with horizon, producing poor sample complexity (Hong et al., 2023).

The proposed remedy is a compact history representation approximating a bisimulation metric. For histories $r$6 and $r$7, an on-policy metric has the form

$r$8

An encoder is trained so that embedding distances approximate reward and next-history distribution differences. If histories can be aggregated into a compact representation space $r$9 with small bisimulation error, the raw-history sample-complexity dependence on $\gamma$0 can be replaced by dependence on $\gamma$1, plus representation error (Hong et al., 2023).

APE-V addresses a different representation problem: uncertainty about which MDP generated the evaluation environment. Its Bayesian formulation treats offline RL as an implicit POMDP whose hidden state includes the environment identity. The optimal policy may depend on the current physical state and an evolving belief over environmental hypotheses:

$\gamma$2

APE-V approximates this structure using an ensemble of value functions, a belief vector, a Bellman-error-based surrogate likelihood, and a belief-conditioned actor. Unlike a static lower-confidence-bound policy, it can revise its hypothesis weights after observing evaluation transitions. In Locked Doors and Procgen Mazes, adaptation improves recovery from incorrect initial hypotheses; on D4RL, it is competitive with strong conservative methods and is most useful where multiple salient hypotheses remain (Ghosh et al., 2022).

Resource-constrained offline RL concerns a train–deployment observation mismatch. The offline dataset may contain rich features generated by expensive language, vision, sensor, or embedding systems, while the deployed policy can use only limited observations. A teacher trained with the rich state produces action targets for a student trained on deployment-available features. The continuous-control student objective combines critic improvement, behavior cloning toward the logged action, and transfer toward the teacher action:

$\gamma$3

with $\gamma$4. The method improves substantially over limited-feature TD3+BC on many medium and medium-replay datasets, but does not remove the irreducible teacher–student gap when omitted features are action-relevant (Regatti et al., 2021).

Offline multitask representation learning exploits a different source of structure. In low-rank MDPs, multiple tasks share $\gamma$5 while having task-specific transition measures $\gamma$6. MORL jointly estimates the shared representation and task-specific transition components from fixed source-task datasets. Its analysis gives average transition error of order $\gamma$7 and supports downstream reward-free, offline, and online RL under partial feature coverage rather than full state-action concentrability (Ishfaq et al., 2024).

6. Model learning, inverse rewards, preferences, and deployment

Model-based offline RL estimates dynamics or reward models from the dataset, then uses them for planning or data augmentation. Its principal risk is model error outside the data support. MOPO penalizes model uncertainty, while ROMI constrains imagination structurally through reverse rollouts. CAMERON extends this framework to offline inverse reinforcement learning, where the reward is unknown and only a small set of expert demonstrations is available.

CAMERON learns a transition model, estimates policy-dependent occupancy distributions using the Off-policy Idle GAN-based procedure, generates short model rollouts, and optimizes a discriminator-based cost with COMBO. Its purpose is to approximate the occupancy distributions required by adversarial IRL without current-policy trajectories from the real environment. The method outperforms ORIL and TGR and is competitive with COMBO on Ant, Half-Cheetah, and Hopper, although its density-ratio estimation, GAN optimization, and model rollouts introduce substantial computational and statistical demands (Jarboui et al., 2021).

Sim-OPRL addresses unknown rewards through offline preference elicitation. It learns a transition model and confidence set, constructs policies that are plausible optima under reward models consistent with existing preferences, and queries comparisons between pessimistically simulated rollouts selected for