A Clean Slate for Offline Reinforcement Learning

Abstract: Progress in offline reinforcement learning (RL) has been impeded by ambiguous problem definitions and entangled algorithmic designs, resulting in inconsistent implementations, insufficient ablations, and unfair evaluations. Although offline RL explicitly avoids environment interaction, prior methods frequently employ extensive, undocumented online evaluation for hyperparameter tuning, complicating method comparisons. Moreover, existing reference implementations differ significantly in boilerplate code, obscuring their core algorithmic contributions. We address these challenges by first introducing a rigorous taxonomy and a transparent evaluation protocol that explicitly quantifies online tuning budgets. To resolve opaque algorithmic design, we provide clean, minimalistic, single-file implementations of various model-free and model-based offline RL methods, significantly enhancing clarity and achieving substantial speed-ups. Leveraging these streamlined implementations, we propose Unifloral, a unified algorithm that encapsulates diverse prior approaches within a single, comprehensive hyperparameter space, enabling algorithm development in a shared hyperparameter space. Using Unifloral with our rigorous evaluation protocol, we develop two novel algorithms - TD3-AWR (model-free) and MoBRAC (model-based) - which substantially outperform established baselines. Our implementation is publicly available at https://github.com/EmptyJackson/unifloral.

• The paper introduces a taxonomy of offline RL evaluation regimes to combat ambiguous benchmarks, highlighting variability, performance, domain dependence, and redundant configurations
• By developing a unified framework, Unifloral, researchers can experiment with common components across diverse methods
• TD3-AWR and MoBRAC show significant improvements on multiple datasets demonstrating the power of a unified algorithmic evaluation framework
• TD3-AWR exemplifies an effective combination of ReBRAC and IQL-centric elements to improve performance, notably on certain benchmark tasks

Problem formulation and central thesis

“A Clean Slate for Offline RL” argues that the principal obstacle to cumulative progress in offline reinforcement learning is not the absence of algorithmic ideas, but the lack of a stable experimental substrate on which those ideas can be compared. The paper identifies two interacting sources of ambiguity: first, the offline RL problem is routinely evaluated with undisclosed or inconsistent online interaction budgets; second, published algorithms combine core methodological contributions with implementation choices, tuning conventions, and boilerplate code that are difficult to disentangle. The resulting literature can report strong benchmark performance without establishing whether gains arise from the algorithm, hyperparameter search, implementation details, or favorable task selection (2504.11453).

The paper responds with three linked contributions. It formalizes a taxonomy of offline RL settings, introduces an evaluation protocol that explicitly measures online policy-selection costs, and provides minimal JAX implementations organized around a unified algorithmic space called Unifloral. The authors then use this framework to construct two new methods, TD3-AWR and MoBRAC. Their empirical results are intended not merely as additional benchmark scores, but as evidence that a transparent evaluation and implementation framework can expose useful combinations of existing components.

The paper’s core methodological claim is strong: an offline RL method should include both an algorithm and a fixed sampling range for each hyperparameter. Under this definition, changing hyperparameter ranges across datasets changes the method itself rather than merely adapting its configuration. This position directly challenges a common practice in which an algorithm is treated as invariant while its tuning range, architecture, and optimization schedule vary substantially from task to task.

A taxonomy that makes online interaction explicit

The paper begins by separating offline training from online deployment and then characterizes four evaluation regimes:

1. Zero-shot offline RL, in which one policy is trained from the static dataset and deployed without online adaptation.
2. Offline RL with pre-deployment policy selection, in which several policies are trained offline and a limited number of online episodes are used to select one before deployment.
3. Offline RL with post-deployment policy selection, in which a set of offline-trained policies is selected using deployment-time feedback.
4. Offline-to-online RL, in which a policy is first trained offline and then fine-tuned using online data.

This taxonomy is important because many studies nominally framed as offline RL implicitly operate in the second regime. Hyperparameter configurations are often selected by evaluating candidate policies on the target environment, sometimes with an effectively unrestricted number of episodes. The resulting score therefore measures a compound of offline policy learning and online model selection. The paper does not claim that such procedures are illegitimate; rather, it argues that their interaction budget must be made explicit for comparisons to be interpretable.

Figure 1: Taxonomy of offline RL settings distinguished by pre-deployment interaction, post-deployment policy selection, and online fine-tuning.

The proposed evaluation procedure targets pre-deployment policy selection. A method defines a fixed hyperparameter range, from which the authors sample $P$ configurations and random seeds. Each configuration produces a policy whose episodic-return distribution is estimated from repeated online evaluations. During evaluation, the procedure subsamples $K$ policies and runs a UCB bandit over their noisy episodic returns. At each online budget $N$, performance is measured by the true expected return of the policy currently selected by the bandit.

This design has two advantages. First, it reports performance as a function of deployment interactions rather than only at the asymptotic limit of extensive tuning. Second, it models policy evaluation as a noisy statistical problem: each bandit pull observes one episodic return, not the policy’s expected return. The latter assumption is operationally consequential because high-variance policies can be favored temporarily by the selection procedure.

Figure 2: Offline policies are generated from a fixed hyperparameter range, after which bootstrapped UCB tuning estimates performance under different online evaluation budgets.

The experimental protocol uses $K=8$ policy arms and averages results over 500 bandit rollouts, with confidence intervals estimated from those rollouts. The recommended benchmark distribution includes locomotion, manipulation, kitchen, maze, and ant-maze tasks rather than concentrating exclusively on MuJoCo locomotion. This recommendation follows from the paper’s finding that algorithm rankings are highly domain-dependent.

Benchmark findings and the distractor-policy effect

The evaluation of prior methods produces a deliberately unfavorable conclusion about universal algorithmic superiority. No evaluated method performs consistently well across all nine datasets. ReBRAC is the strongest method at some evaluation budget on five datasets, while IQL is strongest on four. Both methods nevertheless perform poorly relative to competitors on other tasks. The authors therefore identify ReBRAC and IQL as practical reference baselines, not universally dominant algorithms.

Figure 3: Prior algorithms exhibit budget-dependent performance curves rather than a single stable ranking across datasets.

The model-based results are particularly notable. MOPO, MOReL, and COMBO perform poorly on non-locomotion tasks, never ranking above sixth among the ten evaluated algorithms and failing to outperform behavioral cloning at any evaluation budget on those datasets. The paper interprets this pattern partly as a consequence of historical benchmark concentration: the model-based methods were primarily evaluated on MuJoCo locomotion, so their published hyperparameters and design choices may be overfit to that domain. The implication is direct: benchmark coverage is part of an algorithm’s empirical specification, not an incidental reporting detail.

The most technically distinctive evaluation result concerns “distractor policies.” In the ReBRAC experiments on hopper-medium, the authors identify policies with lower mean return but unusually high maximum episodic return. Under a noisy bandit-selection procedure, these policies can appear attractive after a small number of samples. As the bandit samples more policies, the probability of selecting such a policy can initially increase, producing a temporary decline in the expected return of the selected policy.

Figure 4: Distractor policies combine inferior average return with unusually high episodic maxima, making them attractive under limited noisy evaluation.

Figure 5: Increasing the number of candidate policies can deepen the transient performance dip caused by distractor policies.

This phenomenon contradicts the intuitive expectation that additional policy evaluations should monotonically improve selection quality by reducing estimator variance. The contradiction arises because additional candidate arms also increase the probability of including a high-variance distractor. Consequently, online evaluation is not merely a measurement cost; it changes the statistical decision problem. The result also exposes a limitation of protocols that assume low-variance or effectively exact estimates of candidate-policy performance.

Reimplementation as an experimental intervention

The paper’s second major contribution is a systematic reimplementation effort. The authors argue that published offline RL algorithms are often presented as monolithic packages whose differences include both intended innovations and uncontrolled implementation variation. To address this, they construct a compositional genealogy covering model-free methods such as BC, TD3-BC, ReBRAC, IQL, SAC-N, LB-SAC, EDAC, CQL, and Decision Transformer, as well as model-based methods including MOPO, MOReL, and COMBO.

Figure 6: A compositional genealogy organizes offline RL algorithms according to shared and modified components.

Each method is implemented in a minimal single-file format. The design objective is not abstraction for its own sake, but controlled comparison: algorithms that differ by one conceptual component should differ by correspondingly small amounts of code. This makes implementation differences inspectable and supports component-level ablations.

Figure 7: Code edits across SAC-N, CQL, and EDAC identify which implementation changes correspond to algorithmic modifications.

Figure 8: Full implementation differences remain localized when common training and evaluation code is held fixed.

The implementations are written in end-to-end compiled JAX. On HalfCheetah-medium-expert, using a single L40S GPU and one million update steps, Unifloral is reported as the fastest implementation for every listed algorithm. The aggregate speedups are substantial: an average of $131.5\times$ relative to OfflineRL-Kit and $74.8\times$ relative to CORL. These comparisons are computationally important because faster implementations permit broader hyperparameter sampling, more seeds, and more complete evaluation under a fixed compute budget.

Figure 9: Unifloral’s JAX implementations reduce training time substantially relative to OfflineRL-Kit, CORL, and JAX-CORL.

The reproduction experiments also show that the reimplementations broadly match prior reported performance on locomotion tasks. However, matching prior scores does not establish that the original algorithmic claims are correct; it establishes that the new implementations are sufficiently faithful for controlled comparison. The paper’s deeper claim is methodological: implementation clarity is a prerequisite for determining what an offline RL algorithm actually contributes.

Unifloral and the unified design space

Unifloral combines the principal components of the reimplemented algorithms into one configurable framework. Its hyperparameter space is divided into four categories: model design, critic objective, actor objective, and dynamics modeling.

Model-design choices include network depth and width, normalization, learning rates, discount factors, batch size, Polyak averaging, policy stochasticity, and critic-ensemble size. The critic objective can select between IQL-style expectile value targets and TD3-style target critics, then add behavior-cloning, entropy, and ensemble-diversity terms. The actor objective combines value maximization, behavior cloning, advantage-weighted regression, and entropy regularization. The model-based branch adds learned dynamics ensembles, synthetic rollouts, and uncertainty-penalized rewards.

The resulting space contains prior algorithms as subspaces rather than as isolated implementations. For example, IQL is represented through expectile value learning and AWR-based policy extraction; TD3-BC and ReBRAC occupy deterministic, behavior-regularized actor-critic configurations; SAC-N and EDAC use stochastic actors and critic ensembles, with EDAC additionally activating a diversity penalty. This representation makes it possible to search combinations of components without rewriting training code.

The unification is not equivalent to proving that all methods share a common underlying objective. Some design choices are mutually exclusive, while others require method-specific interactions or inactive parameters. The value of Unifloral is therefore empirical and engineering-oriented: it defines a common coordinate system in which algorithmic combinations and ablations can be evaluated under the same training loop, dataset handling, and reporting procedure.

TD3-AWR: combining value-guided policy improvement with deterministic regularization

TD3-AWR is introduced from an explicit comparison between ReBRAC and IQL. ReBRAC uses a TD3-style actor objective with behavior-cloning regularization, whereas IQL extracts a policy through AWR, weighting dataset actions according to estimated advantage. The paper hypothesizes that replacing ReBRAC’s ordinary behavior-cloning term with AWR will retain the benefits of value-guided deterministic policy improvement while preferentially cloning high-advantage actions.

The method is implemented without new source code: it uses the AWR-related hyperparameters from IQL and the remaining ReBRAC configuration. This construction is an important demonstration of the claimed research workflow. The method is not presented as a new architectural paradigm; it is a controlled composition of existing components in a shared implementation.

Figure 10: TD3-AWR is compared with ReBRAC and IQL under the proposed budget-aware evaluation procedure.

The reported results are strong. TD3-AWR strictly dominates ReBRAC on six of nine datasets and is dominated by ReBRAC on only one. It strictly dominates IQL on seven datasets. The advantage is particularly pronounced under small online evaluation budgets on HalfCheetah-medium-expert and Pen-expert. These results imply that the policy-extraction mechanism can materially affect performance even when the critic and broad actor-critic structure remain similar.

The interpretation should nevertheless account for the tuning protocol. TD3-AWR searches a wider hyperparameter range than its source algorithms, so its dominance reflects both the component combination and the defined method range. This is precisely why the paper insists that tuning ranges be reported as part of the method. The result is persuasive as evidence for the utility of the combined configuration, but it does not isolate the marginal contribution of AWR independently from the enlarged search space.

MoBRAC: replacing the model-based policy optimizer

The model-based contribution, MoBRAC, addresses the paper’s finding that existing model-based methods perform poorly outside locomotion. The authors focus on the policy optimizer rather than proposing a new dynamics model. MOPO supplies the learned dynamics ensemble, uncertainty-based reward penalty, and synthetic rollout procedure; ReBRAC supplies the behavior-regularized actor-critic policy optimizer.

This choice is motivated by an underexplored design axis. The model-based methods evaluated in the paper largely rely on SAC-N or CQL for policy optimization, while ReBRAC had demonstrated stronger model-free behavior on several datasets. MoBRAC therefore tests whether a model-based method can benefit from a more recent behavior-regularized optimizer without changing its world-model machinery.

Figure 11: MoBRAC compares the MOPO dynamics and rollout framework with alternative policy optimizers across offline RL datasets.

MoBRAC outperforms the other model-based methods on all datasets except maze2d-large-v1, where MOPO performs better. Under the paper’s transparent evaluation budget, it is the strongest model-based method on six of nine datasets and ties MOPO on the remaining three. The result supports the claim that model-based offline RL performance is not determined solely by model accuracy or uncertainty penalization; policy optimization is an independent and consequential component.

The result also qualifies the paper’s criticism of model-based offline RL. The poor performance of MOPO, MOReL, and COMBO outside locomotion does not establish that learned dynamics are intrinsically unsuitable for diverse offline tasks. MoBRAC’s performance indicates that part of the deficit can be attributed to the interaction between dynamics modeling and policy optimization. At the same time, the method continues to rely on the same family of benchmark environments and on a hard-coded termination function from the target environment, an assumption the authors acknowledge as unrealistic for many deployments.

Limitations and open questions

The evaluation protocol deliberately studies pre-deployment policy selection rather than all offline RL settings. Its conclusions therefore do not directly characterize zero-shot offline RL, post-deployment policy selection, or offline-to-online fine-tuning. The default choice of UCB and the fixed value $K=8$ for the number of policy arms also introduce design decisions whose effect is not fully explored. Different bandit algorithms, arm counts, risk criteria, or evaluation-episode distributions could yield different rankings.

The protocol additionally requires a large collection of online episodic returns to construct the empirical policy-score dataset, even though those returns are later reused through bootstrapping. This is an efficient experimental simulation of tuning budgets, not a zero-interaction method for selecting hyperparameters. The paper’s results therefore quantify the consequences of online tuning rather than eliminate online evaluation.

The benchmark analysis is broad relative to the methods’ original evaluations, but remains centered on D4RL-style datasets and simulated environments. The conclusions about general offline RL practice may not transfer to partial observability, image-based control, real-world logging distributions, nonstationary environments, or datasets with severe support deficiencies. In particular, the paper does not establish how the distractor-policy phenomenon behaves under safety constraints, risk-sensitive objectives, or structured evaluation costs.

Unifloral also leaves unresolved whether its large unified hyperparameter space improves scientific understanding or merely makes high-budget search more effective. TD3-AWR and MoBRAC are configuration-level discoveries, but their gains are not supported by a complete factorial attribution of every component and interaction. The paper therefore establishes the usefulness of the framework more clearly than it establishes the causal necessity of each selected component.

Finally, model-based experiments use access to the target environment’s termination function for comparability with prior work. The authors explicitly note that this is an unrealistic white-box assumption. Whether MoBRAC retains its advantage when termination must also be learned remains an open empirical question.

Conclusion

“A Clean Slate for Offline RL” reframes offline RL evaluation as a joint problem of algorithm design, online policy selection, implementation control, and benchmark coverage (2504.11453). Its taxonomy makes hidden interaction budgets explicit; its bandit-based protocol demonstrates that noisy policy selection can generate non-monotonic and counterintuitive performance curves; and its single-file JAX implementations provide a common basis for reproducible comparisons. The reported $131.5\times$ and $74.8\times$ average speedups, together with the cross-dataset results for TD3-AWR and MoBRAC, show the practical value of unifying implementation and evaluation.

The paper’s principal contribution is methodological rather than a single universally best algorithm. Its results support a narrower and more defensible conclusion: offline RL claims are difficult to interpret when hyperparameter ranges, online evaluation budgets, implementation differences, and benchmark domains are not treated as part of the method being evaluated.