Action-Free Offline-to-Online RL

• Action-free offline-to-online RL is a framework that utilizes unlabeled state and reward logs, bridging offline observations with efficient online policy learning.
• The approach demonstrates that, under admissibility assumptions, specialized algorithms like Foobar, OSO-DecQN, and AF-Guide can match reset-model sample efficiency despite missing action data.
• Empirical benchmarks in domains such as robotics and high-dimensional control show accelerated learning and improved performance compared to traditional online RL methods.

Action-free offline-to-online reinforcement learning (RL) concerns the integration of offline datasets lacking action annotations—i.e., comprising only state and possibly reward observations—with subsequent online RL in environments where data efficiency and partial observability of underlying agent behaviors are paramount. This scenario, which arises frequently in practice due to privacy, storage, or sensor limitations, calls for new algorithms and theories that can bridge the gap between purely observational offline knowledge and effective online exploration and policy learning. Recent work has delivered formal definitions, sample complexity bounds, algorithms, and empirical benchmarks for this paradigm, demonstrating it is possible to leverage such action-free data for accelerated and performant online RL—albeit subject to critical structural and admissibility assumptions.

1. Formal Problem Setting and Motivations

The action-free offline-to-online RL paradigm is defined by a Markov decision process (MDP), $(S, A, P, r, \gamma)$, with states $S \subseteq \mathbb{R}^M$, continuous (or discrete) actions $A$, transition kernel $P(s'|s,a)$, reward function $r: S \times A \to \mathbb{R}$, and discount factor $\gamma \in (0,1)$. In this setting, the learner is provided an offline dataset $D$, which contains tuples of $(s, r, s')$ or $(s_0, r_0, \ldots, s_T, r_T)$ but omits action annotations. This data may originate from various sources, including passive recordings or partial logs, and is "action-free" in the sense that direct behavioral learning (e.g., behavior cloning, standard offline RL) is inapplicable.

The learner is subsequently granted online access to the environment, typically under the trace model (i.e., only initial state resets and rollout of full episodes are possible, with no mid-episode resets). The overarching objective is to leverage the action-free offline data as prior knowledge to accelerate online learning or improve asymptotic performance, competing with the best policy that is “covered” by the offline distribution (Song et al., 2024, Neggatu et al., 31 Jan 2026, Zhu et al., 2023).

This framework is motivated by the ubiquity of action-free state/reward logs in applications such as robotics, health care, video-based learning, infrastructure monitoring, and scenarios with privacy or storage constraints.

2. Structural Hardness and the Admissibility Barrier

A central theoretical result in the study of action-free offline-to-online RL is the demonstration of an exponential gap in sample complexity between models that permit arbitrary state resets (the reset/generative model) and those that enforce trajectories from the initial state (the trace model) when the offline state data is inadmissible. Specifically, in the trace model, if the offline occupancy measures $\{\mu_h\}$ are not representable as the state occupancies of some actual policy within the considered class (inadmissibility), then no trace-model algorithm can efficiently recover the optimal policy; $T = \Omega(A^H)$ full trajectory episodes are required to even approximate the marginal state occupancy at the terminal time step in total variation, while the reset model admits $O(A)$ sample complexity (Song et al., 2024).

Admissibility thus becomes a critical assumption for feasible action-free offline-to-online RL. Formally, the offline data is admissible if there exists a policy $\overline{\pi}$ in the considered policy class such that, for all $h$, the observed offline distribution $\mu_h$ exactly matches $d_h^{\overline{\pi}}$. When admissibility holds, algorithms can circumvent the hardness barrier.

3. Algorithmic Strategies

Recent work has proposed several families of algorithms to tackle action-free offline-to-online RL; three primary approaches are prominent.

a. Forward-Backward (Foobar) Approach under Admissibility

The "Foobar" algorithm (Song et al., 2024) is a two-phase method under the trace-model and admissibility assumptions:

• Forward Phase: A minmax-imitation process constructs value-like discriminators and matches the offline state occupancy distributions $\mu_h$ using iterative imitation learning, ensuring the learned policy aligns with the offline data in an integral probability metric (IPM) sense.
• Backward Phase: Utilizing regression on value functions (via least-squares temporal difference learning on sampled traces), the algorithm computes Q-functions and greedy policies in a backward dynamic programming fashion.

A key subroutine is resolving minmax objectives of the form

$$\min_{\pi_h \in \Pi} \max_{g \in G_h} \widehat{\mathbb{E}}_{s \sim d_h^f, a \sim \pi_h}[g(s')] - \widehat{\mathbb{E}}_{s \sim \mu_h}[g(s')]$$

to match critical moment features of the state distributions.

b. Discretised State Policy Learning (OSO-DecQN)

"Offline State-Only Decoupled Q-Network" (OSO-DecQN) (Neggatu et al., 31 Jan 2026) circumvents the absence of actions by discretising per-coordinate state changes $\Delta s = s' - s$ into a finite-binned set $A^s$, reducing the state prediction problem to classification. An ensemble of M-dimensional Q-functions, decomposed by state dimension, is learned using Bellman-conservative Q-learning (with regularization to penalize unrealizable state transitions).

After offline training, an inverse dynamics model (IDM) is learned online to map discretised state transitions to real actions. Guided exploration during online RL is implemented via "policy-switching" (probabilistically mixing between guided actions from the state-policy and the agent's own policy), with the guidance annealed over the course of training.

c. Action-Free Decision Transformer and Guided SAC

The AF-Guide framework (Zhu et al., 2023) leverages Transformer architectures (AFDT: Action-Free Decision Transformer) to learn, via supervised regression, a mapping from recent state histories and return-to-go signals to future state predictions, never observing actions. In online RL, a modified Soft Actor-Critic (Guided SAC) agent is shaped by a dense guiding reward, reflecting the similarity between the agent's experienced transition and AFDT's predicted transition.

Unlike naïve reward augmentation, AF-Guide optimizes separate critics for environment and guiding rewards, with the actor maximizing a weighted sum of these, minus entropy.

4. Theoretical Guarantees and Limitations

Theoretical analysis of these algorithms typically proceeds along two axes:

• Statistical Rate Equivalence: When admissibility and sufficient capacity conditions are satisfied, the trace model with observation-only offline data achieves the same sample complexity rate as full-data reset-model algorithms: $$N_\text{for}, N_\text{back} = O\left( \frac{\text{cov}(\pi^*)^2 H^5 A \log(|F||\Pi|/\delta)}{\epsilon^2} \right)$$ for error $\epsilon$ and occupancy coverage factor $\text{cov}(\pi^*)$ (Song et al., 2024).
• Discretisation Error Bounds: For configuration-wise discretisation into $k$ bins, the suboptimality gap due to discretisation is bounded by

$$|V^*(s) - V^\text{disc}(s)| \le O\left( \frac{\gamma H M}{(1-\gamma)k} \right)$$

(Neggatu et al., 31 Jan 2026).

Limitations persist:

• Absence of admissibility yields exponential sample complexity.
• Inverse dynamics modeling introduces a further dependency on the local invertibility of the environment’s dynamics.
• Rewards and next-state transitions in offline data may still bias the offline-learned guidance, especially under distributional shifts or when offline data is of low quality.

5. Empirical Findings

Benchmark evaluations substantiate the theoretical predictions, demonstrating that:

• In sparse-reward settings and high-dimensional control (e.g., Adroit hammer, AntMaze, D4RL MuJoCo, DeepMind Suite), action-free offline-to-online RL algorithms outperform vanilla online RL (e.g., SAC, TD3) in both convergence speed and final performance.
• Discretising state transitions for offline learning is robust to the number of bins and regularisation parameters, and scales to state spaces with up to 78 dimensions (Neggatu et al., 31 Jan 2026).
• Purely online baselines often fail to solve sparse or long-horizon tasks under sample budget constraints, while action-free offline-guided methods do succeed, provided admissibility (or at least no adversarial inadmissibility) holds (Song et al., 2024, Zhu et al., 2023).
• Dual-critic architectures and explicit policy-switching are critical to avoid degradation in performance due to misspecified guiding signals or reward scale mismatch.

6. Integration with Other Paradigms and Future Directions

Action-free offline-to-online RL connects fundamentally to imitation learning under partial observability, unsupervised world model pretraining, and reward shaping via auxiliary signals. The paradigm naturally extends to scenarios with pixel observations, partial rewards, or multimodal input, with open questions concerning the integration of learned or contrastive metrics for high-dimensional guidance, improved inverse model estimation, and handling of stochastic dynamics (Zhu et al., 2023).

Empirical and theoretical limitations point toward the necessity for further research on mechanisms that relax the admissibility assumption, adaptive regularisation of discretisation granularity, and new algorithms for partial resets or trace models with broader classes of offline data distributions.

Paper Title	Main Contributions	arXiv ID
Hybrid Reinforcement Learning from Offline Observation Alone	Formalises trace vs reset model, theory & Foobar algorithm	(Song et al., 2024)
Guiding Online Reinforcement Learning with Action-Free Offline Pretraining	AF-Guide with AFDT and Guided SAC, empirical results	(Zhu et al., 2023)
Action-Free Offline-to-Online RL via Discretised State Policies	OSO-DecQN, theoretical and empirical analysis	(Neggatu et al., 31 Jan 2026)