Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learning

Abstract: In offline-to-online reinforcement learning (O2O-RL), policies are first safely trained offline using previously collected datasets and then further fine-tuned for tasks via limited online interactions. In a typical O2O-RL pipeline, candidate policies trained with offline RL are evaluated via either off-policy evaluation (OPE) or online evaluation (OE). The policy with the highest estimated value is then deployed and continually fine-tuned. However, this setup has two main issues. First, OPE can be unreliable, making it risky to deploy a policy based solely on those estimates, whereas OE may identify a viable policy with substantial online interaction, which could have been used for fine-tuning. Second--and more importantly--it is also often not possible to determine a priori whether a pretrained policy will improve with post-deployment fine-tuning, especially in non-stationary environments. As a result, procedures committing to a single deployed policy are impractical in many real-world settings. Moreover, a naive remedy that exhaustively fine-tunes all candidates would violate interaction budget constraints and is likewise infeasible. In this paper, we propose a novel adaptive approach for policy selection and fine-tuning under online interaction budgets in O2O-RL. Following the standard pipeline, we first train a set of candidate policies with different offline RL algorithms and hyperparameters; we then perform OPE to obtain initial performance estimates. We next adaptively select and fine-tune the policies based on their predicted performance via an upper-confidence-bound approach thereby making efficient use of online interactions. We demonstrate that our approach improves upon O2O-RL baselines with various benchmarks.

• The paper introduces an adaptive policy selection framework that utilizes UCB forecasts to dynamically allocate limited online interactions in offline-to-online RL.
• It models non-monotonic fine-tuning trajectories with a linear AR(2)-ARCH(1) process, addressing the challenge of unpredictable policy improvement.
• Empirical evaluations on D4RL benchmarks demonstrate superior normalized scores, highlighting enhanced sample efficiency and robustness over traditional methods.

Adaptive Policy Selection and Fine-Tuning under Constraints in Offline-to-Online RL

Motivation and Context

Offline-to-online reinforcement learning (O2O-RL) addresses the practical constraints of real-world RL deployments, particularly for high-stakes robotic and autonomous systems, where extensive online interaction is infeasible due to cost, time, or safety concerns. Classic offline RL leverages previously collected data to pretrain policies; O2O-RL further refines these offline-initialized policies through a restricted number of online interactions. However, standard O2O-RL pipelines commit to a single policy based on off-policy evaluation (OPE) or online evaluation (OE), which is often unreliable. The challenge is exacerbated by the observed volatility and unpredictability of policy improvement (or regression) during fine-tuning, especially in the presence of hyperparameter and algorithmic sensitivity, environmental stochasticity, and strict interaction budgets.

Core Contributions

The paper introduces a novel adaptive policy selection and fine-tuning framework for O2O-RL that jointly addresses the intertwined, underexplored trade-off between exhaustive policy selection and efficient fine-tuning under strict interaction budgets. The proposed approach encompasses the following steps:

1. Diverse Candidate Generation: Offline RL is used to train a variety of candidate policies across multiple algorithms and hyperparameter settings, broadening the search space for promising solutions.
2. Initial Ranking via OPE: OPE is employed to provide preliminary (albeit often unreliable) value estimates and rankings for the offline-trained policies.
3. Adaptive Fine-Tuning with UCB: An upper-confidence-bound (UCB) mechanism based on a linear autoregressive (AR) model with conditionally heteroscedastic errors (ARCH) predicts each policy’s future value trajectory under fine-tuning. Policy selection and continuation are driven adaptively: the policy with the best forecasted UCB is selected for fine-tuning; as more data accrues, UCBs are recomputed, and switching occurs if another policy's projected gain justifies a reallocation of the remaining interaction budget.
4. Efficient Allocation of Budget: This mechanism addresses the impossibility of exhaustively fine-tuning all candidates under budget constraints and mitigates the risk of stagnation or regression in policy performance, thereby minimizing regret relative to an oracle with hindsight knowledge of all fine-tuning outcomes.

The entire pipeline is illustrated in (Figure 1).

Figure 1: Overview of the O2O-RL framework, highlighting candidate generation, OPE-based ranking, AR-ARCH-UCB forecasting, and adaptive fine-tuning decision flow.

Characterization of Value Dynamics during Fine-Tuning

Empirical analysis of fine-tuning trajectories across algorithms, seeds, and hyperparameters reveals highly non-monotonic, high-variance value curves. Performance improvements can stall, regress, or display significant randomness, even for policies with similar starting points and configurations. This instability underscores the need for dynamic, data-driven reallocation of online interaction, as static or naive approaches (e.g., exhausting the entire budget on a single selected policy) can routinely yield suboptimal outcomes. The diversity of these dynamics is depicted in (Figure 2).

Figure 2: Fine-tuning trajectories for various pretrained policies on walker-random highlight idiosyncratic, non-monotonic behaviors and dependency on random seeds, motivating the need for adaptivity in O2O-RL.

Methodological Structure

Modeling Value Evolution: The paper models the fine-tuning value trajectory via a linear AR(2)-ARCH(1) process, capturing both short-term trends and volatility clustering in value signals. This enables robust multi-step forecasting of possible fine-tuning outcomes per policy, facilitating a UCB-based exploration-exploitation trade-off across the candidate set.

Policy Selection Mechanism: Policies are ranked and chosen based on the maximal UCB among forecasted values, ensuring priority is given to those with both high potential and high prediction confidence. The queue-based selection protocol ensures continual reevaluation and allows the dynamic allocation of the online interaction budget in response to real-time feedback, rather than a one-shot or rigid schedule.

Integration of OPE and UCB: While OPE estimates are used for initial ranking, subsequent decisions prioritize UCBs from online-initialized forecasts, allowing the framework to self-correct if OPE is inaccurate.

Empirical Evaluation

Experiments are conducted across standard locomotion tasks in the D4RL benchmark using multiple offline RL algorithms (AWAC, IQL, CalQL, ReBRAC) and comprehensive hyperparameter sweeps. Results are reported for varying interaction budgets. The paper systematically compares:

• OPE Selection: No fine-tuning, policy with best OPE estimate.
• Best Oracle: No fine-tuning, best policy by true performance (upper bound for selection alone).
• OE (Online Evaluation): Uniformly distributes online evaluations, no fine-tuning.
• FT (Full Fine-Tuning): One selected policy is fine-tuned using the entire interaction budget.
• Adaptive (Proposed): Adaptive UCB-driven policy selection/fine-tuning as described.

Key findings:

• Across all environments and datasets, the proposed adaptive method attains the highest overall normalized scores under limited budgets.
• OPE and FT baselines suffer from underperformance due to unreliable OPE values and the risk of over-committing to poor fine-tuning trajectories.
• OE is limited in effectiveness as the budget per candidate shrinks with growing candidate sets; exhaustive evaluation is infeasible in practical scenarios with large K.
• The adaptive approach demonstrates clear robustness to initialization and seed sensitivity, as well as to under- or over-performing individual algorithms.

Theoretical and Practical Implications

Practical Implications:

• The approach yields improved deployability for RL in domains where every interaction is costly or hazardous, such as robotics, autonomous vehicles, and safety-critical industrial control.
• The methodology flexibly integrates with any offline RL algorithm and hyperparameter-tuning regime, making it a modular addition to O2O-RL pipelines without requiring bespoke modifications to policy learning codebases.

Theoretical Implications:

• The analysis highlights the deficiency in current OPE/OE-centric strategies for O2O-RL, advocating for joint, adaptive, budget-constrained optimization of both policy selection and fine-tuning schedule.
• The modeling of fine-tuning as a nonstationary, volatile process (captured via time series models) challenges the assumption of smooth improvement and directly informs regret minimization strategies applicable to other resource-bounded RL problems.

Future Directions

The limitations discussed include the considerable online evaluation cost per fine-tuning step and the lack of integration of policy similarity information for budget-sharing efficiency. Prospective research avenues involve:

• Developing in-situ or low-overhead progress monitoring for fine-tuning.
• Leveraging representation or trajectory-level similarities to guide selection, akin to clustering or meta-learning.
• Designing OPE methods specifically attuned to ranking and fine-tuning potential, rather than one-step policy value estimation.
• Extending the setting to more complex deployment regimes (e.g., continual learning, task generalization, or hybrid RL).

Conclusion

The paper provides an adaptive O2O-RL framework that unifies policy selection and fine-tuning under interaction constraints by incorporating predictive UCB modeling over candidate policy reward trajectories. Extensive results substantiate its utility and reliability over established baselines, with clear benefits in regret minimization and sample efficiency. The methodology lays the groundwork for future advances in practical RL deployment, especially in safety- and cost-constrained settings.