ActiveRL Sample-Complexity Analysis

Updated 8 February 2026

The paper presents nonparametric sample-complexity guarantees for ActiveRL using Gaussian Process regression, significantly reducing active queries compared to purely offline methods.
It employs GP concentration inequalities and information gain bounds to tightly control uncertainty, ensuring the policy suboptimality decays proportionally with increased queries.
Empirical validations on benchmarks like D4RL and Maze2D demonstrate that ActiveRL agents require 30–80% fewer active interactions to achieve near-optimal performance.

Active Reinforcement Learning (ActiveRL) refers to a regime in reinforcement learning (RL) where the agent is augmented with the ability to issue a limited number of targeted queries to a generative model or simulator, in addition to access to offline data, with the explicit aim of accelerating policy learning with minimal online interactions. The challenge in ActiveRL is to develop algorithms that learn a near-optimal policy with substantially fewer environment interactions than conventional online or purely offline RL, and to rigorously analyze the sample complexity—the number of active queries required to attain a desired suboptimality $\varepsilon$ . Recent work, particularly "Sample Efficient Active Algorithms for Offline Reinforcement Learning" (Roy et al., 1 Feb 2026), provides the first nonparametric, information-theoretic sample-complexity guarantees for ActiveRL using Gaussian Processes, and reveals substantial improvements over offline-only protocols. This article surveys the core sample-complexity results, algorithmic frameworks, theoretical principles, and implications for RL research.

1. ActiveRL Problem Formulation and Distinctions

ActiveRL considers learning a policy in a (bounded-reward) discounted Markov Decision Process (MDP) $\mathcal{M}=(\mathcal S,\mathcal A,P,r,p,\gamma)$ , with access to a fixed offline dataset $\mathcal{D}_{\mathrm{off}}$ of $N$ transitions and the privilege of issuing up to $M$ active queries to the environment (or a generative model). The goal is to output a policy $\pi$ such that the expected value shortfall $J(\pi^*)-J(\pi)$ is at most $\varepsilon$ , with high probability, using as few active queries $M$ as possible.

This setting interpolates between two classical regimes:

Offline RL: Only a static dataset; no active queries. Policy performance is constrained by support mismatch and distributional shift.
Generative-model (active) RL: Unlimited access to a simulator; arbitrary targeted querying leads to minimax-optimal sample-complexity, but is often impractical.

ActiveRL exploits targeted exploration, focusing each online query where epistemic uncertainty regarding value estimates is highest, facilitating dramatically improved efficiency relative to passive coverage or uniform querying.

2. GP-Based ActiveRL: Algorithmic Structure

A central advance in sample-efficient ActiveRL is the integration of Gaussian Process (GP) regression for value function modeling, uncertainty quantification, and guiding exploration (Roy et al., 1 Feb 2026). The algorithm maintains a GP prior $V\sim\mathcal{GP}(0,k)$ on the optimal value function $V^*$ , updating this model on both offline and active samples.

At each active step $t$ (up to budget $M$ ):

Model update: GP posterior $(\mu_{t-1},\sigma_{t-1})$ is computed on all collected data.
Acquisition: Select $s_t= \arg\max_{s}\sigma_{t-1}(s)$ (maximal GP variance), then pick $a_t$ (possibly $\varepsilon$ -greedy w.r.t.\ $\mu_{t-1}$ ).
Query: Observe $(s_t,a_t,r_t,s_t')$ from the environment, compute $y_t = r_t + \gamma\,\mu_{t-1}(s_t') + \eta_t$ .
Posterior update: Add $(s_t,y_t)$ to the dataset; recompute the GP posterior.
Policy extraction: After all queries, output policy $\pi_T(s)=\arg\max_a \mu_T(s,a)$ .

This procedure, by maximizing the reduction in epistemic uncertainty at each step, yields near-optimal convergence rates under nonparametric function classes.

3. Sample-Complexity Guarantees: Nonparametric PAC Bounds

The main theoretical result of (Roy et al., 1 Feb 2026) asserts that, under standard regularity conditions (RKHS norm bound, Lipschitz transitions, sub-Gaussian noise), the number of active samples $M$ required to find an $\varepsilon$ -optimal policy is

$M = \widetilde O\!\left(\frac{1}{\varepsilon^2(1-\gamma)^2}\right)$

where $\widetilde O(\cdot)$ hides logarithmic factors, and $(1-\gamma)^{-2}$ is the horizon dependence. This is achieved thanks to

GP concentration inequalities: For all $s$ , $|V^*(s)-\mu_{t-1}(s)|\le\beta_t\sigma_{t-1}(s)$ with high probability, for an explicit $\beta_t$ .
Information gain bounds: The sum of GP posterior variances along the query sequence is tightly controlled by the maximum information gain $\Gamma_T$ :

$\sum_{i=1}^T \sigma_{i-1}^2(s_i) \le 2 \Gamma_T, \quad \text{where} \quad \Gamma_T = \max_{A:|A|=T} I(y_A; V)$

Value gap: The performance difference $J(\pi^*)-J(\pi_T)$ contracts proportionally to $\sqrt{\Gamma_T/T}$ , via Bellman contraction and uncertainty decay.

Thus, an $\varepsilon$ -optimal policy is attainable after $T = \widetilde O(\Gamma_T/(\varepsilon^2(1-\gamma)^2))$ queries. For kernels with sublinear information gain growth (e.g., RBF, Matern), the dependence is nearly linear in $1/\varepsilon^2$ .

4. Comparison to Offline and Other Active RL Protocols

A key implication is the substantial improvement over passive, purely offline RL. GP-based offline RL (no active queries) incurs a sample complexity lower bound

$\Omega\left(\frac{1}{\varepsilon^2(1-\gamma)^4}\right)$

reflecting the need to uniformly cover the state-action space. ActiveRL's budget is quadratically smaller in the horizon factor, matching generative-model (KQLearn (Yeh et al., 2023)) up to the unavoidable penalty of not having full simulator access.

Related frameworks include:

Kernel-based active Q-learning: KQLearn builds uncertainty-maximizing sets for active querying in the RKHS, attaining $N(\varepsilon, \delta) = \tilde O(\Gamma/((1-\gamma)^4\varepsilon^2))$ (Yeh et al., 2023), where the difference in $(1-\gamma)$ exponent reflects algorithmic specifics and the impact of querying all transitions per uncertainty locus.
Objective-agnostic sample collection: GOSPRL (Tarbouriech et al., 2020) decouples the target sample prescription from the online transport, yielding time complexity $\tilde O(BD + D^{3/2}S^2A)$ to collect $B$ prescribed samples, modulo MDP diameter $D$ and combinatorial terms, thus providing a general plug-and-play tool for ActiveRL in finite communicating MDPs.
Limited revisiting linear MDPs: ActiveRL can approach generative-model efficiency under strong linear structure and a sufficiently large suboptimality gap, even with only controlled revisits, not full arbitrary queries (Li et al., 2021).

5. Underlying Principles: Information Gain and GP Concentration

The rate-limiting factor in nonparametric function-approximation regimes is the information gain $\Gamma_T$ —the maximal mutual information the entire sample history conveys about the target value function under the GP prior. This metric unifies bandit/active learning statistical complexity and RL sample efficiency:

GP posterior concentration: Score-based exploration maximally reduces $\sum\sigma_{t-1}^2(s_t)$ , which, via the elliptical-potential lemma, contracts at $O(\Gamma_T/T)$ .
Bellman error amplification: Propagation through the Bellman operator inflates errors by $1/(1-\gamma)$ ; uncertainty-guided sampling ensures this is not pathological, unlike in pure offline RL.
Theoretically, these mechanisms bridge Bayesian nonparametrics and RL, and establish PAC-style guarantees unattainable with generic function classes or uniform sampling.

6. Practical Implications and Experimental Validation

Empirical results in (Roy et al., 1 Feb 2026) confirm theoretical predictions:

D4RL continuous-control benchmarks and Maze2D sparse-reward tasks are tackled with GP-sparse approximations and only 5–30% of the original offline data after region-based pruning.
ActiveRL agents require 30–80% fewer active transitions than offline or random exploration variants to surpass baseline performance, with learning curves demonstrating the predicted $O(\Gamma_T/T)$ decay in policy suboptimality and posterior uncertainty.
Scalable approximations—sparse GPs, large-scale kernel regression—are crucial for practical deployment. Sample efficiency gains are robust under varying offline data coverage and kernel choices.

7. Limitations and Scope of Current Analyses

Current ActiveRL sample-complexity analyses require:

RKHS structure containing $V^*$ with bounded norm, and kernels admitting sublinear $\Gamma_T$ growth.
Transition kernels Lipschitz in state-action, bounded rewards, and mild offline data coverage (finite initial posterior variance across states).
Gaussian noise observation models for tractable GP updating.

Computational complexity remains a practical limitation due to GP inference cost, mitigated by sparse approximations or ensembles. The analysis is not yet fully general to classes lacking tractable uncertainty quantification or for environments where the generative model is highly restricted. A plausible implication is that future work will need to address these modeling and scalability gaps, as well as optimality gaps for more complex function classes.

In sum, ActiveRL establishes a new regime for sample-efficient RL by leveraging model-based uncertainty quantification and targeted exploration, achieving nonparametric, information-theoretically grounded guarantees that interpolate between offline learning and full-simulator methods (Roy et al., 1 Feb 2026, Yeh et al., 2023, Tarbouriech et al., 2020, Li et al., 2021). This framework enables accelerated policy learning with provably minimal active interaction, shaping the frontier of efficient RL.