On-Policy Data Collection in RL

• On-policy data collection is the process of gathering data directly by deploying the target policy, ensuring empirical state-action distributions align with theoretical expectations.
• It underpins methods like PPO and actor-critic algorithms, where optimized sampling and variance reduction improve policy updates and overall performance.
• Best practices include maximizing parallel environments, adaptive sampling, and human-in-the-loop corrections to enhance data diversity and reduce sampling errors.

On-policy data collection refers to the process of acquiring data by sampling directly from a policy as it is being learned or evaluated, maintaining alignment between the data-generating mechanism and the policy of interest. This paradigm is distinguished from off-policy data collection, in which data are gathered via a policy other than the current (target) policy and often require statistical correction. On-policy approaches are ubiquitous in reinforcement learning (RL)—notably in policy gradient and actor-critic algorithms—as well as in online contextual bandit policy learning, imitation learning, and certain classes of causal/semi-parametric inference procedures. The technical challenges and innovations in on-policy data collection vary substantially across domains, spanning optimality theory, stability/bias-variance trade-offs, coverage and sampling error, safe and human-in-the-loop collection, and sample-efficient strategies under practical constraints.

1. Formal Definitions and Theoretical Foundations

On-policy data collection formally consists of generating trajectories or samples by deploying the target policy $\pi$ in the environment to obtain empirical distributions that match the induced theoretical state(-action) visitation distributions. In RL, this induces the canonical Markov chain

$H = (S_0, A_0, R_0, S_1, \ldots) \sim P^\pi$

where $A_t \sim \pi(\cdot|S_t)$ and $S_{t+1} \sim P(\cdot|S_t, A_t)$. The empirical return estimator (on-policy Monte Carlo) is

$$\hat{v}^{\mathrm{MC}}(\pi) = \frac{1}{N}\sum_{i=1}^N G(H_i), \qquad H_i \sim \pi,$$

where $G(H_i)$ denotes the trajectory return (Zhong et al., 2021).

The global statistical optimality of on-policy estimators can be compromised by finite-sample effects, coverage deficiencies (the lack of adequate coverage over relevant state-action pairs), and, in adaptive settings, dependencies in the data due to policy evolution. Recent theoretical work decomposes the variance and bias in on-policy evaluation into components attributable to the data-collection policy, the baseline or estimator structure, and the sequential nature of decision-making (Liu et al., 2024).

2. On-Policy Data Collection in Reinforcement Learning

On-policy data collection is foundational for deep RL algorithms such as Proximal Policy Optimization (PPO), A3C, and variants thereof. In each iteration, the agent samples $N$ parallel environments for $T$ timesteps using the current policy $\pi_{\theta}$, collecting a batch $\mathcal{B}$ of transitions that is immediately used for policy and value updates (Mayor et al., 3 Jun 2025):

• Gradient variance is inversely proportional to $H = (S_0, A_0, R_0, S_1, \ldots) \sim P^\pi$0.
• Truncation bias is mitigated by longer rollouts, but increased $H = (S_0, A_0, R_0, S_1, \ldots) \sim P^\pi$1 raises per-trajectory return variance.
• State and action space coverage benefits from increased $H = (S_0, A_0, R_0, S_1, \ldots) \sim P^\pi$2 (more parallel actors), which improves sample diversity and reduces the risk of overfitting (“primacy bias”), particularly when performing multiple stochastic gradient epochs on the same batch.

Empirical results demonstrate that increasing the number of parallel environments $H = (S_0, A_0, R_0, S_1, \ldots) \sim P^\pi$3, even at fixed batch size, yields higher final performance, broader state coverage, richer learned representations (high feature rank, fewer dormant neurons), and greater stability. Short rollouts with large $H = (S_0, A_0, R_0, S_1, \ldots) \sim P^\pi$4 are strictly preferable to long rollouts with small $H = (S_0, A_0, R_0, S_1, \ldots) \sim P^\pi$5, holding sample budget constant (Mayor et al., 3 Jun 2025).

Best practices include maximizing $H = (S_0, A_0, R_0, S_1, \ldots) \sim P^\pi$6 within compute constraints, carefully tuning the number of epochs per batch, and adjusting architecture (e.g., separate actor/critic encoders or feature split) to exploit increased data diversity. Regularization—via entropy bonuses and clipping—is critical to maintain stability as $H = (S_0, A_0, R_0, S_1, \ldots) \sim P^\pi$7 increases.

3. Optimal On-Policy Sampling and Minimizing Estimator Variance

Variance minimization in on-policy evaluation is a central problem, particularly in the context of sequential decision processes where naive i.i.d. sampling is inefficient. The "doubly-optimal" paradigm, as formalized by Liu et al., solves a bi-level optimization: select a behavior policy $H = (S_0, A_0, R_0, S_1, \ldots) \sim P^\pi$8 (possibly diverging from $H = (S_0, A_0, R_0, S_1, \ldots) \sim P^\pi$9) and baseline $A_t \sim \pi(\cdot|S_t)$0 so as to minimize the variance of the (per-decision) importance sampling estimator, subject to unbiasedness and policy coverage (Liu et al., 2024). The closed-form solution prescribes

$A_t \sim \pi(\cdot|S_t)$1

where $A_t \sim \pi(\cdot|S_t)$2 quantifies per-state-action variance contributions recursively. The resulting estimator remains unbiased and incurs strictly lower variance than both on-policy MC and classical doubly robust estimators. Empirically, the doubly-optimal method achieves the same root-MSE as on-policy MC with up to $A_t \sim \pi(\cdot|S_t)$3 fewer samples in MuJoCo benchmarks.

The technical constraint is coverage: $A_t \sim \pi(\cdot|S_t)$4 must allocate nonzero probability to all actions for which $A_t \sim \pi(\cdot|S_t)$5. If coverage is inadequate, unbiasedness fails, necessitating fallback to true on-policy MC.

4. Addressing Sampling Error and Coverage in On-Policy Collection

Finite-sample on-policy data often fails to match the true on-policy distribution, leading to high-variance estimators and slow convergence. The distinction between collecting on-policy data and strictly sampling i.i.d. from the current policy is nontrivial. The robust on-policy sampling (ROS) framework and further extensions such as PROPS recognize that non-i.i.d., adaptively corrected sampling policies can accelerate convergence to the desired empirical distribution (Zhong et al., 2021, Corrado et al., 2023).

• Sampling error is typically measured by $A_t \sim \pi(\cdot|S_t)$6.
• ROS adaptively perturbs action selection to target under-sampled actions, yielding $A_t \sim \pi(\cdot|S_t)$7 convergence of empirical action distributions (vs. $A_t \sim \pi(\cdot|S_t)$8 for i.i.d.).
• PROPS leverages a clipped surrogate (PPO-style) objective and KL-regularization to realize adaptive behavior policies for collecting data in policy gradient contexts, leading to lower sampling error, improved data efficiency, and superior policy returns than pure on-policy PPO, especially in high-dimensional domains (Corrado et al., 2023).

In contextual bandit and preference learning, coverage improvement principles underpin exponential convergence: each on-policy batch with sufficient size yields strictly better empirical feature covariance, shrinking estimation error rates and enabling rapid approach to the statistical optimum (Kim et al., 13 Jan 2026). Hybrid or G-optimal design strategies further enhance coverage and uniformity of exploration.

5. Practical Strategies, Human-in-the-Loop, and Domain-Specific Innovations

On-policy collection methods have been actively developed in domains with operational/human constraints, such as robotics and interactive LLM data curation.

• Human-in-the-loop correction: VR-DAgger and Shared Autonomy systems alternate between autonomous policy rollout and selective expert intervention. Uncertainty-guided sampling (e.g., using MC-dropout) focuses human feedback on failure segments most likely to close coverage gaps and improve robustness. Shared autonomy architectures combine real-time human macro (arm) control with autonomous micro (hand) policy, streamlining high-quality demonstration acquisition (Zurbrügg et al., 26 May 2026, Cui et al., 31 Oct 2025).
• Long-context LLMs: The PolicyLong paradigm closes the “off-policy gap” observed when static data curation (e.g., offline screening/filtering with a fixed model) fails to keep pace with the model’s evolving knowledge. Iterative, on-policy screening, retrieval, and adaptive selection using the current model’s entropy landscape ensures continual alignment of training samples with model uncertainty, yielding consistently harder datasets and monotonic context-extension gains (Jia et al., 9 Apr 2026).
• Semiparametric inference and causal learning: In the Online Moment Selection framework, on-policy data allocation across sources is adaptively optimized using interim estimators of variance, ensuring that sampling proportions approach oracle optimality and minimizing regret (Gupta et al., 2024).
• Safety and resource trade-offs: On-policy collection is adapted to satisfy safety constraints (e.g., bounded cumulative cost), budget limitations, and practical measurement trade-offs by optimizing not just sampling frequency but the allocation of measurement effort and resources (Mukherjee et al., 2024, Opocher, 8 Apr 2026).

6. Limitations, Trade-Offs, and Technical Challenges

• Coverage and exploration: On-policy data collection is vulnerable to poor coverage in high-dimensional or long-horizon tasks. Self-supervised or intrinsic-bonus–driven exploration (e.g., using diversity incentives such as RND) mitigates this to some extent (Endrawis et al., 2021).
• Sample efficiency: The need for fresh data after each policy update exacerbates sample complexity. Innovations such as variance-aware behavior policy design, optimized baselines, and non-i.i.d. adaptive sampling seek to reduce the number of samples required for a given estimation accuracy (Liu et al., 2024, Zhong et al., 2021).
• Bias-variance and overfitting: Reuse of the same on-policy batch for multiple updates (epochs) risks overfitting and loss of network plasticity in deep RL. This necessitates careful balancing of data diversity, batch size, and update frequency (Mayor et al., 3 Jun 2025).
• Human factors: In human-in-the-loop regimes, cognitive load, demonstration fatigue, and interface bottlenecks constrain throughput; shared autonomy, focused corrective labeling, and automation of sub-tasks are effective mitigations (Zurbrügg et al., 26 May 2026, Cui et al., 31 Oct 2025).

7. Empirical Outcomes and Best Practices

Sustained empirical evaluation across domains demonstrates that optimized on-policy data collection can yield significant sample-efficiency and robustness improvements over both naively on-policy and purely off-policy data regimes. Key findings include:

• Up to $A_t \sim \pi(\cdot|S_t)$9 reduction in variance for policy evaluation relative to on-policy MC via doubly-optimal collection (Liu et al., 2024).
• In PPO, higher $S_{t+1} \sim P(\cdot|S_t, A_t)$0 leads to state-coverage gains, improved performance, and more expressive representations for a fixed compute budget (Mayor et al., 3 Jun 2025).
• Non-i.i.d. on-policy sampling and correction (ROS, PROPS) reduce sampling error and estimator MSE orders of magnitude faster than i.i.d. approaches (Zhong et al., 2021, Corrado et al., 2023).
• Shared autonomy methods yield high-quality, scalable demonstration datasets with reduced human cost and greater robustness (Cui et al., 31 Oct 2025).
• On-policy curriculum (e.g. PolicyLong) closes long-context LLM gaps and maintains alignment with model capabilities across successive training epochs (Jia et al., 9 Apr 2026).

In practice, optimal on-policy data collection requires adapting the sampling policy or data allocation to estimated variance, coverage gaps, safety or resource constraints, and domain-specific operational realities. Empirical best practices include maximizing diversity (e.g., parallelism, goal-sampling), employing adaptive or robust sample selection/weighting, measuring and tracking coverage/sampling error, and integrating human feedback efficiently when available.