On-Policy Data Generation Techniques

• On-policy data generation is the process of collecting trajectories using the current policy, ensuring the samples accurately reflect the model’s behavior.
• It enables unbiased, well-conditioned learning signals by using fresh, state-dependent data that captures real-time policy performance.
• Practical approaches include pipelined actor-trainer schemes, adaptive policies, and on-policy corrections that balance sample efficiency with learning stability.

On-policy data generation refers to the process of producing data—typically in the form of trajectories or samples—that are distributed according to a currently deployed or learned policy in a sequential decision-making system such as reinforcement learning (RL) or sequential generative modeling. In contrast to off-policy data, which is generated independently of the current policy (e.g., from previous policies or external agents), on-policy data is synchronized with the behavior of the agent as it is updated, ensuring that learning signals reflect the most recent model parameters. This property is foundational for controlling distributional mismatch, enabling rigorous theoretical guarantees, and stabilizing policy improvement in diverse domains including RL, language modeling, and generative modeling.

1. Definitions and Theoretical Foundations

Formally, an "on-policy" dataset for a policy πθ in an MDP or sequential generative process consists of samples (state, action, reward, …) tuples drawn by executing πθ in the environment, i.e., (s_t, a_t, r_t, s_{t+1}) with a_t ∼ πθ(·|s_t). The empirical distribution of such data approximates the true visitation distribution induced by πθ, denoted d^{π_θ}(s,a). On-policy data is essential for learning algorithms that require unbiased, well-conditioned estimates of value functions, gradients, or surrogate objectives referencing the current policy distribution. The classical Kakade-Achiam policy improvement lemma and generalizations establish that if policy update steps are localized (small KL or TV divergence), improvements in expected return can be guaranteed provided updates are computed from on-policy data (Queeney et al., 2021, Queeney et al., 2022).

However, the empirical data distribution garnered from a finite set of trajectories may deviate from the ideal on-policy distribution, leading to sampling error. Techniques such as Robust On-Policy Sampling (ROS) formally address this gap by adapting the data-collection policy to minimize the KL divergence between empirical and target distributions, achieving faster convergence rates in the estimation error (Zhong et al., 2021). Similarly, in practical RLHF (reinforcement learning from human feedback) for LLM fine-tuning and preference-based vision-LLMs, it is explicitly noted that only genuinely on-policy samples allow direct correction or suppression of the model's current dominant error modes, such as hallucinations (Tang et al., 25 Mar 2025, Yu et al., 30 Nov 2025).

2. Principles and Methodologies of On-Policy Data Generation

On-policy data generation workflows can be instantiated in several canonical forms:

• Agent-Environment Loop: At each iteration, the current policy π_t is deployed in the environment, collecting new episodes or transitions used for policy evaluation or improvement. After each update, all prior data is treated as off-policy and not reused, as in standard PPO/TRPO (Queeney et al., 2021).
• Asynchronous and Pipelined Generation: In large-scale LLM RL fine-tuning, systems such as PipelineRL interleave concurrent sequence generation ("actors") and policy optimization ("trainers"), using rapid in-flight synchronization of policy parameters to ensure generators track policy updates closely. Data staleness is measured using parameter-space distance or effective sample size (ESS), with ESS ≈ 1 corresponding to ideal on-policyness (Piché et al., 23 Sep 2025).
• Corrected Model-Based Rollouts: In model-based RL, hybrid methods employ real environment data for anchoring rollouts and apply time-dependent on-policy corrections to learned model predictions, thereby mitigating bias from compounding model error. This technique, exemplified by on-policy corrections (OPC), ensures synthetic rollouts align with the current policy's local distribution (Fröhlich et al., 2021).
• Adaptive Behavior Policies: PROPS and ROS employ an auxiliary behavior policy πb, adaptively optimized to reduce mismatches between the empirical buffer distribution and the true on-policy distribution πθ, with explicit regularizers on the divergence D_{KL}(πb‖πθ) (Corrado et al., 2023, Zhong et al., 2021).
• Sequential Generative Rollouts: In generative modeling, sequential latent-variable models parameterized as policies (e.g., in data imputation or guided diffusion) execute on-policy rollouts of latent or observable variables, directly generating new data from the learned policy at each step. This is formalized as an MDP and trained via guided policy search (Bachman et al., 2015, Jackson et al., 2024).

3. Algorithmic Realizations and Practical Implementations

The implementation of on-policy data generation varies by context:

• Standard On-Policy RL: At each update step, collect n fresh transitions using the current policy, discard old data after gradient ascent, and restrict updates to data sampled under the immediate policy (Queeney et al., 2021).
• Sample Reuse and Trust Region Control: Modern algorithms such as Generalized Policy Improvement (GPI), GePPO, and related frameworks blend recent on-policy batches with a small window of past data, applying importance weighting and rigorous trust-region constraints to preserve the stability and performance guarantees of pure on-policy approaches while improving sample efficiency (Queeney et al., 2022, Queeney et al., 2021).
• Pipelined Actor-Trainer Schemes: In highly parallelized environments (e.g., distributed LLM RL), actors generate data with "in-flight" weights frequently synchronized with the central trainer's most recent parameters, as in PipelineRL. On-policyness is preserved by bounding the staleness metric and maintaining ESS close to 1, with trade-offs between hardware utilization and strict on-policyness (Piché et al., 23 Sep 2025).
• Correction of Model-Based Rollouts: OPC applies a data-driven correction to model-predicted next states by anchoring rollouts in real transitions combined with a mean prediction offset, avoiding error accumulation and sustaining on-policy fidelity even with learned models (Fröhlich et al., 2021).
• Preference-Based Data Collection in LLMs and LVLMs: For alignment tasks, on-policy samples are generated via rollouts from the current policy, assessed and filtered (e.g., using hallucination classifiers) to construct "preference datasets" that enable effective suppression of current policy error patterns, outperforming off-policy-based preference optimization (Tang et al., 25 Mar 2025, Yu et al., 30 Nov 2025).

4. Empirical Properties and Comparative Analysis

Experiments consistently find that on-policy data generation yields superior policy improvement reliability, sample efficiency (under appropriate reuse), and error suppression capability:

• On-policy AGRO, when compared to standard RLHF and off-policy AGRO variants, demonstrates a 7% faster evaluation accuracy improvement and up to +7% test accuracy uplift under on-policy supervised trajectories for Llama-3-8B fine-tuning (Tang et al., 25 Mar 2025).
• In vision-language hallucination mitigation, on-policy iterative preference optimization combined with filtered data and boundary-focused reweighting reduces hallucination rates by 50.8% (MMHalBench) and 79.5% (Object HalBench), outperforming equivalent off-policy approaches (Yu et al., 30 Nov 2025).
• On-policy corrections in model-based RL outperform standard MBPO in sample efficiency and learning stability, especially under simulator mismatch and high-variance regimes, without extra hyperparameters (Fröhlich et al., 2021).
• PipelineRL achieves a ∼2× reduction in wall-clock training time and maintains ESS ≈ 0.9 compared to 0.4–0.6 for conventional batch sizes, thereby delivering near on-policy guarantees at high hardware utilization (Piché et al., 23 Sep 2025).
• Adaptive sampling methods such as ROS and PROPS provide provably faster convergence of data distribution to true on-policy, with KL error decay as O(1/m²) versus O(1/m) for i.i.d. sampling, leading to lower mean squared error in policy evaluation and up to 50% reductions in environment steps for control benchmarks (Corrado et al., 2023, Zhong et al., 2021).

Comparison Table: On-Policy Data Generation Methods

Method	Core Mechanism	Notable Advantage
RLHF/AGRO	Direct on-policy rollouts	Fast, stable fine-tuning (Tang et al., 25 Mar 2025)
GPI/GePPO	Sample reuse with trust regions	Sample efficiency, stability (Queeney et al., 2022)
OPC	On-policy model correction	Model bias mitigation (Fröhlich et al., 2021)
PipelineRL	Asynchronous pipelined updates	High throughput, ESS ≈ 1 (Piché et al., 23 Sep 2025)
ROS/PROPS	Adaptive empirical correction	Fast sampling error decay (Corrado et al., 2023, Zhong et al., 2021)
Policy-guided Diffusion	Trajectory-level diffusion guidance	On-policy synthetic data in offline RL (Jackson et al., 2024)

5. Broader Impact, Limitations, and Trade-offs

On-policy data generation underpins key algorithmic guarantees in RL, LLM alignment, and generative modeling, specifically in controlling the distributional support and suppressing learned error modes. However, it is intrinsically less sample efficient than off-policy learning due to discarding or underutilizing past experiences unless principled reuse is applied. Reuse (as in GPI, GePPO) is effective but requires rigorous control of trust-region violations and careful weighting to avoid bias. Asynchronous pipelining (PipelineRL) offers hardware efficiency but introduces token staleness, manageable through frequent synchronization and bounded lag. Model-based and synthetic approaches (OPC, policy-guided diffusion) must control compounding model errors or hallucinations but can produce on-policy-like synthetic data, increasing coverage without direct environmental interaction.

Limitations include sensitivity to rapid policy change (which invalidates buffered or synthetic data), requirement of trustworthy filtering/classification for preference-based learning, and environment settings (e.g., POMDPs, stochastic noise) impacting the effectiveness of on-policy correction or reuse strategies (Fröhlich et al., 2021, Yu et al., 30 Nov 2025). Theoretical analyses require assumptions of smoothness, bounded variance, or slowly evolving policies.

6. Recent Innovations and Outlook

Recent work emphasizes bridging the strict dichotomy between on-policy and off-policy paradigms via theoretically principled sample reuse, empirical KL minimization, and hybrid offline/online synthetic data generation. Dynamic sample reweighting, classifier filtering, diffusion-guided synthesis, and adaptive actor-trainer architectures exemplify the growing sophistication and flexibility of on-policy data generation systems.

Empirical and theoretical findings indicate that on-policy data—whether from real environment rollouts, adaptive behavior policies, or properly guided simulation—remains crucial for maximizing return improvement guarantees and addressing distribution shift, even as hardware bottlenecks and data constraints intensify with model scale. Ongoing directions include generalizing strong finite-sample error bounds to high-dimensional, function-approximate policies, automatic adaptation of sample reuse windows, and integrating robust on-policy sampling into advanced RL pipelines for large-scale, multi-modal systems (Piché et al., 23 Sep 2025, Yu et al., 30 Nov 2025, Zhong et al., 2021).