On-Policy Consistency Training (OPCT)

Updated 3 July 2026

OPCT is an optimization method in reinforcement learning that enforces self-consistency by evaluating groups of model outputs.
It blends local and global reward signals to address challenges like reward collapse, sample wastage, and poor generalization.
Empirical results in math, medical reasoning, and safety alignment demonstrate enhanced robustness and performance over traditional fine-tuning.

On-Policy Consistency Training (OPCT) is a class of optimization techniques in the alignment and reinforcement learning (RL) paradigm for LLMs and large multimodal models (LMMs). OPCT algorithms directly optimize, on-policy, for outcome-invariance or self-consistency over groups of responses sampled from the current model, typically by contrasting these outputs with those arising from clean or reference prompts or by explicitly reinforcing consistency among the model’s own rollouts. This approach addresses failures of off-policy supervised fine-tuning (SFT) and group-based RL methods, specifically tackling degradation under reward collapse, sample wastage, and poor generalization for alignment and safety-critical objectives (Han et al., 6 Aug 2025, Zhang et al., 1 Feb 2026, Han et al., 20 May 2026).

1. Theoretical Foundations of OPCT

OPCT operates in the on-policy RL regime. Core to the method is the formulation of the policy $\pi_\theta$ as a generative model over trajectories (complete model outputs or token sequences) conditioned on inputs $x$ or $(I, Q)$ (image, question), with rewards constructed to reflect correctness and/or logical consistency. For a batch of $B$ prompts or input pairs, each is associated with a group $\mathcal{O}_q$ or $\mathcal{G}_j$ of $G$ sampled outputs under the current model.

In mathematical reasoning settings, COPO defines a local group-based advantage using rule-based rewards $R(o_i) = \mathbf{1}_{\text{correct}}$ and group normalization:

$\hat A_i^{\text{local}} = \frac{R(o_i) - \frac{1}{G}\sum_{j=1}^G R(o_j)}{\sqrt{ \frac{1}{G}\sum_{j=1}^G (R(o_j) - \overline{R})^2 } + \epsilon}$

However, when all outputs in a group are correct (or incorrect), variance collapses, $\hat{A}_i^{\text{local}} \to 0$ , causing gradients to vanish and learning to halt.

OPCT introduces a structured global reward:

$x$ 0

and computes a global advantage over the batch:

$x$ 1

In LMMs for medical reasoning, a “consistency reward” $x$ 2 judges whether the model’s reasoning sequence justifies (via an external evaluator or similarity metric) its answer, directly optimizing for transparent CoT consistency (Zhang et al., 1 Feb 2026).

In alignment and safety domains, OPCT frames “consistency” as invariance under contrastive prompt perturbations (e.g., sycophancy, jailbreak, safety fact elision) by aligning the on-policy student distribution $x$ 3 to a frozen teacher’s output $x$ 4 over each sampled continuation (Han et al., 20 May 2026). This is implemented via a reverse KL objective over the model’s sampled responses.

2. Algorithms and Objective Structures

OPCT generally blends local (group-level) and global (batch-level) optimization signals. In COPO, this is realized by a soft entropy-based blending function:

For each prompt $x$ 5 with $x$ 6 unique outputs, consistency entropy $x$ 7 is given by:

$x$ 8

where $x$ 9 is the empirical frequency.

Blending weights:

$(I, Q)$ 0

interpolate between exploration (local advantage) and convergence (global consistency).

The overall surrogate objective in COPO is:

$(I, Q)$ 1

where $(I, Q)$ 2 is the usual importance ratio and $(I, Q)$ 3 is the KL weight (Han et al., 6 Aug 2025).

In MedAD-R1 (medical anomaly detection), OPCT is instantiated as Consistency Group Relative Policy Optimization (Con-GRPO), where each trajectory $(I, Q)$ 4 receives:

$(I, Q)$ 5 for factual accuracy
$(I, Q)$ 6 if the model’s reasoning process justifies its answer, 0 otherwise

with the final objective

$(I, Q)$ 7

and gradient

$(I, Q)$ 8

(Zhang et al., 1 Feb 2026).

For the alignment/safety regime, OPCT is formulated by aligning student and teacher per-token likelihoods across contrastive pairs:

$(I, Q)$ 9

This loss is minimized for every on-policy sample from $B$ 0 (Han et al., 20 May 2026).

3. Implementation Regimes and Training Procedures

OPCT implementations rely on multi-sample rollouts per prompt/input, group-based aggregation, and optionally external evaluators for reward construction. Procedures typically include:

Sampling $B$ 1 outputs (rollouts) per prompt for local and consistency reward estimation.
Group-based standardization and baseline subtraction for local advantage estimation.
Batch-level normalization for global consistency signals.
Entropy computation and adaptive blending for dynamic exploration-convergence tradeoff.

Key commonalities in the published recipes:

Setting	Rollouts per input	Optimizer	Policy Update	Baseline
COPO (reasoning, math) (Han et al., 6 Aug 2025)	G = 6	AdamW (lr $B$ 2)	PPO w/ KL regularizer	Group + batch mean
MedAD-R1 (medical reasoning) (Zhang et al., 1 Feb 2026)	G (small, e.g. 4)	AdamW	Con-GRPO/PPO-style	Group mean
OPCT (alignment/safety) (Han et al., 20 May 2026)	k (varies by task)	AdamW (via LoRA)	Gradient of KL divergence, on-policy	Teacher

Rollout parameters, group sizes, and blending hyperparameters (e.g., $B$ 3 in COPO) are typically tuned empirically. All methods require periodic synchronization of past policies (PPO “old policy” snapshot) to control importance ratio drift.

4. Comparative Analysis with Other Consistency Approaches

OPCT strategies directly address limitations of prior group-based RL approaches:

GRPO (Group Relative Policy Optimization) optimizes intra-group ranking but fails under reward collapse (identical outputs) (Han et al., 6 Aug 2025).
DAPO (Dynamic Advantage Policy Optimization) filters “all-correct” or “all-incorrect” groups, causing high sample wastage (>50%) and data inefficiency.
Off-policy consistency training (e.g., standard SFT on reference targets) induces memorization, poor out-of-distribution generalization, and capability collapse, especially in safety and alignment-sensitive regimes (Han et al., 20 May 2026).

OPCT, by supervision on the current model’s own rollouts (“on-policy”) and adaptive global-local blending, ensures continued gradient flow, promotes sample efficiency, and enables robust generalization beyond the training distribution.

In MedAD-R1, ablation results demonstrate that explicitly optimizing for consistency yields higher diagnostic performance than optimizing for accuracy alone or off-policy SFT, with the largest gains observed in cognitively demanding tasks (Zhang et al., 1 Feb 2026).

5. Empirical Results and Impact

OPCT has been empirically validated across mathematical reasoning, medical anomaly detection, and alignment/safety benchmarks.

Mathematics: On MATH-500 (mean@8), COPO achieves 60.38% (Qwen2.5 3B) and 65.80% (7B), outperforming GRPO by +4.55 and +2.22 percentage points and avoiding late-stage performance collapse (Han et al., 6 Aug 2025).
Medical Reasoning: MedAD-R1 (3B) reaches 85.15% on MedAD-38K overall, compared to ~77% for SFT-only baselines. Gains are especially pronounced on Anomaly Detection (+18.3%) and Lesion Localization (+18.4%) (Zhang et al., 1 Feb 2026).
Alignment/Safety: OPCT halves sycophancy rates compared to baseline SFT (e.g., 8.6% vs. 12.9% on Qwen3-8B). Under adversarial jailbreak attacks, defense rates remain near 99% (vs. 87% for SFT). Safety awareness improves on SAGE-Eval and Safe-SAGE (AUSC up to 0.78) with minimal if any capability degradation on general benchmarks like MATH-500 (Han et al., 20 May 2026).

Ablation studies confirm the necessity of global/batch-level consistency terms and adaptive blending; omitting these induces gradient vanishing or over-rewarding of degenerate samples. Optimizing for consistency alone can outperform accuracy-only rewards in some tasks.

6. Limitations and Open Problems

Several limitations are noted in OPCT implementations:

Teacher bottleneck: In the alignment paradigm, OPCT does not outperform its teacher; learning is restricted to the coverage and correctness of the reference model (Han et al., 20 May 2026).
Compute cost: On-policy regimes require multiple on-line rollouts per sample and, for teacher-guided OPCT, repeated teacher forward passes, increasing training compute relative to off-policy SFT.
Domain scope: Published results emphasize mathematics, medical reasoning, and safety alignment. Tasks such as code generation, conversational QA, and long-context reasoning remain untested (Zhang et al., 1 Feb 2026, Han et al., 20 May 2026).

Future work is suggested in reducing compute via importance sampling, enhancing reward signals via RL-based safety objectives, coverage expansion to new domains, and automatic curriculum learning for contrastive data selection.

7. Significance and Broader Implications

OPCT introduces a principled framework for learning robustness and self-consistency in LLMs and LMMs. By maintaining training strictly on the policy’s own output distribution and adaptively reinforcing invariants or logical coherence, OPCT achieves state-of-the-art results in domains demanding reliability, safety, and verifiability. This paradigm represents a systematic departure from static SFT alignment and opens avenues for advanced RL-based alignment protocols with provable generalization properties. The systematic avoidance of sample and gradient wastage, combined with adaptability to on-policy drift, constitutes the central empirical and theoretical advantage of OPCT over prior methods (Han et al., 6 Aug 2025, Zhang et al., 1 Feb 2026, Han et al., 20 May 2026).