---
title: 'ACPO: Adaptive Curriculum Policy Optimization'
url: https://www.emergentmind.com/topics/adaptive-curriculum-policy-optimization-acpo
type: topic
---

# ACPO: Adaptive Curriculum Policy Optimization

Adaptive Curriculum Policy Optimization (ACPO) denotes a line of policy optimization methods in which curriculum or optimization strength is dynamically tuned in response to evolving model competence. In the cited literature, this label covers mechanisms such as adaptive KL regularization, frontier-centered sample selection, adaptive clipping, staged reuse schedules, and automated problem curation, all designed to improve exploration, sample efficiency, training stability, or generalization. ACPO-style methods appear in online combinatorial optimization, autonomous driving, truck dispatching, reinforcement finetuning of large language models, vision-language model alignment, and text-to-image generation [2605.11403][2510.00690].

## 1. Foundations and scope

A theoretical precursor to ACPO appears in “Understanding Curriculum Learning in Policy Optimization for Online Combinatorial Optimization” [2202.05423]. That work formulates online combinatorial optimization problems as latent Markov Decision Processes and proves convergence bounds on natural policy gradient for solving LMDPs. Its central theoretical claim is that curriculum learning can find a strong sampling policy and reduce the distribution shift, a critical quantity that governs the convergence rate. For the Best Choice Problem, the paper formally proves that distribution shift is reduced exponentially with curriculum learning even if the curriculum is a randomly generated BCP on a smaller scale, and it further shows that multi-step curricula can be simplified to a single-step scheme [2202.05423].

A more explicit policy-optimization realization appears in “Curriculum Proximal Policy Optimization with Stage-Decaying Clipping for Self-Driving at Unsignalized Intersections” [2308.16445]. CPPO integrates curriculum learning into PPO by training across three stages, $\mathbf{\Omega}=\{\Omega_1,\Omega_2,\Omega_3\}$, and coupling stage progression to a decaying clipping parameter. The stated purpose is to let the vehicle first rapidly search for an approximate optimal policy or its neighborhood with a large parameter, and then converge to the optimal policy with a small one [2308.16445].

By 2025–2026, ACPO-style ideas became prominent in RLVR and related post-training settings. Several papers identify fixed KL coefficients, fixed clipping bounds, uniform question or prompt sampling, and static on-/off-policy schedules as core inefficiencies, then replace them with competence-aware controllers or curricula [2605.11403][2504.05520][2605.17807][2510.00690].

| Method | Adaptive signal | Intervention |
|---|---|---|
| FG-ExPO | Batch mean accuracy; EMA-smoothed pass-rate | KL scaling; Gaussian curriculum sampling |
| AdaRFT | Average reward; target difficulty | Batch selection near dynamic target difficulty |
| Actor-Curator | Estimated policy improvement utility | OSMD-trained neural curator |
| CGPO | Group reward variance; category reward | Prompt sampling; category calibration |
| ACPO for VLMs | Training step; normalized advantage | Sample reuse schedule; adaptive clipping |

## 2. Frontier-centered data curricula

A defining ACPO pattern is to target data near the model’s current learning frontier rather than to rely on uniform sampling. In FG-ExPO, Gaussian Curriculum Sampling assigns sampling weights to questions following a Gaussian distribution centered at a moderate accuracy level around $0.5$, focusing training on the model’s learning frontier [2605.11403]. Each question $q$ tracks an EMA-smoothed pass-rate
\[
\tilde{p}_q^{(t)}=\alpha \tilde{p}_q^{(t-1)}+(1-\alpha)p_q^{(t)},
\]
and the sampling weight is
\[
w_q^{(t)}=\exp\left(-\frac{(\tilde{p}_q^{(t)}-0.5)^2}{2\sigma^2}\right).
\]
Very easy questions with pass rate near $1$ and very hard questions with pass rate near $0$ are smoothly downweighted, while frontier questions with pass rate near $0.5$ are upweighted [2605.11403].

AdaRFT adopts a related but scalar target-difficulty formulation. It maintains a target difficulty $T$ and updates it from recent batch reward:
\[
T'=\mathrm{clip}\left(T+\eta\cdot \tanh\left(\alpha\cdot (R_{\text{avg}}-\beta)\right),\, d_{\min}, d_{\max}\right).
\]
At each step, the algorithm computes $\Delta_i=|d_i-T|$ and selects the batch of $B$ problems closest to the current target difficulty [2504.05520]. If $R_{\text{avg}}>\beta$, difficulty is increased; if $R_{\text{avg}}<\beta$, difficulty is decreased. This implements a “challenging but solvable” regime without changing the reward function, policy optimizer, or model architecture [2504.05520].

In text-to-image RL, CGPO uses group reward variance as an online proxy for prompt inconsistency. For a prompt $p$ with a group of images scored by a reward model,
\[
V_p=\mathrm{Var}(\{R(x_1),\ldots,R(x_G)\})=\frac{1}{G}\sum_{i=1}^G (R(x_i)-\mu_x)^2.
\]
High variance indicates that the model has partially captured the prompt requirements but has not yet achieved stable mastery, so such prompts are sampled more heavily [2605.17807]. CGPO also adds category calibration based on proportional fairness optimization, with
\[
q_i=\frac{1+\lambda v_i}{c+\lambda},
\]
to address imbalance across prompt categories [2605.17807].

SPO reaches a similar frontier effect through a persistent value tracker rather than group baselines. For each prompt $x$, it estimates a success probability $\hat v(x)$ and samples prompts with weight
\[
w_i(x)\propto \sqrt{\hat v_{-1}(x)(1-\hat v_{-1}(x))}+\epsilon.
\]
This prioritizes prompts with highest Bernoulli uncertainty, which again peaks near half-solved instances [2509.13232]. A plausible implication is that ACPO in current practice is usually not a simple easy-to-hard schedule; it is more often a frontier-centered allocation rule based on uncertainty, inconsistency, or recent success.

## 3. Adaptive regularization and optimization schedules

A second ACPO axis concerns how aggressively the policy is allowed to move. FG-ExPO’s Accuracy-Conditioned KL Scaling replaces a fixed KL coefficient with
\[
\beta_{\text{eff}}(a)=\beta\cdot \rho(a), \qquad \rho(a)=\frac{\tanh(a)+1}{2}, \quad a\in[0,1],
\]
where $a$ is batch mean accuracy [2605.11403]. When the model struggles, $\beta_{\text{eff}}$ is reduced, allowing greater deviation from the reference; when the model succeeds, $\beta_{\text{eff}}$ is increased, tying the policy more tightly to the reference and preventing overfitting or catastrophic forgetting [2605.11403].

CPPO expresses the same principle through PPO clipping rather than KL scaling. Its stage-decaying clipping uses
\[
\varepsilon=
\begin{cases}
0.25, & \text{if in } \Omega_1\\
0.20, & \text{if in } \Omega_2\\
0.15, & \text{if in } \Omega_3
\end{cases}
\]
inside the PPO clipped surrogate objective
\[
J_{\text{clip}}(\theta)=\mathbb{E}_t\left[\min\left(\rho_t(\theta)\hat A_t,\operatorname{clip}(\rho_t(\theta),1-\varepsilon,1+\varepsilon)\hat A_t\right)\right].
\]
Large $\varepsilon$ in early curriculum stages enables rapid exploration and large policy updates; smaller $\varepsilon$ in later stages stabilizes training and ensures fine policy refinement [2308.16445].

A more explicit formulation appears in the vision-language paper titled “ACPO: Adaptive Curriculum Policy Optimization for Aligning Vision-Language Models in Complex Reasoning” [2510.00690]. There, the curriculum orchestrates a transition from on-policy exploration to off-policy exploitation by progressively increasing sample reuse:
\[
K(t)=\max\left(1,\left\lceil \frac{N\cdot t}{T}\right\rceil\right).
\]
The same framework introduces Advantage-Aware Adaptive Clipping, in which the upper clipping bound becomes
\[
\epsilon_{\text{high}}(\hat A_{i,t})=\epsilon_{\text{high}}^0+\delta\cdot \tilde A_{i,t},
\]
with normalized advantage
\[
\tilde A_{i,t}=\frac{1}{2}\left(1+\operatorname{erf}\left(\frac{\hat A_{i,t}}{\sqrt{2}\sigma_A}\right)\right).
\]
High-advantage tokens receive a wider clipping range, whereas low- or negative-advantage tokens are more tightly constrained [2510.00690].

AGPO generalizes this adaptive-control view by using a shared probe-derived statistical state to control both clipping and sampling temperature. Its adaptive clipping controller is
\[
\varepsilon_{\mathrm{adaptive}}=\mathrm{clip}\left(
\varepsilon_{\text{base}}\,
\frac{H(\pi_{\theta_{\text{old}}})\cdot (1+\delta\,\mathcal{E}_{\text{probe}})}
{1+\alpha\,\hat{\sigma}+\zeta\,|\tilde{\kappa}_3(r)|+\gamma\,\widehat{D}_{\text{KL}}^{\text{step}}},
\varepsilon_{\min},\varepsilon_{\max}\right),
\]
while its adaptive temperature sampling centers batch uncertainty $U_t$ against a running baseline and sets
\[
\tau_t=\mathrm{clip}\left(\tau_{\text{base}}\cdot (1+\lambda \widetilde U_t),\,\tau_{\min},\tau_{\max}\right).
\]
This places exploration control and trust-region control under the same adaptive statistics [2605.20722].

## 4. Co-adaptive curation, restructuring, and teacher guidance

Some ACPO variants move beyond reweighting to learn a separate curator or to alter the training problem itself. Actor-Curator formulates problem selection as a non-stationary stochastic bandit problem and trains a neural curator to directly optimize expected policy performance improvement [2602.20532]. Its central learning signal is
\[
u^t=J(\pi^{t+1})-J(\pi^t),
\]
and its tabular OSMD update has the form
\[
p^{t+1}(x)\propto p^t(x)\exp(\eta \hat U_x^t).
\]
The framework uses a two-stage scheme in practice: sample a candidate batch from a proposal and then reweight or select via the curator, with partial-feedback regret guarantees [2602.20532].

CLPO introduces what the paper calls a dynamic pedagogical feedback loop. For each problem $q$, the current policy generates $G$ solutions and computes empirical accuracy
\[
\mathrm{Acc}(q,\pi_\theta)=\frac{1}{G}\sum_{i=1}^G \mathbb{I}(\mathrm{Verifier}(y_i,a)=\mathrm{True}).
\]
Problems are partitioned by thresholds into hard and medium cases, after which an Adaptive Problem Restructuring mechanism simplifies hard problems and diversifies medium-difficulty problems [2509.25004]. CLPO also applies difficulty-aware KL regularization, using lower regularization on hard problems and higher regularization otherwise [2509.25004]. This is a stronger intervention than mere resampling: the curriculum changes the effective training distribution by rewriting problems into forms that remain “not mastered but not impossible” [2509.25004].

In truck dispatching, Curriculum-inspired Adaptive Direct Policy Guidance implements curriculum through a teacher policy rather than through environment stages. A Shortest Processing Time teacher policy regularizes the student with
\[
\text{GuideLoss}=\frac{1}{N}\sum_{i=1}^{N}\log \pi_\theta(a_i^{\text{sug}}|s_i),
\]
and the guidance coefficient is
\[
\text{guide\_coef}=
\begin{cases}
0, & \text{if tons}\ge \text{base\_tons}\\
\alpha(1-c_{\text{teacher}}), & \text{otherwise}
\end{cases}.
\]
The total loss is
\[
\text{Total Loss}=\text{PPO Loss}+\text{guide\_coef}\times \text{GuideLoss}.
\]
Early training is heavily regularized toward the teacher; guidance then fades automatically once the student matches or surpasses the baseline [2502.20845].

## 5. Theoretical analyses

The most explicit ACPO-oriented theory in the cited literature concerns distribution shift and non-stationary selection. In the LMDP analysis of online combinatorial optimization, the natural policy gradient convergence bound depends on a relative condition number $\kappa$, which captures the distribution shift between the current or sampling policy and the optimal policy. The paper’s main theorem gives
\[
\mathbb{E}\left[\min_{0\le t\le T} V^\star-V^t\right]
\le
\frac{\Phi(\pi_0)}{\eta T}
+\eta \frac{B^2G^2}{2}
+\sqrt{H\epsilon_{\textup{bias}}}
+\sqrt{H\kappa \epsilon_{\textup{stat}}}.
\]
For the Best Choice Problem, the paper proves that curriculum can reduce $\kappa$ exponentially relative to naive sampling, which explains the acceleration effect [2202.05423].

Actor-Curator contributes a different theoretical strand. Because each problem arm has non-stationary utility and only sampled problems reveal feedback, the paper analyzes dynamic regret and establishes
\[
\mathrm{Reg}_T \le O(T^{2/3}V_T^{1/3}),
\]
where
\[
V_T=\sum_{t=2}^{T}\max_{x\in \mathcal{X}} |u_x^t-u_x^{t-1}|
\]
measures the non-stationarity of per-arm utility over time [2602.20532]. This result is presented as optimal for non-stationary online bandits and provides a formal account of adaptive curriculum design under partial feedback.

These two analyses suggest two complementary theoretical views of ACPO. One view treats curriculum as a way to reduce harmful distribution mismatch during policy optimization; the other treats curriculum as a sequential decision problem whose objective is cumulative policy improvement. Both are consistent with later empirical ACPO systems that repeatedly estimate competence and then allocate sampling or update budget accordingly.

## 6. Empirical behavior, applications, and terminological issues

Empirical results across the cited literature consistently report gains from adaptive curricula over fixed schedules. FG-ExPO, evaluated on DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-8B-Base across six mainstream mathematical reasoning benchmarks, reports an absolute improvement of $13.34$ on the AIME 2025 pass@32 metric, rising from $63.33$ percent to $76.67$ percent, and an average pass@32 gain of $2.66$ on the 8B model [2605.11403]. Actor-Curator reports relative gains of $28.6\%$ on AIME2024 and $30.5\%$ on ARC-1D over the strongest baseline and up to $80\%$ speedup [2602.20532]. AdaRFT reports up to $2\times$ reduction in training time and improved accuracy across AMC, AIME, and IMO-style problems [2504.05520]. CGPO reports an overall score of $0.96$ on GenEval, compared with $0.94$ for Flow-GRPO, and reaches high-performance benchmarks in about half the GPU hours of Flow-GRPO [2605.17807]. SPO reports an average maj@32 improvement of $+3.4$ percentage points over GRPO across five hard math benchmarks, including $+7.3$ points on BRUMO 25 and $+4.4$ on AIME 25 [2509.13232]. In autonomous driving, CPPO completes training in $3$ hr $34$ min, compared with $6$ hr $45$ min for PPO with fixed $\varepsilon=0.15$ and $5$ hr $58$ min for PPO with fixed $\varepsilon=0.25$ [2308.16445].

A recurrent misconception is that ACPO is synonymous with a fixed easy-to-hard curriculum. The cited methods do not support that reduction. They instead emphasize moderately difficult questions with pass rate near $0.5$, prompts with high reward variance, batches closest to a dynamically updated target difficulty, or samples whose normalized advantage warrants broader clipping. This suggests that the contemporary ACPO literature is organized less by a single schedule template than by a general principle: learning should concentrate where policy improvement is currently most likely.

A second source of confusion is terminological. “ACPO” is also used for several unrelated acronyms: “Adaptive Cognition Policy Optimization” for efficient hybrid reasoning in LRMs [2505.16315], “Attribution-based Contribution to Policy Optimization” for step-level credit assignment in RLVR [2510.08899], “Average-Constrained Policy Optimization” for average-reward CMDPs [2302.00808], “Adversarial Constrained Policy Optimization” for adapting cost budgets in constrained RL [2410.20786], and “Automatic Constraint Policy Optimization” for offline RL under a continuous constraint interpolation framework [2601.23010]. In encyclopedia usage, “Adaptive Curriculum Policy Optimization” therefore refers not to a single universally standardized algorithm, but to a family of competence-adaptive policy optimization methods whose common feature is the joint shaping of training data exposure and update dynamics.

Source: https://www.emergentmind.com/topics/adaptive-curriculum-policy-optimization-acpo