ACPO: Adaptive Curriculum Policy Optimization

• Adaptive Curriculum Policy Optimization is a strategy that dynamically adjusts training data and optimization parameters based on a model’s evolving competence.
• It employs adaptive techniques such as KL scaling, adaptive clipping, and frontier-centered sampling to reduce distribution shift and enhance convergence.
• ACPO methods have shown significant efficiency and performance gains in applications like autonomous driving, vision-language alignment, and text-to-image generation.

Adaptive Curriculum Policy Optimization (ACPO) denotes a line of policy optimization methods in which curriculum or optimization strength is dynamically tuned in response to evolving model competence. In the cited literature, this label covers mechanisms such as adaptive KL regularization, frontier-centered sample selection, adaptive clipping, staged reuse schedules, and automated problem curation, all designed to improve exploration, sample efficiency, training stability, or generalization. ACPO-style methods appear in online combinatorial optimization, autonomous driving, truck dispatching, reinforcement finetuning of LLMs, vision-LLM alignment, and text-to-image generation (Lin et al., 12 May 2026, Wang et al., 1 Oct 2025).

1. Foundations and scope

A theoretical precursor to ACPO appears in “Understanding Curriculum Learning in Policy Optimization for Online Combinatorial Optimization” (Zhou et al., 2022). That work formulates online combinatorial optimization problems as latent Markov Decision Processes and proves convergence bounds on natural policy gradient for solving LMDPs. Its central theoretical claim is that curriculum learning can find a strong sampling policy and reduce the distribution shift, a critical quantity that governs the convergence rate. For the Best Choice Problem, the paper formally proves that distribution shift is reduced exponentially with curriculum learning even if the curriculum is a randomly generated BCP on a smaller scale, and it further shows that multi-step curricula can be simplified to a single-step scheme (Zhou et al., 2022).

A more explicit policy-optimization realization appears in “Curriculum Proximal Policy Optimization with Stage-Decaying Clipping for Self-Driving at Unsignalized Intersections” (Peng et al., 2023). CPPO integrates curriculum learning into PPO by training across three stages, $\mathbf{\Omega}=\{\Omega_1,\Omega_2,\Omega_3\}$, and coupling stage progression to a decaying clipping parameter. The stated purpose is to let the vehicle first rapidly search for an approximate optimal policy or its neighborhood with a large parameter, and then converge to the optimal policy with a small one (Peng et al., 2023).

By 2025–2026, ACPO-style ideas became prominent in RLVR and related post-training settings. Several papers identify fixed KL coefficients, fixed clipping bounds, uniform question or prompt sampling, and static on-/off-policy schedules as core inefficiencies, then replace them with competence-aware controllers or curricula (Lin et al., 12 May 2026, Shi et al., 7 Apr 2025, Li et al., 18 May 2026, Wang et al., 1 Oct 2025).

Method	Adaptive signal	Intervention
FG-ExPO	Batch mean accuracy; EMA-smoothed pass-rate	KL scaling; Gaussian curriculum sampling
AdaRFT	Average reward; target difficulty	Batch selection near dynamic target difficulty
Actor-Curator	Estimated policy improvement utility	OSMD-trained neural curator
CGPO	Group reward variance; category reward	Prompt sampling; category calibration
ACPO for VLMs	Training step; normalized advantage	Sample reuse schedule; adaptive clipping

2. Frontier-centered data curricula

A defining ACPO pattern is to target data near the model’s current learning frontier rather than to rely on uniform sampling. In FG-ExPO, Gaussian Curriculum Sampling assigns sampling weights to questions following a Gaussian distribution centered at a moderate accuracy level around $0.5$, focusing training on the model’s learning frontier (Lin et al., 12 May 2026). Each question $q$ tracks an EMA-smoothed pass-rate

$$\tilde{p}_q^{(t)}=\alpha \tilde{p}_q^{(t-1)}+(1-\alpha)p_q^{(t)},$$

and the sampling weight is

$$w_q^{(t)}=\exp\left(-\frac{(\tilde{p}_q^{(t)}-0.5)^2}{2\sigma^2}\right).$$

Very easy questions with pass rate near $1$ and very hard questions with pass rate near $0$ are smoothly downweighted, while frontier questions with pass rate near $0.5$ are upweighted (Lin et al., 12 May 2026).

AdaRFT adopts a related but scalar target-difficulty formulation. It maintains a target difficulty $T$ and updates it from recent batch reward: $$T'=\mathrm{clip}\left(T+\eta\cdot \tanh\left(\alpha\cdot (R_{\text{avg}}-\beta)\right),\, d_{\min}, d_{\max}\right).$$ At each step, the algorithm computes $0.5$0 and selects the batch of $0.5$1 problems closest to the current target difficulty (Shi et al., 7 Apr 2025). If $0.5$2, difficulty is increased; if $0.5$3, difficulty is decreased. This implements a “challenging but solvable” regime without changing the reward function, policy optimizer, or model architecture (Shi et al., 7 Apr 2025).

In text-to-image RL, CGPO uses group reward variance as an online proxy for prompt inconsistency. For a prompt $0.5$4 with a group of images scored by a reward model,

$0.5$5

High variance indicates that the model has partially captured the prompt requirements but has not yet achieved stable mastery, so such prompts are sampled more heavily (Li et al., 18 May 2026). CGPO also adds category calibration based on proportional fairness optimization, with

$0.5$6

to address imbalance across prompt categories (Li et al., 18 May 2026).

SPO reaches a similar frontier effect through a persistent value tracker rather than group baselines. For each prompt $0.5$7, it estimates a success probability $0.5$8 and samples prompts with weight

$0.5$9

This prioritizes prompts with highest Bernoulli uncertainty, which again peaks near half-solved instances (Xu et al., 16 Sep 2025). A plausible implication is that ACPO in current practice is usually not a simple easy-to-hard schedule; it is more often a frontier-centered allocation rule based on uncertainty, inconsistency, or recent success.

3. Adaptive regularization and optimization schedules

A second ACPO axis concerns how aggressively the policy is allowed to move. FG-ExPO’s Accuracy-Conditioned KL Scaling replaces a fixed KL coefficient with

$q$0

where $q$1 is batch mean accuracy (Lin et al., 12 May 2026). When the model struggles, $q$2 is reduced, allowing greater deviation from the reference; when the model succeeds, $q$3 is increased, tying the policy more tightly to the reference and preventing overfitting or catastrophic forgetting (Lin et al., 12 May 2026).

CPPO expresses the same principle through PPO clipping rather than KL scaling. Its stage-decaying clipping uses

$q$4

inside the PPO clipped surrogate objective

$q$5

Large $q$6 in early curriculum stages enables rapid exploration and large policy updates; smaller $q$7 in later stages stabilizes training and ensures fine policy refinement (Peng et al., 2023).

A more explicit formulation appears in the vision-language paper titled “ACPO: Adaptive Curriculum Policy Optimization for Aligning Vision-LLMs in Complex Reasoning” (Wang et al., 1 Oct 2025). There, the curriculum orchestrates a transition from on-policy exploration to off-policy exploitation by progressively increasing sample reuse: $q$8 The same framework introduces Advantage-Aware Adaptive Clipping, in which the upper clipping bound becomes

$q$9

with normalized advantage

$$\tilde{p}_q^{(t)}=\alpha \tilde{p}_q^{(t-1)}+(1-\alpha)p_q^{(t)},$$0

High-advantage tokens receive a wider clipping range, whereas low- or negative-advantage tokens are more tightly constrained (Wang et al., 1 Oct 2025).

AGPO generalizes this adaptive-control view by using a shared probe-derived statistical state to control both clipping and sampling temperature. Its adaptive clipping controller is

$$\tilde{p}_q^{(t)}=\alpha \tilde{p}_q^{(t-1)}+(1-\alpha)p_q^{(t)},$$1

while its adaptive temperature sampling centers batch uncertainty $$\tilde{p}_q^{(t)}=\alpha \tilde{p}_q^{(t-1)}+(1-\alpha)p_q^{(t)},$$2 against a running baseline and sets

$$\tilde{p}_q^{(t)}=\alpha \tilde{p}_q^{(t-1)}+(1-\alpha)p_q^{(t)},$$3

This places exploration control and trust-region control under the same adaptive statistics (Hu et al., 20 May 2026).

4. Co-adaptive curation, restructuring, and teacher guidance

Some ACPO variants move beyond reweighting to learn a separate curator or to alter the training problem itself. Actor-Curator formulates problem selection as a non-stationary stochastic bandit problem and trains a neural curator to directly optimize expected policy performance improvement (Gu et al., 24 Feb 2026). Its central learning signal is

$$\tilde{p}_q^{(t)}=\alpha \tilde{p}_q^{(t-1)}+(1-\alpha)p_q^{(t)},$$4

and its tabular OSMD update has the form

$$\tilde{p}_q^{(t)}=\alpha \tilde{p}_q^{(t-1)}+(1-\alpha)p_q^{(t)},$$5

The framework uses a two-stage scheme in practice: sample a candidate batch from a proposal and then reweight or select via the curator, with partial-feedback regret guarantees (Gu et al., 24 Feb 2026).

CLPO introduces what the paper calls a dynamic pedagogical feedback loop. For each problem $$\tilde{p}_q^{(t)}=\alpha \tilde{p}_q^{(t-1)}+(1-\alpha)p_q^{(t)},$$6, the current policy generates $$\tilde{p}_q^{(t)}=\alpha \tilde{p}_q^{(t-1)}+(1-\alpha)p_q^{(t)},$$7 solutions and computes empirical accuracy

$$\tilde{p}_q^{(t)}=\alpha \tilde{p}_q^{(t-1)}+(1-\alpha)p_q^{(t)},$$8

Problems are partitioned by thresholds into hard and medium cases, after which an Adaptive Problem Restructuring mechanism simplifies hard problems and diversifies medium-difficulty problems (Zhang et al., 29 Sep 2025). CLPO also applies difficulty-aware KL regularization, using lower regularization on hard problems and higher regularization otherwise (Zhang et al., 29 Sep 2025). This is a stronger intervention than mere resampling: the curriculum changes the effective training distribution by rewriting problems into forms that remain “not mastered but not impossible” (Zhang et al., 29 Sep 2025).

In truck dispatching, Curriculum-inspired Adaptive Direct Policy Guidance implements curriculum through a teacher policy rather than through environment stages. A Shortest Processing Time teacher policy regularizes the student with

$$\tilde{p}_q^{(t)}=\alpha \tilde{p}_q^{(t-1)}+(1-\alpha)p_q^{(t)},$$9

and the guidance coefficient is

$$w_q^{(t)}=\exp\left(-\frac{(\tilde{p}_q^{(t)}-0.5)^2}{2\sigma^2}\right).$$0

The total loss is

$$w_q^{(t)}=\exp\left(-\frac{(\tilde{p}_q^{(t)}-0.5)^2}{2\sigma^2}\right).$$1

Early training is heavily regularized toward the teacher; guidance then fades automatically once the student matches or surpasses the baseline (Meng et al., 28 Feb 2025).

5. Theoretical analyses

The most explicit ACPO-oriented theory in the cited literature concerns distribution shift and non-stationary selection. In the LMDP analysis of online combinatorial optimization, the natural policy gradient convergence bound depends on a relative condition number $$w_q^{(t)}=\exp\left(-\frac{(\tilde{p}_q^{(t)}-0.5)^2}{2\sigma^2}\right).$$2, which captures the distribution shift between the current or sampling policy and the optimal policy. The paper’s main theorem gives

$$w_q^{(t)}=\exp\left(-\frac{(\tilde{p}_q^{(t)}-0.5)^2}{2\sigma^2}\right).$$3

For the Best Choice Problem, the paper proves that curriculum can reduce $$w_q^{(t)}=\exp\left(-\frac{(\tilde{p}_q^{(t)}-0.5)^2}{2\sigma^2}\right).$$4 exponentially relative to naive sampling, which explains the acceleration effect (Zhou et al., 2022).

Actor-Curator contributes a different theoretical strand. Because each problem arm has non-stationary utility and only sampled problems reveal feedback, the paper analyzes dynamic regret and establishes

$$w_q^{(t)}=\exp\left(-\frac{(\tilde{p}_q^{(t)}-0.5)^2}{2\sigma^2}\right).$$5

where

$$w_q^{(t)}=\exp\left(-\frac{(\tilde{p}_q^{(t)}-0.5)^2}{2\sigma^2}\right).$$6

measures the non-stationarity of per-arm utility over time (Gu et al., 24 Feb 2026). This result is presented as optimal for non-stationary online bandits and provides a formal account of adaptive curriculum design under partial feedback.

These two analyses suggest two complementary theoretical views of ACPO. One view treats curriculum as a way to reduce harmful distribution mismatch during policy optimization; the other treats curriculum as a sequential decision problem whose objective is cumulative policy improvement. Both are consistent with later empirical ACPO systems that repeatedly estimate competence and then allocate sampling or update budget accordingly.

6. Empirical behavior, applications, and terminological issues

Empirical results across the cited literature consistently report gains from adaptive curricula over fixed schedules. FG-ExPO, evaluated on DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-8B-Base across six mainstream mathematical reasoning benchmarks, reports an absolute improvement of $$w_q^{(t)}=\exp\left(-\frac{(\tilde{p}_q^{(t)}-0.5)^2}{2\sigma^2}\right).$$7 on the AIME 2025 pass@32 metric, rising from $$w_q^{(t)}=\exp\left(-\frac{(\tilde{p}_q^{(t)}-0.5)^2}{2\sigma^2}\right).$$8 percent to $$w_q^{(t)}=\exp\left(-\frac{(\tilde{p}_q^{(t)}-0.5)^2}{2\sigma^2}\right).$$9 percent, and an average pass@32 gain of $1$0 on the 8B model (Lin et al., 12 May 2026). Actor-Curator reports relative gains of $1$1 on AIME2024 and $1$2 on ARC-1D over the strongest baseline and up to $1$3 speedup (Gu et al., 24 Feb 2026). AdaRFT reports up to $1$4 reduction in training time and improved accuracy across AMC, AIME, and IMO-style problems (Shi et al., 7 Apr 2025). CGPO reports an overall score of $1$5 on GenEval, compared with $1$6 for Flow-GRPO, and reaches high-performance benchmarks in about half the GPU hours of Flow-GRPO (Li et al., 18 May 2026). SPO reports an average maj@32 improvement of $1$7 percentage points over GRPO across five hard math benchmarks, including $1$8 points on BRUMO 25 and $1$9 on AIME 25 (Xu et al., 16 Sep 2025). In autonomous driving, CPPO completes training in $0$0 hr $0$1 min, compared with $0$2 hr $0$3 min for PPO with fixed $0$4 and $0$5 hr $0$6 min for PPO with fixed $0$7 (Peng et al., 2023).

A recurrent misconception is that ACPO is synonymous with a fixed easy-to-hard curriculum. The cited methods do not support that reduction. They instead emphasize moderately difficult questions with pass rate near $0$8, prompts with high reward variance, batches closest to a dynamically updated target difficulty, or samples whose normalized advantage warrants broader clipping. This suggests that the contemporary ACPO literature is organized less by a single schedule template than by a general principle: learning should concentrate where policy improvement is currently most likely.

A second source of confusion is terminological. “ACPO” is also used for several unrelated acronyms: “Adaptive Cognition Policy Optimization” for efficient hybrid reasoning in LRMs (Cheng et al., 22 May 2025), “Attribution-based Contribution to Policy Optimization” for step-level credit assignment in RLVR (Yin et al., 10 Oct 2025), “Average-Constrained Policy Optimization” for average-reward CMDPs (Agnihotri et al., 2023), “Adversarial Constrained Policy Optimization” for adapting cost budgets in constrained RL (Ma et al., 2024), and “Automatic Constraint Policy Optimization” for offline RL under a continuous constraint interpolation framework (Han et al., 30 Jan 2026). In encyclopedia usage, “Adaptive Curriculum Policy Optimization” therefore refers not to a single universally standardized algorithm, but to a family of competence-adaptive policy optimization methods whose common feature is the joint shaping of training data exposure and update dynamics.