Actor-Curator: Automated RL Curriculum Learning

• Actor-Curator is a fully automated curriculum learning framework that coordinates an actor LLM and a neural curator using a non-stationary bandit formulation.
• It leverages online stochastic mirror descent for curator updates, ensuring unbiased utility estimates and providing theoretical regret guarantees.
• Empirical evaluations on complex reasoning benchmarks show significant gains in stability and efficiency, achieving target accuracy with up to 80% fewer training steps.

Actor-Curator (AC) is a fully automated curriculum learning framework designed for reinforcement learning (RL) post-training of LLMs. The method introduces a co-adaptive interaction between two components: the actor, a policy model (typically a pretrained LLM), and the curator, a neural network that selects training problems from a large, heterogeneous problem bank. By formulating the curriculum selection as a non-stationary stochastic bandit problem, Actor-Curator directly optimizes for expected policy improvement, is robust to large unstructured problem banks, and features theoretical regret guarantees under partial feedback. Actor-Curator demonstrates significant empirical gains in training stability, efficiency, and final performance over baseline sampling or hand-engineered curricula, with particular efficacy on complex reasoning benchmarks (Gu et al., 24 Feb 2026).

1. Problem Formulation and Bandit Perspective

Actor-Curator is situated in the context of RL post-training on vast, heterogeneous task distributions. The setup involves:

• Problem bank $\mathcal{X}$: a large, diverse set of problems (e.g., math, algorithmic, symbolic reasoning tasks).
• Actor $\pi^t$: the RL policy, such as a partially or fully pretrained LLM; for an input problem $x\in\mathcal{X}$, generates a solution $y \sim \pi^t(\cdot|x)$.
• Curator $C_t$: a trainable selector with parameters $\phi^t$, assigning scores $w_{\phi}(x)$ which induce a sampling distribution $$p_\phi(x)=w_\phi(x)/\sum_{x'\in\mathcal{X}}w_\phi(x')$$.

At each iteration $t$:

• The curator samples a mini-batch $\mathcal{X}^t\subset\mathcal{X}$;
• The actor is rolled out on each $\pi^t$0;
• The actor is updated to $\pi^t$1 using a reward model and gathered trajectories.

The overarching objective is to maximize expected reward:

$\pi^t$2

The per-iteration gain is $\pi^t$3.

Bandit formulation: Each problem $\pi^t$4 is conceptualized as a non-stationary bandit arm with utility:

$\pi^t$5

where $\pi^t$6 is the policy advantage. Importance sampling enables unbiased utility estimation even under partial batchwise feedback.

2. Curator Optimization via Online Stochastic Mirror Descent

The curator’s update is grounded in online stochastic mirror descent (OSMD):

• Tabular OSMD: For the tabular regime, $\pi^t$7 is updated by exponentiated gradient:

$\pi^t$8

where $\pi^t$9 is an unbiased utility estimate.

• Neural function approximation: When using a neural curator $x\in\mathcal{X}$0, the objective is to minimize

$x\in\mathcal{X}$1

• Two-stage sampling: In practical large-scale settings, a candidate set $x\in\mathcal{X}$2 is first sampled, followed by subsampling according to $x\in\mathcal{X}$3. The two-stage unbiased estimator and loss function account for proposal bias and importance weighting.
• Stabilization via proximal clipping: A Proximal Clipping Objective (PCO), inspired by PPO, stabilizes the curator’s update by clipping the importance-weighted policy ratio, providing robustness to volatility in the curation distribution.

3. Theoretical Guarantees

Actor-Curator provides rigorous guarantees under mild assumptions:

• Unbiasedness: Both single- and two-stage bandit estimators are proven to be unbiased: $x\in\mathcal{X}$4.
• Dynamic regret bound: Under OSMD with negative-entropy mirror map and partial-feedback estimation, the best-in-hindsight dynamic regret is

$x\in\mathcal{X}$5

where $x\in\mathcal{X}$6 quantifies non-stationarity of per-problem utilities.

• Assumptions: The guarantees assume bounded utility functions, estimator unbiasedness, a minimum inclusion probability $x\in\mathcal{X}$7, entropy regularization ($x\in\mathcal{X}$8), and mild Lipschitzness.

The proof approach decomposes the time horizon into near-stationary blocks, uses mirror descent inequalities, and strategically balances estimation error with non-stationary drift via block and step-size optimization.

4. Algorithmic Structure

The Actor-Curator algorithm (Algorithm 1 in the paper) operates as follows:

• Initialize the problem bank, actor, reward model, neural curator, and proposal distribution.
• For each training step $x\in\mathcal{X}$9:
  • Candidate sampling: Draw candidate problems $y \sim \pi^t(\cdot|x)$0 of size $y \sim \pi^t(\cdot|x)$1.
  • Selection: Subsample batch $y \sim \pi^t(\cdot|x)$2 of size $y \sim \pi^t(\cdot|x)$3 from $y \sim \pi^t(\cdot|x)$4 using the curator.
  • Data collection: Run $y \sim \pi^t(\cdot|x)$5 policy rollouts per selected problem, compute rewards.
  • Actor update: Update $y \sim \pi^t(\cdot|x)$6 using collected data (e.g., guided by GSPO/GRPO).
  • Utility estimation: Compute per-problem advantage estimates $y \sim \pi^t(\cdot|x)$7, then unbiased utility $y \sim \pi^t(\cdot|x)$8 for each $y \sim \pi^t(\cdot|x)$9.
  • Curator update: Update curator via gradient step on $C_t$0.

Utility computations require both pre- and post-update policies and leverage importance sampling for unbiasedness.

5. Empirical Evaluation

Experiments address diverse curriculum learning settings for LLM post-training:

• Benchmarks:
  • Symbolic: Countdown (and hard), Zebra (and hard)
  • Visual: ARC-1D (and ARC-hard)
  • Mathematics: MATH500, AIME2024
• Setup:
  • Actor: Qwen2.5-3B or Llama3.2-3B-IT, trained with GSPO.
  • Curator: Qwen3-0.6B.
• Metrics: Held-out test accuracy (peak within 100 steps).
• Results:
  • ARC-1D: Uniform 26.7% $C_t$1 Actor-Curator 36.4% (+30.5%)
  • AIME24: Uniform 23.3% $C_t$2 Actor-Curator 30.0% (+28.6%)
  • Largest gains observed on hard subsets (e.g., ARC-hard +59%).
• Speed: Actor-Curator achieves target test accuracy in $C_t$380% fewer training steps compared to uniform sampling.
• Ablation studies: Removing the policy-improvement target or OSMD in favor of regression degrades performance; method remains robust to actor update mode and curator/candidate batch sizes.

6. Strengths, Limitations, and Future Directions

Strengths:

• Direct optimization of a policy-improvement-driven curriculum objective
• Explicit handling of bandit feedback, exploration–exploitation, and non-stationarity
• Scalability to large, unstructured task banks via neural curation
• Theoretical guarantees with minimal assumptions
• Empirical gains in convergence and final accuracy across disparate reasoning tasks

Limitations:

• Efficacy relies on the existence of a reliable reward model (critical for verifiable correctness tasks)
• Curator training incurs a modest (9–15%) wall-clock overhead; offset by resultant speedups
• Stability of actor updates is necessary; RL collapse risk persists
• Neural function approximation only approximately captures utility cross-interference between problems

Future Work:

• Extension to continuous/environment-parameterized task spaces
• Joint adaptation of actor update hyperparameters using the bandit signal
• Theoretical quantification of cross-problem interference with deep nets
• Integration with search-based rollout methods (e.g., Tree-of-Thoughts) for further performance improvement

Actor-Curator establishes a practical, theoretically sound methodology for scalable, automated curriculum discovery in RL-finetuning of LLMs, delivering faster convergence and higher performance on reasoning-intensive benchmarks (Gu et al., 24 Feb 2026).