---
title: 'Actor-Curator: Automated RL Curriculum Learning'
url: https://www.emergentmind.com/topics/actor-curator-dd5d9e31-ab22-4887-8f9f-3d5237dcbe20
type: topic
---

# Actor-Curator: Automated RL Curriculum Learning

Actor-Curator (AC) is a fully automated curriculum learning framework designed for reinforcement learning (RL) post-training of large language models (LLMs). The method introduces a co-adaptive interaction between two components: the actor, a policy model (typically a pretrained LLM), and the curator, a neural network that selects training problems from a large, heterogeneous problem bank. By formulating the curriculum selection as a non-stationary stochastic bandit problem, Actor-Curator directly optimizes for expected policy improvement, is robust to large unstructured problem banks, and features theoretical regret guarantees under partial feedback. Actor-Curator demonstrates significant empirical gains in training stability, efficiency, and final performance over baseline sampling or hand-engineered curricula, with particular efficacy on complex reasoning benchmarks [2602.20532].

## 1. Problem Formulation and Bandit Perspective

Actor-Curator is situated in the context of RL post-training on vast, heterogeneous task distributions. The setup involves:

- **Problem bank $\mathcal{X}$**: a large, diverse set of problems (e.g., math, algorithmic, symbolic reasoning tasks).
- **Actor $\pi^t$**: the RL policy, such as a partially or fully pretrained LLM; for an input problem $x\in\mathcal{X}$, generates a solution $y \sim \pi^t(\cdot|x)$.
- **Curator $C_t$**: a trainable selector with parameters $\phi^t$, assigning scores $w_{\phi}(x)$ which induce a sampling distribution $p_\phi(x)=w_\phi(x)/\sum_{x'\in\mathcal{X}}w_\phi(x')$.

At each iteration $t$:
- The curator samples a mini-batch $\mathcal{X}^t\subset\mathcal{X}$;
- The actor is rolled out on each $x\in\mathcal{X}^t$;
- The actor is updated to $\pi^{t+1}$ using a reward model and gathered trajectories.

The overarching objective is to maximize expected reward:
$$
J(\pi) = \mathbb{E}_{x \sim p_\mathcal{X}, y \sim \pi(\cdot|x)}[R(y|x)]
$$
The *per-iteration gain* is $u^t = J(\pi^{t+1}) - J(\pi^t)$.

**Bandit formulation:** Each problem $x$ is conceptualized as a non-stationary bandit arm with utility:
$$
u^t_x = p_\mathcal{X}(x)\, \mathbb{E}_{y\sim\pi^t(\cdot|x)}\left[\frac{\pi^{t+1}(y|x)}{\pi^t(y|x)}A_{\pi^t}(y|x)\right]
$$
where $A_{\pi^t}(y|x) = R(y|x) - \mathbb{E}_{y'\sim\pi^t(\cdot|x)}[R(y'|x)]$ is the policy advantage. Importance sampling enables unbiased utility estimation even under partial batchwise feedback.

## 2. Curator Optimization via Online Stochastic Mirror Descent

The curator’s update is grounded in online stochastic mirror descent (OSMD):

- **Tabular OSMD:** For the tabular regime, $p^{t+1}$ is updated by exponentiated gradient:
  $$
  p^{t+1}(x) \propto p^t(x)\exp(\eta \hat{U}^t_x)
  $$
  where $\hat{U}^t_x$ is an unbiased utility estimate.

- **Neural function approximation:** When using a neural curator $C_\phi$, the objective is to minimize
  $$
  \mathcal{L}_\mathrm{cur}(\phi) = \mathrm{KL}(p_\phi \| p^t) - \eta \sum_{x\in\mathcal{X}} p_\phi(x)\hat{U}^t_x
  $$

- **Two-stage sampling:** In practical large-scale settings, a candidate set $\tilde{\mathcal{X}}^t$ is first sampled, followed by subsampling according to $p_\phi|_{\tilde{\mathcal{X}}^t}$. The two-stage unbiased estimator and loss function account for proposal bias and importance weighting.

- **Stabilization via proximal clipping:** A Proximal Clipping Objective (PCO), inspired by PPO, stabilizes the curator’s update by clipping the importance-weighted policy ratio, providing robustness to volatility in the curation distribution.

## 3. Theoretical Guarantees

Actor-Curator provides rigorous guarantees under mild assumptions:

- **Unbiasedness:** Both single- and two-stage bandit estimators are proven to be unbiased: $\mathbb{E}[\hat{U}^t_x]=u^t_x$.
- **Dynamic regret bound:** Under OSMD with negative-entropy mirror map and partial-feedback estimation, the best-in-hindsight dynamic regret is
  $$
  \mathrm{Reg}_T = O\left(T^{2/3}V_T^{1/3}\right)
  $$
  where $V_T = \sum_{t=2}^T \max_x |u^t_x - u^{t-1}_x|$ quantifies non-stationarity of per-problem utilities.

- **Assumptions:** The guarantees assume bounded utility functions, estimator unbiasedness, a minimum inclusion probability $q(x)\geq q_{\min}>0$, entropy regularization ($p \geq \alpha$), and mild Lipschitzness.

The proof approach decomposes the time horizon into near-stationary blocks, uses mirror descent inequalities, and strategically balances estimation error with non-stationary drift via block and step-size optimization.

## 4. Algorithmic Structure

The Actor-Curator algorithm (Algorithm 1 in the paper) operates as follows:

- Initialize the problem bank, actor, reward model, neural curator, and proposal distribution.
- For each training step $t$:
    - **Candidate sampling:** Draw candidate problems $\tilde{\mathcal{X}}^t$ of size $M$.
    - **Selection:** Subsample batch $\mathcal{X}^t$ of size $N$ from $\tilde{\mathcal{X}}^t$ using the curator.
    - **Data collection:** Run $m$ policy rollouts per selected problem, compute rewards.
    - **Actor update:** Update $\pi^{t+1}$ using collected data (e.g., guided by GSPO/GRPO).
    - **Utility estimation:** Compute per-problem advantage estimates $\hat{A}^t(x)$, then unbiased utility $\hat{U}^t_\mathrm{two}(x)$ for each $x\in \mathcal{X}^t$.
    - **Curator update:** Update curator via gradient step on $\mathcal{L}_\mathrm{PCO}(\phi)$.

Utility computations require both pre- and post-update policies and leverage importance sampling for unbiasedness.

## 5. Empirical Evaluation

Experiments address diverse curriculum learning settings for LLM post-training:

- **Benchmarks:**
    - Symbolic: Countdown (and hard), Zebra (and hard)
    - Visual: ARC-1D (and ARC-hard)
    - Mathematics: MATH500, AIME2024

- **Setup:**
    - Actor: Qwen2.5-3B or Llama3.2-3B-IT, trained with GSPO.
    - Curator: Qwen3-0.6B.

- **Metrics:** Held-out test accuracy (peak within 100 steps).

- **Results:**
    - ARC-1D: Uniform 26.7% $\rightarrow$ Actor-Curator 36.4% (+30.5%)
    - AIME24: Uniform 23.3% $\rightarrow$ Actor-Curator 30.0% (+28.6%)
    - Largest gains observed on hard subsets (e.g., ARC-hard +59%).

- **Speed:** Actor-Curator achieves target test accuracy in $\sim$80% fewer training steps compared to uniform sampling.

- **Ablation studies:** Removing the policy-improvement target or OSMD in favor of regression degrades performance; method remains robust to actor update mode and curator/candidate batch sizes.

## 6. Strengths, Limitations, and Future Directions

**Strengths:**
- Direct optimization of a policy-improvement-driven curriculum objective
- Explicit handling of bandit feedback, exploration–exploitation, and non-stationarity
- Scalability to large, unstructured task banks via neural curation
- Theoretical guarantees with minimal assumptions
- Empirical gains in convergence and final accuracy across disparate reasoning tasks

**Limitations:**
- Efficacy relies on the existence of a reliable reward model (critical for verifiable correctness tasks)
- Curator training incurs a modest (9–15%) wall-clock overhead; offset by resultant speedups
- Stability of actor updates is necessary; RL collapse risk persists
- Neural function approximation only *approximately* captures utility cross-interference between problems

**Future Work:**
- Extension to continuous/environment-parameterized task spaces
- Joint adaptation of actor update hyperparameters using the bandit signal
- Theoretical quantification of cross-problem interference with deep nets
- Integration with search-based rollout methods (e.g., Tree-of-Thoughts) for further performance improvement

Actor-Curator establishes a practical, theoretically sound methodology for scalable, automated curriculum discovery in RL-finetuning of LLMs, delivering faster convergence and higher performance on reasoning-intensive benchmarks [2602.20532].

Source: https://www.emergentmind.com/topics/actor-curator-dd5d9e31-ab22-4887-8f9f-3d5237dcbe20