---
title: 'Actor-Critic with Action Chunking (AC2): Reinforcement Learning for Language Models'
url: https://www.emergentmind.com/topics/actor-critic-with-action-chunking-ac2
type: topic
---

# Actor-Critic with Action Chunking (AC2): Reinforcement Learning for Language Models

Actor-Critic with Action Chunking (AC2) is a reinforcement-learning framework that assigns policy credit to temporally extended action chunks rather than requiring every sampled trajectory to reach a terminal reward. In the formulation introduced in “Trust the Critic More,” an action is a continuation of 10,000 tokens generated from a replayed language-model prefix, while a learned critic estimates the value of the nonterminal state reached at the chunk endpoint. AC2 combines chunk-level advantage estimation, selective critic use, local reliability tests, reference solutions, and partial rollouts to reduce terminal-verification requirements in language-model reinforcement learning [2609.39247].

## 1. Conceptual foundations

In language-model reinforcement learning with verifiable rewards, a problem $x$ is followed by a response

$$
y=(y_1,\ldots,y_T).
$$

The state after the first $t-1$ response tokens is the prefix

$$
s_t=x\cdot y_{<t},
$$

and the action is the next token $a_t=y_t$. Intermediate rewards are absent:

$$
r_t=0\quad (t<T),
$$

while the terminal verifier supplies

$$
r_T=R(x,y),
$$

where $R(x,y)\in[0,1]$ is defined only for a complete response.

The policy objective is

$$
J(\theta)=\mathbb E_{x\sim D,\;y\sim\pi_\theta(\cdot\mid x)}[R(x,y)].
$$

The value and action-value functions are

$$
V^\pi(s)=\mathbb E_\pi[R(x,y)\mid s],
$$

and

$$
Q^\pi(s,a)=\mathbb E_\pi[R(x,y)\mid s,a].
$$

Because language-model transitions are deterministic,

$$
Q^\pi(s,a)=V^\pi(s\cdot a),
$$

and the advantage becomes

$$
A^\pi(s,a)=V^\pi(s\cdot a)-V^\pi(s).
$$

AC2 treats a short continuation of a prefix as the effective action. Instead of assigning one terminal advantage to every token in a complete response, it compares the value of several continuations from the same prefix. A continuation may terminate and receive a verifier reward, or stop at a nonterminal chunk boundary and receive a critic prediction.

This formulation differs from ordinary token-level actor-critic learning in both temporal resolution and rollout requirements. The critic is not merely a variance-reducing baseline: when it is sufficiently reliable for a particular problem, it supplies the endpoint value used for policy improvement.

## 2. Relation to GRPO and conventional actor-critic learning

Group Relative Policy Optimization (GRPO) samples complete responses and assigns every token in response $i$ the same group-relative advantage:

$$
\widehat A_{i,t}
=
R(x,y_i)
-
\frac{1}{g}\sum_{j=1}^{g}R(x,y_j).
$$

This procedure requires every response to reach a terminal verifier outcome. It provides trajectory-level credit: a successful response gives the same positive signal to all of its tokens, while an unsuccessful response gives the same negative signal even if only a late portion caused failure.

With a value function, generalized advantage estimation can interpolate between local critic-based and terminal-reward estimators. Let

$$
\delta_t^V=V(s_{t+1})-V(s_t).
$$

Then

$$
\widehat A_t^{\mathrm{GAE}(\lambda)}
=
\sum_{l=0}^{T-t}\lambda^l\delta_{t+l}^V.
$$

The endpoints are

$$
\widehat A_t^{\mathrm{GAE}(0)}
=
V(s_t\cdot a_t)-V(s_t)
$$

and

$$
\widehat A_t^{\mathrm{GAE}(1)}
=
R(x,y)-V(s_t).
$$

Any $\lambda>0$ requires eventual access to the terminal reward because the residuals telescope to the terminal state. AC2 instead uses a chunk-level analogue of the $\lambda=0$ estimator for problems whose critics pass problem-specific readiness tests.

For a replayed prefix $s$, AC2 samples continuations $c_1,\ldots,c_g$ and defines endpoint values by

$$
v_i=
\begin{cases}
r_i,&s\cdot c_i\text{ is complete},\\
V^\pi_\theta(s\cdot c_i),&\text{otherwise}.
\end{cases}
$$

The group estimate of the prefix value is

$$
\widehat V(s)=\frac{1}{g}\sum_{j=1}^{g}v_j,
$$

and the chunk advantage is

$$
\widehat A_i
=
v_i-\frac{1}{g}\sum_{j=1}^{g}v_j.
$$

On unready problems, all continuations are generated to termination and the estimator is terminal-reward based. On ready problems, most continuations stop after a 10,000-token chunk and use critic endpoint values. Thus, AC2 combines terminal and bootstrapped credit assignment according to problem-specific critic reliability.

## 3. Chunk construction and training workflow

AC2 maintains a replay buffer of complete prior rollouts, a set of problems whose critics are trusted, a FIFO buffer of prefix-target pairs for critic training, and a bank of known-correct reference solutions.

The reported workflow is:

1. Sample $n_{\mathrm{refill}}=192$ fresh problems.
2. Generate one complete response for each fresh problem, judge it, and insert it into replay regardless of correctness.
3. Sample $n_{\mathrm{batch}}=192$ stored trajectories.
4. Select a random cut point after the problem statement.
5. Use the resulting prefix as the conditioning state.
6. Sample a group of $g=16$ continuations.
7. Use terminal rewards for complete continuations and critic values for eligible incomplete chunks.
8. Compute group-relative chunk advantages.
9. Apply the actor update only to newly generated chunk tokens.
10. Store valid prefix-target pairs for critic training.

Cuts lie on a 10,000-token grid and occur no later than 90% of the trajectory length. Empty-response prefixes are allowed. The replayed prefix is conditioning context only; its tokens receive no actor loss.

For unready problems, every continuation is generated to termination. For ready problems, a continuation generates at most

$$
b=10{,}000
$$

new tokens. A fraction

$$
\alpha=\frac14
$$

of ready problems still receive full rollouts as audits. The overall response budget is 50,000 tokens, counting both the replayed prefix and newly generated tokens.

A ready continuation is critic-scored only if it reaches the chunk limit without ending the proof or exhausting the total budget. If it completes, it is judged normally. The method therefore does not eliminate terminal rollouts; it uses them selectively for replay-buffer refill, critic calibration, auditing, reference discovery, and readiness determination.

The actor uses a PPO-style token-level clipped objective, but the advantage is constant over the newly generated tokens in a chunk:

$$
\rho_t(\theta)
=
\frac{\pi_\theta(a_t\mid s_t)}
{\pi_{\theta_{\mathrm{old}}}(a_t\mid s_t)}.
$$

The actor loss is

$$
L(\theta)
=
-\mathbb E
\left[
\sum_t
\min\left(
\rho_t(\theta)\widehat A_t,\,
\operatorname{clip}
\left(
\rho_t(\theta),1-\epsilon_{\mathrm{low}},1+\epsilon_{\mathrm{high}}
\right)\widehat A_t
\right)
\right].
$$

Reported settings include an actor learning rate of $2\times10^{-6}$, $\epsilon_{\mathrm{low}}=0.2$, adaptive $\epsilon_{\mathrm{high}}$, gradient clipping at $0.3$, weight decay $0.01$, no KL penalty or entropy bonus, and a dual-clip coefficient of 3. Each iteration performs two actor updates with a critic update interleaved between them.

The critic predicts values using a generative language-model prompt. It outputs a value on the grid

$$
\{0,0.1,0.2,\ldots,1.0\}.
$$

It receives the problem, partial response, and, when available, a known-correct reference solution. It greedily generates at most four tokens, and the first valid grid value is parsed. A continuation with an invalid endpoint value receives zero actor weight. For critic training, a prefix is retained only when at least eight of its sixteen continuations have valid rewards or values.

The critic target is the group mean

$$
\bar v=\frac{1}{g}\sum_{j=1}^{g}v_j,
$$

rounded to the nearest value on the value grid, with ties rounded down. Prefix-target pairs are stored in a FIFO buffer of capacity 1,920, and each critic update samples up to 768 pairs.

## 4. Local readiness and reference-conditioned value estimation

The principal reliability mechanism is local readiness. A critic that is accurate on average may still be inaccurate for a particular problem, so AC2 evaluates critic error separately by problem.

Let

$$
\varepsilon(s)
=
V^\pi_\theta(s)
-
\frac{1}{g}\sum_{j=1}^{g}v_j
$$

denote the critic error measured before the current update. A problem becomes ready when:

1. the mean critic error over every prefix sampled in the most recent five steps is below $\tau_{\mathrm{global}}=0.20$;
2. the problem’s error from its previous sampling is below $\tau_{\mathrm{local}}=0.18$;
3. the policy has produced at least one correct trajectory for the problem.

The implementation also requires a nonzero valid critic prediction at the current or an earlier step. Once a problem becomes ready, it remains ready. Invalid critic predictions count as error 1 in readiness measurements, and the relevant absolute error is used.

Local readiness determines whether incomplete chunks may be trusted. Unready problems always use terminal rollouts. Ready problems may use critic-scored chunks, subject to endpoint validity and sufficient valid values in the group. This creates a problem-specific switch between terminal-reward learning and critic-based learning.

AC2 also supplies the critic with a reference solution when one is available. A previous rollout becomes a reference when it receives at least six of seven judge points. The critic receives the problem, partial response, and reference solution in context. Critic mean absolute error was lower with a reference at every tested checkpoint. Critic training also uses prompted variants with and without the reference at equal weight, so the model does not become dependent on references at inference.

The 10,000-token chunk length is another reliability mechanism. Token-level critic estimates are described as highly noisy because an individual token usually provides little information about the eventual correctness of a long proof. A 10,000-token continuation offers more evaluable reasoning context while remaining substantially shorter than a 50,000-token complete response. Reducing the chunk length to 2,000 tokens degraded performance: the corresponding variant peaked at 16.62% and later fell to 13.57%, whereas the 10,000-token configuration continued improving.

## 5. Empirical evaluation and computational efficiency

The authors train Qwen3-4B-Thinking-2507 on approximately 5,200 olympiad proof problems from FineProofs-RL and evaluate on IMO-ProofBench, which contains 60 curated olympiad problems. Validation responses are judged by DeepSeek-V4-Flash using the problem and a reference solution or rubric. Training itself does not provide the reference solution or rubric to the judge.

The main AC2 settings are:

- group size: $g=16$;
- chunk size: $b=10{,}000$;
- full-response budget: 50,000 tokens;
- replay capacity: 256 trajectories;
- fresh rollouts per iteration: 192;
- replayed prefixes per iteration: 192;
- audit fraction: $1/4$;
- evaluation interval: 10 steps;
- validation samples per problem: 16 responses.

The principal baselines are GRPO, which starts from the problem statement and rolls every response to completion, and Prefix GRPO, which starts from replayed prefixes but still rolls every continuation to completion.

AC2 first exceeds GRPO’s peak validation score of 18.50% at approximately

$$
0.79\times10^{20}
$$

decoding FLOPs, while GRPO reaches that score at approximately

$$
1.99\times10^{20}
$$

FLOPs. AC2 therefore reaches the GRPO peak with approximately 2.5 times less decoding compute and in 90 rather than 120 training steps, a 25% reduction. AC2’s higher peak mean validation score is 20.57%, compared with 18.50% for GRPO.

The efficiency gain has two sources. AC2 reaches the target score in fewer training steps, and each AC2 step generates fewer tokens because most ready trajectories stop after one chunk instead of continuing to completion. Prefix replay alone does not explain the result: Prefix GRPO, despite using replayed prefixes, performs substantially worse than AC2 at matched decoding compute.

At one diagnostic checkpoint, group-mean endpoint values for 256 prefixes and groups of 16 continuations had mean absolute error 0.065 relative to mean terminal reward. Direct prefix-value predictions had mean absolute error 0.211. Critic-based advantages had mean absolute error 0.150 relative to terminal GRPO advantages and Pearson correlation 0.388. These results indicate that the group mean is a more reliable critic-training target than direct sparse supervision of the prefix value.

The critic’s global readiness threshold was first crossed around step 7, and approximately 70% of sampled problems were ready by step 200. Removing auditing still reached 18.66%, although with reduced stability. A singleton variant using one chunk per prefix and

$$
\widehat A=V^\pi_\theta(s\cdot c)-V^\pi_\theta(s)
$$

performed comparably. A highly stale replay buffer refreshed every ten steps reached 17.90%, indicating that AC2 can use substantially off-policy prefixes.

Several variants degraded substantially. Two-thousand-token chunks peaked at 16.62% and later fell to 13.57%. Removing local readiness caused learning to stall and scores to decline. A correct-only replay buffer initially reached 17.35% but later fell to 14.67%, showing that retaining only successful trajectories reduces useful diversity and can lead to degradation.

## 6. Limitations, interpretation, and extensions

The principal limitation is critic error. An inaccurate critic can misrank continuations and produce harmful policy updates. AC2 mitigates this through local readiness, terminal calibration on unready problems, audits, reference solutions, group averaging, validity checks, and relatively long action chunks. These mechanisms do not make the critic unbiased: early diagnostic results show nonzero error, and individual cut-state values can be considerably less accurate than group means.

Local readiness also imposes a data requirement. Determining whether a critic is reliable requires periodic terminal outcomes, making the method most suitable for settings with repeated access to the same problems. In a strictly single-pass or continually expanding data regime, there may be insufficient feedback to establish problem-specific readiness.

The method introduces additional computation for critic inference, terminal audits, replay-buffer refills, and critic training. Shared actor-critic parameters can also create optimization interference. The approach depends on repeated problem access, available reference solutions, and continued terminal feedback for calibration.

AC2’s 10,000-token action granularity creates a tradeoff. Larger chunks improve evaluability and critic reliability but provide coarser credit assignment. Smaller chunks permit finer updates but make endpoint value prediction harder. Complete trajectories provide terminal supervision but are computationally expensive and supply only trajectory-level credit.

Within the broader actor-critic literature, the term “action chunking” has several meanings. Interval-based sample selection in “Context-aware Active Multi-Step Reinforcement Learning” uses chunks for training-sample selection rather than temporally extended action execution [1911.04107]. TAAC creates adaptive closed-loop chunks through act-or-repeat decisions [2104.06521]. “Chunking the Critic” introduces temporal sequences primarily on the critic side while retaining a stepwise actor [2503.03660]. AC3, SEAR, ACSAC, DWS, PAC-ACT, and ACPPO represent distinct actor-side, critic-side, adaptive-horizon, implicit, post-training, and feedback-corrected variants of chunked actor-critic learning [2508.11143] [2603.01891] [2605.11009] [2605.19592] [2607.09590] [2609.36250].

AC2 in the LLM-RL formulation is distinguished by its use of language prefixes as states, long continuations as actions, critic-scored nonterminal endpoints, local readiness, and selective avoidance of terminal rollouts. Its broader implication is that a critic can serve not only as a baseline but also as a conditional substitute for terminal verification when its accuracy has been demonstrated on the specific problem. This enables replayed-prefix training, selective terminal evaluation, chunk-level branching, value-guided search, adaptive chunk lengths, and other forms of partial-rollout policy optimization.

Source: https://www.emergentmind.com/topics/actor-critic-with-action-chunking-ac2