---
title: Cost-Aware Policy Optimization (CAPO)
url: https://www.emergentmind.com/topics/cost-aware-policy-optimization-capo
type: topic
---

# Cost-Aware Policy Optimization (CAPO)

Cost-Aware Policy Optimization (CAPO) denotes policy-optimization methods in which cost is an explicit optimization variable rather than a secondary implementation concern. In the formulation instantiated by "Cost-Aware Learning" [2604.28020], the objective is to reach a target error or benchmark performance while minimizing total cost, with cost arising from heterogeneous component evaluations in finite-sum optimization and, in LLM reinforcement learning, from sequence-length-dependent policy-gradient computation. A broader cost-aware control instantiation appears in opportunistic spectrum access, where the policy maximizes cumulative expected net reward, i.e., reward minus sensing and transmission costs, via structured threshold rules [1804.04048]. The acronym is not unique; several recent papers use CAPO for unrelated methods, so the cost-aware usage requires terminological disambiguation.

## 1. Terminology and scope

In current arXiv usage, CAPO is an overloaded acronym. In the cost-aware sense relevant here, the term refers to optimizing policy updates under heterogeneous costs, especially when those costs are measurable and can be incorporated into sampling or decision rules. In "Cost-Aware Learning," the reinforcement-learning instantiation is called "Cost-Aware GRPO (CA-GRPO)," and the paper explicitly presents it as an instance of Cost-Aware Policy Optimization [2604.28020].

| Meaning of CAPO | Source | Scope |
|---|---|---|
| Cost-Aware Policy Optimization | [2604.28020] | Cost-aware sampling and importance weighting in policy optimization |
| Credit Assignment Policy Optimization | [2508.02298] | Verifiable token-level credit assignment for reasoning LLMs |
| Consensus Aggregation for Policy Optimization | [2603.12596] | Width-over-depth PPO consensus aggregation |
| Calibration-Aware Policy Optimization | [2604.12632] | AUC-consistent calibration optimization for reasoning LLMs |

This terminological overlap matters because the cost-aware variant does not target token-level process rewards, PPO consensus aggregation, or relative calibration. Its central concern is the total cost required to achieve a target optimization error or policy-improvement threshold.

## 2. Finite-sum cost-aware optimization framework

The formal starting point is a finite-sum objective
$$
f(x) = \frac{1}{n} \sum_{i=1}^n f_i(x),
$$
where each component $f_i$ is convex and $G_i$-Lipschitz, and each component has an evaluation cost $c_i \ge 0$. Querying sample $i$ at point $x$ returns the exact gradient $\nabla f_i(x)$ and incurs cost $c_i$. The stated goal is to obtain $\hat x$ such that
$$
\mathbb{E}[f(\hat x) - f(x^*)] \le \varepsilon
$$
while minimizing total expected cost [2604.28020].

The corresponding stochastic method is Cost-Aware SGD, an importance-weighted projected SGD scheme. With sampling distribution $p \in \Delta^n$, the stochastic gradient estimator is
$$
\tilde g_t = \frac{1}{n p_{i_t}} \nabla f_{i_t}(x_t),
$$
which is unbiased:
$$
\mathbb{E}[\tilde g_t] = \nabla f(x_t).
$$
Its second moment is bounded by
$$
S(p) \le \frac{1}{n^2} \sum_{i=1}^n \frac{G_i^2}{p_i},
$$
and the expected per-iteration cost is
$$
C(p) = \sum_i p_i c_i.
$$
If $T(\varepsilon,p)$ is the number of iterations required to reach error $\varepsilon$, then the total expected cost is
$$
K(\varepsilon,p) = T(\varepsilon,p)\cdot C(p).
$$

For general convex functions, the paper states
$$
\mathbb{E}[f(\bar x_T) - f(x^*)] \le \frac{D \sqrt{S(p)}}{\sqrt{T}},
$$
so achieving error $\varepsilon$ requires
$$
T = \frac{D^2 S(p)}{\varepsilon^2},
$$
and hence
$$
K(\varepsilon,p) = \frac{D^2}{\varepsilon^2} S(p)\cdot C(p).
$$
For $\mu$-strongly convex functions with $\eta_t = 1/(\mu t)$, the paper gives
$$
T = \frac{4 S(p)}{\mu \varepsilon},
\qquad
K(\varepsilon,p) = \frac{4}{\mu \varepsilon} S(p)\cdot C(p).
$$
In both regimes, minimizing total cost reduces to minimizing the product
$$
J(p) = S(p)\cdot C(p).
$$

Theorem 4.1 identifies the optimal sampling distribution:
$$
p_i^* = \frac{G_i / \sqrt{c_i}}{\sum_{j=1}^n G_j / \sqrt{c_j}}.
$$
Under this distribution, the minimum expected cost becomes
$$
K_{\mathrm{conv}}(\varepsilon,p^*) =
\frac{D^2}{\varepsilon^2 n^2}
\Big(\sum_{i=1}^n G_i \sqrt{c_i}\Big)^2
$$
for the convex case, and
$$
K_{\mathrm{str\text{-}conv}}(\varepsilon,p^*) =
\frac{4}{\mu \varepsilon n^2}
\Big(\sum_{i=1}^n G_i \sqrt{c_i}\Big)^2
$$
for the strongly convex case. The comparison point is traditional variance-only importance sampling, which uses $p_i \propto G_i$ and ignores costs. Theorem 4.2 states that the optimal cost-aware strategy is never worse than uniform or variance-only sampling in total cost to reach $\varepsilon$-accuracy.

A plausible implication is that CAPO changes the optimization target from “minimize iteration count” to “minimize cost-adjusted convergence time.” In the LLM setting, this alters which trajectories are most valuable to revisit during policy updates.

## 3. Optimality, proxy quality, and subset selection

The theory in "Cost-Aware Learning" does not stop at the optimal distribution. It also characterizes lower bounds, proxy error, and biased subset restriction [2604.28020].

The lower-bound result states, informally, that for any algorithm achieving $\mathbb{E}[F(\hat x)-\min F(x)] \le \varepsilon$, there exists a convex, $G$-Lipschitz finite-sum instance over $X=[-1,1]$ such that the expected total query cost is at least
$$
\Omega\!\Big(\frac{G^2}{\varepsilon^2 n} \sum_{i \in S^*} \sqrt{c_i}\Big),
$$
for any subset $S^*$ satisfying the stated cost-uniformity condition. The paper’s interpretation is that the dependence on $\sum_i G_i \sqrt{c_i}$ in the upper bounds is fundamental in heterogeneous-cost regimes.

Because exact $G_i$ values are often unavailable, the paper gives an exact proxy sub-optimality identity. Let $\tilde p_i = p_i c_i / C(p)$ be the cost-biased version of a sampling distribution. Then Theorem 4.3 states
$$
\frac{J(p')}{J(p^*)} = 1 + D_{\chi^2}(\tilde p^* \| \tilde p').
$$
Thus the gap between a practical sampler and the optimal sampler is determined exactly by the Pearson $\chi^2$-divergence between their cost-biased distributions. Theorem 4.4 further analyzes noisy proxies $G'_i = G_i + \varepsilon_i$ and gives an approximate dependence on the proxy–truth Pearson correlation $\rho$.

The subset-selection analysis introduces a bias–cost trade-off. Restricting optimization to a subset $T \subseteq \{1,\dots,n\}$ yields
$$
f_T(x) = \frac{1}{|T|}\sum_{i\in T} f_i(x),
$$
which generally converges to $x_T^* \ne x^*$. With bias upper bound $B_T$, the convex-case guarantee becomes
$$
\mathbb{E}[f(\bar x_T)-f(x^*)]
\le
\frac{D\sqrt{\sigma_T^2(q)}}{\sqrt{T}} + D B_T.
$$
Theorem 5.1 states that, for a fixed bias budget $I$ with requirement $D B_T \le I$, the optimal subset is equivalent to a Min-Knapsack problem:
$$
\min_{z \in \{0,1\}^n} \sum_{i=1}^n z_i w_i
\quad \text{s.t.} \quad
\sum_{i=1}^n z_i v_i \ge V_{\mathrm{req}},
$$
with item cost $w_i = G_i \sqrt{c_i}$, item value $v_i = G_i$, and
$$
V_{\mathrm{req}} = \sum_{j=1}^n G_j - \frac{n I}{D}.
$$
The greedy approximation sorts by density $v_i/w_i = 1/\sqrt{c_i}$ and therefore selects the cheapest samples first.

This suggests a two-level CAPO picture: first optimize sampling probabilities inside the full dataset, and then, under stronger budget pressure, restrict the support itself through an explicit bias-aware selection rule.

## 4. CAPO in GRPO-based LLM policy optimization

The LLM instantiation applies the cost-aware finite-sum analysis to Group Relative Policy Optimization. GRPO alternates between data collection and policy-gradient updates, uses a token-level PPO-like objective with clipping and KL regularization, and normalizes advantages as
$$
A_i = \frac{r_i - \mathrm{mean}(\{r_k\}_{k=1}^M)}{\mathrm{std}(\{r_k\}_{k=1}^M)}.
$$
The empirical finite-sum GRPO objective is written as
$$
\hat J_{\mathrm{GRPO}}(\theta)
=
\frac{1}{nM}\sum_{i=1}^n \sum_{g=1}^M \ell(o_{i,g};\theta,q_i),
$$
with rollout-level contributions $\ell(o;\theta,q)$ defined by the clipped ratio term and the KL regularizer [2604.28020].

The cost model is sequence-length based:
$$
c_i = \mathrm{len}(\text{prompt}_i) + \mathrm{len}(\text{response}_i),
$$
namely the number of tokens processed during the policy-gradient step, which the paper states is proportional to FLOPs. This converts policy optimization into a heterogeneous-cost finite-sum problem, because longer trajectories are more expensive to reuse during the update stage.

Cost-Aware GRPO instantiates CAPO by importance sampling rollouts during policy updates using the cost-aware rule. Since exact per-sample gradient norms are not computed, the paper uses the magnitude of the normalized advantage, $|A_i|$, as a proxy for $G_i$. Sampling probabilities for rollout $u$ in the step dataset $D_{\mathrm{step}}$ are therefore set to
$$
p_u \propto \frac{|A_u|}{\sqrt{c_u}},
$$
with an optional smoothed distribution
$$
p_{\mathrm{smooth}}(\alpha) = (1-\alpha)p^* + \alpha u,
$$
where $u$ is uniform and $\alpha \in [0,1]$. The bias induced by nonuniform sampling is corrected by the importance-weighted objective
$$
\hat J_{\mathrm{GRPO}}(\theta)
=
\frac{1}{nM}\sum_{i=1}^n \sum_{g=1}^M \frac{1}{p_{i,g}} \ell(o_{i,g};\theta,q_i).
$$
An implementation detail recentered the importance weights around $1$ in order to maintain stable gradient norms.

Algorithm 2 proceeds in three phases per iteration: rollout collection, computation of the sampling distribution from $|A_u|/\sqrt{c_u}$, and multinomial mini-batch sampling for policy updates. The practical guidance is explicit that data collection is not reweighted; only the policy-gradient stage is cost-aware. The same section also notes that smoothing values such as $\alpha=0.01$, $0.05$, and $0.1$ were explored, and that the method is robust across GRPO, GRPO+ZVF, and CISPO.

## 5. Empirical results in reasoning LLMs

The empirical study uses Qwen2.5-Math-1.5B-Instruct and Qwen3-8B Base, trains on the DAPO dataset with verifiable math rewards, and evaluates on MATH500, AMC, GSM8K, and AIME1983–2024 using pass@1 or mean@32. The implementation uses Verl, vLLM for inference, and FlashAttention for efficiency. For Qwen3-8B runs, the hardware is 8× H100 GPUs with approximately 5 days per run; the 1.5B runs take approximately 1–1.5 days [2604.28020].

The main reported outcome is a reduction in cumulative tokens used in policy optimization while matching or exceeding baseline accuracy. The paper also reports that the proxy $|A_i|$ is empirically well aligned with true gradient norms: Pearson correlations are approximately $0.92$–$0.98$, and cost-biased $\chi^2$-divergences are approximately $0.02$–$0.06$ across training steps.

| Setting | Token savings | Accuracy summary |
|---|---|---|
| Qwen2.5-Math-1.5B-Instruct, standard GRPO on AIME | $p^*$ and $p_{\mathrm{smooth}}(0.01)$ match baseline peak using 28% and 32% fewer tokens | Both ultimately exceed baseline by approximately 5% peak accuracy |
| Qwen2.5-Math-1.5B-Instruct, GRPO+ZVF on AIME | $p_{\mathrm{smooth}}(0.01)$ achieves baseline peak with 13% fewer tokens | Exceeds baseline by approximately 5% |
| Qwen3-8B, AIME | $p^*$ reaches baseline peak using 51% fewer tokens; at 99% of peak accuracy, 17% fewer tokens | Peak accuracy preserved or slightly improved |
| Qwen3-8B, AMC | $p^*$ reaches baseline maximum using 47% fewer tokens | Matches baseline maximum |

The reported peak accuracy averages further emphasize that cost reduction does not require an accuracy sacrifice. For the 1.5B model under GRPO, the best averaged checkpoint over four benchmarks is 71.2% with no sampling, 74.0% with $p^*$, and 72.9% with $p_{\mathrm{smooth}}(0.01)$. Under GRPO+ZVF, the same comparison is 70.6%, 69.8%, and 72.5%. For the 8B model under GRPO, the corresponding values are 78.5% with no sampling, 78.8% with $p^*$, and 78.5% with both $p_{\mathrm{smooth}}(0.01)$ and $p_{\mathrm{smooth}}(0.05)$.

The ablations are important for interpretation. Under CISPO, $p^*$ and $p_{\mathrm{smooth}}(0.01)$ require 31% and 34% fewer tokens to reach baseline peak, respectively. By contrast, the length-only distribution $p^*$–LEN, i.e., $G_i \equiv 1$, underperforms. This directly supports the claim that CAPO is not merely “prefer shorter sequences”; it is signal-aware sampling proportional to estimated information-to-cost ratio.

## 6. Broader cost-aware control formulation

A distinct but conceptually related instantiation appears in cost-aware opportunistic spectrum access, where the objective is to maximize cumulative expected net reward over $T$ frames under random sensing costs, random transmission costs, and stochastic transmission rewards [1804.04048]. Each frame has a sensing phase followed by a transmission phase. Channel $i$ is idle with probability $\theta_i$, and if transmission occurs on an idle channel the reward is a random variable with mean $b_0$; sensing and transmission incur random costs with means $c_0$ and $p_0$.

The per-frame decision process admits a Bellman equation
$$
V(i,X_i,S) = \max \Big\{
X_i b_0 - p_0,\;
\max_{j\in S}\big[-c_0 + \mathbb{E}[V(j,X_j,S\setminus\{j\})]\big],\;
\max_{j\in S}[\theta_j b_0 - p_0],\;
0
\Big\}.
$$
When the last sensed channel is busy, the value recursion becomes
$$
E_{i-1}
=
\max \Big\{
-c_0 + (b_0-p_0)\theta_i + E_i(1-\theta_i),\;
\theta_i b_0 - p_0,\;
0
\Big\},
$$
with $E_K=0$. The resulting optimal offline policy has a recursive double-threshold structure. For each position $i$, there are thresholds
$$
u_i = \max \Big\{\frac{p_0}{b_0},\; 1 - \frac{c_0}{p_0 + E_i}\Big\},
\qquad
l_i = \min \Big\{\frac{p_0}{b_0},\; 1 - \frac{b_0-p_0-c_0}{b_0-p_0-E_i}\Big\},
$$
such that:
- if $\theta_i \ge u_i$, the policy guesses and transmits without sensing;
- if $l_i \le \theta_i < u_i$, it senses channel $i$;
- if $\theta_i < l_i$, it quits the frame.

The online version estimates $\theta_i$, $b_0$, $c_0$, and $p_0$, uses an exploration threshold $D(t)=L\log t + D$, and alternates between exploration and plug-in exploitation. The regret guarantee is
$$
\mathbb{E}[R(T)] \le C_1 \log T + C_2,
$$
with the stated constants
$$
C_1 = K L (b_0 + K c_0),
\qquad
C_2 = (\pi^2 + D K + 1)(b_0 + K c_0),
$$
and the paper also establishes a matching $\Omega(\log T)$ lower bound under mild conditions.

This earlier line of work differs from LLM CAPO in mechanism, but not in principle: in both cases, costs affect the optimal policy directly rather than appearing only as after-the-fact efficiency metrics.

## 7. Limitations and recurring misconceptions

The LLM-oriented CAPO framework comes with several explicit caveats. The underlying theory assumes convex or strongly convex finite-sum objectives, whereas LLM policy optimization is non-convex and non-stationary because rollout pools change across training rounds. The paper argues that the $J(p)=S(p)\cdot C(p)$ objective remains structurally optimal due to variance-controlled iteration complexity, but this is not a convex-style global guarantee [2604.28020].

The method also relies on gradient proxies rather than exact gradient norms. In the reported math-reasoning setting, $|A_i|$ works well, but the paper notes that long-horizon credit assignment and sparse rewards can make advantage magnitudes noisy; smoothing and ZVF help, but more robust proxies may be needed. Distribution shift across training rounds further means that guarantees are per rollout pool rather than global. Subset selection introduces bias and therefore requires a carefully chosen bias budget. A related misconception is to treat length-only sampling as sufficient; the reported underperformance of $p^*$–LEN shows that explicit signal terms remain necessary.

Finally, CAPO should not be treated as a unique algorithm name. In recent work, the acronym also denotes Credit Assignment Policy Optimization, Consensus Aggregation for Policy Optimization, and Calibration-Aware Policy Optimization [2508.02298] [2603.12596] [2604.12632]. In the cost-aware sense, the defining feature is not verifiable process supervision, optimizer-path consensus, or AUC-consistent calibration; it is the direct optimization of performance relative to heterogeneous cost. A plausible implication is that future CAPO variants for RLHF will be judged less by raw iteration counts than by how effectively they convert compute, measured in tokens or FLOPs, into policy improvement.

Source: https://www.emergentmind.com/topics/cost-aware-policy-optimization-capo