Calibration-Aware Policy Optimization (CAPO)

• CAPO is a family of methods that modulate policy updates using calibrated signals (e.g., confidence, uncertainty, or evidence) to improve learning from noisy data.
• It integrates techniques such as relative reward margins, AUC-consistent ranking, and geometry-aware uncertainty weighting to enhance multilingual and reasoning LLM performance.
• Experimental results indicate that CAPO methods can significantly improve reward accuracy and calibration, leading to safer and more reliable policy optimization.

Calibration-Aware Policy Optimization (CAPO) denotes a line of policy optimization methods in which reward maximization is explicitly modulated by a calibrated signal rather than by reward or likelihood alone. Across recent work, that calibrated signal has taken the form of a relative reward margin in multilingual preference learning, an AUC-oriented surrogate for relative calibration in reasoning LLMs, a decoupled confidence objective in RLVR, geometry-aware and reward-calibrated uncertainty weights in critic-free post-training, evidence-calibrated step-level credit for long-horizon agents, and a conformal regulator that constrains deployment-time deviation from a safe reference policy (Pokharel et al., 10 Nov 2025, Wang et al., 14 Apr 2026, Ma et al., 10 Mar 2026, Zhang et al., 20 May 2026, Li et al., 4 Jun 2026, Prinster et al., 2 Mar 2026). A plausible synthesis is that CAPO is best understood as an umbrella viewpoint: policy updates should be scaled, filtered, or constrained by a signal that is calibrated to preference reliability, correctness probability, gradient variance, statistical evidence, or safety risk.

1. Scope, terminology, and acronym usage

Recent arXiv usage does not assign a single expansion to the acronym “CAPO.” In calibration-centered work, the label appears directly as “Calibration-Aware Policy Optimization” for reasoning LLMs (Wang et al., 14 Apr 2026), as “Confidence Aware Preference Optimization” for multilingual preference learning (Pokharel et al., 10 Nov 2025), and as a broader design perspective instantiated by methods such as DCPO, GCPO, ECPO, and Conformal Policy Control (Ma et al., 10 Mar 2026, Zhang et al., 20 May 2026, Li et al., 4 Jun 2026, Prinster et al., 2 Mar 2026). The acronym is also used for unrelated methods, including “Coordinate Ascent Policy Optimization” (Su et al., 2022), “Consensus Aggregation for Policy Optimization” (Su et al., 13 Mar 2026), “Counterfactual Advantage Policy Optimization” (Deshmukh et al., 20 Apr 2026), “Credit Assignment Policy Optimization” (Xie et al., 4 Aug 2025), and “Consistency-Aware Preference Optimization” in Med-VQA (Jiang et al., 15 Jun 2025).

A plausible implication is that the calibration-aware sense of CAPO is now a methodological family rather than a single algorithm. What unifies that family is not a shared optimizer, but a shared intervention point: the learning signal is altered so that policy updates reflect calibrated confidence, uncertainty, evidence strength, or deployment risk.

Instantiation	Setting	Calibration mechanism
CAPO (Pokharel et al., 10 Nov 2025)	Multilingual preference optimization	Relative reward margin
CAPO (Wang et al., 14 Apr 2026)	RLVR for reasoning LLMs	Logistic AUC surrogate and noise masking
DCPO (Ma et al., 10 Mar 2026)	RLVR with verbalized confidence	Decoupled reasoning and calibration blocks
GCPO (Zhang et al., 20 May 2026)	GRPO-style post-training	Geometry-aware uncertainty and reward dispersion
ECPO (Li et al., 4 Jun 2026)	Long-horizon LLM agents	Shrinkage and variance-gated step credit
CPC (Prinster et al., 2 Mar 2026)	Safe deployment and exploration	Conformal calibration around a safe reference policy

2. Core optimization principle

The recurring claim in CAPO-style work is that standard post-training objectives are uncertainty-agnostic. In multilingual preference optimization, fixed-weight DPO-style objectives treat each pair equally and therefore ignore whether a comparison is noisy, ambiguous, or distorted by language-specific tokenization and log-likelihood scale effects (Pokharel et al., 10 Nov 2025). In reasoning RLVR, GRPO-style objectives optimize correctness but can degrade relative calibration, producing over-confident incorrect answers (Wang et al., 14 Apr 2026). In group-based critic-free post-training, entropy-based uncertainty estimators can be misaligned with both gradient variance and reward informativeness (Zhang et al., 20 May 2026). In long-horizon agents, dense anchor-level credit can still be statistically unreliable when it is built from low-count or high-variance samples (Li et al., 4 Jun 2026).

One formalization of the calibration target is relative calibration via AUC. For responses $o_i,o_j$ with binary rewards $R_i,R_j$ and scalar confidence score $f$, the relative-calibration metric used for reasoning LLMs is

$$\mathrm{AUC}(\pi,q,f)=\mathbb{E}_{o_i,o_j\sim\mathcal{D}}\Big[\mathbb{I}\big((R_i-R_j)(f(o_i)-f(o_j))>0\big)\Big],$$

so that correct responses should be ranked above incorrect ones by the model’s own confidence score (Wang et al., 14 Apr 2026). In that setting, CAPO replaces GRPO’s reward-only linear surrogate with a logistic AUC surrogate,

$$J_{\text{logistic}}(\theta) = -\mathbb{E}_{o_1,o_2\sim\mathcal{D}} \Big[ \log\big(1+\exp(-t/\tau)\big) \Big],$$

where $t$ is the signed margin between average log-probabilities of two responses scaled by their reward difference (Wang et al., 14 Apr 2026). The stated rationale is that the logistic surrogate is AUC-consistent and admits a regret bound, so optimization is explicitly aligned with relative calibration.

A second recurring principle is per-example modulation. CAPO-style methods do not merely add auxiliary losses; they change how strongly each example, query, anchor, or deployment decision affects the policy. In the preference setting, the per-pair margin is adjusted by a relative-confidence term (Pokharel et al., 10 Nov 2025). In reasoning RLVR, the advantage becomes uncertainty-aware and is further filtered by a reference-model-based noise mask (Wang et al., 14 Apr 2026). In GCPO, query-level weights depend jointly on geometry-aware semantic disagreement and reward dispersion (Zhang et al., 20 May 2026). In ECPO, step-level advantages are shrunk toward a prior and then downweighted when within-action noise dominates between-action signal (Li et al., 4 Jun 2026). In CPC, the policy itself is clipped toward a safe reference policy and the clipping aggressiveness is calibrated from data to satisfy a user-declared risk tolerance (Prinster et al., 2 Mar 2026).

3. Multilingual preference optimization

The paper “CAPO: Confidence Aware Preference Optimization Learning for Multilingual Preferences” formulates CAPO as a reference-free alternative to DPO for multilingual preference learning (Pokharel et al., 10 Nov 2025). The setting uses preference triples

$\mathcal{D}=\{(x,y_w,y_\ell)\},$

where $x$ is a prompt, $y_w$ is the preferred response, and $y_\ell$ is the dispreferred response. The stated motivation is that multilingual preference data often contains low-margin comparisons, language-specific tokenization differences, and mis-calibrated log-likelihood scales, so fixed-weight objectives can overfit noise or favor languages whose scoring patterns produce larger absolute gaps.

Its CAPO loss is

$R_i,R_j$0

where the second term is the Relative Reward Margin-driven dynamic margin and $R_i,R_j$1 controls how much the relative preference affects the update (Pokharel et al., 10 Nov 2025). The paper’s text emphasizes that CAPO modulates the learning signal according to confidence in each preference pair, aims to normalize across language-specific reward scales, and is intended to be more robust to noisy or low-margin multilingual comparisons.

Experimentally, the paper reports that CAPO improves average reward accuracy by at least $R_i,R_j$2 over DPO and $R_i,R_j$3 over DPONLL, and that it has the highest reward accuracy in all languages except English (Pokharel et al., 10 Nov 2025). On multilingual MT-Bench, average scores are reported as $R_i,R_j$4 for Base Llama, $R_i,R_j$5 for DPO, $R_i,R_j$6 for SimPO, and $R_i,R_j$7 for CAPO. On XLSum, CAPO wins in $R_i,R_j$8 languages against DPO, especially ar, uk, zh, es $R_i,R_j$9, while DPO is slightly better in en and vi. On M-IFEval, CAPO consistently matches or outperforms DPO in both strict and loose metrics, with a clear improvement in Japanese from $f$0 in strict and $f$1 in loose scoring (Pokharel et al., 10 Nov 2025).

The multilingual paper also explicitly addresses the naming issue: “Confidence-Aware Preference Optimization” and “Calibration-Aware Policy Optimization” are presented as two perspectives on the same algorithm, with “preference optimization” emphasizing the supervision format and “policy optimization” emphasizing the direct optimization of $f$2 (Pokharel et al., 10 Nov 2025).

4. Reasoning LLMs and RLVR

In reasoning-focused RLVR, CAPO is formulated as an explicit response to the empirical finding that GRPO can improve accuracy while degrading relative calibration as measured by AUC-mean (Wang et al., 14 Apr 2026). The paper “Calibration-Aware Policy Optimization for Reasoning LLMs” argues that reward-only advantage estimators correspond to an AUC-inconsistent linear surrogate, creating a mismatch between accuracy optimization and calibration optimization (Wang et al., 14 Apr 2026). CAPO addresses this by adopting the logistic AUC surrogate above, constructing an uncertainty-aware advantage that emphasizes misranked correct–incorrect pairs, and applying a reference-model-based noise mask that suppresses suspicious successes and overly plausible failures. Empirically, it reports that CAPO-1.5B improves calibration by up to $f$3 while achieving accuracy comparable to or better than GRPO, further boosts accuracy on downstream inference-time scaling tasks by up to $f$4, and achieves a Pareto-optimal precision-coverage trade-off when abstention is allowed (Wang et al., 14 Apr 2026).

A related line, “Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards,” argues that there exists a fundamental gradient conflict between maximizing policy accuracy and minimizing calibration error when the model is over-confident (Ma et al., 10 Mar 2026). DCPO is presented as a CAPO framework that decouples reasoning and calibration by structuring the rollout as

$f$5

using correctness reward for $f$6, a confidence reward

$f$7

and masked gradients so that reasoning advantages update only reasoning tokens and calibration advantages update only confidence tokens (Ma et al., 10 Mar 2026). On five math benchmarks with Qwen3-8B, DCPO reports overall performance of Acc $f$8, ECE $f$9, PCE $$\mathrm{AUC}(\pi,q,f)=\mathbb{E}_{o_i,o_j\sim\mathcal{D}}\Big[\mathbb{I}\big((R_i-R_j)(f(o_i)-f(o_j))>0\big)\Big],$$0, and AUROC $$\mathrm{AUC}(\pi,q,f)=\mathbb{E}_{o_i,o_j\sim\mathcal{D}}\Big[\mathbb{I}\big((R_i-R_j)(f(o_i)-f(o_j))>0\big)\Big],$$1, compared with GRPO at Acc $$\mathrm{AUC}(\pi,q,f)=\mathbb{E}_{o_i,o_j\sim\mathcal{D}}\Big[\mathbb{I}\big((R_i-R_j)(f(o_i)-f(o_j))>0\big)\Big],$$2, ECE $$\mathrm{AUC}(\pi,q,f)=\mathbb{E}_{o_i,o_j\sim\mathcal{D}}\Big[\mathbb{I}\big((R_i-R_j)(f(o_i)-f(o_j))>0\big)\Big],$$3, PCE $$\mathrm{AUC}(\pi,q,f)=\mathbb{E}_{o_i,o_j\sim\mathcal{D}}\Big[\mathbb{I}\big((R_i-R_j)(f(o_i)-f(o_j))>0\big)\Big],$$4, and AUROC $$\mathrm{AUC}(\pi,q,f)=\mathbb{E}_{o_i,o_j\sim\mathcal{D}}\Big[\mathbb{I}\big((R_i-R_j)(f(o_i)-f(o_j))>0\big)\Big],$$5. Removing decoupling yields Acc $$\mathrm{AUC}(\pi,q,f)=\mathbb{E}_{o_i,o_j\sim\mathcal{D}}\Big[\mathbb{I}\big((R_i-R_j)(f(o_i)-f(o_j))>0\big)\Big],$$6, ECE $$\mathrm{AUC}(\pi,q,f)=\mathbb{E}_{o_i,o_j\sim\mathcal{D}}\Big[\mathbb{I}\big((R_i-R_j)(f(o_i)-f(o_j))>0\big)\Big],$$7, and PCE $$\mathrm{AUC}(\pi,q,f)=\mathbb{E}_{o_i,o_j\sim\mathcal{D}}\Big[\mathbb{I}\big((R_i-R_j)(f(o_i)-f(o_j))>0\big)\Big],$$8, which the paper treats as empirical confirmation of the gradient-conflict analysis (Ma et al., 10 Mar 2026).

Taken together, these papers suggest two distinct calibration-aware strategies for reasoning LLMs. One treats calibration as a ranking problem over correct and incorrect responses and alters the advantage accordingly (Wang et al., 14 Apr 2026). The other treats calibration as a separate prediction problem and enforces structural decoupling between the reasoning policy and the confidence output (Ma et al., 10 Mar 2026).

5. Uncertainty- and evidence-calibrated variants

The paper “Why Semantic Entropy Fails: Geometry-Aware and Calibrated Uncertainty for Policy Optimization” frames CAPO as regulation of gradient variance by uncertainty signals that are themselves calibrated to learning-signal quality (Zhang et al., 20 May 2026). Its diagnosis is that entropy-based estimators suffer from two defects: the anisotropic gap, because semantic entropy depends only on cluster masses and discards intra-cluster variance, and the calibration gap, because entropy ignores reward dispersion and therefore suppresses high-variance, high-signal samples together with genuinely noisy ones (Zhang et al., 20 May 2026). GCPO addresses this by combining geometry-aware disagreement measures—Cosine Dispersion or Barycentric Transport—with Reward Dispersion into a query-level weight

$$\mathrm{AUC}(\pi,q,f)=\mathbb{E}_{o_i,o_j\sim\mathcal{D}}\Big[\mathbb{I}\big((R_i-R_j)(f(o_i)-f(o_j))>0\big)\Big],$$9

which rescales normalized advantages in GRPO-style training (Zhang et al., 20 May 2026). On NarrativeQA, the reported F1 scores are $$J_{\text{logistic}}(\theta) = -\mathbb{E}_{o_1,o_2\sim\mathcal{D}} \Big[ \log\big(1+\exp(-t/\tau)\big) \Big],$$0 for GRPO, $$J_{\text{logistic}}(\theta) = -\mathbb{E}_{o_1,o_2\sim\mathcal{D}} \Big[ \log\big(1+\exp(-t/\tau)\big) \Big],$$1 for GRPO+SE, and $$J_{\text{logistic}}(\theta) = -\mathbb{E}_{o_1,o_2\sim\mathcal{D}} \Big[ \log\big(1+\exp(-t/\tau)\big) \Big],$$2 for GRPO+BoT; on HotpotQA EM, the corresponding numbers are $$J_{\text{logistic}}(\theta) = -\mathbb{E}_{o_1,o_2\sim\mathcal{D}} \Big[ \log\big(1+\exp(-t/\tau)\big) \Big],$$3, $$J_{\text{logistic}}(\theta) = -\mathbb{E}_{o_1,o_2\sim\mathcal{D}} \Big[ \log\big(1+\exp(-t/\tau)\big) \Big],$$4, and $$J_{\text{logistic}}(\theta) = -\mathbb{E}_{o_1,o_2\sim\mathcal{D}} \Big[ \log\big(1+\exp(-t/\tau)\big) \Big],$$5. The paper states that geometry-aware measures correlate more strongly with sample-level gradient variance than entropy or semantic entropy and that benefits are larger when semantic variation aligns tightly with optimizer-relevant variance (Zhang et al., 20 May 2026).

The paper “When Denser Credit Is Not Enough: Evidence-Calibrated Policy Optimization for Long-Horizon LLM Agent Training” pushes the calibration theme from query weighting to step-level credit assignment (Li et al., 4 Jun 2026). Its claim is that dense anchor-based credit, as in GiGPO, can still be statistically unreliable because rare but lucky actions can receive overly large advantages, producing divergent anchor bias and late-stage training oscillation (Li et al., 4 Jun 2026). ECPO introduces Evidence-Calibrated Action Advantage, which groups rollouts by canonical actions and shrinks low-count estimates toward an anchor mean, and Variance-Gated Credit Weighting, which assigns each anchor a reliability weight based on between-action versus within-action variance. The final advantage is

$$J_{\text{logistic}}(\theta) = -\mathbb{E}_{o_1,o_2\sim\mathcal{D}} \Big[ \log\big(1+\exp(-t/\tau)\big) \Big],$$6

so step-level credit is used only after evidence calibration (Li et al., 4 Jun 2026). On Qwen2.5-1.5B, ECPO improves GiGPO by $$J_{\text{logistic}}(\theta) = -\mathbb{E}_{o_1,o_2\sim\mathcal{D}} \Big[ \log\big(1+\exp(-t/\tau)\big) \Big],$$7 success points on ALFWorld and $$J_{\text{logistic}}(\theta) = -\mathbb{E}_{o_1,o_2\sim\mathcal{D}} \Big[ \log\big(1+\exp(-t/\tau)\big) \Big],$$8 success points on WebShop while adding only $$J_{\text{logistic}}(\theta) = -\mathbb{E}_{o_1,o_2\sim\mathcal{D}} \Big[ \log\big(1+\exp(-t/\tau)\big) \Big],$$9 additional advantage-computation overhead (Li et al., 4 Jun 2026).

A plausible implication is that CAPO has expanded from confidence-aware reward shaping to a more general doctrine: uncertainty signals should be trusted only when they are calibrated to actual optimizer-relevant evidence.

6. Safety, deployment, broader applications, and limitations

Conformal Policy Control extends the CAPO idea from training-time weighting to deployment-time regulation (Prinster et al., 2 Mar 2026). Its setting assumes a safe reference policy $t$0 and a newly optimized but untested policy $t$1. CPC constructs a constrained family

$t$2

then uses conformal calibration on data from the safe policy to choose the most aggressive $t$3 that still satisfies the user’s declared risk tolerance $t$4 under finite-sample guarantees, including for non-monotonic bounded constraint functions (Prinster et al., 2 Mar 2026). The paper’s experiments in question answering, active learning, and biomolecular sequence optimization argue that safe exploration is possible from the first moment of deployment and can also improve performance (Prinster et al., 2 Mar 2026). In CAPO terms, this is a calibration layer wrapped around arbitrary policy optimization rather than a new optimizer.

Calibration-aware policy optimization has also appeared outside language-model alignment. “Graph Reinforcement Learning for Calibration-Aware Quantum Circuit Routing” trains a graph-based PPO router that uses same-day IBM Heron r2 calibration data in both state representation and reward (Tomar et al., 11 Jun 2026). Across nine MQT Bench circuits and three calibration snapshots, the reported pooled mean exact fidelity is $t$5, compared with $t$6 for SABRE-best20 and $t$7 for target-aware SABRE, although gains are concentrated in the 5q and 8q circuit families and all 10q families favor SABRE-best20 under the fixed tree action graph (Tomar et al., 11 Jun 2026). This suggests that CAPO need not be limited to epistemic confidence about text generation; it can also denote direct optimization against hardware calibration data.

Across these papers, the main limitations are heterogeneous but structurally related. The multilingual preference paper is limited to $t$8 training pairs, $t$9 English-source directions, and a single 8B model with sensitivity to the hyperparameter $\mathcal{D}=\{(x,y_w,y_\ell)\},$0 (Pokharel et al., 10 Nov 2025). The reasoning-RLVR line assumes either reliable relative-calibration ranking or a confidence channel that can be optimized separately, and it remains tied to verifiable correctness settings (Wang et al., 14 Apr 2026, Ma et al., 10 Mar 2026). GCPO introduces a biased, reweighted gradient estimator and depends on semantic–gradient alignment, with weaker gains when that alignment is weak (Zhang et al., 20 May 2026). ECPO depends on repeated or canonicalizable anchor states and can underestimate truly rare but high-value actions because shrinkage trades variance for bias (Li et al., 4 Jun 2026). CPC provides marginal expected-risk guarantees rather than conditional guarantees and assumes access to evaluable policy likelihoods, bounded losses, and stability of the hyperparameter-selection procedure (Prinster et al., 2 Mar 2026). The quantum-routing variant is trained on three same-day snapshots and a fixed 10-qubit tree, so calibration drift and topology dependence remain open concerns (Tomar et al., 11 Jun 2026).

A frequent misconception is that CAPO names a single standardized algorithm. The record is more plural. Recent work uses CAPO to describe a family of interventions—confidence-aware margin shaping, AUC-consistent ranking losses, decoupled confidence heads, geometry-aware uncertainty weighting, evidence-calibrated credit shrinkage, and conformal risk regulation—that all modify policy optimization by asking not only whether a sample is useful, but whether the signal governing its update is itself trustworthy.