---
title: Proportionate Credit Policy Optimization (PCPO)
url: https://www.emergentmind.com/topics/proportionate-credit-policy-optimization-pcpo
type: topic
---

# Proportionate Credit Policy Optimization (PCPO)

Searching arXiv for the relevant PCPO variants and closely related token-credit papers.
Proportionate Credit Policy Optimization (PCPO) denotes a class of policy-optimization methods in which credit is assigned in proportion to the estimated contribution of individual tokens, spans, or timesteps to task success, rather than being broadcast uniformly across an entire sampled trajectory. In current arXiv usage, the term appears in at least two technically distinct but conceptually aligned settings: language-model reasoning, where token credit is derived from counterfactual effects on final-answer probability, and text-to-image alignment, where timestep credit is reweighted to correct non-proportional feedback induced by diffusion or flow samplers [2602.09331], [2509.25774]. A related formulation, Guidance Contrastive Policy Optimization (GCPO), explicitly presents itself as a concrete PCPO instantiation for discrete policies, using contrastive positive-versus-negative conditioning to produce per-token advantages [2605.29198].

## 1. Concept and scope

PCPO is motivated by the observation that standard group-advantage policy-gradient methods such as GRPO and DAPO often apply the same advantage to all actions in a completion or rollout, even when those actions play sharply different causal or semantic roles. In the language-model setting, this means that a filler phrase such as “Let me think” can receive the same update as a critical computation such as “23 + 45 = 68,” which dilutes gradient updates with non-causal content, slows learning, and reinforces scaffolding [2602.09331]. In the text-to-image setting, the analogous problem is disproportionate timestep feedback induced by the sampler’s mathematical structure, which creates volatile and non-proportional gradient magnitudes across denoising or flow steps [2509.25774].

The common principle is proportionate credit: the update magnitude should track the contribution of the local action to the outcome. In the formal definition given for reasoning models, PCPO is policy optimization where per-token credit is proportionate to the estimated causal impact of that token, or of its containing span, on success probability [2602.09331]. In the broader framing used by GCPO, PCPO is a general design principle: align per-token credit to each token’s relative contribution to task success, compute token-wise scaling weights from a task-relevant guidance signal, and retain the underlying trust-region optimization while replacing uniform broadcasting with token-wise scaling [2605.29198].

This shared principle should be distinguished from unrelated uses of the acronym “PCPO” in constrained reinforcement learning. In arXiv, “PCPO” also denotes “Proactive Constrained Policy Optimization” [2508.01883] and “Projection-Based Constrained Policy Optimization” [2010.03152], neither of which addresses proportionate credit assignment.

## 2. Counterfactual PCPO for reasoning models

In the reasoning formulation introduced in “Beyond Uniform Credit: Causal Credit Assignment for Policy Optimization” [2602.09331], PCPO is a policy-gradient framework for training language models on reasoning tasks in which the credit assigned to each generated token or action is proportional to its estimated causal impact on the probability of producing the correct final answer. The method replaces the uniform token credit used by GRPO and DAPO with counterfactual importance weights derived directly from the policy model by masking candidate reasoning spans and measuring the drop in answer probability.

The setup uses an input prompt $x$, a state sequence $s_{1:T}$, an autoregressive policy $\pi_\theta$ over token-actions $a_{1:T}$, and a terminal verifiable reward $R \in \{0,1\}$ indicating whether the final answer is correct. The standard REINFORCE gradient with baseline $b$ is

$$
\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].
$$

PCPO injects token-level weights $w_t$:

$$
\nabla_\theta J_{\text{PCPO}}(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T w_t \;\nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].
$$

In the DAPO instantiation, the per-completion advantage $A_i$ replaces $(R-b)$, and the corresponding batch loss is

$$
\mathcal{L}_{\text{PCPO}} = -\frac{1}{\sum_i T_i} \sum_{i=1}^{G} \sum_{t=1}^{T_i} w_t^{(i)} \cdot A_i \cdot \log \pi_\theta\big(y_t^{(i)} \mid x, y_{<t}^{(i)}\big).
$$

The central operation is “mask-and-measure” counterfactual estimation. Let $y^*$ be the correct final answer string, and let a generated completion be decomposed into a reasoning prefix $r$ and an answer span. For a candidate reasoning span $S \subset r$, a masked prefix $r_{-S}$ is constructed by replacing the tokens in $S$ with a fixed placeholder string of equal length. The counterfactual effect of $S$ on answer probability is

$$
\tau(S) = p_\theta(y^* \mid x, r) - p_\theta\big(y^* \mid x, r_{-S}\big).
$$

In log-space, which is used in the experiments,

$$
D(S) = \log p_\theta\big(y^* \mid x, r_{-S}\big) - \log p_\theta(y^* \mid x, r), \quad I(S) = -D(S) \ge 0.
$$

Large drops imply that the span is causally necessary for the answer and should receive more credit; positive $D(S)$ indicates a distractor. Span-level importance is normalized within each completion and mapped to token weights:

$$
w_t = w_{\min} + \hat{I}(S)\,\big(w_{\max} - w_{\min}\big), \quad \hat{I}(S) = \frac{I(S) - \min_j I(S_j)}{\max_j I(S_j) - \min_j I(S_j) + \epsilon}.
$$

Tokens outside detected spans receive baseline weight $w_t = 1$, and final answer tokens receive a small boost, with the reported experiments using $w_{\text{ans}} = 1.5$ and $[w_{\min}, w_{\max}] = [0.5, 4.0]$ [2602.09331].

The implementation estimates importance without auxiliary models. Reasoning spans are detected with regex-based heuristics tuned for math, including arithmetic expressions, intermediate calculations, and sentence-like boundaries. Detection is capped at $K_{\max} = 10$ spans per completion. Counterfactual answer probabilities are computed under teacher forcing, all masked forward passes run under `torch.no_grad()`, and masking uses replacement rather than deletion to preserve positional structure and avoid length-change confounds [2602.09331].

## 3. Algorithmic structure and empirical behavior in reasoning

The GSM8K experiments reported for counterfactual PCPO use Qwen3-1.7B, Qwen2.5-3B, and Llama3.2-3B with LoRA $r = 32$, $\alpha = 32$, exact-match numeric reward $r \in \{0,1\}$, group size $G = 8$, temperature $0.6$, top-$p$ $0.95$, learning rate $2.5 \times 10^{-5}$, batch size $16$, gradient accumulation $4$, $500$ steps, and $3$ random seeds [2602.09331].

The training loop proceeds by sampling $G$ completions per prompt, computing exact-match rewards on GSM8K, forming DAPO group-normalized advantages
$$
A_i = \frac{r_i - \mu_r}{\sigma_r + \epsilon},
$$
splitting each completion into a reasoning prefix and a final numeric answer, detecting up to $K_{\max}$ reasoning spans, computing the original log answer probability, computing one masked log answer probability per span, converting the resulting drops into normalized importance scores, assigning token weights, and then accumulating the importance-weighted DAPO loss. The optimizer step can optionally use DAPO’s decoupled clipping, specifically Clip-Higher, on the token-level loss [2602.09331].

On GSM8K, the method yields consistent gains over uniform-credit DAPO at step $500$. For test accuracy, the reported mean $\pm$ standard deviation over seeds is Qwen3-1.7B: CF $84.3 \pm 0.5$ versus DAPO $83.4 \pm 0.6$; Qwen2.5-3B: CF $86.7 \pm 0.2$ versus DAPO $85.6 \pm 0.2$ with $p = 0.002$; and Llama3.2-3B: CF $78.9 \pm 0.3$ versus DAPO $78.2 \pm 0.6$ [2602.09331]. For cumulative accuracy across training, the reported AUC values are Qwen3-1.7B: CF $83.2 \pm 0.4$ versus DAPO $82.5 \pm 0.5$ with $p = 0.035$; Qwen2.5-3B: CF $85.8 \pm 0.4$ versus DAPO $85.3 \pm 0.4$ with $p = 0.049$; and Llama3.2-3B: CF $77.3 \pm 0.3$ versus DAPO $76.6 \pm 0.6$ [2602.09331].

The ablations are designed to test whether the learned importance signal tracks genuine causal structure rather than arbitrary non-uniformity. Inverted weights, defined as $w_t \leftarrow 1 - \hat{I}$, consistently underperform DAPO by $1.5$–$1.8$ percentage points, while random weights are broadly neutral or slightly worse than DAPO [2602.09331]. The paper interprets this as evidence that directionality matters and that non-uniform credit alone does not explain the gains.

Analysis of gradient allocation is explicitly quantitative. High-importance tokens, defined as $\hat{I} > 0.8$, comprise $26.3\%$ of tokens but receive $42.5\%$ of gradient mass under PCPO, a $1.6\times$ concentration. Low-importance tokens, comprising $53.9\%$, receive only $32.2\%$ of gradient mass, a $0.6\times$ dilution [2602.09331]. Content analysis further reports that calculation chains are $11.2\times$ more prevalent in critical spans, multiply/divide is $6.5\times$ more prevalent, and proportion/rate is $3.9\times$ more prevalent, whereas setup text such as “let…” and step headers is $0.37$–$0.62\times$ depleted. Distractors constitute $3.5\%$ of spans, defined by positive $D(S)$ where masking improves answer probability, and are dominated by verbose headers and redundant restatements [2602.09331].

A negative result is also reported: on MBPP+, PCPO provides no consistent benefit over DAPO, with pass@1 fluctuating within $\pm 1$–$2\%$. The stated interpretation is that the method is best suited to tasks with a concentrated “answer sink,” specifically a single numeric target, rather than tasks with distributed correctness across many structural elements [2602.09331].

## 4. Diffusion and flow PCPO for image-generation alignment

A distinct use of the same name appears in “PCPO: Proportionate Credit Policy Optimization for Aligning Image Generation Models” [2509.25774]. Here the problem is not token-level reasoning credit but disproportionate timestep credit in reinforcement learning for text-to-image diffusion and flow models. The paper follows the MDP formulation used for diffusion-based text-to-image alignment, with states $s_t = (c, t, x_t)$, actions $a_t = x_{t-1}$, and a terminal reward $r(x_0, c)$ assigned at the final step.

The baseline PPO-style clipped objective is written as

$$
\mathcal{L}_{\mathrm{PPO}}(\theta) := \mathbb{E}_{\tau \sim p_{\theta_\text{old}}}
\left[ \sum_{t=1}^{T} \max
\left( -\rho_t A , -\mathrm{clip}_\xi(\rho_t) A \right) \right],
$$

where $A$ is the normalized terminal reward and $\rho_t := p_\theta^{(t)} / p_{\theta_\text{old}}^{(t)}$. The reported diagnosis is that the mathematical structure of generative samplers inherently induces volatile, non-proportional feedback across timesteps: $\log \rho_t$ decomposes into terms scaled by a native weight $w(t)$ spanning orders of magnitude, which yields inconsistent per-timestep gradient magnitudes, preferential clipping of amplified steps, and high-variance updates that destabilize learning and drive model collapse [2509.25774].

The first component of this PCPO variant is a stable objective reformulation. Following a hinge-loss view of PPO, the gradient of the PPO loss is taken to be equivalent to that of

$$
\mathcal{L}_{\text{hinge}} := \mathbb{E} [\sum_t \max\{0, \xi|A|-A(\rho_t-1)\}],
$$

and the method replaces $(\rho_t-1)$ by $\log \rho_t$ under tight clipping, giving the base PCPO objective

$$
\mathcal{L}_{\text{PCPO-base}}(\theta) := \mathbb{E}
\left[ \sum_{t=1}^{T} \max \bigl\{0,\; \xi|A|-A\,\log\rho_t \bigr\} \right].
$$

The empirical approximation error of $\log \rho_t \approx \rho_t - 1$ is reported as never exceeding $1.2\%$ during training [2509.25774].

The second component is timestep reweighting to enforce proportionate credit. For DDIM diffusion, the paper derives

$$
\log\rho_t =
-\Bigl[w(t) (\hat{\varepsilon}_\theta^{(t)}-\hat{\varepsilon}_{\text{old}}^{(t)})
\!\cdot\! \epsilon_{\text{old}}^{(t)} \;+\; \tfrac12
\bigl\| w(t)( \hat{\varepsilon}_\theta^{(t)} - \hat{\varepsilon}_{\text{old}}^{(t)} )
\bigr\|^{2}
\Bigr], \qquad w(t)=\frac{C(t)}{\sigma_t}.
$$

This makes the native weighting explicit. The method then sets $w(t)=w^\star$ constant for all diffusion timesteps by re-engineering the DDIM variance schedule $\tilde{\sigma}_t$, with a reported example value $w^\star = 4.5$, rescaled to preserve the mean of the original weights [2509.25774]. For flow models, where timestep shifting induces non-uniform $\Delta t_i$, the objective is reweighted directly with

$$
w(t_i) = \zeta \Delta t_i,
$$

so that credit is proportional to the integration interval without altering the sampler [2509.25774].

This variant therefore uses “proportionate credit” in a timestep sense rather than a token sense. The underlying idea remains aligned with the general PCPO principle: credit should track the mathematically relevant local contribution rather than be distorted by the optimization parameterization.

## 5. Empirical results in text-to-image alignment

The image-generation PCPO paper reports experiments on DDPO with Stable Diffusion 1.5 and on DanceGRPO with SD1.4 and FLUX.1-dev, using Aesthetics, BERTScore, and HPSv2.1 rewards [2509.25774]. Evaluation is conducted on $50$k images with FID, Inception Score, LPIPS Diversity, per-prompt Linear Mixed Model analysis, and a human preference study.

Across all reported settings, PCPO reduces clipping and accelerates convergence. The stated speedups in epochs to reach primary targets are DDPO Aesthetics: $24.6\%$ for target $6.90$ ($147 \rightarrow 118$), BERTScore: $30.8\%$ for target $0.52$ ($191 \rightarrow 146$), DanceGRPO SD1.4: $25.5\%$ for HPS $= 0.370$ ($236 \rightarrow 188$), and DanceGRPO FLUX: $41.2\%$ for HPS $= 0.360$ ($209 \rightarrow 148$). For FLUX at HPS $= 0.35$, an additional threshold gives a $59.1\%$ speedup ($148 \rightarrow 93$) [2509.25774].

The quality metrics also improve. For DDPO Aesthetics at batch size $256$, the reported values are FID $31.72 \rightarrow 27.86$, IS* $26.35 \rightarrow 24.12$, and LPIPS $0.6208 \rightarrow 0.6262$; at batch size $512$, FID $24.09 \rightarrow 22.06$, IS* $25.67 \rightarrow 25.65$, and LPIPS $0.6321 \rightarrow 0.6525$ [2509.25774]. For DanceGRPO HPS, the reported SD1.4 values are FID $90.34 \rightarrow 84.74$, IS* $7.61 \rightarrow 7.50$, and LPIPS $0.4948 \rightarrow 0.4894$; the FLUX values are FID $46.23 \rightarrow 40.38$, IS* $12.66 \rightarrow 11.90$, and LPIPS $0.5736 \rightarrow 0.5708$ [2509.25774]. The paper interprets lower IS in this setting as reduced mode collapse and states that the improvement in image quality is a direct result of mitigating model collapse in recursive training.

The component ablation is also framed in terms of credit proportionality. Sequentially adding the log-$\rho_t$ objective, $\varepsilon$-matching, and proportional weighting lowers clipping progressively; only full PCPO achieves a zero on-policy clipping ratio [2509.25774]. Heuristic alternatives fare worse. Timestep subsampling reduces wall-clock per epoch but degrades quality and diversity at matched rewards, and a uniform reweighting scheme that emphasizes high-noise timesteps underperforms vanilla $\sqrt{\Delta t_i}$ weighting, which the paper presents as supporting the proportionality principle [2509.25774].

A plausible implication is that, in this literature, “proportionate credit” has broadened from token-level attribution to a more general requirement that the policy-gradient signal be aligned with the effective local contribution induced by the model’s generation dynamics.

## 6. Related formulations, acronym ambiguity, and limitations

GCPO provides the clearest bridge between token-level credit assignment and a broader PCPO perspective. In “Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization” [2605.29198], the group advantage $\widehat{A}_i$ is scaled by a per-token guidance factor derived from the KL divergence between predictive distributions under positive and negative prompts:

$$
\eta_{i,t}=\mathbb{D}_\text{KL}\Big( \pi_{\theta} (y_{i,t} | x, y_{i,<t})\big\|\pi_{\theta} (y_{i,t} | x^-, y_{i,<t}) \Big).
$$

After sequence-wise CDF normalization, GCPO defines

$$
\widehat{A}^{\text{GCPO}}_{i,t} = \eta_{\text{normalized},i,t}\,\widehat{A}_i.
$$

The paper explicitly states that this makes GCPO a proportionate-credit method and presents the relation
$$
\widehat{A}^{\text{PCPO}}_{i,t} = \gamma_{i,t}\,\widehat{A}_i
\quad\text{with}\quad
\gamma_{i,t}=\eta_{\text{normalized},i,t},
$$
thereby treating PCPO as the general principle and GCPO as one instantiation [2605.29198]. Empirically, GCPO outperforms GRPO and DAPO on text-to-image and multimodal reasoning benchmarks, with KL divergence and histogram-CDF normalization performing best among the tested contrast metrics and normalization schemes [2605.29198].

At the same time, acronym ambiguity is substantial. “PCPO” also names “Proactive Constrained Policy Optimization,” a safe-RL method based on a preemptive extended log-barrier and a constraint-aware intrinsic reward [2508.01883], and “Projection-Based Constrained Policy Optimization,” a two-step CMDP algorithm consisting of a reward-improvement step followed by projection onto a linearized feasible set [2010.03152]. Both papers explicitly clarify that their “PCPO” does not denote proportionate credit assignment.

The current proportionate-credit literature also identifies several limitations. In counterfactual PCPO for reasoning, importance is defined solely via final-answer likelihood, masking with pad or end-of-text may induce distribution shift, regex span detection is tuned for math rather than code, and experiments extend only up to $3$B parameters, with benefits at $7$B and above left untested [2602.09331]. In GCPO, applicability depends on prompt-dependent rewards and on the model’s ability to respond meaningfully to negative prompts; weaker models may produce unreliable contrast signals [2605.29198]. In text-to-image PCPO, the diffusion formulation assumes equal integration intervals and approximately constant dot-product statistics across timesteps, while the flow formulation depends on accurate accounting of non-uniform $\Delta t_i$ and does not eliminate all off-policy clipping [2509.25774].

These limitations indicate that PCPO is presently better understood as a methodological direction than as a single closed-form algorithm. The recurring claim across its variants is narrower but technically precise: uniform credit is often misaligned with causal or dynamical contribution, and explicit proportionalization of credit can improve optimization behavior when the weighting signal is task-faithful [2602.09331], [2509.25774], [2605.29198].

Source: https://www.emergentmind.com/topics/proportionate-credit-policy-optimization-pcpo