Proportionate Credit Policy Optimization (PCPO)

• PCPO is an optimization paradigm that assigns credit proportional to individual token or timestep contributions to task success.
• It refines policy gradients by weighting actions based on counterfactual impacts and diffusion dynamics.
• Empirical studies show PCPO accelerates convergence and improves performance versus uniform credit methods in language and image tasks.

Searching arXiv for the relevant PCPO variants and closely related token-credit papers. Proportionate Credit Policy Optimization (PCPO) denotes a class of policy-optimization methods in which credit is assigned in proportion to the estimated contribution of individual tokens, spans, or timesteps to task success, rather than being broadcast uniformly across an entire sampled trajectory. In current arXiv usage, the term appears in at least two technically distinct but conceptually aligned settings: language-model reasoning, where token credit is derived from counterfactual effects on final-answer probability, and text-to-image alignment, where timestep credit is reweighted to correct non-proportional feedback induced by diffusion or flow samplers (Khandoga et al., 10 Feb 2026, Lee et al., 30 Sep 2025). A related formulation, Guidance Contrastive Policy Optimization (GCPO), explicitly presents itself as a concrete PCPO instantiation for discrete policies, using contrastive positive-versus-negative conditioning to produce per-token advantages (Li et al., 28 May 2026).

1. Concept and scope

PCPO is motivated by the observation that standard group-advantage policy-gradient methods such as GRPO and DAPO often apply the same advantage to all actions in a completion or rollout, even when those actions play sharply different causal or semantic roles. In the language-model setting, this means that a filler phrase such as “Let me think” can receive the same update as a critical computation such as “23 + 45 = 68,” which dilutes gradient updates with non-causal content, slows learning, and reinforces scaffolding (Khandoga et al., 10 Feb 2026). In the text-to-image setting, the analogous problem is disproportionate timestep feedback induced by the sampler’s mathematical structure, which creates volatile and non-proportional gradient magnitudes across denoising or flow steps (Lee et al., 30 Sep 2025).

The common principle is proportionate credit: the update magnitude should track the contribution of the local action to the outcome. In the formal definition given for reasoning models, PCPO is policy optimization where per-token credit is proportionate to the estimated causal impact of that token, or of its containing span, on success probability (Khandoga et al., 10 Feb 2026). In the broader framing used by GCPO, PCPO is a general design principle: align per-token credit to each token’s relative contribution to task success, compute token-wise scaling weights from a task-relevant guidance signal, and retain the underlying trust-region optimization while replacing uniform broadcasting with token-wise scaling (Li et al., 28 May 2026).

This shared principle should be distinguished from unrelated uses of the acronym “PCPO” in constrained reinforcement learning. In arXiv, “PCPO” also denotes “Proactive Constrained Policy Optimization” (Yang et al., 3 Aug 2025) and “Projection-Based Constrained Policy Optimization” (Yang et al., 2020), neither of which addresses proportionate credit assignment.

2. Counterfactual PCPO for reasoning models

In the reasoning formulation introduced in “Beyond Uniform Credit: Causal Credit Assignment for Policy Optimization” (Khandoga et al., 10 Feb 2026), PCPO is a policy-gradient framework for training LLMs on reasoning tasks in which the credit assigned to each generated token or action is proportional to its estimated causal impact on the probability of producing the correct final answer. The method replaces the uniform token credit used by GRPO and DAPO with counterfactual importance weights derived directly from the policy model by masking candidate reasoning spans and measuring the drop in answer probability.

The setup uses an input prompt $x$, a state sequence $s_{1:T}$, an autoregressive policy $\pi_\theta$ over token-actions $a_{1:T}$, and a terminal verifiable reward $R \in \{0,1\}$ indicating whether the final answer is correct. The standard REINFORCE gradient with baseline $b$ is

$$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$

PCPO injects token-level weights $w_t$:

$$\nabla_\theta J_{\text{PCPO}}(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T w_t \;\nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$

In the DAPO instantiation, the per-completion advantage $A_i$ replaces $s_{1:T}$0, and the corresponding batch loss is

$s_{1:T}$1

The central operation is “mask-and-measure” counterfactual estimation. Let $s_{1:T}$2 be the correct final answer string, and let a generated completion be decomposed into a reasoning prefix $s_{1:T}$3 and an answer span. For a candidate reasoning span $s_{1:T}$4, a masked prefix $s_{1:T}$5 is constructed by replacing the tokens in $s_{1:T}$6 with a fixed placeholder string of equal length. The counterfactual effect of $s_{1:T}$7 on answer probability is

$s_{1:T}$8

In log-space, which is used in the experiments,

$s_{1:T}$9

Large drops imply that the span is causally necessary for the answer and should receive more credit; positive $\pi_\theta$0 indicates a distractor. Span-level importance is normalized within each completion and mapped to token weights:

$\pi_\theta$1

Tokens outside detected spans receive baseline weight $\pi_\theta$2, and final answer tokens receive a small boost, with the reported experiments using $\pi_\theta$3 and $\pi_\theta$4 (Khandoga et al., 10 Feb 2026).

The implementation estimates importance without auxiliary models. Reasoning spans are detected with regex-based heuristics tuned for math, including arithmetic expressions, intermediate calculations, and sentence-like boundaries. Detection is capped at $\pi_\theta$5 spans per completion. Counterfactual answer probabilities are computed under teacher forcing, all masked forward passes run under torch.no_grad(), and masking uses replacement rather than deletion to preserve positional structure and avoid length-change confounds (Khandoga et al., 10 Feb 2026).

3. Algorithmic structure and empirical behavior in reasoning

The GSM8K experiments reported for counterfactual PCPO use Qwen3-1.7B, Qwen2.5-3B, and Llama3.2-3B with LoRA $\pi_\theta$6, $\pi_\theta$7, exact-match numeric reward $\pi_\theta$8, group size $\pi_\theta$9, temperature $a_{1:T}$0, top-$a_{1:T}$1 $a_{1:T}$2, learning rate $a_{1:T}$3, batch size $a_{1:T}$4, gradient accumulation $a_{1:T}$5, $a_{1:T}$6 steps, and $a_{1:T}$7 random seeds (Khandoga et al., 10 Feb 2026).

The training loop proceeds by sampling $a_{1:T}$8 completions per prompt, computing exact-match rewards on GSM8K, forming DAPO group-normalized advantages

$a_{1:T}$9

splitting each completion into a reasoning prefix and a final numeric answer, detecting up to $R \in \{0,1\}$0 reasoning spans, computing the original log answer probability, computing one masked log answer probability per span, converting the resulting drops into normalized importance scores, assigning token weights, and then accumulating the importance-weighted DAPO loss. The optimizer step can optionally use DAPO’s decoupled clipping, specifically Clip-Higher, on the token-level loss (Khandoga et al., 10 Feb 2026).

On GSM8K, the method yields consistent gains over uniform-credit DAPO at step $R \in \{0,1\}$1. For test accuracy, the reported mean $R \in \{0,1\}$2 standard deviation over seeds is Qwen3-1.7B: CF $R \in \{0,1\}$3 versus DAPO $R \in \{0,1\}$4; Qwen2.5-3B: CF $R \in \{0,1\}$5 versus DAPO $R \in \{0,1\}$6 with $R \in \{0,1\}$7; and Llama3.2-3B: CF $R \in \{0,1\}$8 versus DAPO $R \in \{0,1\}$9 (Khandoga et al., 10 Feb 2026). For cumulative accuracy across training, the reported AUC values are Qwen3-1.7B: CF $b$0 versus DAPO $b$1 with $b$2; Qwen2.5-3B: CF $b$3 versus DAPO $b$4 with $b$5; and Llama3.2-3B: CF $b$6 versus DAPO $b$7 (Khandoga et al., 10 Feb 2026).

The ablations are designed to test whether the learned importance signal tracks genuine causal structure rather than arbitrary non-uniformity. Inverted weights, defined as $b$8, consistently underperform DAPO by $b$9–$$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$0 percentage points, while random weights are broadly neutral or slightly worse than DAPO (Khandoga et al., 10 Feb 2026). The paper interprets this as evidence that directionality matters and that non-uniform credit alone does not explain the gains.

Analysis of gradient allocation is explicitly quantitative. High-importance tokens, defined as $$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$1, comprise $$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$2 of tokens but receive $$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$3 of gradient mass under PCPO, a $$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$4 concentration. Low-importance tokens, comprising $$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$5, receive only $$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$6 of gradient mass, a $$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$7 dilution (Khandoga et al., 10 Feb 2026). Content analysis further reports that calculation chains are $$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$8 more prevalent in critical spans, multiply/divide is $$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$9 more prevalent, and proportion/rate is $w_t$0 more prevalent, whereas setup text such as “let…” and step headers is $w_t$1–$w_t$2 depleted. Distractors constitute $w_t$3 of spans, defined by positive $w_t$4 where masking improves answer probability, and are dominated by verbose headers and redundant restatements (Khandoga et al., 10 Feb 2026).

A negative result is also reported: on MBPP+, PCPO provides no consistent benefit over DAPO, with pass@1 fluctuating within $w_t$5–$w_t$6. The stated interpretation is that the method is best suited to tasks with a concentrated “answer sink,” specifically a single numeric target, rather than tasks with distributed correctness across many structural elements (Khandoga et al., 10 Feb 2026).

4. Diffusion and flow PCPO for image-generation alignment

A distinct use of the same name appears in “PCPO: Proportionate Credit Policy Optimization for Aligning Image Generation Models” (Lee et al., 30 Sep 2025). Here the problem is not token-level reasoning credit but disproportionate timestep credit in reinforcement learning for text-to-image diffusion and flow models. The paper follows the MDP formulation used for diffusion-based text-to-image alignment, with states $w_t$7, actions $w_t$8, and a terminal reward $w_t$9 assigned at the final step.

The baseline PPO-style clipped objective is written as

$$\nabla_\theta J_{\text{PCPO}}(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T w_t \;\nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$0

where $$\nabla_\theta J_{\text{PCPO}}(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T w_t \;\nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$1 is the normalized terminal reward and $$\nabla_\theta J_{\text{PCPO}}(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T w_t \;\nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$2. The reported diagnosis is that the mathematical structure of generative samplers inherently induces volatile, non-proportional feedback across timesteps: $$\nabla_\theta J_{\text{PCPO}}(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T w_t \;\nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$3 decomposes into terms scaled by a native weight $$\nabla_\theta J_{\text{PCPO}}(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T w_t \;\nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$4 spanning orders of magnitude, which yields inconsistent per-timestep gradient magnitudes, preferential clipping of amplified steps, and high-variance updates that destabilize learning and drive model collapse (Lee et al., 30 Sep 2025).

The first component of this PCPO variant is a stable objective reformulation. Following a hinge-loss view of PPO, the gradient of the PPO loss is taken to be equivalent to that of

$$\nabla_\theta J_{\text{PCPO}}(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T w_t \;\nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$5

and the method replaces $$\nabla_\theta J_{\text{PCPO}}(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T w_t \;\nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$6 by $$\nabla_\theta J_{\text{PCPO}}(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T w_t \;\nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$7 under tight clipping, giving the base PCPO objective

$$\nabla_\theta J_{\text{PCPO}}(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T w_t \;\nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$8

The empirical approximation error of $$\nabla_\theta J_{\text{PCPO}}(\theta) = \mathbb{E}_{\pi_\theta}\Big[ \sum_{t=1}^T w_t \;\nabla_\theta \log \pi_\theta(a_t \mid s_t)\, (R - b) \Big].$$9 is reported as never exceeding $A_i$0 during training (Lee et al., 30 Sep 2025).

The second component is timestep reweighting to enforce proportionate credit. For DDIM diffusion, the paper derives

$A_i$1

This makes the native weighting explicit. The method then sets $A_i$2 constant for all diffusion timesteps by re-engineering the DDIM variance schedule $A_i$3, with a reported example value $A_i$4, rescaled to preserve the mean of the original weights (Lee et al., 30 Sep 2025). For flow models, where timestep shifting induces non-uniform $A_i$5, the objective is reweighted directly with

$A_i$6

so that credit is proportional to the integration interval without altering the sampler (Lee et al., 30 Sep 2025).

This variant therefore uses “proportionate credit” in a timestep sense rather than a token sense. The underlying idea remains aligned with the general PCPO principle: credit should track the mathematically relevant local contribution rather than be distorted by the optimization parameterization.

5. Empirical results in text-to-image alignment

The image-generation PCPO paper reports experiments on DDPO with Stable Diffusion 1.5 and on DanceGRPO with SD1.4 and FLUX.1-dev, using Aesthetics, BERTScore, and HPSv2.1 rewards (Lee et al., 30 Sep 2025). Evaluation is conducted on $A_i$7k images with FID, Inception Score, LPIPS Diversity, per-prompt Linear Mixed Model analysis, and a human preference study.

Across all reported settings, PCPO reduces clipping and accelerates convergence. The stated speedups in epochs to reach primary targets are DDPO Aesthetics: $A_i$8 for target $A_i$9 ($s_{1:T}$00), BERTScore: $s_{1:T}$01 for target $s_{1:T}$02 ($s_{1:T}$03), DanceGRPO SD1.4: $s_{1:T}$04 for HPS $s_{1:T}$05 ($s_{1:T}$06), and DanceGRPO FLUX: $s_{1:T}$07 for HPS $s_{1:T}$08 ($s_{1:T}$09). For FLUX at HPS $s_{1:T}$10, an additional threshold gives a $s_{1:T}$11 speedup ($s_{1:T}$12) (Lee et al., 30 Sep 2025).

The quality metrics also improve. For DDPO Aesthetics at batch size $s_{1:T}$13, the reported values are FID $s_{1:T}$14, IS* $s_{1:T}$15, and LPIPS $s_{1:T}$16; at batch size $s_{1:T}$17, FID $s_{1:T}$18, IS* $s_{1:T}$19, and LPIPS $s_{1:T}$20 (Lee et al., 30 Sep 2025). For DanceGRPO HPS, the reported SD1.4 values are FID $s_{1:T}$21, IS* $s_{1:T}$22, and LPIPS $s_{1:T}$23; the FLUX values are FID $s_{1:T}$24, IS* $s_{1:T}$25, and LPIPS $s_{1:T}$26 (Lee et al., 30 Sep 2025). The paper interprets lower IS in this setting as reduced mode collapse and states that the improvement in image quality is a direct result of mitigating model collapse in recursive training.

The component ablation is also framed in terms of credit proportionality. Sequentially adding the log-$s_{1:T}$27 objective, $s_{1:T}$28-matching, and proportional weighting lowers clipping progressively; only full PCPO achieves a zero on-policy clipping ratio (Lee et al., 30 Sep 2025). Heuristic alternatives fare worse. Timestep subsampling reduces wall-clock per epoch but degrades quality and diversity at matched rewards, and a uniform reweighting scheme that emphasizes high-noise timesteps underperforms vanilla $s_{1:T}$29 weighting, which the paper presents as supporting the proportionality principle (Lee et al., 30 Sep 2025).

A plausible implication is that, in this literature, “proportionate credit” has broadened from token-level attribution to a more general requirement that the policy-gradient signal be aligned with the effective local contribution induced by the model’s generation dynamics.

6. Related formulations, acronym ambiguity, and limitations

GCPO provides the clearest bridge between token-level credit assignment and a broader PCPO perspective. In “Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization” (Li et al., 28 May 2026), the group advantage $s_{1:T}$30 is scaled by a per-token guidance factor derived from the KL divergence between predictive distributions under positive and negative prompts:

$s_{1:T}$31

After sequence-wise CDF normalization, GCPO defines

$s_{1:T}$32

The paper explicitly states that this makes GCPO a proportionate-credit method and presents the relation

$s_{1:T}$33

thereby treating PCPO as the general principle and GCPO as one instantiation (Li et al., 28 May 2026). Empirically, GCPO outperforms GRPO and DAPO on text-to-image and multimodal reasoning benchmarks, with KL divergence and histogram-CDF normalization performing best among the tested contrast metrics and normalization schemes (Li et al., 28 May 2026).

At the same time, acronym ambiguity is substantial. “PCPO” also names “Proactive Constrained Policy Optimization,” a safe-RL method based on a preemptive extended log-barrier and a constraint-aware intrinsic reward (Yang et al., 3 Aug 2025), and “Projection-Based Constrained Policy Optimization,” a two-step CMDP algorithm consisting of a reward-improvement step followed by projection onto a linearized feasible set (Yang et al., 2020). Both papers explicitly clarify that their “PCPO” does not denote proportionate credit assignment.

The current proportionate-credit literature also identifies several limitations. In counterfactual PCPO for reasoning, importance is defined solely via final-answer likelihood, masking with pad or end-of-text may induce distribution shift, regex span detection is tuned for math rather than code, and experiments extend only up to $s_{1:T}$34B parameters, with benefits at $s_{1:T}$35B and above left untested (Khandoga et al., 10 Feb 2026). In GCPO, applicability depends on prompt-dependent rewards and on the model’s ability to respond meaningfully to negative prompts; weaker models may produce unreliable contrast signals (Li et al., 28 May 2026). In text-to-image PCPO, the diffusion formulation assumes equal integration intervals and approximately constant dot-product statistics across timesteps, while the flow formulation depends on accurate accounting of non-uniform $s_{1:T}$36 and does not eliminate all off-policy clipping (Lee et al., 30 Sep 2025).

These limitations indicate that PCPO is presently better understood as a methodological direction than as a single closed-form algorithm. The recurring claim across its variants is narrower but technically precise: uniform credit is often misaligned with causal or dynamical contribution, and explicit proportionalization of credit can improve optimization behavior when the weighting signal is task-faithful (Khandoga et al., 10 Feb 2026, Lee et al., 30 Sep 2025, Li et al., 28 May 2026).