Contrastive Evidence Policy Optimization (CEPO)

• The paper introduces CEPO, which distinguishes decisive tokens by contrasting correct and wrong answer signals to refine credit assignment.
• It employs a stop-gradient contrastive mechanism that avoids vocabulary-wide gradient leakage and boosts sample efficiency over GRPO.
• CEPO achieves notable accuracy gains on multimodal reasoning benchmarks, outperforming GRPO by +2.26% at 2B and +3.13% at 4B scales.

Searching arXiv for the CEPO paper and closely related work to ground citations. Contrastive Evidence Policy Optimization (CEPO) is a reinforcement-learning-with-verifiable-rewards (RLVR) self-distillation method introduced in “CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimization” (Heakl et al., 19 May 2026). It addresses the token-level credit-assignment problem that arises when a correct rollout receives a single sequence-level reward and every token in that rollout is updated identically, regardless of whether the token was a decisive reasoning step or merely grammatical filler. CEPO conditions the model on both a correct answer and a wrong answer, then asks at each token whether the correct answer favors that token while the wrong answer disfavors it. The wrong-answer teacher is constructed from rejected rollouts already present in the batch, so no additional sampling is required. The method is presented as inheriting the structural safety guarantees of prior RLVR self-distillation while sharpening credit at decisive tokens and leaving the improvement to vanish at filler positions; empirically, it reports higher average accuracy than GRPO on five multimodal mathematical reasoning benchmarks at both 2B and 4B scale (Heakl et al., 19 May 2026).

1. RLVR setting and the credit-assignment bottleneck

In the RLVR formulation used by CEPO, a policy $\pi_\theta$ receives a problem prompt $x$ and samples a sequence, or chain of thought, $y=(y_1,\dots,y_T)$. A deterministic verifier assigns a binary outcome,

$R(x,y)\in\{0,1\},$

labeling the rollout correct or wrong (Heakl et al., 19 May 2026).

A practical instantiation discussed in the CEPO paper is Group Relative Policy Optimization (GRPO), which samples $G$ rollouts per prompt and computes sequence-level advantages

$A^{(i)}=\frac{R(x,y^{(i)})-\mu_G}{\sigma_G},$

where $\mu_G$ and $\sigma_G$ are the mean and standard deviation of the rewards across the $G$ samples. GRPO then applies that same scalar advantage uniformly to every token in a rollout: all tokens in a correct rollout receive $+A^{(i)}$, and all tokens in a wrong rollout receive $x$0 (Heakl et al., 19 May 2026).

The CEPO paper identifies the resulting limitation as uniform credit assignment over long chains. On this account, grammar, framing, and other filler tokens receive the same reinforcement signal as key arithmetic steps or logical inferences. As chains grow, the signal allocated to decisive reasoning steps becomes diluted. The paper characterizes the consequence as reduced sample efficiency and impaired convergence in long-horizon reasoning settings (Heakl et al., 19 May 2026).

2. Self-distillation in RLVR and the leakage problem

The CEPO formulation introduces three next-token distributions that share the same parameters $x$1 but differ in conditioning context:

$x$2

$x$3

$x$4

where $x$5 is the student, $x$6 is a teacher conditioned on the correct answer $x$7, and $x$8 is a teacher conditioned on a wrong answer $x$9 (Heakl et al., 19 May 2026).

The paper contrasts CEPO with on-policy self-distillation methods such as OPSD and SDPO, which minimize

$y=(y_1,\dots,y_T)$0

As described in the paper, the gradient of this objective contains a vocabulary-wide term,

$y=(y_1,\dots,y_T)$1

and prior work is cited there as showing that this term inevitably leaks the answer into the gradient and degrades generalization. The CEPO paper further states that distribution-matching self-distillation methods, specifically OPSD and SDPO, empirically fall below the untrained baseline, which it interprets as confirmation of the predicted information leakage (Heakl et al., 19 May 2026).

The immediate precursor presented in the CEPO paper is RLSD. Its remedy is to evaluate only the sampled token under stop-gradient, yielding the evidence ratio

$y=(y_1,\dots,y_T)$2

and then reweight the sequence-level advantage according to

$y=(y_1,\dots,y_T)$3

In the presentation of the method, this removes the vocabulary-wide sum and anchors the update direction to $y=(y_1,\dots,y_T)$4 (Heakl et al., 19 May 2026).

3. Contrastive evidence formulation

CEPO sharpens the token-level signal by replacing the one-sided question “does the correct answer favor this token?” with the contrastive question “does the correct answer favor it while the wrong answer disfavors it?” (Heakl et al., 19 May 2026).

Its token-level contrastive quantity is the raw stop-gradiented delta

$y=(y_1,\dots,y_T)$5

The paper also gives an equivalent probability-difference form,

$y=(y_1,\dots,y_T)$6

These quantities distinguish tokens that are positively supported by the correct-answer teacher and negatively supported by the wrong-answer teacher from tokens for which the two teachers behave similarly (Heakl et al., 19 May 2026).

Given a normalized sequence-level advantage $y=(y_1,\dots,y_T)$7, CEPO defines the contrastive weight

$y=(y_1,\dots,y_T)$8

and constructs token-level advantages

$y=(y_1,\dots,y_T)$9

where $R(x,y)\in\{0,1\},$0 is annealed from $R(x,y)\in\{0,1\},$1 over $R(x,y)\in\{0,1\},$2 steps and $R(x,y)\in\{0,1\},$3 is a clipping bound (Heakl et al., 19 May 2026).

Because $R(x,y)\in\{0,1\},$4 is under stop-gradient, the policy-gradient loss is written as

$R(x,y)\in\{0,1\},$5

The paper’s interpretation is asymmetric in the sign of $R(x,y)\in\{0,1\},$6. For correct rollouts, tokens with large $R(x,y)\in\{0,1\},$7 receive $R(x,y)\in\{0,1\},$8, whereas filler tokens with $R(x,y)\in\{0,1\},$9 remain near the original sequence-level weight. For wrong rollouts, the same mechanism sharpens blame on tokens favored by the wrong-answer teacher (Heakl et al., 19 May 2026).

4. Wrong-answer teachers, computational overhead, and algorithmic realization

A central operational feature of CEPO is that the wrong-answer teacher is obtained from rejected rollouts already computed within each GRPO group. If $G$0 denotes the rejected subset, the method chooses $G$1 as the final answer of the lowest-reward rollout. When all rollouts are correct, the pseudocode specifies the fallback $G$2 (Heakl et al., 19 May 2026).

The paper emphasizes that no extra sampling is needed. One forward pass of $G$3 conditioned on $G$4 yields $G$5, and together with $G$6 conditioned on the ground-truth answer, this amounts to one extra teacher pass per trajectory, which the paper states is identical overhead to RLSD (Heakl et al., 19 May 2026).

The training loop is correspondingly simple. For each $G$7 in the batch, the policy samples $G$8 rollouts, computes $G$9, selects $A^{(i)}=\frac{R(x,y^{(i)})-\mu_G}{\sigma_G},$0 from the lowest-reward rollout, evaluates $A^{(i)}=\frac{R(x,y^{(i)})-\mu_G}{\sigma_G},$1 and $A^{(i)}=\frac{R(x,y^{(i)})-\mu_G}{\sigma_G},$2, forms per-token contrastive deltas $A^{(i)}=\frac{R(x,y^{(i)})-\mu_G}{\sigma_G},$3, computes weights $A^{(i)}=\frac{R(x,y^{(i)})-\mu_G}{\sigma_G},$4, converts sequence-level advantages into token-level advantages $A^{(i)}=\frac{R(x,y^{(i)})-\mu_G}{\sigma_G},$5, and finally applies a PPO-clipped surrogate update using those token-level advantages (Heakl et al., 19 May 2026).

This suggests that CEPO is designed as a drop-in modification to existing RLVR pipelines built around grouped rollouts and PPO-style policy optimization, rather than as a separate sampling or verification framework.

5. Structural guarantees and discriminative sharpness

The CEPO paper states a theorem labeled “CEPO Safety” for $A^{(i)}=\frac{R(x,y^{(i)})-\mu_G}{\sigma_G},$6 and $A^{(i)}=\frac{R(x,y^{(i)})-\mu_G}{\sigma_G},$7 with three properties (Heakl et al., 19 May 2026).

First, direction anchoring states that

$A^{(i)}=\frac{R(x,y^{(i)})-\mu_G}{\sigma_G},$8

so privileged information cannot flip the update direction. Second, leakage-free gradients state that $A^{(i)}=\frac{R(x,y^{(i)})-\mu_G}{\sigma_G},$9 contains no sum over $\mu_G$0; the correct and wrong answers enter only as scalars at the sampled token under stop-gradient. Third, RLSD containment states that if

$\mu_G$1

then

$\mu_G$2

and CEPO is exactly RLSD (Heakl et al., 19 May 2026).

The paper also gives a proposition labeled “Discriminative Sharpness.” On a correct trajectory with $\mu_G$3,

$\mu_G$4

Accordingly, the paper concludes that CEPO strictly amplifies tokens that the wrong-answer teacher disfavors, identifying these as genuine reasoning steps, while CEPO and RLSD coincide near filler positions where $\mu_G$5 (Heakl et al., 19 May 2026).

Within the internal logic of the method, this proposition is the main justification for the term “contrastive evidence.” The additional teacher is not used to introduce a second target distribution for matching; it is used to sharpen token-level discrimination relative to the student baseline while preserving the stop-gradient safety structure.

6. Empirical profile and nomenclature

The empirical evaluation in the CEPO paper uses Qwen3-VL-2B-Instruct and Qwen3-VL-4B-Instruct, LoRA-tuned for 50 steps on Geo3k, a set of 3 K geometry problems. Evaluation is conducted on held-out multimodal reasoning benchmarks: DynaMath, LogicVista, MathVision$\mu_G$6, MMMU, and WeMath. Baselines under identical budgets are GRPO sequence-level RL, OPSD, SDPO, and RLSD (Heakl et al., 19 May 2026).

Method	2B average accuracy	4B average accuracy
Base	39.73%	58.36%
GRPO	41.17%	57.43%
RLSD	40.05%	58.51%
OPSD	34.96%	56.23%
SDPO	35.70%	55.75%
CEPO	43.43%	60.56%

On the averages reported over the five benchmarks, CEPO improves over GRPO by $\mu_G$7 percentage points at 2B and $\mu_G$8 percentage points at 4B. The paper further states that OPSD and SDPO collapse below the base model, whereas CEPO yields the best sample efficiency across scales and tasks while sharply localizing credit on decisive reasoning tokens and preserving the safety guarantees of prior RLVR self-distillation. The code release is listed at https://github.com/ahmedheakl/CEPO (Heakl et al., 19 May 2026).

The acronym “CEPO” is not unique in the literature. An earlier, unrelated paper, “Policy Optimization with Sparse Global Contrastive Explanations” (Yao et al., 2022), uses CEPO to denote a constrained-MDP framework in which a target policy $\mu_G$9 maximizes expected return subject to a budget on the expected number of deviations from a behavior policy $\sigma_G$0. In that setting, explanation length is the number of diverging states or regions included in a global contrastive explanation, and the optimization is posed with a hard cap $\sigma_G$1 on

$\sigma_G$2

That 2022 usage concerns sparse policy modification and interpretable contrastive explanations in discrete and continuous MDPs, whereas the 2026 usage concerns RLVR self-distillation and token-level contrastive credit assignment (Yao et al., 2022). A common misconception is therefore to treat the acronym as denoting a single method family; in the arXiv record, it refers to two distinct formulations that share only the abbreviation.