Progressively Ascending Confidence Reward (PACR)

• PACR is a reward-shaping method that transforms a model’s evolving confidence in the ground-truth answer into process-level supervision for reasoning.
• It offers two variants—Sparse-PACR and Dense-PACR—that respectively use binary indicators and magnitude of stepwise log-probability gains to adjust rewards.
• PACR improves exploration and convergence in reinforcement learning, demonstrating significant performance gains on mathematical reasoning benchmarks.

Progressively Ascending Confidence Reward (PACR) is a reward-shaping method for reinforcement learning with verifiable rewards (RLVR) that converts the model’s own evolving belief in the ground-truth answer into an intermediate supervision signal for reasoning. In its canonical formulation, PACR assumes that along a faithful reasoning trajectory, the probability of the correct final answer should exhibit a generally ascending trend. Rather than relying only on a sparse terminal reward indicating whether the final answer is correct, PACR assigns process-level credit from stepwise changes in the log-probability of the ground-truth answer, thereby targeting both exploration and credit assignment (Yoon et al., 25 Oct 2025).

1. Definition and conceptual premise

PACR was introduced to address a structural limitation of standard RLVR for reasoning: the usual verifiable reward is terminal and binary, so it gives no direct guidance about which intermediate reasoning steps were helpful. PACR reframes this by treating the model’s answer-conditioned confidence trajectory as a dense, model-intrinsic signal. The central inductive bias is explicit: along a well-formed reasoning path, confidence in the correct answer should generally rise (Yoon et al., 25 Oct 2025).

The method is step-based in how it parses reasoning, but answer-based in what it scores. A generated reasoning trace $H=(h_1,\ldots,h_T)$ is segmented by a simple heuristic: start a new step at a newline (\n) or at a period followed by a space (.); fragments shorter than five tokens are merged with the preceding step. At each such prefix $H_{\le k}$, PACR evaluates the probability of the entire ground-truth answer rather than assigning token-level rewards to the chain-of-thought itself (Yoon et al., 25 Oct 2025).

Two variants are defined. Sparse-PACR compresses stepwise confidence changes into a trajectory-level shaping term, while Dense-PACR turns the magnitude of confidence gains into a step-level return.

Variant	Process signal	RL integration
Sparse-PACR	Fraction of steps with $C_k>0$	Trajectory-level shaped reward
Dense-PACR	Discounted sum of confidence gains	Per-step mixed advantage

Sparse-PACR measures whether confidence ascent is consistent across the trajectory. Dense-PACR instead uses the actual signed confidence increments and therefore preserves magnitude information. This distinction is central to the later empirical finding that Dense-PACR is consistently stronger than Sparse-PACR on the reported benchmarks (Yoon et al., 25 Oct 2025).

2. Mathematical construction

Let $q$ be a question, $H_{\le k}$ the reasoning prefix up to step $k$, and $$Y_{\mathrm{gt}}=(y^1_{\mathrm{gt}},\ldots,y^L_{\mathrm{gt}})$$ the verifiable answer, preceded by a short answer-format prefix $y^0_{\mathrm{gt}}$ such as “So the final answer is $\backslash boxed\{$”. PACR defines ground-truth confidence at step $k$ as

$H_{\le k}$0

The stepwise confidence gain is then

$H_{\le k}$1

A positive $H_{\le k}$2 means that adding step $H_{\le k}$3 makes the correct answer more likely under the model; a negative $H_{\le k}$4 means the opposite (Yoon et al., 25 Oct 2025).

Sparse-PACR aggregates only the sign of these gains:

$H_{\le k}$5

and mixes that with the original GRPO reward:

$H_{\le k}$6

Its group-centered advantage is

$H_{\le k}$7

Dense-PACR uses the magnitude of each stepwise gain. For step $H_{\le k}$8 in trajectory $H_{\le k}$9, the discounted return is

$C_k>0$0

These returns are then normalized across the group using Min-Max normalization:

$C_k>0$1

For trajectories shorter than step $C_k>0$2, $C_k>0$3 is treated as zero. The final Dense-PACR step-level advantage is

$C_k>0$4

Because the dense process advantage is Min-Max scaled, it lies in $C_k>0$5 and is therefore nonnegative. This is a deliberate design choice: the process-level shaping term is used to encourage ascent rather than to impose explicit negative penalties on intermediate steps (Yoon et al., 25 Oct 2025).

In the reported implementation, PACR is used with groups of 8 rollouts per prompt, temperature $C_k>0$6, one PPO epoch per rollout, pass@1 greedy decoding for evaluation, total batch size $C_k>0$7, AdamW with learning rate $C_k>0$8, maximum prompt length $C_k>0$9, maximum generation length $q$0, $q$1, and $q$2. The reported description specifies that both sparse and dense settings use these $q$3 values; the chosen numerical value of $q$4 is not stated in the reported formulation (Yoon et al., 25 Oct 2025).

3. Integration with GRPO and theoretical justification

PACR is integrated into an RLVR pipeline built on GRPO, specifically Dr.GRPO. The underlying verifiable reward remains the usual binary terminal signal,

$q$5

and standard GRPO forms a group-relative advantage from this reward. PACR does not introduce a separate auxiliary loss. Instead, it reshapes the reward or advantage used inside the RL update: Sparse-PACR replaces the trajectory reward with a shaped mixture, while Dense-PACR supplies a mixed step-level advantage (Yoon et al., 25 Oct 2025).

The main theoretical argument is built from an oracle-policy assumption. Suppose a capable model conditioned on the correct final answer generates the next step from $q$6. Then the expected confidence gain is nonnegative:

$q$7

Using Bayes’ rule, the paper rewrites the confidence gain as

$q$8

so that

$q$9

This gives PACR a formal interpretation: if a step is sampled from the model’s own ground-truth-conditioned reasoning distribution, then in expectation that step should preserve or increase belief in the correct answer. The argument is statistical rather than absolute. It does not imply that every good step must raise confidence, nor that every confidence-raising step is logically sound, but it does justify ascending confidence as an inductive bias for exploration (Yoon et al., 25 Oct 2025).

4. Empirical behavior and reported gains

PACR is evaluated on three open-source models—Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Qwen3-4B—and on five mathematical reasoning benchmarks: MATH500, Minerva-Math, OlympiadBench, AIME 2024, and AMC 2023. Accuracy is reported as pass@1. Across all three model scales, both PACR variants outperform Dr.GRPO on average, and Dense-PACR is consistently better than Sparse-PACR (Yoon et al., 25 Oct 2025).

Model	Dr.GRPO avg	Dense-PACR avg
Qwen2.5-Math-1.5B	41.7	44.2
Qwen2.5-Math-7B	49.6	52.6
Qwen3-4B	54.6	57.6

The strongest gains are concentrated on harder competition-style benchmarks, especially AIME 2024. For Qwen2.5-Math-1.5B, Dense-PACR improves AIME24 from $H_{\le k}$0 to $H_{\le k}$1 and raises the overall average from $H_{\le k}$2 to $H_{\le k}$3. For Qwen2.5-Math-7B, AIME24 increases from $H_{\le k}$4 to $H_{\le k}$5, with average performance moving from $H_{\le k}$6 to $H_{\le k}$7. For Qwen3-4B, AIME24 rises from $H_{\le k}$8 to $H_{\le k}$9, and OlympiadBench from $k$0 to $k$1, yielding an average improvement from $k$2 to $k$3 (Yoon et al., 25 Oct 2025).

The training-dynamics analysis is as important as the endpoint numbers. PACR is reported to improve the early rate of accuracy gain and to converge to higher final accuracy than Dr.GRPO, which the authors interpret as faster exploration and earlier reward saturation. The paper does not provide a separate table of exact rollout counts to a fixed target reward, but it repeatedly emphasizes that PACR reaches reward saturation with fewer trajectories (Yoon et al., 25 Oct 2025).

A key ablation concerns Dense-PACR’s normalization. The authors compare Min-Max normalization with a leave-one-out baseline,

$k$4

LOO yields negative process advantages, accelerates learning early, and then causes premature convergence and plateauing. Min-Max normalization, by contrast, keeps the process signal nonnegative and produces better final performance. The practical implication is that PACR’s process reward is not merely “dense”; its specific normalization strategy is part of the method’s exploration behavior (Yoon et al., 25 Oct 2025).

5. Related formulations, extensions, and common confusions

PACR has already been extended and contrasted within adjacent confidence-based RL literature. In multimodal reasoning, "Perception-Decomposed Confidence Reward" (PDCR) treats PACR as the baseline dense confidence-growth method and argues that PACR’s global Min-Max normalization is suboptimal for heterogeneous vision-language trajectories. PDCR keeps the same core confidence-growth signal—stepwise changes in $k$5—but replaces global normalization with separate normalization inside visual and textual skill clusters inferred from a Visual Dependence Score and Otsu thresholding. In reported benchmarks, PACR remains a strong baseline, but PDCR improves average performance from $k$6 to $k$7 on Qwen2.5-VL-3B, from $k$8 to $k$9 on Qwen2.5-VL-7B, and from $$Y_{\mathrm{gt}}=(y^1_{\mathrm{gt}},\ldots,y^L_{\mathrm{gt}})$$0 to $$Y_{\mathrm{gt}}=(y^1_{\mathrm{gt}},\ldots,y^L_{\mathrm{gt}})$$1 on Qwen3-VL-8B (Yoon et al., 13 May 2026).

PACR is also frequently confused with APART’s “Ascending Reward.” The APART paper explicitly does not define PACR. Its “ascending” component is a time-ascending weighting or Bernoulli dropout over intrinsic reward within an episode, with $$Y_{\mathrm{gt}}=(y^1_{\mathrm{gt}},\ldots,y^L_{\mathrm{gt}})$$2, and its confidence-sensitive part comes from the minimum signed pairwise output of an all-pairs discriminator. That mechanism is therefore a timestep-based stochastic masking scheme, not a progressively ascending confidence reward in the PACR sense (Galler et al., 2023).

Other nearby methods use confidence as reward without implementing PACR’s stepwise confidence-growth construction. C3RL uses a fixed threshold $$Y_{\mathrm{gt}}=(y^1_{\mathrm{gt}},\ldots,y^L_{\mathrm{gt}})$$3 over verbalized confidence and adds a threshold-guided calibration reward that gives $$Y_{\mathrm{gt}}=(y^1_{\mathrm{gt}},\ldots,y^L_{\mathrm{gt}})$$4 for certain correct or uncertain incorrect responses and $$Y_{\mathrm{gt}}=(y^1_{\mathrm{gt}},\ldots,y^L_{\mathrm{gt}})$$5 for certain incorrect or uncertain correct responses, plus a dataset-informed reference accuracy reward. This is confidence-aware reward shaping, but not a progressive stepwise ascent mechanism (Yang et al., 2 Jul 2026). CRew, by contrast, uses the mean probability of final-answer tokens as a training-free reward proxy for close-ended tasks, and CRew-DPO constructs preference pairs from confidence scores combined with correctness signals; again, the scoring is answer-level rather than PACR’s intermediate-step process reward (Du et al., 15 Oct 2025).

Taken together, these papers place PACR within a broader class of model-intrinsic confidence methods, but PACR’s distinguishing feature remains its use of prefix-conditioned, ground-truth-answer confidence growth as the reward-generating object.

6. Limitations, scope, and acronym ambiguity

PACR is tightly coupled to RLVR settings with a known verifiable answer available during training, because it explicitly evaluates

$$Y_{\mathrm{gt}}=(y^1_{\mathrm{gt}},\ldots,y^L_{\mathrm{gt}})$$6

This makes it natural for mathematical reasoning and other tasks with a unique checkable target, but it limits direct applicability to subjective or non-verifiable domains. The reported experiments are entirely on math-style reasoning benchmarks; no evidence is provided for open-ended alignment or non-verifiable tasks (Yoon et al., 25 Oct 2025).

Several methodological caveats follow from the construction. First, PACR depends on the model’s own confidence quality. The empirical case for PACR is based on the observed correlation between ascending confidence and correct or coherent reasoning, but the paper does not present a formal calibration analysis. Second, the step segmentation rule is simple and deliberately heuristic; different segmentation policies could change the measured $$Y_{\mathrm{gt}}=(y^1_{\mathrm{gt}},\ldots,y^L_{\mathrm{gt}})$$7 values and therefore the reward. Third, Dense-PACR requires confidence evaluation at every intermediate reasoning prefix, so its computational overhead is likely nontrivial, even though the paper does not quantify wall-clock cost in detail (Yoon et al., 25 Oct 2025).

A further caution concerns terminology. In current arXiv usage, “PACR” is not unique to LLM reasoning. An unrelated 2026 paper in power systems uses PACR to mean “Parameter-Optimized AC Power Flow Restoration,” a differentiable AC feasibility recovery method for DCOPF dispatches. That PACR is a post-DCOPF restoration framework and has no connection to confidence-based reward shaping in RLVR (Boateng et al., 27 Jun 2026).

In the reasoning literature proper, PACR is therefore best defined narrowly: a GRPO-compatible reward-shaping framework in which stepwise increases in the model’s log-probability of the ground-truth answer become process supervision. Its significance lies less in any single heuristic than in a specific synthesis: answer-level confidence measured at intermediate prefixes, converted into either a trajectory-level ascent score or a step-level discounted return, and then injected directly into RL optimization as shaped reward or advantage (Yoon et al., 25 Oct 2025).