Process-Calibrated Advantage Rescaling (PCAR)

• PCAR is a GRPO-based method that recalibrates token-level policy updates via self-reported retrieval confidence, addressing credit-assignment failures in long-horizon tasks.
• It employs a strict Search-to-Evaluate protocol where each retrieval is immediately scored, generating fine-grained process signals for segment-level gradient modulation.
• By amplifying reliable segments and tempering updates for less confident ones, PCAR improves multi-hop question answering performance and stabilizes learning.

Process-Calibrated Advantage Rescaling (PCAR) is a GRPO-based optimization mechanism for retrieval-augmented agents that rescales policy advantages at the segment level using explicit self-evaluation scores produced by the agent during interaction. It was introduced within the EvalAct framework, where every retrieval action is immediately followed by an evaluation action, yielding dense process signals aligned with the trajectory rather than only an outcome-level reward at the end. In this formulation, PCAR addresses the credit-assignment failure of outcome-only reinforcement learning in long-horizon, noise-prone retrieval by amplifying gradients for reliable retrieval–reasoning segments and tempering updates for dubious ones (Shu et al., 10 Mar 2026).

1. Problem setting and conceptual role

PCAR arises in retrieval-augmented multi-step question answering modeled as a sequential decision process with external tool calls interleaved with free-form reasoning. The underlying difficulty is that noisy retrieval can derail multi-hop reasoning early, while outcome-only reinforcement learning signals such as final-answer correctness are too coarse to distinguish helpful from misleading intermediate steps. Under standard GRPO or PPO, a single rollout-level advantage is broadcast nearly uniformly across the tokens of a trajectory, so both productive and unproductive intermediate actions are reinforced or penalized together. The stated consequence is poor credit assignment, especially in long-horizon settings where an early retrieval mistake can determine the final answer while many intermediate tokens remain locally reasonable (Shu et al., 10 Mar 2026).

In EvalAct, the agent’s interaction history is represented as

$h_t = [\,x, a_0, o_0, \ldots, a_{t-1}, o_{t-1}\,],$

with observations determined by the action type:

$$o_t \sim \begin{cases} \mathcal{R}(q), & \text{if } a_t = \text{Search}(q),\ \Phi(z), & \text{if } a_t = \text{Evaluate}(c,z),\ \emptyset, & \text{otherwise}. \end{cases}$$

Within this formulation, PCAR is not a separate reward model. It is an optimizer-side mechanism that consumes process signals generated during the trajectory and converts them into token-level gradient modulation. Its purpose is therefore narrower than full stepwise supervision but more structured than pure outcome-only RL.

2. Search-to-Evaluate coupling and the generation of process signals

The process signals used by PCAR are produced by the coupled Search-to-Evaluate protocol in EvalAct. After any $\text{Search}(q)$ with retrieved documents $o_t$, the agent must immediately invoke $\text{Evaluate}(c,z)$, where $c$ is a short textual assessment and $z \in [0,10]$ is a scalar self-reported reliability or confidence score. The environment maps $z$ to a discrete control cue:

$$\Phi(z)= \begin{cases} \mathcal{I}_{\text{low}}, & z \in [0,3],\ \mathcal{I}_{\text{mid}}, & z \in (3,7],\ \mathcal{I}_{\text{high}}, & z \in (7,10]. \end{cases}$$

The cue is appended to the context and modulates subsequent decisions. The environment does not parse $c$ or inspect the evidence; supervision remains purely self-evaluated and is gated by protocol compliance rather than external verification (Shu et al., 10 Mar 2026).

This architecture establishes a specific notion of “process alignment.” Because each retrieval is followed immediately by an explicit evaluation, every retrieval–evaluation pair becomes a localized unit whose reliability is contemporaneously scored. A common misconception is to treat PCAR as a form of external process reward modeling. In the EvalAct formulation, that is not the case: no oracle verifies the textual critique, and the environment never scores the retrieved documents directly. Instead, the system enforces a protocol in which the agent must expose its own retrieval assessment in a structured form, after which PCAR calibrates learning updates using those self-reported signals.

The final outcome reward is still answer-based and gated by formatting and protocol constraints:

$$o_t \sim \begin{cases} \mathcal{R}(q), & \text{if } a_t = \text{Search}(q),\ \Phi(z), & \text{if } a_t = \text{Evaluate}(c,z),\ \emptyset, & \text{otherwise}. \end{cases}$$0

Here, $$o_t \sim \begin{cases} \mathcal{R}(q), & \text{if } a_t = \text{Search}(q),\ \Phi(z), & \text{if } a_t = \text{Evaluate}(c,z),\ \emptyset, & \text{otherwise}. \end{cases}$$1 is extracted from <answer> tags, and the format indicator enforces both proper > enclosure and a strict one-to-one Search$$o_t \sim \begin{cases} \mathcal{R}(q), & \text{if } a_t = \text{Search}(q),\ \Phi(z), & \text{if } a_t = \text{Evaluate}(c,z),\ \emptyset, & \text{otherwise}. \end{cases}$$2Evaluate pairing. PCAR therefore supplements rather than replaces outcome reward: final credit remains tied to EM or F1 answer correctness, but the optimizer no longer treats all intermediate tokens as equally trustworthy.

3. Segment-level advantage rescaling in the GRPO backbone

The GRPO backbone computes a rollout-level group-relative advantage across $$o_t \sim \begin{cases} \mathcal{R}(q), & \text{if } a_t = \text{Search}(q),\ \Phi(z), & \text{if } a_t = \text{Evaluate}(c,z),\ \emptyset, & \text{otherwise}. \end{cases}$$3 rollouts for the same prompt:

$$o_t \sim \begin{cases} \mathcal{R}(q), & \text{if } a_t = \text{Search}(q),\ \Phi(z), & \text{if } a_t = \text{Evaluate}(c,z),\ \emptyset, & \text{otherwise}. \end{cases}$$4

PCAR retains this GRPO structure and changes only how that scalar advantage is distributed over the trajectory. The central object is the “segment,” defined as the atomic unit spanning one Search$$o_t \sim \begin{cases} \mathcal{R}(q), & \text{if } a_t = \text{Search}(q),\ \Phi(z), & \text{if } a_t = \text{Evaluate}(c,z),\ \emptyset, & \text{otherwise}. \end{cases}$$5Evaluate pair. For trajectory $$o_t \sim \begin{cases} \mathcal{R}(q), & \text{if } a_t = \text{Search}(q),\ \Phi(z), & \text{if } a_t = \text{Evaluate}(c,z),\ \emptyset, & \text{otherwise}. \end{cases}$$6, if there are $$o_t \sim \begin{cases} \mathcal{R}(q), & \text{if } a_t = \text{Search}(q),\ \Phi(z), & \text{if } a_t = \text{Evaluate}(c,z),\ \emptyset, & \text{otherwise}. \end{cases}$$7 such pairs, the $$o_t \sim \begin{cases} \mathcal{R}(q), & \text{if } a_t = \text{Search}(q),\ \Phi(z), & \text{if } a_t = \text{Evaluate}(c,z),\ \emptyset, & \text{otherwise}. \end{cases}$$8-th segment is denoted $$o_t \sim \begin{cases} \mathcal{R}(q), & \text{if } a_t = \text{Search}(q),\ \Phi(z), & \text{if } a_t = \text{Evaluate}(c,z),\ \emptyset, & \text{otherwise}. \end{cases}$$9 and carries the Evaluate score $\text{Search}(q)$0. The implementation may include the immediately contiguous reasoning tokens in the segment.

Calibration begins with intra-trajectory standardization of the self-evaluation scores:

$\text{Search}(q)$1

where $\text{Search}(q)$2 and $\text{Search}(q)$3 are the mean and standard deviation of the segment scores in trajectory $\text{Search}(q)$4. The stated purpose is to suppress trivial constant scoring and make the signal relative to the trajectory rather than absolute.

The standardized score is then converted into a gain coefficient through a linear map:

$\text{Search}(q)$5

with $\text{Search}(q)$6. For any token $\text{Search}(q)$7 that belongs to segment $\text{Search}(q)$8, the rescaled advantage is

$\text{Search}(q)$9

Equivalently, if a segment-level advantage $o_t$0 is the GRPO advantage shared by tokens in $o_t$1, then

$o_t$2

with

$o_t$3

The clamp lower bound $o_t$4 prevents sign inversions and near-zero gradients; the reported setting is $o_t$5 (Shu et al., 10 Mar 2026).

The policy objective remains GRPO-style clipped optimization with KL regularization against a reference policy. Thus PCAR does not alter the underlying clipped objective structure; it alters the per-token advantage weights entering that objective. This design is important for positioning: PCAR is not a new reward term and not a separate verifier, but a segment-wise advantage transformation embedded inside a standard GRPO update.

The worked example supplied in the source illustrates the mechanism numerically. With two segments scored $o_t$6 and $o_t$7, intra-trajectory standardization gives $o_t$8 and $o_t$9. Using $\text{Evaluate}(c,z)$0 and $\text{Evaluate}(c,z)$1, the gain coefficients are $\text{Evaluate}(c,z)$2 and $\text{Evaluate}(c,z)$3, yielding multipliers $\text{Evaluate}(c,z)$4 and $\text{Evaluate}(c,z)$5. If the rollout-level GRPO advantage is approximately $\text{Evaluate}(c,z)$6, then the token-level calibrated advantages become $\text{Evaluate}(c,z)$7 for tokens in the first segment and $\text{Evaluate}(c,z)$8 for tokens in the second. The intended effect is explicit: highly reliable segments receive larger gradient magnitudes, while partially useful or dubious segments are updated more conservatively.

4. Training procedure, empirical behavior, and component attribution

The training loop for EvalAct with PCAR follows a fixed sequence. A batch of queries is sampled from the dataset; for each query, $\text{Evaluate}(c,z)$9 protocol-compliant trajectories are generated with strict Search$c$0Evaluate coupling; gated rewards are computed; each trajectory is segmented into retrieval–evaluation units; GRPO group statistics produce rollout-level advantages; segment scores are standardized and mapped to gain factors; and the clipped GRPO objective with KL penalty is optimized using the rescaled token-level advantages. The appendix description identifies this as “EvalAct Training with PCAR (GRPO Backbone)” (Shu et al., 10 Mar 2026).

The empirical evaluation covers seven open-domain QA benchmarks: single-hop tasks NQ, TriviaQA, and PopQA, and multi-hop tasks HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle. On Qwen2.5-3B-Instruct, EvalAct achieves 44.0% average EM and surpasses the second-best AutoRefine at 40.5% by 3.5 points; the largest multi-hop gains are reported on 2Wiki at 50.0, Bamboogle at 48.0, Hotpot at 44.3, and MuSiQue at 21.6. On Qwen2.5-7B-Instruct, EvalAct reaches 47.1% average EM versus AutoRefine at 45.5%, a gain of 1.6 points; the multi-hop scores are 52.1 on 2Wiki, 56.0 on Bamboogle, 48.8 on Hotpot, and 25.3 on MuSiQue (Shu et al., 10 Mar 2026).

A crucial interpretive point comes from the ablations. Removing the evaluation loop causes average EM on multi-hop tasks to fall from 41.0% to 33.5%, a drop of 7.5 points, with declines of 8.6 on 2Wiki and 10.8 on Bamboogle. By contrast, removing PCAR while keeping Evaluate but using standard GRPO reduces average EM from 41.0% to 39.8%, so PCAR contributes an additional 1.2 points on average. The per-dataset improvements attributed to PCAR are +1.8 on 2Wiki, +1.8 on Bamboogle, +0.2 on Hotpot, and +0.8 on MuSiQue. The source explicitly states that the dominant contribution comes from the explicit evaluation structure, while PCAR provides consistent optimization benefit on top of that structure; statistical significance is not reported, and the effect sizes are described as small-to-moderate but robust across tasks.

This component attribution is central to understanding PCAR. It is not presented as the sole driver of EvalAct’s performance. Rather, the explicit Search-to-Evaluate protocol creates the dense aligned signals, and PCAR converts those signals into calibrated gradient scaling. The system’s gains therefore depend on both the representational intervention in the interaction protocol and the optimizer-side rescaling.

5. Hyperparameters, implementation details, and stability mechanisms

The reported GRPO settings are a learning rate of $c$1, global batch size 256, 5 rollouts per prompt, 2 epochs, temperature 1.0, KL coefficient $c$2, and clip ratio $c$3. Group-relative advantage normalization uses within-group mean and standard deviation with small constants $c$4 or $c$5 for numerical stability. The PCAR-specific parameters are $c$6, $c$7, and clamp lower bound $c$8. Scores are drawn from $c$9 and standardized within each trajectory, with no explicit external smoothing beyond standardization and clamp. A sensitivity study reports that moderate rescaling intensity, denoted as $z \in [0,10]$0, performs best, while overly aggressive rescaling hurts stability (Shu et al., 10 Mar 2026).

The training setup uses Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct as backbones. Retrieval is BM25 over the December 2018 Wikipedia dump, with top-$z \in [0,10]$1 per Search and a tool budget of up to 20 Search calls per question. The system includes an SFT warm-up of 2k synthesized protocol-compliant trajectories and an RL dataset formed by filtering ASearcherBase35K to approximately 27k examples. Training is performed with 8 NVIDIA A100 GPUs under full-parameter optimization with gradient checkpointing.

Several stability mechanisms are stated explicitly. Group normalization of rewards reduces variance across rollouts. The clamp prevents gradient inversion and vanishing on low-confidence segments. KL regularization stabilizes learning against a reference policy. Intra-trajectory standardization of the self-evaluation scores is said to curb reward hacking by making relative reliability matter rather than absolute score magnitude. These features indicate that PCAR was designed not only to improve credit assignment but also to avoid pathological optimizer behavior when the self-scored signals are noisy or strategically manipulated.

6. Relation to CalibAdv and the broader PCAR principle

A second 2026 paper, “Negative Advantage Is a Double-Edged Sword: Calibrating Advantage in GRPO for Deep Search,” does not use the term PCAR, but the supplied explanatory mapping describes its method, CalibAdv, as exactly a process-calibrated rescaling of advantages. In deep-search GRPO, CalibAdv addresses coarse-grained rollout-level credit assignment and imbalance between positive and negative advantages by using process-level correctness signals to attenuate negative intermediate-step advantages and by rebalancing the final-answer-step positives against negatives (Wu et al., 20 Apr 2026).

CalibAdv’s process signal differs from EvalAct’s self-evaluation score. For a given question, the method constructs a set of “silver documents” from correct rollouts and defines an intermediate-step correctness score

$z \in [0,10]$2

If the original advantage at step $z \in [0,10]$3 is negative, it is softened according to

$z \in [0,10]$4

For the final answer step, CalibAdv aggregates groupwise positive and negative advantages, defines

$z \in [0,10]$5

and rescales positive final-answer-step advantages by $z \in [0,10]$6 while leaving negatives unchanged. It also assigns zero advantage to the special <think> token to prevent format-related collapse.

In conceptual terms, both PCAR and CalibAdv are process-calibrated because they introduce step-sensitive modulation of policy updates instead of broadcasting a single scalar advantage to all tokens. Their main difference lies in the source and use of the calibration signal. EvalAct’s PCAR relies on explicit self-evaluation scores $z \in [0,10]$7 emitted after every retrieval and standardizes them within each trajectory before applying a bidirectional multiplier that can amplify or dampen segment updates. CalibAdv derives correctness proxies from cross-rollout document overlap, focuses on reducing misattributed negative updates at intermediate steps, and adds a separate groupwise rebalance for final-answer positives. The explanatory mapping notes that a broader PCAR scope could also amplify intermediate-step positives, whereas CalibAdv leaves such positives unchanged.

The reported empirical behavior is also notable. Across three models and seven benchmarks, CalibAdv improves over standard GRPO by 11.80% relative on average and avoids collapse. On Qwen2.5-7B-Base, Search-R1 achieves 49.15 F1 and collapses, whereas CalibAdv reaches 56.70 F1 with no collapse; analogous gains are reported for Qwen2.5-3B-Base and Llama-3.2-3B-Instruct. The ablations attribute improved stability first to decoupling <think>, then to soft penalization of negative intermediate-step advantages, and finally to positive–negative answer-step rebalancing. The validation of the silver-document proxy reports that steps with $z \in [0,10]$8 are helpful 89% by human evaluation and 83% by LLM evaluation on average across checkpoints, while replacing the proxy with an LLM judge gives nearly identical average accuracy but incurs 67% more wall time and 200% more GPU cost (Wu et al., 20 Apr 2026).

7. Limitations, edge cases, and prospective generalization

The principal limitation stated for PCAR is score miscalibration. If the self-reported $z \in [0,10]$9 values are uninformative or biased, rescaling can misguide gradients by over-amplifying poor segments or suppressing corrective updates. Intra-trajectory standardization and the clamp mitigate this risk but do not eliminate it. Sensitivity to segment definition is also explicit: which tokens are assigned to $z$0, how strong the rescaling is through the $z$1 parameters, and how noisy retrieval is can all affect optimization stability. The source warns that aggressive rescaling may starve low-reliability segments of corrective gradients (Shu et al., 10 Mar 2026).

A second limitation is protocol rigidity. The strict one-to-one Search$z$2Evaluate coupling is described as a heuristic that may constrain autonomy. The stated future direction is to learn when evaluation should occur rather than requiring it after every search. This is significant because the effectiveness of PCAR in EvalAct depends on the existence of aligned segment boundaries. If evaluation actions become optional or asynchronous, the calibration problem becomes more complex.

The broader generalization claim is cautiously framed. Although demonstrated on open-domain QA, the underlying idea of explicit evaluation actions combined with process-calibrated advantage rescaling is said to transfer to other tool-using tasks such as web navigation, code generation, and memory management, where intermediate tool outputs can be self-scored and used to calibrate advantages (Shu et al., 10 Mar 2026). This suggests that PCAR should be understood less as a retrieval-specific trick than as a general optimizer pattern for interactive agents: obtain process-level signals that are local to meaningful action segments, calibrate those signals within trajectories or groups, and use them to reshape the advantages entering policy optimization.

Across both EvalAct and the CalibAdv interpretation, the core claim remains consistent. Standard GRPO assigns one coarse advantage to an entire rollout. PCAR modifies that assignment by using process-level evidence—self-evaluation scores in EvalAct or silver-document correctness signals in CalibAdv—to decide where negative updates should be weakened, where reliable segments should be emphasized, and how instability caused by poorly calibrated trajectory-level credit can be reduced.