Conditional Advantage Estimation (CANON)

• CANON is an adaptive advantage estimation method in reinforcement learning that uses auxiliary metrics without presuming a fixed directional preference.
• It conditionally splits responses into high- and low-metric groups and balances inter-group with intra-group comparisons to determine beneficial trends.
• Empirical results show CANON improves accuracy on math and logic reasoning tasks while optimizing token efficiency in large language models.

Conditional advANtage estimatiON (CANON) is an advantage-estimation method for Reinforcement Learning with Verifiable Rewards (RLVR) in LLMs that amplifies the impact of auxiliary training metrics such as response entropy and response length without presuming whether higher or lower values are preferable. Introduced in "Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models" (Chen et al., 28 Sep 2025), CANON conditionally regroups sampled responses for each query into high- and low-metric groups, uses inter-group comparison to determine which metric trend contributes to better performance, and uses intra-group comparison to identify the better response within the same group. It was proposed to address the brittleness of reward or advantage shaping methods that encode directional priors through hand-crafted bonuses or penalties.

1. Problem setting and motivation

CANON was developed for RLVR settings in which LLMs are optimized on tasks with clear correctness criteria, such as mathematical reasoning tasks. In this setting, auxiliary metrics including response entropy and response length have been observed to correlate with different reasoning behaviors and eventual performance (Chen et al., 28 Sep 2025).

The motivating problem is not the existence of useful metrics, but the instability of using them through fixed directional assumptions. Prior reward or advantage shaping methods typically encode preferences such as “higher entropy is better,” “lower entropy is better,” “shorter is better,” or “longer is better,” often through hand-crafted penalties or bonuses. The limitation identified by CANON is that such directional priors can be overly biased and brittle. A metric trend that helps in one regime may be harmful in another: for example, higher entropy may aid exploration on very hard problems, whereas lower entropy may yield higher accuracy where the model is already capable. This creates substantial hyperparameter sensitivity.

A central misconception addressed by CANON is that a useful auxiliary metric must be used with a fixed preferred direction. CANON rejects that premise. Its objective is to amplify the impact of specific metric trends on learning without imposing an a priori “higher is better” or “lower is better” assumption. The method is therefore data-driven in a precise sense: it attempts to discover, from grouped reward comparisons, which trend is beneficial for a given query, model, or training regime.

2. Conditional regrouping and advantage formulation

The core construction begins with a set of sampled responses per query, denoted $G_q$, together with a target metric $m$ such as entropy or length. CANON sorts the responses according to $m$ and splits $G_q$ into two equal-sized groups:

$$G_q^+ := \{ o \in G_q \mid o \text{ satisfy condition } c \}, \qquad G_q^- := G_q \setminus G_q^+ .$$

The condition $c$ is typically whether the metric value is above or below the median. The paper states that splitting at the median, yielding equal sizes, is typically optimal per theory.

For a response $o \in G_q$ with reward $R_o$, CANON defines an inter-group advantage and an intra-group advantage. The inter-group term compares a response against the mean reward of the opposite group:

$$\hat{A}^{\text{inter}}_{q,o,t} = \begin{cases} R_o - \text{mean}(\{R_{o'} \mid o' \in G_q^+\}) & \text{if } o \in G_q^- \ R_o - \text{mean}(\{R_{o'} \mid o' \in G_q^-\}) & \text{if } o \in G_q^+ \end{cases}$$

The intra-group term compares a response against the mean reward of its own group:

$$\hat{A}^{\text{intra}}_{q,o,t} = \begin{cases} R_o - \text{mean}(\{R_{o'} \mid o' \in G_q^+\}) & \text{if } o \in G_q^+ \ R_o - \text{mean}(\{R_{o'} \mid o' \in G_q^-\}) & \text{if } o \in G_q^- \end{cases}$$

The unified CANON advantage is a convex combination of the two:

$m$0

Here $m$1 is a scheduling coefficient balancing inter-group and intra-group comparison. In the formulation given in the paper, inter-group comparison identifies which metric trend is associated with higher rewards, whereas intra-group comparison promotes the stronger response within a given trend (Chen et al., 28 Sep 2025).

3. Directionality, special cases, and steering

The distinctive property of CANON is that it does not assume a preferred direction for any metric. By splitting responses into high- and low-metric groups and comparing rewards across them, it discovers whether lower or higher values of the metric are advantageous. If the lower-entropy group has higher reward, CANON exploits certainty; if the higher-entropy group yields higher reward, CANON exploits exploration. The same logic applies to response length.

This mechanism can be understood as separating two roles. Inter-group comparison identifies which trend direction is more aligned with reward, while intra-group comparison preserves competition within that direction and thereby supports exploration and avoids collapse. The paper describes $m$2 as a means of balancing aggressive exploitation and more diverse exploration and reflection.

CANON also subsumes a standard baseline under a specific setting. If $m$3 and the groups are equal-sized, DR.GRPO is recovered as a special case:

$m$4

The framework also permits weighted inter-group comparison through a coefficient $m$5, used to further steer trends:

$m$6

The paper’s summary of the main formulations is as follows (Chen et al., 28 Sep 2025):

Method	Formulation (per sample)	Description
DR.GRPO	$m$7	Group-level baseline; ignores metric
CANON-Inter	$m$8	Exploit metric trend
CANON-Intra	$m$9	Explore within current trend
CANON (general)	$m$0	Adaptive balance

4. Relation to prior methods and to advantage-estimation research

CANON is positioned against two prior families of methods. The first is baseline-based methods such as DR.GRPO, GRPO, and RLOO, which use the same baseline, namely the mean of all rewards in the sampled group. The second is reward or advantage shaping based on hand-designed penalties or preferences, such as length penalties or directional entropy preferences. CANON differs from both by conditioning the baseline on an auxiliary metric without hard-coding the direction that should be preferred.

Empirically, the paper compares CANON against DR.GRPO, ReMax, RLOO, REINFORCE++, length-clipping, and various length-reward formulations. In implementation terms, the stated integration requirement is minimal: advantage estimation is the only change required over standard PPO, GRPO, and DR.GRPO pipelines (Chen et al., 28 Sep 2025).

Within the broader literature on advantage estimation, CANON belongs to a distinct line from methods such as Direct Advantage Estimation (DAE). DAE models the advantage function directly from on-policy data, interprets advantage as a causal effect, and minimizes the variance of the return without requiring the action-value function (Pan et al., 2021). CANON does not attempt to replace the general problem of estimating advantages from trajectories; instead, it modifies groupwise advantage construction in RLVR by conditioning on auxiliary metrics. This suggests a division of labor within advantage-estimation research: DAE focuses on direct estimation of $m$1, whereas CANON focuses on how auxiliary response-level metrics should shape comparative baselines in LLM reinforcement learning.

5. Empirical evaluation in mathematical and logical reasoning

The reported experiments cover 6 math reasoning datasets—MATH-500, GSM8K, AMC, OlympiadBench, and AIME 24/25—and 3 logic reasoning tasks from ZebraLogic. The evaluated models are Qwen2.5-Math-7B, Llama3.1-8B, and Qwen2.5-Math-1.5B. The metrics tracked include accuracy, entropy, response length, and reflection gains (Chen et al., 28 Sep 2025).

For math reasoning, the principal result is that CANON-Inter based on entropy consistently outperforms previous methods, with a 1.9 point average accuracy gain over DR.GRPO. It achieves higher accuracy on 4/6 math benchmarks, and the paper characterizes these math tasks as favoring exploitation, where low entropy often wins. The same experiments report that response lengths are maintained or reduced.

For high-complexity logic tasks, the strongest gains come from CANON-Intra based on entropy. The reported improvement is up to 5.2 points higher than DR.GRPO on the hardest subset. The paper attributes this to effective reflection and exploration, especially when the reward signal is sparse or tasks require creative strategies. The contrast between the math and logic results is central to the method’s motivation: the helpful trend is task- and regime-dependent, so fixed directional priors are unreliable.

When CANON is applied to response length rather than entropy, the focus shifts from accuracy alone to token-efficient reasoning. The reported gains include up to 2.63× the performance of DR.GRPO under low token budgets and a 45.5% reduction in token cost for the same performance. The paper further states that CANON outperforms length-clip and reward-penalty baselines across the accuracy/efficiency tradeoff, producing a more favorable Pareto frontier.

Taken together, these findings support the paper’s claim that CANON can span the spectrum from “precise, deterministic answers” to “expansive, reflective reasoning” through its choice of conditioning metric and its balance between inter-group and intra-group comparison.

6. Scheduling, robustness, and significance

The general CANON formulation allows dynamic scheduling of $m$2, either by accuracy or by training steps, using linear, cosine-annealing, or other smooth transitions (Chen et al., 28 Sep 2025). In the paper’s interpretation, this scheduling controls the balance between exploitation and exploration: larger emphasis on inter-group comparison biases learning toward the currently better-performing trend, whereas larger emphasis on intra-group comparison favors diversity, reflection, and within-trend competition.

The method also includes a robustness claim that is unusually important for practical RLVR. CANON does not degrade when the metric trend is not predictive: random grouping yields baseline performance. The paper therefore characterizes the method as having no risk of harming performance through mis-specified directional priors. This is significant because the central failure mode of prior shaping methods is precisely the incorrect specification of a preferred metric direction.

A broader implication is that CANON operationalizes auxiliary metrics as conditional structure in the advantage estimator rather than as hand-crafted reward engineering. This suggests a more conservative way to exploit observed correlations between training metrics and reasoning behavior. Instead of asserting that entropy or length should always move in one direction, CANON asks whether the high- or low-metric group is currently better, then learns accordingly. In the context of large reasoning models, that design makes CANON a method for adaptive metric utilization rather than metric prescription.