SPAR++: Span-Level Reward Modeling

• The paper introduces SPAR++ as a method that transforms outcome-only feedback into span-level reward signals using hidden state divergence.
• It employs Wasserstein distance and Sinkhorn approximations to quantify divergence between correct and incorrect reasoning spans.
• The SHEAR algorithm reweights token-level advantages in GRPO, yielding improved credit assignment and empirical performance on math and code benchmarks.

Searching arXiv for the specified paper and closely related reward-modeling context. tool call: arxiv_search tool result: arxiv_search Span-Level Reward Modeling, described in the formulation termed “SPAR++” in the context of "Hidden States Know Where Reasoning Diverges: Credit Assignment via Span-Level Wasserstein Distance" (Chen et al., 25 Apr 2026), denotes a span-level credit-assignment approach for reinforcement learning with verifiable rewards (RLVR) that derives a per-span reward proxy directly from the policy’s hidden-state geometry. In this setting, a policy $\pi_\theta$ generates a rollout $y = (y_1, \ldots, y_T)$ for an input $x$ and receives an outcome-level verifiable reward $r \in \{0,1\}$ indicating correctness. The core claim is that, within a Group Relative Policy Optimization (GRPO) group containing both correct and incorrect rollouts for the same problem, hidden-state distributions diverge around spans where local reasoning quality differs. SHEAR—Span-level Hidden state Enabled Advantage Reweighting—operationalizes that observation by scaling token-level advantages with span-level Wasserstein distances, thereby turning outcome-only RLVR into span-level credit assignment without step labels or an auxiliary reward model (Chen et al., 25 Apr 2026).

1. RLVR, GRPO, and the span-level credit gap

RLVR assumes that supervision is available only at the level of final correctness. For inputs such as math or code problems, a policy $\pi_\theta$ produces a full rollout and receives a binary reward. In GRPO, for each $x$, one samples a group of $G$ rollouts $\mathcal{G}(x) = \{y^{(1)}, \ldots, y^{(G)}\}$ and computes a group-normalized scalar advantage $A^{(i)}$ per rollout using the group’s rewards. The GRPO policy gradient assigns the same advantage to every token in the rollout:

$$\nabla_\theta J = \mathbb{E}_{y^{(i)} \sim \pi_\theta} \left[\sum_{t=1}^{T_i} A^{(i)} \nabla_\theta \log \pi_\theta(y_t^{(i)} \mid y_{<t}^{(i)}, x)\right].$$

This is a coarse-grained credit signal. Identical feedback is applied to all tokens in a rollout even when only a small subset of steps determines correctness. The paper identifies this mismatch as a span-level credit gap: process reward models can provide step-level scores, but they require step annotations or training an auxiliary reward model, whereas the proposed method works with outcome-only supervision and extracts a span-level signal directly from hidden states (Chen et al., 25 Apr 2026).

Conceptually, this places SPAR++ in a distinct position relative to traditional span-level reward modeling. Traditional SPAR relies on span-level reward models or annotations. The formulation here instead mines a span-level reward proxy from the policy’s own representations, using only the correctness labels already present in RLVR. A plausible implication is that span-level supervision can be induced from internal geometry rather than externally supplied at the step level.

2. Hidden-state divergence as a self-supervised span signal

The central observation is that correct-vs-incorrect hidden-state spans remain similar when reasoning is aligned, and separate where reasoning diverges. Let $y = (y_1, \ldots, y_T)$0 denote the last-layer hidden state, pre-LM-head, at token $y = (y_1, \ldots, y_T)$1 in rollout $y = (y_1, \ldots, y_T)$2. For a span $y = (y_1, \ldots, y_T)$3, the hidden states in that span are

$y = (y_1, \ldots, y_T)$4

Within a GRPO group, rollouts are partitioned into correct and incorrect sets, $y = (y_1, \ldots, y_T)$5 and $y = (y_1, \ldots, y_T)$6. The paper compares span-level hidden-state distributions across these opposing sets using the $y = (y_1, \ldots, y_T)$7-Wasserstein distance:

$y = (y_1, \ldots, y_T)$8

In practice, it uses the entropically regularized Sinkhorn approximation $y = (y_1, \ldots, y_T)$9 with the guarantee

$x$0

where $x$1 is the number of samples in the span. The stated motivation is fast GPU computation.

The empirical signal is reported at both global and local scales. Along trajectories, aggregate continuation success, estimated by re-completing from prefixes, declines as W-distance increases. Locally, decreases in continuation accuracy align with local increases in W-distance. Across spans, higher $x$2 correlates with lower success and discriminates correct-vs-incorrect spans increasingly well, with $x$3 at high $x$4 (Chen et al., 25 Apr 2026).

These observations motivate the use of span-level Wasserstein distance as a self-supervision signal. The method does not require token-wise alignment, step labels, or an additional scoring network. Instead, it assumes that hidden-state geometry already encodes where local reasoning quality separates.

3. Formalization through the separation theorem

The paper formalizes divergence with an operational notion based on matched correct and incorrect rollouts for the same input. There exists a divergence point $x$5 such that, up to $x$6, the hidden states are aligned in distribution, while after $x$7 the incorrect trajectory samples from a distinct distribution. The assumptions are:

• (A1) Divergence structure: for $x$8, $x$9; for $r \in \{0,1\}$0, $r \in \{0,1\}$1 and $r \in \{0,1\}$2.
• (A2) Bounded support: $r \in \{0,1\}$3.
• (A3) Finite-sample concentration: for independent bounded-support samples over a span of length $r \in \{0,1\}$4,

$r \in \{0,1\}$5

inducing a noise floor

$r \in \{0,1\}$6

For a span $r \in \{0,1\}$7 of length $r \in \{0,1\}$8, the population mixtures are

$r \in \{0,1\}$9

with population distance

$\pi_\theta$0

The pairwise separation theorem states that under (A1)–(A3):

• Pre-divergence ($\pi_\theta$1):

$\pi_\theta$2

• Post-divergence ($\pi_\theta$3):

$\pi_\theta$4

• If $\pi_\theta$5, expected post-divergence distance exceeds expected pre-divergence distance.

The proof intuition is summarized as replacing empirical measures by their population mixtures, applying triangle and reverse-triangle inequalities, and using (A3) to control empirical-to-population deviations. The key condition is that the population-level gap $\pi_\theta$6 dominates finite-sample noise.

The group-level extension compares an incorrect rollout to multiple correct rollouts. If $\pi_\theta$7 is the divergence point for each pair and $\pi_\theta$8, the minimum empirical discrepancy for an incorrect span $\pi_\theta$9 is

$x$0

with $x$1 ranging over spans in the correct rollouts. Then:

• If $x$2, $x$3.
• If $x$4, $x$5, where

$x$6

• $x$7 implies strict separation.

This theorem justifies the use of span-level Wasserstein distances as fine-grained credit signals: provided the population-level separation exceeds finite-sample noise, spans after hidden-state divergence are provably more distant from the opposing group than pre-divergence spans (Chen et al., 25 Apr 2026).

4. SHEAR as a SPAR++ algorithm

SHEAR modifies GRPO by reweighting token-level advantages according to span-level hidden-state divergence. For each rollout $x$8 of length $x$9, it generates overlapping spans with window size $G$0 and stride $G$1:

$G$2

For each span, the empirical hidden-state distribution is

$G$3

For rollout $G$4, the opposing set is defined as $G$5 if the rollout is correct and $G$6 if it is incorrect. The per-span discrepancy is the minimum Sinkhorn distance to all opposing spans:

$G$7

The “min over opposing spans” is explicitly described as conservative with multiple valid solution paths: a span is marked as divergent only if it is far from all alternatives in the opposing set.

To stabilize magnitudes, SHEAR uses a group-level hidden-state norm:

$G$8

Token weights are obtained by max-pooling over overlapping spans:

$G$9

Max-pooling is used so that tokens near divergence boundaries inherit the strongest local signal rather than being diluted by pre-divergence spans.

The rollout-level advantage $\mathcal{G}(x) = \{y^{(1)}, \ldots, y^{(G)}\}$0 is then replaced by a token-weighted advantage

$\mathcal{G}(x) = \{y^{(1)}, \ldots, y^{(G)}\}$1

yielding the GRPO update

$\mathcal{G}(x) = \{y^{(1)}, \ldots, y^{(G)}\}$2

In the main implementation, $\mathcal{G}(x) = \{y^{(1)}, \ldots, y^{(G)}\}$3 is the identity after normalization, so $\mathcal{G}(x) = \{y^{(1)}, \ldots, y^{(G)}\}$4. The paper notes that one can wrap $\mathcal{G}(x) = \{y^{(1)}, \ldots, y^{(G)}\}$5 with a monotone $\mathcal{G}(x) = \{y^{(1)}, \ldots, y^{(G)}\}$6, such as $\mathcal{G}(x) = \{y^{(1)}, \ldots, y^{(G)}\}$7 or a clipped linear map, but reports stable results without extra shaping. It also reports that an ablation using per-rollout normalization, which removes cross-rollout effects, retains most gains; this is taken as evidence that the core value lies in within-rollout span credit rather than cross-rollout magnitude modulation (Chen et al., 25 Apr 2026).

5. Implementation profile and empirical performance

The implementation uses the last transformer layer before the LM head at every generated token, with token-level granularity and hidden dimension equal to the model hidden size. Default span parameters are window $\mathcal{G}(x) = \{y^{(1)}, \ldots, y^{(G)}\}$8 and stride $\mathcal{G}(x) = \{y^{(1)}, \ldots, y^{(G)}\}$9. Spans containing the final answer tokens are excluded to avoid trivial leakage. Distance computation uses Sinkhorn-regularized optimal transport over span-level hidden-state point clouds. Hyperparameters explored include $A^{(i)}$0, stride $A^{(i)}$1, and group size $A^{(i)}$2 with additional tests at $A^{(i)}$3. Training settings are batch size $A^{(i)}$4, learning rate $A^{(i)}$5, clip range $A^{(i)}$6–$A^{(i)}$7, rollout length up to $A^{(i)}$8k tokens, mini-batch $A^{(i)}$9, and temperature $$\nabla_\theta J = \mathbb{E}_{y^{(i)} \sim \pi_\theta} \left[\sum_{t=1}^{T_i} A^{(i)} \nabla_\theta \log \pi_\theta(y_t^{(i)} \mid y_{<t}^{(i)}, x)\right].$$0 during training, with no entropy or KL regularization (Chen et al., 25 Apr 2026).

Relative to GRPO, total training time increases by $$\nabla_\theta J = \mathbb{E}_{y^{(i)} \sim \pi_\theta} \left[\sum_{t=1}^{T_i} A^{(i)} \nabla_\theta \log \pi_\theta(y_t^{(i)} \mid y_{<t}^{(i)}, x)\right].$$1 on Qwen2.5-Math-7B, $$\nabla_\theta J = \mathbb{E}_{y^{(i)} \sim \pi_\theta} \left[\sum_{t=1}^{T_i} A^{(i)} \nabla_\theta \log \pi_\theta(y_t^{(i)} \mid y_{<t}^{(i)}, x)\right].$$2 on Llama3.1-8B-Instruct, and $$\nabla_\theta J = \mathbb{E}_{y^{(i)} \sim \pi_\theta} \left[\sum_{t=1}^{T_i} A^{(i)} \nabla_\theta \log \pi_\theta(y_t^{(i)} \mid y_{<t}^{(i)}, x)\right].$$3 on Qwen2.5-14B-Base. The stated explanation is that Sinkhorn cost is fixed per-span while forward/backward cost dominates at larger scales.

The reported empirical results span five mathematical reasoning benchmarks and five code generation benchmarks. For math, the metrics are avg@32 for AIME24/25, avg@8 for AMC23, and avg@1 for MATH500 and OlympiadBench. For code, the metric is avg@4 on HumanEval, HumanEval+, MBPP, MBPP+, and LiveCodeBench v6 (Feb–May 2025).

On math, Qwen2.5-Math-7B improves from GRPO $$\nabla_\theta J = \mathbb{E}_{y^{(i)} \sim \pi_\theta} \left[\sum_{t=1}^{T_i} A^{(i)} \nabla_\theta \log \pi_\theta(y_t^{(i)} \mid y_{<t}^{(i)}, x)\right].$$4 to SHEAR $$\nabla_\theta J = \mathbb{E}_{y^{(i)} \sim \pi_\theta} \left[\sum_{t=1}^{T_i} A^{(i)} \nabla_\theta \log \pi_\theta(y_t^{(i)} \mid y_{<t}^{(i)}, x)\right].$$5 averaged over five benchmarks, with component changes AIME24 $$\nabla_\theta J = \mathbb{E}_{y^{(i)} \sim \pi_\theta} \left[\sum_{t=1}^{T_i} A^{(i)} \nabla_\theta \log \pi_\theta(y_t^{(i)} \mid y_{<t}^{(i)}, x)\right].$$6, AIME25 $$\nabla_\theta J = \mathbb{E}_{y^{(i)} \sim \pi_\theta} \left[\sum_{t=1}^{T_i} A^{(i)} \nabla_\theta \log \pi_\theta(y_t^{(i)} \mid y_{<t}^{(i)}, x)\right].$$7, AMC23 $$\nabla_\theta J = \mathbb{E}_{y^{(i)} \sim \pi_\theta} \left[\sum_{t=1}^{T_i} A^{(i)} \nabla_\theta \log \pi_\theta(y_t^{(i)} \mid y_{<t}^{(i)}, x)\right].$$8, MATH500 $$\nabla_\theta J = \mathbb{E}_{y^{(i)} \sim \pi_\theta} \left[\sum_{t=1}^{T_i} A^{(i)} \nabla_\theta \log \pi_\theta(y_t^{(i)} \mid y_{<t}^{(i)}, x)\right].$$9, and Olympiad $y = (y_1, \ldots, y_T)$00. Llama3.1-8B-Instruct improves from $y = (y_1, \ldots, y_T)$01 to $y = (y_1, \ldots, y_T)$02, with the largest gains on AMC23 $y = (y_1, \ldots, y_T)$03. Qwen2.5-14B-Base improves from $y = (y_1, \ldots, y_T)$04 to $y = (y_1, \ldots, y_T)$05, and SHEAR outperforms PRM-based variants on that backbone, where the PRM variants underperform GRPO. On code, Qwen2.5-Coder-7B improves from $y = (y_1, \ldots, y_T)$06 to $y = (y_1, \ldots, y_T)$07, with LiveCodeBench $y = (y_1, \ldots, y_T)$08; Llama3.1-8B-Instruct improves from $y = (y_1, \ldots, y_T)$09 to $y = (y_1, \ldots, y_T)$10; and Qwen2.5-14B-Base improves from $y = (y_1, \ldots, y_T)$11 to $y = (y_1, \ldots, y_T)$12, with LiveCodeBench $y = (y_1, \ldots, y_T)$13 (Chen et al., 25 Apr 2026).

The ablations emphasize that distribution-aware metrics matter. Wasserstein, Chamfer, and MMD all improve over GRPO, but Wasserstein is best. Cosine distance between span means harms performance, plateauing at approximately $y = (y_1, \ldots, y_T)$14 versus GRPO at approximately $y = (y_1, \ldots, y_T)$15, which the paper interprets as evidence that mean-only signals are insufficient. Performance is reported as robust to window size in $y = (y_1, \ldots, y_T)$16, while smaller stride, such as $y = (y_1, \ldots, y_T)$17, consistently helps by increasing overlap and coverage of local divergences. Increasing group size from $y = (y_1, \ldots, y_T)$18 to $y = (y_1, \ldots, y_T)$19 benefits both GRPO and SHEAR, but SHEAR benefits more, which is attributed to richer opposing sets yielding better Wasserstein estimates. Reliability diagnostics include aggregate Spearman $y = (y_1, \ldots, y_T)$20 between continuation success and $y = (y_1, \ldots, y_T)$21 along trajectories, local stride-wise correlation $y = (y_1, \ldots, y_T)$22, and a monotonic rise in AUC separating spans from originally correct vs. incorrect rollouts, exceeding $y = (y_1, \ldots, y_T)$23 at high $y = (y_1, \ldots, y_T)$24 (Chen et al., 25 Apr 2026).

6. Relation to SPAR, limitations, and deployment conditions

The paper frames SHEAR as “SPAR++” because it provides span-level credit without any step-level labels or extra model. In that sense, it is aligned with span-level reward modeling while differing technically from typical SPAR in three respects: there is no auxiliary process reward model, the supervision is outcome-only, and the span signal is self-calibrated because the hidden-state geometry evolves with the policy. This is presented as a way to avoid mismatch problems seen with fixed PRMs.

The advantages listed for this formulation are annotation-free and model-free span signals, the use of $y = (y_1, \ldots, y_T)$25 to capture beyond-mean distributional shape, and conservative min-to-opposing aggregation for handling multiple valid reasoning paths. The trade-offs are equally explicit. The method depends on the policy’s hidden-state geometry being informative; if representations do not separate reasoning quality, the signal weakens. It requires span construction and optimal transport computation, though the reported overhead remains below $y = (y_1, \ldots, y_T)$26. It is also sensitive to span segmentation and group composition and therefore needs enough opposing examples within a group (Chen et al., 25 Apr 2026).

The theorem and algorithm are conditional on several assumptions. The separation theorem assumes a clean divergence point, bounded-support hidden states, and a population-level gap $y = (y_1, \ldots, y_T)$27 large enough to dominate finite-sample noise $y = (y_1, \ldots, y_T)$28. Real trajectories may contain multiple divergence and correction events; the paper states that the group-level theorem and piecewise reasoning partially address this, but guarantees remain conditional. Weak divergence regimes, such as short chains or superficial errors, may produce small $y = (y_1, \ldots, y_T)$29 and therefore limited weight contrast. Distribution shift and multi-path equivalence can also suppress weights when incorrect spans remain close to some correct spans, because the conservative minimum-to-opposing aggregation intentionally yields low weights in that case.

Several mitigations are named for over-amplification risks when spurious high $y = (y_1, \ldots, y_T)$30 occurs in noisy regions: conservative min-to-opposing aggregation, optional monotone $y = (y_1, \ldots, y_T)$31 with clipping, per-rollout normalization, smoothing weights across neighboring spans, and excluding final-answer spans to avoid leakage. For integration into an RLVR+GRPO pipeline, the recommended workflow is to sample $y = (y_1, \ldots, y_T)$32 rollouts with last-layer hidden states recorded, partition them into $y = (y_1, \ldots, y_T)$33 and $y = (y_1, \ldots, y_T)$34 by correctness, fall back to vanilla GRPO if either set is empty, construct overlapping spans with $y = (y_1, \ldots, y_T)$35 and $y = (y_1, \ldots, y_T)$36, compute Sinkhorn distances to opposing spans and take the minimum, normalize with $y = (y_1, \ldots, y_T)$37, form $y = (y_1, \ldots, y_T)$38 by max-pooling, and apply GRPO updates with tokenwise weighted advantages. The paper recommends Wasserstein distance via Sinkhorn with Euclidean ground cost, and suggests considering $y = (y_1, \ldots, y_T)$39 if budget allows (Chen et al., 25 Apr 2026).

In this formulation, SPAR++ is not a separate reward model but a mechanism for deriving span-level reward or credit signals from hidden-state divergence under outcome-only supervision. This suggests a broader interpretation of span-level reward modeling in RLVR: fine-grained credit need not originate from explicit step labels if the policy’s internal representations already encode where reasoning trajectories separate.