---
title: 'SPAR++: Span-Level Reward Modeling'
url: https://www.emergentmind.com/topics/span-level-reward-modeling-spar
type: topic
---

# SPAR++: Span-Level Reward Modeling

Searching arXiv for the specified paper and closely related reward-modeling context.
tool call: arxiv_search
tool result: arxiv_search
Span-Level Reward Modeling, described in the formulation termed “SPAR++” in the context of "Hidden States Know Where Reasoning Diverges: Credit Assignment via Span-Level Wasserstein Distance" [2604.23318], denotes a span-level credit-assignment approach for reinforcement learning with verifiable rewards (RLVR) that derives a per-span reward proxy directly from the policy’s hidden-state geometry. In this setting, a policy $\pi_\theta$ generates a rollout $y = (y_1, \ldots, y_T)$ for an input $x$ and receives an outcome-level verifiable reward $r \in \{0,1\}$ indicating correctness. The core claim is that, within a Group Relative Policy Optimization (GRPO) group containing both correct and incorrect rollouts for the same problem, hidden-state distributions diverge around spans where local reasoning quality differs. SHEAR—Span-level Hidden state Enabled Advantage Reweighting—operationalizes that observation by scaling token-level advantages with span-level Wasserstein distances, thereby turning outcome-only RLVR into span-level credit assignment without step labels or an auxiliary reward model [2604.23318].

## 1. RLVR, GRPO, and the span-level credit gap

RLVR assumes that supervision is available only at the level of final correctness. For inputs such as math or code problems, a policy $\pi_\theta$ produces a full rollout and receives a binary reward. In GRPO, for each $x$, one samples a group of $G$ rollouts $\mathcal{G}(x) = \{y^{(1)}, \ldots, y^{(G)}\}$ and computes a group-normalized scalar advantage $A^{(i)}$ per rollout using the group’s rewards. The GRPO policy gradient assigns the same advantage to every token in the rollout:
$$
\nabla_\theta J = \mathbb{E}_{y^{(i)} \sim \pi_\theta} \left[\sum_{t=1}^{T_i} A^{(i)} \nabla_\theta \log \pi_\theta(y_t^{(i)} \mid y_{<t}^{(i)}, x)\right].
$$

This is a coarse-grained credit signal. Identical feedback is applied to all tokens in a rollout even when only a small subset of steps determines correctness. The paper identifies this mismatch as a span-level credit gap: process reward models can provide step-level scores, but they require step annotations or training an auxiliary reward model, whereas the proposed method works with outcome-only supervision and extracts a span-level signal directly from hidden states [2604.23318].

Conceptually, this places SPAR++ in a distinct position relative to traditional span-level reward modeling. Traditional SPAR relies on span-level reward models or annotations. The formulation here instead mines a span-level reward proxy from the policy’s own representations, using only the correctness labels already present in RLVR. A plausible implication is that span-level supervision can be induced from internal geometry rather than externally supplied at the step level.

## 2. Hidden-state divergence as a self-supervised span signal

The central observation is that correct-vs-incorrect hidden-state spans remain similar when reasoning is aligned, and separate where reasoning diverges. Let $h_t^{(i)} \in \mathbb{R}^d$ denote the last-layer hidden state, pre-LM-head, at token $t$ in rollout $y^{(i)}$. For a span $s = [i,j]$, the hidden states in that span are
$$
H_s^{(i)} = \{h_t^{(i)} : t \in [i,j]\}.
$$

Within a GRPO group, rollouts are partitioned into correct and incorrect sets, $\mathcal{G}^+$ and $\mathcal{G}^-$. The paper compares span-level hidden-state distributions across these opposing sets using the $1$-Wasserstein distance:
$$
W_1(P,Q) = \inf_{\gamma \in \Pi(P,Q)} \mathbb{E}_{(x,y)\sim\gamma}[\|x-y\|_2].
$$
In practice, it uses the entropically regularized Sinkhorn approximation $W_\epsilon$ with the guarantee
$$
W_1(P,Q) \le W_\epsilon(P,Q) \le W_1(P,Q) + \epsilon \log n,
$$
where $n$ is the number of samples in the span. The stated motivation is fast GPU computation.

The empirical signal is reported at both global and local scales. Along trajectories, aggregate continuation success, estimated by re-completing from prefixes, declines as W-distance increases. Locally, decreases in continuation accuracy align with local increases in W-distance. Across spans, higher $W$ correlates with lower success and discriminates correct-vs-incorrect spans increasingly well, with $\mathrm{AUC} > 0.97$ at high $W$ [2604.23318].

These observations motivate the use of span-level Wasserstein distance as a self-supervision signal. The method does not require token-wise alignment, step labels, or an additional scoring network. Instead, it assumes that hidden-state geometry already encodes where local reasoning quality separates.

## 3. Formalization through the separation theorem

The paper formalizes divergence with an operational notion based on matched correct and incorrect rollouts for the same input. There exists a divergence point $\tau$ such that, up to $\tau$, the hidden states are aligned in distribution, while after $\tau$ the incorrect trajectory samples from a distinct distribution. The assumptions are:

- **(A1) Divergence structure**: for $t \le \tau$, $h_t^+, h_t^- \sim P_t$; for $t > \tau$, $h_t^+ \sim P_t$ and $h_t^- \sim Q_t \ne P_t$.
- **(A2) Bounded support**: $\|h_t\|_2 \le M$.
- **(A3) Finite-sample concentration**: for independent bounded-support samples over a span of length $n$,
  $$
  \mathbb{E}[W_1(\hat P_n, \bar P_n)] \le \tilde C_d M n^{-1/d},
  $$
  inducing a noise floor
  $$
  \eta(n,d) := 2 \tilde C_d M n^{-1/d}.
  $$

For a span $S=[a,b]$ of length $n$, the population mixtures are
$$
\bar P_S^+ = \frac{1}{n}\sum_{t=a..b} P_t, \qquad \bar P_S^- = \frac{1}{n}\sum_{t=a..b} Q_t,
$$
with population distance
$$
D(S) = W_1(\bar P_S^-, \bar P_S^+).
$$

The pairwise separation theorem states that under (A1)–(A3):

- **Pre-divergence** ($b \le \tau$):
  $$
  \mathbb{E}[W_1(\hat P_S^-, \hat P_S^+)] \le \eta(n,d).
  $$
- **Post-divergence** ($a > \tau$):
  $$
  \mathbb{E}[W_1(\hat P_S^-, \hat P_S^+)] \ge D(S) - \eta(n,d).
  $$
- If $D(S) > 2\eta(n,d)$, expected post-divergence distance exceeds expected pre-divergence distance.

The proof intuition is summarized as replacing empirical measures by their population mixtures, applying triangle and reverse-triangle inequalities, and using (A3) to control empirical-to-population deviations. The key condition is that the population-level gap $D(S)$ dominates finite-sample noise.

The group-level extension compares an incorrect rollout to multiple correct rollouts. If $\tau_k$ is the divergence point for each pair and $\tau^* = \max_k \tau_k$, the minimum empirical discrepancy for an incorrect span $S$ is
$$
d_{\min}(S) = \min_{k,j} W_1(\hat P_S^-, \hat P_{S'^{+},k}),
$$
with $S'$ ranging over spans in the correct rollouts. Then:

- If $b \le \tau^*$, $\mathbb{E}[d_{\min}(S)] \le \eta(n,d)$.
- If $a > \tau^*$, $\mathbb{E}[d_{\min}(S)] \ge D^*(S) - \eta(n,d)$, where
  $$
  D^*(S) = \min_{k,j} W_1(\bar P_S^-, \bar P_{S'^{+},k}).
  $$
- $D^*(S) > 2\eta$ implies strict separation.

This theorem justifies the use of span-level Wasserstein distances as fine-grained credit signals: provided the population-level separation exceeds finite-sample noise, spans after hidden-state divergence are provably more distant from the opposing group than pre-divergence spans [2604.23318].

## 4. SHEAR as a SPAR++ algorithm

SHEAR modifies GRPO by reweighting token-level advantages according to span-level hidden-state divergence. For each rollout $y^{(i)}$ of length $T_i$, it generates overlapping spans with window size $w$ and stride $s$:
$$
\mathcal{S}^{(i)} = \{S_k^{(i)} = [a_k,b_k] : a_k = 1 + (k-1)s,\; b_k = \min(a_k + w - 1, T_i)\}.
$$
For each span, the empirical hidden-state distribution is
$$
\hat P_k^{(i)} = \frac{1}{|S_k^{(i)}|}\sum_{t \in S_k^{(i)}} \delta_{h_t^{(i)}}.
$$

For rollout $y^{(i)}$, the opposing set is defined as $\mathcal{O}^{(i)} = \mathcal{G}^-$ if the rollout is correct and $\mathcal{O}^{(i)} = \mathcal{G}^+$ if it is incorrect. The per-span discrepancy is the minimum Sinkhorn distance to all opposing spans:
$$
d_k^{(i)} = \min_{y^{(j)} \in \mathcal{O}^{(i)},\, S_\ell^{(j)} \in \mathcal{S}^{(j)}} W_\epsilon(\hat P_k^{(i)}, \hat P_\ell^{(j)}).
$$
The “min over opposing spans” is explicitly described as conservative with multiple valid solution paths: a span is marked as divergent only if it is far from all alternatives in the opposing set.

To stabilize magnitudes, SHEAR uses a group-level hidden-state norm:
$$
N_{\mathcal G} = \sum_i T_i, \qquad \bar n = \frac{1}{N_{\mathcal G}} \sum_i \sum_t \|h_t^{(i)}\|_2.
$$
Token weights are obtained by max-pooling over overlapping spans:
$$
\omega_t^{(i)} = \frac{1}{\bar n}\max_{k:\, t \in S_k^{(i)}} d_k^{(i)}.
$$
Max-pooling is used so that tokens near divergence boundaries inherit the strongest local signal rather than being diluted by pre-divergence spans.

The rollout-level advantage $A^{(i)}$ is then replaced by a token-weighted advantage
$$
\tilde A_t^{(i)} = A^{(i)} \cdot \omega_t^{(i)},
$$
yielding the GRPO update
$$
\nabla_\theta J = \mathbb{E}_{y^{(i)}\sim \pi_\theta}\left[\sum_t \tilde A_t^{(i)} \nabla_\theta \log \pi_\theta(y_t^{(i)} \mid y_{<t}^{(i)}, x)\right].
$$

In the main implementation, $g(\cdot)$ is the identity after normalization, so $\tilde A_t = A \cdot \omega_t$. The paper notes that one can wrap $\omega$ with a monotone $g$, such as $g(W)=\mathrm{softplus}(\alpha(W-\beta))$ or a clipped linear map, but reports stable results without extra shaping. It also reports that an ablation using per-rollout normalization, which removes cross-rollout effects, retains most gains; this is taken as evidence that the core value lies in within-rollout span credit rather than cross-rollout magnitude modulation [2604.23318].

## 5. Implementation profile and empirical performance

The implementation uses the last transformer layer before the LM head at every generated token, with token-level granularity and hidden dimension equal to the model hidden size. Default span parameters are window $w=100$ and stride $s=25$. Spans containing the final answer tokens are excluded to avoid trivial leakage. Distance computation uses Sinkhorn-regularized optimal transport over span-level hidden-state point clouds. Hyperparameters explored include $w \in \{80,100,120\}$, stride $s \in \{25,\ldots,75\}$, and group size $G=8$ with additional tests at $G=16$. Training settings are batch size $512$, learning rate $1\times 10^{-6}$, clip range $0.2$–$0.28$, rollout length up to $3$k tokens, mini-batch $32$, and temperature $1.0$ during training, with no entropy or KL regularization [2604.23318].

Relative to GRPO, total training time increases by $+15.5\%$ on Qwen2.5-Math-7B, $+10.4\%$ on Llama3.1-8B-Instruct, and $+7.4\%$ on Qwen2.5-14B-Base. The stated explanation is that Sinkhorn cost is fixed per-span while forward/backward cost dominates at larger scales.

The reported empirical results span five mathematical reasoning benchmarks and five code generation benchmarks. For math, the metrics are avg@32 for AIME24/25, avg@8 for AMC23, and avg@1 for MATH500 and OlympiadBench. For code, the metric is avg@4 on HumanEval, HumanEval+, MBPP, MBPP+, and LiveCodeBench v6 (Feb–May 2025).

On math, Qwen2.5-Math-7B improves from GRPO $49.6 \pm 0.8$ to SHEAR $51.2 \pm 0.8$ averaged over five benchmarks, with component changes AIME24 $33.6 \to 35.2$, AIME25 $15.5 \to 16.8$, AMC23 $66.5 \to 70.3$, MATH500 $84.9 \to 83.3$, and Olympiad $47.8 \to 48.7$. Llama3.1-8B-Instruct improves from $24.7 \pm 0.4$ to $26.2 \pm 0.1$, with the largest gains on AMC23 $(32.7 \to 37.2)$. Qwen2.5-14B-Base improves from $45.4 \pm 0.4$ to $46.9 \pm 1.0$, and SHEAR outperforms PRM-based variants on that backbone, where the PRM variants underperform GRPO. On code, Qwen2.5-Coder-7B improves from $63.1$ to $63.9$, with LiveCodeBench $17.8 \to 18.3$; Llama3.1-8B-Instruct improves from $54.7$ to $55.8$; and Qwen2.5-14B-Base improves from $67.1$ to $67.7$, with LiveCodeBench $19.7 \to 21.1$ [2604.23318].

The ablations emphasize that distribution-aware metrics matter. Wasserstein, Chamfer, and MMD all improve over GRPO, but Wasserstein is best. Cosine distance between span means harms performance, plateauing at approximately $0.4837$ versus GRPO at approximately $0.4884$, which the paper interprets as evidence that mean-only signals are insufficient. Performance is reported as robust to window size in $\{80,100,120\}$, while smaller stride, such as $s=25$, consistently helps by increasing overlap and coverage of local divergences. Increasing group size from $8$ to $16$ benefits both GRPO and SHEAR, but SHEAR benefits more, which is attributed to richer opposing sets yielding better Wasserstein estimates. Reliability diagnostics include aggregate Spearman $\rho \approx -0.96$ between continuation success and $W$ along trajectories, local stride-wise correlation $\rho \approx -0.42$, and a monotonic rise in AUC separating spans from originally correct vs. incorrect rollouts, exceeding $0.97$ at high $W$ [2604.23318].

## 6. Relation to SPAR, limitations, and deployment conditions

The paper frames SHEAR as “SPAR++” because it provides span-level credit without any step-level labels or extra model. In that sense, it is aligned with span-level reward modeling while differing technically from typical SPAR in three respects: there is no auxiliary process reward model, the supervision is outcome-only, and the span signal is self-calibrated because the hidden-state geometry evolves with the policy. This is presented as a way to avoid mismatch problems seen with fixed PRMs.

The advantages listed for this formulation are annotation-free and model-free span signals, the use of $W_1$ to capture beyond-mean distributional shape, and conservative min-to-opposing aggregation for handling multiple valid reasoning paths. The trade-offs are equally explicit. The method depends on the policy’s hidden-state geometry being informative; if representations do not separate reasoning quality, the signal weakens. It requires span construction and optimal transport computation, though the reported overhead remains below $16\%$. It is also sensitive to span segmentation and group composition and therefore needs enough opposing examples within a group [2604.23318].

The theorem and algorithm are conditional on several assumptions. The separation theorem assumes a clean divergence point, bounded-support hidden states, and a population-level gap $D(S)$ large enough to dominate finite-sample noise $\eta$. Real trajectories may contain multiple divergence and correction events; the paper states that the group-level theorem and piecewise reasoning partially address this, but guarantees remain conditional. Weak divergence regimes, such as short chains or superficial errors, may produce small $D(S)$ and therefore limited weight contrast. Distribution shift and multi-path equivalence can also suppress weights when incorrect spans remain close to some correct spans, because the conservative minimum-to-opposing aggregation intentionally yields low weights in that case.

Several mitigations are named for over-amplification risks when spurious high $W$ occurs in noisy regions: conservative min-to-opposing aggregation, optional monotone $g(W)$ with clipping, per-rollout normalization, smoothing weights across neighboring spans, and excluding final-answer spans to avoid leakage. For integration into an RLVR+GRPO pipeline, the recommended workflow is to sample $G$ rollouts with last-layer hidden states recorded, partition them into $\mathcal{G}^+$ and $\mathcal{G}^-$ by correctness, fall back to vanilla GRPO if either set is empty, construct overlapping spans with $w=100$ and $s=25$, compute Sinkhorn distances to opposing spans and take the minimum, normalize with $\bar n$, form $\omega_t^{(i)}$ by max-pooling, and apply GRPO updates with tokenwise weighted advantages. The paper recommends Wasserstein distance via Sinkhorn with Euclidean ground cost, and suggests considering $G=16$ if budget allows [2604.23318].

In this formulation, SPAR++ is not a separate reward model but a mechanism for deriving span-level reward or credit signals from hidden-state divergence under outcome-only supervision. This suggests a broader interpretation of span-level reward modeling in RLVR: fine-grained credit need not originate from explicit step labels if the policy’s internal representations already encode where reasoning trajectories separate.

Source: https://www.emergentmind.com/topics/span-level-reward-modeling-spar