Relative Surprisal Index (RSI)

• RSI is a token-level metric that quantifies surprise by comparing the selected token's log-probability to the average, normalized by the predictive entropy.
• It captures both uncertainty and token rarity, reconciling high-entropy signals with low-probability token drawbacks in reinforcement learning with verifiable rewards.
• RSI selection (RSI-S) applies asymmetric bounds to filter tokens adaptively, leading to improved model performance and reduced response length on math benchmarks.

Searching arXiv for the specified paper to ground the article in the cited source. “Reverse Surprise Index” is a misnomer. In the RLVR literature, the correct term is Relative Surprisal Index (RSI), introduced in “Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Index” (Lv et al., 30 Jun 2026). RSI is a token-level score for reinforcement learning with verifiable rewards that couples two signals previously used separately in token selection and weighting: predictive entropy at a token position and the probability of the specific token that was actually sampled. The stated motivation is that evaluating sampled-token probability or entropy in isolation is insufficient to capture policy optimization dynamics. RSI therefore measures the selected token’s log-probability relative to the distribution’s average log-probability, normalized by entropy, and serves as the basis for RSI Selection (RSI-S), an entropy-adaptive token filtering method for GRPO-style RLVR (Lv et al., 30 Jun 2026).

1. Conceptual definition and nomenclature

The paper explicitly clarifies that Relative Surprisal Index is the correct expansion of RSI and that “Reverse Surprise Index” should not be used (Lv et al., 30 Jun 2026). In context, RSI is designed for RL with Verifiable Rewards, a setting in which token-level training decisions affect policy updates during reasoning-oriented optimization.

RSI is positioned as a response to two empirical paradigms that appear contradictory when stated independently. One line of inquiry emphasizes high-entropy token positions during training, whereas another warns against allowing low-probability tokens to dominate gradient updates. The paper argues that these positions are only superficially inconsistent, because high entropy and low sampled-token probability are correlated but not interchangeable descriptors of the optimization state. This suggests that the operative quantity is not uncertainty alone, nor improbability alone, but the relationship between the realized token and the uncertainty scale of the whole predictive distribution (Lv et al., 30 Jun 2026).

The paper’s core intuition is threefold. Entropy alone indicates where the model is uncertain but does not indicate whether the realized token was typical at that uncertainty level. Probability alone indicates how likely the realized token was, but not whether low probability reflects informative exploration or an unstable tail event. RSI combines both by comparing the selected token’s log-probability to the distribution’s average log-probability and normalizing by entropy. Positive RSI indicates “lower-than-average surprisal,” negative RSI indicates “higher-than-average surprisal,” and moderate magnitudes around zero correspond to tokens aligned with the distribution’s uncertainty scale (Lv et al., 30 Jun 2026).

2. Formal definition and basic properties

At step $t$, the language-model policy is defined as the softmax over logits $l_t \in \mathbb{R}^V$, where the vocabulary size is $V$ and

$$p_{t,i} = \mathrm{Softmax}(l_t)_i = \frac{\exp(l_{t,i})}{\sum_{j=1}^V \exp(l_{t,j})}.$$

The predictive entropy at position $t$ is

$$\mathcal{H}_t = -\sum_{s=1}^V p_{t,s}\,\log p_{t,s}.$$

The distributional average log-probability is

$$\mathbb{E}_{p}[\log p] = \sum_s p_{t,s}\log p_{t,s} = -\mathcal{H}_t.$$

If the sampled token is $o_t$ with probability $p_{t,n} = \pi_\theta(o_t \mid q, o_{<t})$, then the deviation of the selected log-probability from the distributional average is

$$\delta_t = \log p_{t,n} - \mathbb{E}_{p}[\log p] = \log p_{t,n} + \mathcal{H}_t.$$

The Relative Surprisal Index is then defined as

$l_t \in \mathbb{R}^V$0

It is defined for $l_t \in \mathbb{R}^V$1. For degenerate distributions with $l_t \in \mathbb{R}^V$2, the paper states that RSI is defined by continuity, with

$l_t \in \mathbb{R}^V$3

(Appendix C) (Lv et al., 30 Jun 2026).

Two normalization properties are emphasized. First, the expectation under the policy distribution is

$l_t \in \mathbb{R}^V$4

Second, the range is

$l_t \in \mathbb{R}^V$5

with heavy empirical mass near $l_t \in \mathbb{R}^V$6 according to Figure 1. These properties matter because they establish RSI as a centered, bounded-above token statistic whose sign and magnitude are directly interpretable in terms of relative surprisal rather than absolute surprisal (Lv et al., 30 Jun 2026).

3. Theoretical interpretation via local sensitivity

The theoretical contribution is framed through a perturbation of the selected logit,

$l_t \in \mathbb{R}^V$7

and through the gradient norm of the selected-token log-probability with respect to logits. The gradient is

$l_t \in \mathbb{R}^V$8

where $l_t \in \mathbb{R}^V$9 is the one-hot basis vector and $V$0 is the probability vector. Its $V$1 norm is

$V$2

Under mild conditions, specifically $V$3 and $V$4, Theorem 1 states that the ratio of first-order variations of $V$5 and $V$6 with respect to the selected-logit perturbation equals the reciprocal of RSI:

$V$7

The proof sketch proceeds from the softmax derivative

$V$8

the entropy derivative

$V$9

and the gradient-norm derivative

$$p_{t,i} = \mathrm{Softmax}(l_t)_i = \frac{\exp(l_{t,i})}{\sum_{j=1}^V \exp(l_{t,j})}.$$0

which combine algebraically into the stated ratio (Lv et al., 30 Jun 2026).

The interpretation given in the paper is that RSI characterizes the local sensitivity coupling between changes in entropy and changes in the logit-gradient norm when nudging the selected logit. Smaller $$p_{t,i} = \mathrm{Softmax}(l_t)_i = \frac{\exp(l_{t,i})}{\sum_{j=1}^V \exp(l_{t,j})}.$$1 implies stronger coupling, since the magnitude of the ratio becomes larger, while extreme negative RSI is associated with instability from tail tokens. A plausible implication is that RSI is not merely a descriptive score for filtering but also a diagnostic of how token-level perturbations map into optimization-relevant quantities in RLVR (Lv et al., 30 Jun 2026).

4. RSI Selection as entropy-adaptive token filtering

Building on RSI, the paper proposes RSI Selection (RSI-S), which retains tokens whose RSI lies within a stable interval. The method chooses asymmetric bounds $$p_{t,i} = \mathrm{Softmax}(l_t)_i = \frac{\exp(l_{t,i})}{\sum_{j=1}^V \exp(l_{t,j})}.$$2 with larger magnitude on the negative side, $$p_{t,i} = \mathrm{Softmax}(l_t)_i = \frac{\exp(l_{t,i})}{\sum_{j=1}^V \exp(l_{t,j})}.$$3, and keeps token $$p_{t,i} = \mathrm{Softmax}(l_t)_i = \frac{\exp(l_{t,i})}{\sum_{j=1}^V \exp(l_{t,j})}.$$4 iff

$$p_{t,i} = \mathrm{Softmax}(l_t)_i = \frac{\exp(l_{t,i})}{\sum_{j=1}^V \exp(l_{t,j})}.$$5

This admits an equivalent entropy-adaptive probability form:

$$p_{t,i} = \mathrm{Softmax}(l_t)_i = \frac{\exp(l_{t,i})}{\sum_{j=1}^V \exp(l_{t,j})}.$$6

The paper emphasizes that this differs from static probability thresholds because the gate tightens in low-entropy regimes and relaxes in high-entropy regimes (Lv et al., 30 Jun 2026).

The intended effect is to filter both ends of the token spectrum. Redundant low-surprisal tokens, particularly high-probability tokens in low-entropy regimes, are limited through the upper bound $$p_{t,i} = \mathrm{Softmax}(l_t)_i = \frac{\exp(l_{t,i})}{\sum_{j=1}^V \exp(l_{t,j})}.$$7. Unstable high-surprisal tail tokens, corresponding to very low-probability realizations, are limited through the lower bound $$p_{t,i} = \mathrm{Softmax}(l_t)_i = \frac{\exp(l_{t,i})}{\sum_{j=1}^V \exp(l_{t,j})}.$$8. The paper presents this as the mechanism by which RSI-S reconciles the empirical preference for high-entropy positions with the caution against tail-token dominance. This suggests that the “middle band” of RSI values is treated as the region in which tokens are simultaneously informative and comparatively stable for policy optimization (Lv et al., 30 Jun 2026).

The paper also includes concrete examples. For a 3-token distribution $$p_{t,i} = \mathrm{Softmax}(l_t)_i = \frac{\exp(l_{t,i})}{\sum_{j=1}^V \exp(l_{t,j})}.$$9, the entropy is approximately $t$0. If the selected token has probability $t$1, then $t$2 and $t$3, which is retained under bounds $t$4. If the selected token has probability $t$5, then $t$6 and $t$7; this remains within $t$8 but would be excluded under a tighter lower bound such as $t$9. In a low-surprisal regime with $$\mathcal{H}_t = -\sum_{s=1}^V p_{t,s}\,\log p_{t,s}.$$0 and $$\mathcal{H}_t = -\sum_{s=1}^V p_{t,s}\,\log p_{t,s}.$$1, selecting the $$\mathcal{H}_t = -\sum_{s=1}^V p_{t,s}\,\log p_{t,s}.$$2 token yields $$\mathcal{H}_t = -\sum_{s=1}^V p_{t,s}\,\log p_{t,s}.$$3, so lowering $$\mathcal{H}_t = -\sum_{s=1}^V p_{t,s}\,\log p_{t,s}.$$4 can filter overly predictable positions (Lv et al., 30 Jun 2026).

5. Integration into GRPO-style RLVR

The integration target is GRPO, described in the paper as group relative policy optimization with a KL term to a reference policy. GRPO’s per-token term is written as $$\mathcal{H}_t = -\sum_{s=1}^V p_{t,s}\,\log p_{t,s}.$$5 and includes PPO-style clipping and a KL penalty. The sequence-level advantage is

$$\mathcal{H}_t = -\sum_{s=1}^V p_{t,s}\,\log p_{t,s}.$$6

and the policy ratio is

$$\mathcal{H}_t = -\sum_{s=1}^V p_{t,s}\,\log p_{t,s}.$$7

The per-token objective term is

$$\mathcal{H}_t = -\sum_{s=1}^V p_{t,s}\,\log p_{t,s}.$$8

RSI-S introduces an indicator mask $$\mathcal{H}_t = -\sum_{s=1}^V p_{t,s}\,\log p_{t,s}.$$9 and the kept-token count

$$\mathbb{E}_{p}[\log p] = \sum_s p_{t,s}\log p_{t,s} = -\mathcal{H}_t.$$0

The resulting masked objective is

$$\mathbb{E}_{p}[\log p] = \sum_s p_{t,s}\log p_{t,s} = -\mathcal{H}_t.$$1

If $$\mathbb{E}_{p}[\log p] = \sum_s p_{t,s}\log p_{t,s} = -\mathcal{H}_t.$$2, the sequence is dropped from the update to avoid ill-defined normalization (Lv et al., 30 Jun 2026).

The algorithmic steps are given explicitly. For each batch of queries, the system generates $$\mathbb{E}_{p}[\log p] = \sum_s p_{t,s}\log p_{t,s} = -\mathcal{H}_t.$$3 sequences with the old policy $$\mathbb{E}_{p}[\log p] = \sum_s p_{t,s}\log p_{t,s} = -\mathcal{H}_t.$$4. For each sequence and each position, it computes current-policy logits, probabilities, entropy, and RSI. Tokens inside $$\mathbb{E}_{p}[\log p] = \sum_s p_{t,s}\log p_{t,s} = -\mathcal{H}_t.$$5 are included and the remainder are masked. The method then computes $$\mathbb{E}_{p}[\log p] = \sum_s p_{t,s}\log p_{t,s} = -\mathcal{H}_t.$$6 for included tokens, normalizes by the number of kept tokens, aggregates across tokens and sequences, and updates $$\mathbb{E}_{p}[\log p] = \sum_s p_{t,s}\log p_{t,s} = -\mathcal{H}_t.$$7 (Lv et al., 30 Jun 2026).

The practical computation is lightweight. Per-token RSI requires softmax probabilities, entropy $$\mathbb{E}_{p}[\log p] = \sum_s p_{t,s}\log p_{t,s} = -\mathcal{H}_t.$$8, and the selected-token log-probability. The paper states that entropy computation is a vector dot product of complexity $$\mathbb{E}_{p}[\log p] = \sum_s p_{t,s}\log p_{t,s} = -\mathcal{H}_t.$$9 and that the overall overhead is negligible relative to forward and backward passes. It further recommends log-softmax for numerical stability, notes that no second-order method or value network is required, and comments that temperature or logit scaling affects entropy and selected-token log-probability in a manner that makes selection more permissive in high-uncertainty regimes and stricter in low-uncertainty regimes (Lv et al., 30 Jun 2026).

6. Empirical behavior, baselines, and ablations

The empirical study uses Qwen2.5-1.5B, 3B, and 7B base models without math-specific SFT, trained on DAPO-MATH-17K and evaluated by avg@32 accuracy on AIME (2024–2026) and AMC (2022–2024). The headline result is that RSI-S improves avg@32 accuracy by 2–3 percentage points over GRPO across model scales and shortens responses (Lv et al., 30 Jun 2026).

Model	Average avg@32: GRPO → RSI-S	Length: GRPO → RSI-S
Qwen2.5-1.5B	12.15 → 14.25	1109.03 → 1000.33
Qwen2.5-3B	18.53 → 21.83	1204.98 → 1022.12
Qwen2.5-7B	29.05 → 31.24	1167.11 → 901.16

The paper also reports benchmark-specific changes. For Qwen2.5-1.5B, AIME changes from 2.36 to 2.47 and AMC from 21.95 to 26.04. For Qwen2.5-3B, AIME changes from 5.07 to 5.73 and AMC from 31.98 to 37.93. For Qwen2.5-7B, AIME changes from 10.69 to 11.25 and AMC from 47.41 to 51.24 (Lv et al., 30 Jun 2026).

The baseline set consists of GRPO, EB, and PB. EB is described as an entropy baseline that keeps the top 20% highest-entropy tokens. PB is described as the probability baseline of “Do Not Let Low-Probability Tokens Over-Dominate,” which reweights tokens to avoid dominance by low-probability tokens. RSI-S is reported to consistently outperform GRPO, EB, and PB across scales (Lv et al., 30 Jun 2026).

The ablations are central to the paper’s argument. Static probability masks fail: for Qwen2.5-3B, fixed ranges P1 $o_t$0, P2 $o_t$1, and P3 $o_t$2 underperform GRPO, whereas RSI-S improves by $o_t$3 percentage points. Balanced bounds are also necessary: removing the upper bound (RSI-SU) or lower bound (RSI-SL) degrades performance relative to the balanced interval. Interval sensitivity varies by scale, with best $o_t$4 values shifting downward as models grow larger: $o_t$5 for 1.5B, $o_t$6 for 3B, and $o_t$7 for 7B. Finally, Figure 2 shows that RSI covers more high Jensen–Shannon divergence tokens than entropy selection and is less sensitive to threshold choice; the normalized variance of coverage is reported as $o_t$8, $o_t$9, and $p_{t,n} = \pi_\theta(o_t \mid q, o_{<t})$0 the sensitivity of entropy at 1.5B, 3B, and 7B respectively (Lv et al., 30 Jun 2026).

These results support the paper’s claim that entropy-adaptive selection, rather than static probability gating, is the relevant mechanism. A plausible implication is that RSI-S is acting less as a coarse sparsification heuristic than as a token-level variance-control device tuned to the local uncertainty geometry of the policy.

7. Scope, implementation details, and open questions

The implementation described in the paper uses EasyR1 on top of verl with Qwen2.5-1.5B, 3B, and 7B base models. Training uses a unified prompt template and samples responses with temperature $p_{t,n} = \pi_\theta(o_t \mid q, o_{<t})$1 and top-$p_{t,n} = \pi_\theta(o_t \mid q, o_{<t})$2 $p_{t,n} = \pi_\theta(o_t \mid q, o_{<t})$3. GRPO specifics include group size $p_{t,n} = \pi_\theta(o_t \mid q, o_{<t})$4, learning rate $p_{t,n} = \pi_\theta(o_t \mid q, o_{<t})$5, batch size $p_{t,n} = \pi_\theta(o_t \mid q, o_{<t})$6, mini-batch size $p_{t,n} = \pi_\theta(o_t \mid q, o_{<t})$7, ten epochs corresponding to $p_{t,n} = \pi_\theta(o_t \mid q, o_{<t})$8 outer steps and $p_{t,n} = \pi_\theta(o_t \mid q, o_{<t})$9 mini-batch updates, and a KL term to a reference policy with coefficient $$\delta_t = \log p_{t,n} - \mathbb{E}_{p}[\log p] = \log p_{t,n} + \mathcal{H}_t.$$0 together with PPO-style clipping $$\delta_t = \log p_{t,n} - \mathbb{E}_{p}[\log p] = \log p_{t,n} + \mathcal{H}_t.$$1. The reported RSI-S intervals are $$\delta_t = \log p_{t,n} - \mathbb{E}_{p}[\log p] = \log p_{t,n} + \mathcal{H}_t.$$2 for 1.5B, $$\delta_t = \log p_{t,n} - \mathbb{E}_{p}[\log p] = \log p_{t,n} + \mathcal{H}_t.$$3 for 3B, and $$\delta_t = \log p_{t,n} - \mathbb{E}_{p}[\log p] = \log p_{t,n} + \mathcal{H}_t.$$4 for 7B. Evaluation uses avg@32 with temperature $$\delta_t = \log p_{t,n} - \mathbb{E}_{p}[\log p] = \log p_{t,n} + \mathcal{H}_t.$$5 and top-$$\delta_t = \log p_{t,n} - \mathbb{E}_{p}[\log p] = \log p_{t,n} + \mathcal{H}_t.$$6 $$\delta_t = \log p_{t,n} - \mathbb{E}_{p}[\log p] = \log p_{t,n} + \mathcal{H}_t.$$7 on AIME 2024–2026 and AMC 2022–2024 (Lv et al., 30 Jun 2026).

The paper states several limitations. The theoretical link depends on first-order perturbations of the selected logit and requires $$\delta_t = \log p_{t,n} - \mathbb{E}_{p}[\log p] = \log p_{t,n} + \mathcal{H}_t.$$8 and $$\delta_t = \log p_{t,n} - \mathbb{E}_{p}[\log p] = \log p_{t,n} + \mathcal{H}_t.$$9. In highly degenerate distributions, $l_t \in \mathbb{R}^V$00 by continuity, and selection effectively reduces to trimming extreme negatives via $l_t \in \mathbb{R}^V$01. Overly aggressive bounds can prune too many tokens, especially for smaller models, whereas overly permissive bounds may fail to reduce variance. The reported results are restricted to math benchmarks with base models, so gains may differ on other domains or under data contamination. The paper also states that RSI-S is complementary to other signals, such as future KL and polarity-aware weighting, but that the interactions require study (Lv et al., 30 Jun 2026).

The future directions outlined are specific. They include replacing hard masks with soft weighting, potentially using a bell-shaped function around zero; learning $l_t \in \mathbb{R}^V$02 online or tying bounds to batch-level entropy statistics or reward variance; combining RSI with future-KL, polarity-aware advantages, or sequence-level divergence; exploring applications beyond RLVR for math in instruction-following, coding, or dialogue; and extending the theory to multi-logit perturbations, second-order terms, value-regularized objectives, and explicit variance-reduction analysis (Lv et al., 30 Jun 2026).

Within the scope of the paper, the significance of RSI is therefore narrowly defined but technically precise. It is a token-level, entropy-normalized relative surprisal measure with a local sensitivity interpretation and a practical role as the gating criterion in RSI-S. The terminology “Reverse Surprise Index” obscures that role; the established term in the source is Relative Surprisal Index (Lv et al., 30 Jun 2026).