Token Hidden Reward in Group-Relative RL

• Token Hidden Reward (THR) is a token-level metric that quantifies each token’s influence on the likelihood of correct responses in Group Relative Policy Optimization.
• Positive THR tokens boost confidence by favoring exploitation, while negative THR tokens retain uncertainty to encourage exploration.
• THR-guided reweighting techniques have demonstrated performance improvements in metrics like Pass@1 and Pass@K across different model architectures.

Searching arXiv for the specified THR and closely related token-level reward papers. Token Hidden Reward (THR) denotes a token-level signal intended to expose how local generation events affect future success. In its explicit formulation for group-relative reinforcement learning, THR is a token-level metric that quantifies each token’s influence on the likelihood of correct responses under Group Relative Policy Optimization (GRPO); tokens with positive THR strengthen confidence in correct outputs and favor exploitation, whereas tokens with negative THR preserve probability mass for alternative outputs and favor exploration (Deng et al., 4 Oct 2025). In adjacent work, closely related objects appear as token-local latent outcome signals in hidden activations, continuous token rewards for RLHF, temporally coherent reward/value trajectories, discriminative token-level Q-like rewards, and reward-gated token routing during inference (Zur et al., 6 Nov 2025, Yoon et al., 2024, Nikulkov, 24 Apr 2026, Chen et al., 29 May 2025, MS et al., 2024).

1. Formal definition in group-relative reinforcement learning

The explicit THR construction is introduced for reinforcement learning with verifiable rewards in GRPO and GSPO settings, where a prompt $x$ induces a group of $G$ sampled responses $\{y_i\}_{i=1}^G$ with binary rewards $r_i \in \{0,1\}$. In GRPO, the normalized advantage is constant across tokens within a response, and the training objective is a clipped policy-gradient objective defined over all tokens in the sampled group. THR is then defined as the token-level quantity governing how a token in one sampled response affects the likelihood dynamics of correct responses (Deng et al., 4 Oct 2025).

For a correct response $y_i^+$, another response $y_j$, and token position $k'$ in $y_j$, the definition is

$$\mathrm{THR}(y_i^+, y_j, k') = (2r_j-1)\cdot \sum_{k=1}^{|y_i^+|} \alpha_{k,k'} \cdot \langle \mathbf{h}_{\mathbf{x},y_{i,<k}^+}, \mathbf{h}_{\mathbf{x},y_{j,<k'}} \rangle.$$

Here, $(2r_j-1)$ gives a positive sign for correct responses and a negative sign for incorrect responses, the hidden-state inner product measures context alignment with the correct-response trajectory, and the coefficient $G$0 captures similarity between prediction errors. When a group contains multiple correct responses, the paper marginalizes over positives as

$G$1

This definition places THR at the level of token-token interaction across sampled reasoning trajectories rather than at the level of sequence outcomes alone. The object being measured is not token likelihood in isolation, but each token’s contribution to the change in log-likelihood of correct responses during RL updates.

2. Positive and negative THR as an exploration–exploitation signal

The central interpretation of THR is operational. Positive THR increases the likelihood of correct responses and is therefore treated as increasing confidence in the observed correct answer. Emphasizing such tokens drives probability mass toward high-confidence outputs and favors exploitation, with the expected effect of improving greedy decoding accuracy or Pass@1. Negative THR, by contrast, preserves uncertainty or probability mass for alternative continuations, thereby favoring exploration and improving Pass@K behavior (Deng et al., 4 Oct 2025).

A second empirical claim is that the training dynamics are dominated by a small subset of tokens with high absolute THR values. THR density plots show that most tokens are clustered near zero and only a small subset has high $G$2. The paper further reports that high-THR tokens overlap strongly with high-entropy tokens, often around 90% overlap, and qualitative analyses associate high-THR tokens with reasoning-critical language such as stating givens, transformations, constraints, decisions, and verification or conclusion (Deng et al., 4 Oct 2025).

This interpretation distinguishes THR from uniform sequence-level credit assignment. Under ordinary GRPO, all tokens within a trajectory inherit the same response-level advantage. THR instead treats reasoning traces as internally heterogeneous and identifies tokens that are disproportionately responsible for changing the learning dynamics.

3. THR-guided reweighting and observed effects

The main intervention built on THR is token-level advantage reweighting. The first variant keeps only dominant tokens: $G$3 The second variant adds sign-aware reweighting: $G$4 With $G$5, positive-THR tokens are amplified and negative-THR tokens are damped, biasing training toward exploitation. With $G$6, the reverse occurs, biasing training toward exploration (Deng et al., 4 Oct 2025).

The empirical results follow this interpretation closely. Vanilla THR with $G$7 and only dominant tokens performs very similarly to GRPO, supporting the claim that a small high-$G$8 subset largely drives learning. On Qwen2.5-Math-1.5B, THR($G$9) improves total greedy average over vanilla THR by 1.9% and over GRPO by 1.1%. On Qwen2.5-Math-7B, THR($\{y_i\}_{i=1}^G$0) improves total average greedy accuracy over vanilla THR by 3.5% and over GRPO by 4.0%. For exploration, THR($\{y_i\}_{i=1}^G$1) on Qwen2.5-Math-1.5B beats the best baseline by 2.4% at Pass@128 and 5.0% at Pass@256, while on Qwen2.5-Math-7B, THR($\{y_i\}_{i=1}^G$2) gives about 1% average gains across $\{y_i\}_{i=1}^G$3 (Deng et al., 4 Oct 2025).

The method also outperforms coarser alternatives. Compared with question-level Pass@K reweighting, THR usually achieves better Pass@K, especially on the Qwen models, by roughly $\{y_i\}_{i=1}^G$4 on average across many $\{y_i\}_{i=1}^G$5 values. THR($\{y_i\}_{i=1}^G$6) consistently beats Cov-KL across Pass@K settings, which the authors attribute to modeling cross-token interactions rather than only self-entropy effects. The approach transfers to GSPO-token, where THR($\{y_i\}_{i=1}^G$7) improves Pass@K by about 0.9% versus THR and 1.4% versus GSPO, and to Llama3.2-3B-Instruct, where it yields up to 7% Pass@K improvement over GRPO (Deng et al., 4 Oct 2025).

4. Related token-level hidden reward formulations

Although the exact THR definition above is specific to group-relative RL, several adjacent lines of work study token-local signals that play an analogous role.

In hidden-state analysis of chain-of-thought reasoning, a token-wise future outcome distribution is estimated by Forking Paths Analysis as

$\{y_i\}_{i=1}^G$8

and Bayesian Change Point Detection is applied to estimate $\{y_i\}_{i=1}^G$9. The same work uses residual-stream steering vectors

$r_i \in \{0,1\}$0

and linear separators

$r_i \in \{0,1\}$1

to predict or steer eventual outcomes. It reports a moderate correlation between steering success and base outcome probability of $r_i \in \{0,1\}$2 in the main text and an average correlation of $r_i \in \{0,1\}$3 across four examples, as well as probing results with KL loss $r_i \in \{0,1\}$4 versus $r_i \in \{0,1\}$5 at layer 8 for original-vs-separate model probes. The interpretation is that hidden activations encode alternate reasoning paths before commitment (Zur et al., 6 Nov 2025).

Token-level dense reward modeling for RLHF appears in several forms. TLCR trains a token preference discriminator $r_i \in \{0,1\}$6 from sequence-level preference data converted into token labels via GPT-4 revisions and Levenshtein edits, then maps discriminator confidence to a continuous reward

$r_i \in \{0,1\}$7

It reports an MT-Bench score of 5.04, an AlpacaEval win rate of 84.89%, a human preference share of 45.23%, and 81.4% token-level preference prediction accuracy for the discriminator (Yoon et al., 2024).

Temporally Coherent Reward Modeling argues that reward models should satisfy

$r_i \in \{0,1\}$8

so that prefix scores are conditional expectations of terminal reward. Its regularizers make reward models behave like value functions over prefixes, improving middle-token pairwise accuracy from 50% to 88.9% while preserving final-token accuracy, achieving 44.9% average F1 on ProcessBench among methods trained only on outcome data, and reducing PPO peak GPU memory by 27% and step time by 19% with matching LLM quality (Nikulkov, 24 Apr 2026).

Q-RM derives token-level reward from a discriminative policy

$r_i \in \{0,1\}$9

and treats the learned logits $y_i^+$0 as token rewards. Integrated into PPO or REINFORCE, Q-RM improves average Pass@1 by 5.85/4.70 points on mathematical reasoning tasks compared to the ORM baseline and by 4.56/5.73 points compared to token-level PRM counterparts, while converging 12 times faster than ORM on GSM8K and 11 times faster than step-level PRM on MATH (Chen et al., 29 May 2025).

TGDPO lifts preference optimization from the sequence level to the token level by deriving a token-level reward-guided DPO objective in which different tokens can deviate from the reference policy by different amounts according to token-wise reward guidance. It reports win-rate gains of up to 7.5 points on MT-Bench, 6.2 points on AlpacaEval 2, and 4.3 points on Arena-Hard (Zhu et al., 17 Jun 2025). GTPO and GRPO-S instead redistribute reward by detached policy entropy; the paper states that both methods significantly outperform the strong DAPO baseline on early runs in mathematical reasoning and code generation (Tan et al., 6 Aug 2025).

Inference-time analogues also exist. Reward Transformer adds token-level self-reward modeling to the policy model itself and uses Streaming Looking Ahead for search, achieving an overall win rate of 79.7% against greedy decoding with a frozen policy model and 89.4% when combined with DPO (Zhang et al., 24 Feb 2025). Efficient Hybrid Inference for LLMs uses a reward-based token modelling gate to decide whether an SLM token is accepted or whether a cloud LLM is invoked; on GSM8K, the cloud activation ratio is 56%, 78%, and 87% at thresholds 1.0, 2.0, and 4.0, with hybrid accuracies 66.48, 74.75, and 77.78 against SLM and LLM baselines of 50.7 and 77.8 (MS et al., 2024).

5. Critiques, limitations, and common misunderstandings

THR and related hidden token-level rewards are not universally reliable semantic signals. A direct critique comes from implicit reward modeling, where the reward is defined as

$y_i^+$1

That work argues that implicit reward models rely more heavily on superficial token-level cues than explicit reward heads and therefore generalize worse under token-level distribution shifts. On UltraFeedback paraphrased evaluation, EX-RM scores 0.687 versus IM-RM at 0.579, and on RewardMATH token-shift evaluation, EX-RM scores 0.988 versus IM-RM at 0.515. The same paper also rejects the idea that the main problem is generator–verifier coupling: in a Hamiltonian-cycle task, IM-RMs achieve near-perfect verification accuracy while failing to generate even a single correct Hamiltonian cycle (Razin et al., 10 Jul 2025).

A second misconception is that token-level reward should be treated as token-level ground truth. Reward Transformer explicitly states that “A token-level reward is an estimate of the final reward rather than an absolute ground truth,” and uses it as a search signal rather than as local truth (Zhang et al., 24 Feb 2025). TCRM similarly argues that ordinary reward models produce noisy intermediate token scores unless temporal coherence is explicitly enforced (Nikulkov, 24 Apr 2026).

A third limitation concerns heuristic surrogates for token importance. GTPO uses token entropy for reward shaping but notes that entropy is only a heuristic for importance; simple templated steps may still have high entropy due to lexical variability, and the paper leaves open how best to define the relativity of entropy across positions, within sequences, or across responses (Tan et al., 6 Aug 2025). By contrast, THR in group-relative RL is intended to capture cross-token influence across responses rather than token uncertainty alone (Deng et al., 4 Oct 2025).

The term itself is also not used uniformly. In group-relative RL, THR names a hidden-state interaction metric over sampled trajectories (Deng et al., 4 Oct 2025). In hybrid inference, closely related language describes a per-token reward-based routing mechanism that accepts or rejects SLM tokens according to a threshold (MS et al., 2024). In broader RLHF and reward-modeling work, analogous objects are described as continuous token rewards, token-level Q-functions, process reward trajectories, or implicit rewards rather than THR proper (Yoon et al., 2024, Chen et al., 29 May 2025, Nikulkov, 24 Apr 2026).

6. Significance and unifying interpretation

Across these literatures, a plausible unifying interpretation is that THR names a family resemblance rather than a single formal object. In the narrow sense, it is the signed token influence metric used to steer exploration and exploitation in GRPO-like training (Deng et al., 4 Oct 2025). In a broader mechanistic sense, it refers to token-local hidden quantities that estimate future correctness, preference, controllability, or reward before those quantities become explicit in the emitted text or in the final sequence score (Zur et al., 6 Nov 2025, Nikulkov, 24 Apr 2026).

This broader perspective is supported by several converging observations. Hidden activations can predict future outcome distributions before a model has committed to an answer, and activation steering is most effective when uncertainty remains high (Zur et al., 6 Nov 2025). Temporally coherent reward models can be trained so that token-level outputs behave like prefix value estimates rather than noise (Nikulkov, 24 Apr 2026). Discriminative token reward models and token-guided DPO variants show that token-level credit assignment can improve both sample efficiency and final alignment quality without requiring hand-labeled process supervision (Chen et al., 29 May 2025, Zhu et al., 17 Jun 2025). Inference-time systems such as Reward Transformer and reward-gated hybrid decoding further show that token-level hidden reward signals can guide search or routing online, not only policy optimization offline (Zhang et al., 24 Feb 2025, MS et al., 2024).

The significance of THR therefore lies in its shift of granularity. Instead of treating a response as the atomic unit of reward, it treats the reasoning process itself as structured, locally heterogeneous, and dynamically steerable. For reasoning-intensive LLMs, that shift enables direct intervention on exploration–exploitation trade-offs, denser credit assignment, earlier detection of commitment, and finer integration of reward information into both training and inference. At the same time, the literature also shows that token-level hidden rewards can be brittle, model-specific, or surface-form-sensitive, so their usefulness depends on how they are parameterized, regularized, and interpreted (Deng et al., 4 Oct 2025, Razin et al., 10 Jul 2025).