Group-Level Update Scale in TEPO

• Group-Level Update Scale is the framework that translates a coarse, sequence-level reward into finely calibrated token-level updates using sequence likelihood aggregation.
• TEPO mitigates instability by normalizing rewards over tokens with length normalization, clipping, and batch-level scaling to ensure consistent gradient updates.
• Selective KL-Divergence masking is applied to tokens with positive advantage and decreasing entropy, preventing abrupt shifts and entropy collapse in policy updates.

Searching arXiv for the primary paper and closely related work on TEPO and group-level update scaling. Group-Level Update Scale denotes the problem of converting a group-relative, sequence-level reward into token-level policy updates whose magnitude and distribution remain well calibrated over long chain-of-thought trajectories. In the TEPO line of work, the issue arises because Group Relative Policy Optimization (GRPO) uses a group-level normalized reward, while chain-of-thought reasoning exposes sparse token-level rewards: the final outcome may indicate whether a full response is good, but intermediate tokens do not receive explicit token-specific supervision. TEPO addresses this mismatch by linking group-level rewards to tokens through sequence-level likelihood and token-level aggregation, and, in its later form, by adding a token-level KL-Divergence mask constraint that targets tokens with positive advantages and decreasing entropy to mitigate abrupt policy updates (Lin et al., 14 Apr 2026).

1. Problem formulation in critic-free chain-of-thought RL

In TEPO, the starting point is a group-relative reward computed from multiple sampled responses to the same prompt. The group-level advantage is

$$A_t = \frac{r(\boldsymbol y)-\operatorname{mean}(r(\boldsymbol y^{1:G}))}{\operatorname{std}(r(\boldsymbol y^{1:G}))}.$$

Although the notation is token-indexed, this quantity originates from the sequence reward $r(\boldsymbol y)$ and normalization across $G$ sampled responses. The supervision is therefore fundamentally coarse: it determines whether a whole response is better or worse than its group, but not which particular tokens deserve credit (Lin et al., 14 Apr 2026).

The paper characterizes the resulting difficulty as a group-level update scale problem in several senses. First, the signal is too coarse, because a single sequence-level or group-level advantage must supervise many tokens. Second, the signal is unevenly distributed across tokens, because naive token-level importance sampling or uniform entropy or KL regularization does not properly reflect the global shift of the whole reasoning trajectory. Third, updates can become abrupt or unstable in critic-free GRPO, where sparse-reward trajectories produce high-variance gradients and undifferentiated token-level entropy regularization can induce entropy collapse or model collapse (Lin et al., 14 Apr 2026).

This framing is continuous with the earlier TEPO version based on “Markov Likelihood,” which argued that token-level ratios in GRPO “do not fully reflect how the whole sequence distribution changes” and therefore yield noisy gradient estimates under sparse rewards (Lin et al., 10 Oct 2025). A plausible implication is that the term “group-level update scale” refers not merely to reward normalization across sampled responses, but to the entire mapping from one group-normalized scalar to many token-level logit updates.

2. Sequence-level likelihood as the bridge from group reward to token optimization

TEPO’s central mathematical move is to use the autoregressive factorization of sequence probability,

$$\pi_\theta(y\mid x)=\prod_t \pi_\theta(y_t\mid x,y_{<t}),$$

and therefore

$$\log \pi_\theta(y\mid x)=\sum_t \log \pi_\theta(y_t\mid x,y_{<t}).$$

This decomposition is the bridge from sequence-level reward to token-level optimization: a sequence objective can still generate token gradients because sequence log-likelihood is a sum of token log-probabilities (Lin et al., 14 Apr 2026).

The paper further derives

$J(\theta)=\mathbb E_{\tau\sim p_\theta}[A(\tau)],$

and

$$\nabla J(\theta) = \mathbb E_{\tau\sim p_{\theta_{\text{old}}}} \left[ \frac{p_\theta(\tau)}{p_{\theta_{\text{old}}}(\tau)} \nabla \log p_\theta(\tau)\, A_{\theta_{\text{old}}}(\tau) \right].$$

For sequential generation, trajectory probability factorizes over token steps, so the full importance ratio is a product over token-level ratios. TEPO’s criticism of GRPO is that token-level importance sampling omits this cross-token product structure and therefore mismatches the actual whole-trajectory shift (Lin et al., 14 Apr 2026).

The TEPO sequence-level likelihood weight is

$$w_i(\theta) = \exp\left( \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \log \frac{\pi_\theta(y_{i,t}\mid x,y_{i,<t})} {\pi_{\theta_{\text{old}}}(y_{i,t}\mid x,y_{i,<t})} \right),$$

or equivalently

$$w_i(\theta) = \left( \frac{\pi_\theta(y_i\mid x)} {\pi_{\theta_{\text{old}}}(y_i\mid x)} \right)^{1/|y_i|}.$$

This is the mechanism by which a sequence-level or group-level reward becomes a sequence-level update multiplier that is then redistributed to tokens. TEPO does not assign a distinct importance ratio to each token. Instead, it computes a single sequence-level weight and shares it across the tokens of that sequence through token-level aggregation (Lin et al., 14 Apr 2026).

The earlier TEPO formulation presented the same construction as Markov Likelihood importance sampling and emphasized the role of the geometric mean: it transforms a potentially exponentially length-dependent product into an average log-ratio exponentiated back, making the sequence-level term comparable across different sequence lengths (Lin et al., 10 Oct 2025).

3. Token-level aggregation and the scale law of updates

Operationally, TEPO redistributes the sequence-level signal over tokens through an aggregation rule. In the paper’s notation, each token contribution takes the form

$$L_{i,t}(\theta)=w_i(\theta)\cdot A_{i,t}\cdot \mathrm{Mask}_{i,t},$$

and the full objective averages such contributions across valid tokens in the sampled group (Lin et al., 14 Apr 2026).

The paper’s backward expression makes the update scale explicit: $r(\boldsymbol y)$0 Accordingly, the magnitude of a token update is governed primarily by

$r(\boldsymbol y)$1

plus clipping and possible KL masking effects (Lin et al., 14 Apr 2026).

Each factor has a distinct role. The sequence-level likelihood weight $r(\boldsymbol y)$2 amplifies or attenuates all token updates in a sequence according to whether the new policy increases the likelihood of the whole sequence relative to the old policy. The length normalization $r(\boldsymbol y)$3 prevents long responses from acquiring disproportionately large updates merely because they contain more tokens. The batch or token normalization $r(\boldsymbol y)$4 stabilizes aggregate gradient magnitude across batches with different token counts. The advantage $r(\boldsymbol y)$5 sets the direction and scale of reinforcement. Clipping $r(\boldsymbol y)$6 bounds sequence-weight-induced update magnitude (Lin et al., 14 Apr 2026).

Relative to GRPO or DAPO, this makes updates more global, more normalized, more uniform within a sequence, and more stable. The comparison is explicit in the paper: GRPO uses token-level importance ratios,

$r(\boldsymbol y)$7

whereas TEPO uses the sequence-level geometric-mean ratio

$r(\boldsymbol y)$8

The paper interprets the latter as better aligned with whole-trajectory shift, less jagged than token-level importance sampling, and less sensitive to local token-probability noise (Lin et al., 14 Apr 2026).

The earlier Markov-Likelihood version made the same point in a slightly different form. It showed that when a group-derived advantage is effectively broadcast to tokens, the sequence-length factor in the backward pass approximately cancels the explicit sequence-length dependence of the repeated reward term. This suggests that TEPO is intended to make per-token scaling approximately length-neutral, rather than allowing update magnitude to drift with trajectory length (Lin et al., 10 Oct 2025).

4. Selective KL-Divergence masking as direct scale control

The later TEPO formulation adds a token-level KL-Divergence mask constraint: $r(\boldsymbol y)$9 KL is therefore applied only to tokens that simultaneously have positive advantage and decreasing entropy (Lin et al., 14 Apr 2026).

The paper motivates this through entropy-gradient and policy-gradient alignment. For optimal actions $G$0, it derives

$G$1

and therefore

$G$2

This is the regime in which policy improvement tends to make the policy more deterministic. The paper treats that tendency as potentially useful but dangerous if uncontrolled, and it is precisely this subset of tokens that the mask targets (Lin et al., 14 Apr 2026).

The KL term is $G$3, and the appendix gives the KL-regularized update

$G$4

or equivalently

$G$5

In this view, $G$6 is an inverse update-temperature or trust-region parameter: weaker effective KL control permits larger shifts (Lin et al., 14 Apr 2026).

This selective masking is TEPO’s most direct answer to scale control. If $G$7, a token receives no KL restraint. If $G$8, the token receives a subtractive KL penalty scaled by $G$9. TEPO therefore does not regularize all token updates equally. It restrains only the subset of tokens for which positive reward is already pushing the policy toward lower entropy, that is, the tokens most exposed to abrupt local policy shifts or entropy collapse (Lin et al., 14 Apr 2026).

This design also marks a methodological difference from the earlier Markov-Likelihood TEPO formulation, which argued that maximum-entropy and KL-Divergence regularization hurt performance in sparse-reward chain-of-thought settings. The later paper does not revert to undifferentiated KL; it introduces masked KL only on the vulnerable regime $$\pi_\theta(y\mid x)=\prod_t \pi_\theta(y_t\mid x,y_{<t}),$$0, thereby making regularization selective rather than global (Lin et al., 10 Oct 2025).

5. Empirical evidence on calibration, stability, and convergence

The empirical case for TEPO’s handling of group-level update scale is built around clipping behavior, reward progression, gradient norms, convergence speed, and ablations. Figure 1(a) reports a lower clip ratio for TEPO, which the paper interprets as effectively mitigating gradient bias. Because clipping is triggered when importance weights become too extreme, a lower clip ratio is presented as evidence that the sequence-level geometric-mean weight produces better-scaled updates than token-level importance sampling (Lin et al., 14 Apr 2026).

Figure 2(a) shows steadier and higher rewards across training steps, and Figure 2(b) reports consistently higher gradient norms for TEPO, which the authors interpret as “more active and effective parameter reasoning.” The paper itself cautions against over-interpretation: it does not provide a formal variance decomposition or a direct per-token gradient-scale histogram. Still, the combination of steadier rewards, lower clip ratio, and faster convergence is presented as evidence that TEPO preserves useful update magnitude while reducing harmful spikes (Lin et al., 14 Apr 2026).

The most explicit quantitative stability result is convergence speed. TEPO reaches peak performance in $$\pi_\theta(y\mid x)=\prod_t \pi_\theta(y_t\mid x,y_{<t}),$$1 steps, whereas GRPO or DAPO require about $$\pi_\theta(y\mid x)=\prod_t \pi_\theta(y_t\mid x,y_{<t}),$$2 steps for comparable performance, corresponding to roughly a $$\pi_\theta(y\mid x)=\prod_t \pi_\theta(y_t\mid x,y_{<t}),$$3 reduction in convergence time. Table 5 reports:

• GRPO/DAPO (72-step mean): $$\pi_\theta(y\mid x)=\prod_t \pi_\theta(y_t\mid x,y_{<t}),$$4
• GRPO/DAPO (132-step mean): $$\pi_\theta(y\mid x)=\prod_t \pi_\theta(y_t\mid x,y_{<t}),$$5
• TEPO (72-step mean): $$\pi_\theta(y\mid x)=\prod_t \pi_\theta(y_t\mid x,y_{<t}),$$6

The ablations sharpen the interpretation. Global KL-Divergence is described as highly fragile: $$\pi_\theta(y\mid x)=\prod_t \pi_\theta(y_t\mid x,y_{<t}),$$7 leads to model collapse in $$\pi_\theta(y\mid x)=\prod_t \pi_\theta(y_t\mid x,y_{<t}),$$8 steps, $$\pi_\theta(y\mid x)=\prod_t \pi_\theta(y_t\mid x,y_{<t}),$$9 gives a large performance drop, and smaller $$\log \pi_\theta(y\mid x)=\sum_t \log \pi_\theta(y_t\mid x,y_{<t}).$$0 remains inferior to TEPO-w/o-KL. By contrast, masking only the $$\log \pi_\theta(y\mid x)=\sum_t \log \pi_\theta(y_t\mid x,y_{<t}).$$1 regime gives the best result among tested masking scopes:

• no KL: $$\log \pi_\theta(y\mid x)=\sum_t \log \pi_\theta(y_t\mid x,y_{<t}).$$2
• KL on $$\log \pi_\theta(y\mid x)=\sum_t \log \pi_\theta(y_t\mid x,y_{<t}).$$3: $$\log \pi_\theta(y\mid x)=\sum_t \log \pi_\theta(y_t\mid x,y_{<t}).$$4
• KL on both mismatch conditions: $$\log \pi_\theta(y\mid x)=\sum_t \log \pi_\theta(y_t\mid x,y_{<t}).$$5
• KL only on $$\log \pi_\theta(y\mid x)=\sum_t \log \pi_\theta(y_t\mid x,y_{<t}).$$6: $$\log \pi_\theta(y\mid x)=\sum_t \log \pi_\theta(y_t\mid x,y_{<t}).$$7

Aggregation ablations are also directly relevant:

• GSPO sequence-mean token-mean: $$\log \pi_\theta(y\mid x)=\sum_t \log \pi_\theta(y_t\mid x,y_{<t}).$$8
• sequence-mean token-sum: $$\log \pi_\theta(y\mid x)=\sum_t \log \pi_\theta(y_t\mid x,y_{<t}).$$9
• TEPO design: $J(\theta)=\mathbb E_{\tau\sim p_\theta}[A(\tau)],$0 without KL, and $J(\theta)=\mathbb E_{\tau\sim p_\theta}[A(\tau)],$1 with KL

The paper interprets these results as evidence that the TEPO aggregation better balances token-level sparsity and sequence-level consistency, and that token-mean normalization prevents over-scaling from long responses or dense token counts (Lin et al., 14 Apr 2026).

The earlier TEPO paper reported the same qualitative pattern without masked KL. It described more stable and consistently higher rewards, sustained higher gradient norms, and improved benchmark accuracy, including average accuracy rising from $J(\theta)=\mathbb E_{\tau\sim p_\theta}[A(\tau)],$2 to $J(\theta)=\mathbb E_{\tau\sim p_\theta}[A(\tau)],$3 and MATH-500 increasing from $J(\theta)=\mathbb E_{\tau\sim p_\theta}[A(\tau)],$4 to $J(\theta)=\mathbb E_{\tau\sim p_\theta}[A(\tau)],$5. That version also argued that sequence-level Markov Likelihood reduces gradient bias and balances group-level credit with token-level updates via a sentence likelihood (Lin et al., 10 Oct 2025).

6. Relation to adjacent formulations and remaining limitations

Group-Level Update Scale is not confined to sequence-level chain-of-thought optimization. A closely related formulation appears in A$J(\theta)=\mathbb E_{\tau\sim p_\theta}[A(\tau)],$6TGPO, which studies multi-turn agentic RL and defines the scale problem in terms of turn heterogeneity rather than token sparsity. Its solution combines turn-group normalization within each $J(\theta)=\mathbb E_{\tau\sim p_\theta}[A(\tau)],$7 group, variance-rescaled discounted accumulation using $J(\theta)=\mathbb E_{\tau\sim p_\theta}[A(\tau)],$8, and adaptive turn-level clipping based on normalized Information Gain. In that setting, update magnitudes are calibrated within turn groups and then propagated in a way that remains comparable across trajectory depths (Chen et al., 7 May 2026).

The parallel is substantive. TEPO calibrates how a group-normalized sequence reward becomes token-level updates through sequence-level likelihood aggregation, length normalization, clipping, and masked KL. A$J(\theta)=\mathbb E_{\tau\sim p_\theta}[A(\tau)],$9TGPO calibrates how a turn-level intrinsic signal becomes PPO-style updates through same-depth normalization, square-root rescaling, and adaptive clipping. This suggests that “group-level update scale” has become a broader design question in RL for LLMs: the issue is not merely whether rewards are sparse, but how a coarse supervisory signal should be normalized, accumulated, and constrained before it is consumed by local policy updates (Chen et al., 7 May 2026).

At the same time, the limitations of TEPO are explicit. The paper does not present a richer token-specific credit decomposition beyond the sequence-level weight plus token-mean aggregation, and it does not introduce a learned token critic. The earlier version states this even more directly: TEPO improves update scale and stability, but still mostly distributes one sequence-level signal across all tokens rather than identifying which reasoning steps were causally important (Lin et al., 10 Oct 2025). This suggests that TEPO is best understood as a calibration mechanism for coarse reward propagation, not as a full solution to fine-grained token credit assignment.

In summary, the TEPO framework makes group-level updates more localized than global regularization, more stable than token-level importance sampling, and more selective than uniform token penalties. The effective scale of a token update is explicitly shaped by $$\nabla J(\theta) = \mathbb E_{\tau\sim p_{\theta_{\text{old}}}} \left[ \frac{p_\theta(\tau)}{p_{\theta_{\text{old}}}(\tau)} \nabla \log p_\theta(\tau)\, A_{\theta_{\text{old}}}(\tau) \right].$$0, $$\nabla J(\theta) = \mathbb E_{\tau\sim p_{\theta_{\text{old}}}} \left[ \frac{p_\theta(\tau)}{p_{\theta_{\text{old}}}(\tau)} \nabla \log p_\theta(\tau)\, A_{\theta_{\text{old}}}(\tau) \right].$$1, $$\nabla J(\theta) = \mathbb E_{\tau\sim p_{\theta_{\text{old}}}} \left[ \frac{p_\theta(\tau)}{p_{\theta_{\text{old}}}(\tau)} \nabla \log p_\theta(\tau)\, A_{\theta_{\text{old}}}(\tau) \right].$$2, batch normalization, clipping, and, in the later formulation, masked KL. Within this line of work, Group-Level Update Scale therefore refers to the principled control of how a group-relative reward is translated into usable token-level optimization under sparse-reward chain-of-thought training (Lin et al., 14 Apr 2026).