SofT-GRPO: Variants in Policy Optimization

• SofT-GRPO is a family of GRPO-derived methods that modify training pipelines using soft gating, soft-thinking, or a two-stage SFT→GRPO adaptation, each altering clipping rules and reward signals.
• The approaches improve policy optimization by addressing off-policy adjustments and behavioral correction, as evidenced by reduced WER and enhanced alignment in ASR applications.
• Variants explore continuous advantage aggregation, sequence-coherent soft gates, and pairwise preference objectives, offering flexible trade-offs between bias, variance, and computational efficiency.

SofT-GRPO is not a single canonical algorithm in the current arXiv literature, but a name applied to several GRPO-derived procedures that retain some aspect of Group Relative Policy Optimization while changing the training pipeline, the clipping mechanism, the action representation, or the preference signal. In one usage, it denotes a two-stage automatic speech recognition adaptation pipeline consisting of supervised fine-tuning followed by GRPO with a word-error-rate reward (Kumar et al., 9 Jul 2026). In other usages, it denotes soft-gated GRPO objectives that replace PPO-style hard clipping with smooth temperature-controlled attenuation (Gao et al., 25 Nov 2025, Denisov et al., 22 Feb 2026), a sequence-coherent soft variant based on geometric aggregation of token gates (Glazyrina et al., 22 Feb 2026), a Gumbel-reparameterized method for soft-thinking trajectories (Zheng et al., 9 Nov 2025), or a pairwise-preference variant that substitutes binary pair rewards for group-normalized scalar rewards (Yu, 7 May 2026). The shared substrate is GRPO’s group-relative normalization and critic-free policy optimization.

1. Nomenclature and scope

The term is used in multiple, non-identical senses. The main usages documented in the cited papers are summarized below.

Usage of “SofT-GRPO”	Defining modification	Paper
SFT then GRPO for ASR	two-stage adaptation with a WER-based reward	(Kumar et al., 9 Jul 2026)
Soft-gated GRPO / SAPO	hard token-level clipping replaced by a smooth, temperature-controlled gate	(Gao et al., 25 Nov 2025, Denisov et al., 22 Feb 2026)
Soft Sequence Policy Optimization	soft token gates aggregated geometrically into a sequence-level weight	(Glazyrina et al., 22 Feb 2026)
Soft-thinking policy optimization	Gumbel noise and Gumbel-Softmax for continuous soft-thinking trajectories	(Zheng et al., 9 Nov 2025)
Soft-Pair-GRPO	binary pairwise preference rewards inside the GRPO surrogate	(Yu, 7 May 2026)

This terminological spread matters because the underlying design choices differ substantially. Some formulations alter only the trust-region mechanism, as in SAPO and related soft-gated objectives. Others alter the learning signal, as in Soft-Pair-GRPO. Still others alter the latent representation being optimized, as in soft-thinking, or use “SofT-GRPO” simply as the name of an SFT→GRPO training schedule in ASR (Kumar et al., 9 Jul 2026).

A common misconception is that the phrase always refers to “soft clipping.” The literature does not support that reading. In the ASR paper, “SofT-GRPO” refers specifically to “the two-stage adaptation pipeline the paper studies: first supervised fine-tuning (SFT), then Group Relative Policy Optimization (GRPO) on the same model, with a WER-based reward” (Kumar et al., 9 Jul 2026).

2. GRPO as the common substrate

Across these variants, the baseline substrate is Group Relative Policy Optimization. GRPO samples a group of responses or hypotheses for the same input, computes a scalar reward for each, and forms a group-relative advantage by standardizing rewards within the group rather than learning a separate value function. In the ASR formulation, for an input utterance $x$ and sampled hypotheses $\{o_i\}_{i=1}^G$, the advantage is

$$A_i = \frac{r_i - \mathrm{mean}(\{r_1,\ldots,r_G\})}{\mathrm{std}(\{r_1,\ldots,r_G\}) + \epsilon_{\mathrm{std}}},$$

which “replaces a learned baseline (‘critic’) with the group mean (and scale), yielding a critic-free advantage” (Kumar et al., 9 Jul 2026).

The standard surrogate is PPO-like and operates at token granularity while reusing a sequence-level advantage. In one common form,

$$\mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}\left[ \frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \min\!\left( r_{i,t}(\theta)\,\widehat{A}_i,\; \mathrm{clip}(r_{i,t}(\theta),1-\varepsilon,1+\varepsilon)\,\widehat{A}_i \right) \right],$$

with token-level importance ratio

$$r_{i,t}(\theta)= \frac{\pi_\theta(y_{i,t}\mid q,y_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(y_{i,t}\mid q,y_{i,<t})}.$$

This formulation underlies SAPO, SSPO, and the pairwise variants even when the clipping, sequence weighting, or reward definition is changed (Gao et al., 25 Nov 2025, Glazyrina et al., 22 Feb 2026).

Theoretical analysis of GRPO’s stationary objective shows that its aggregation is not standard logarithmic pooling. Under stationary analysis, the reference-policy penalty behaves “like a reverse KL divergence,” and the fixed-point solution rescales $\pi_{\mathrm{ref}}$ by a rational factor rather than by an exponential tilt (Vojnovic et al., 25 Feb 2025). That distinction is consequential because it separates GRPO-style optimization from the direct-KL aggregation commonly associated with RLHF. The same analysis also shows that for groups of size two, normalized advantages reduce to pairwise comparison signs, which helps explain why pairwise reformulations can remain close to GRPO’s descent direction (Vojnovic et al., 25 Feb 2025).

3. SofT-GRPO as SFT→GRPO for synthetic-speech ASR

In "When Synthetic Speech Is All You Have: Better Call GRPO" (Kumar et al., 9 Jul 2026), SofT-GRPO denotes a specific two-stage adaptation procedure for privacy-constrained ASR in banking-domain customer–agent telephone calls. The motivation is that “real speech is costly and legally constrained to collect,” while synthetic text-to-speech remains acoustically mismatched with real recordings. The paper therefore studies adaptation using only synthetic speech, or small mixtures with real data, under GDPR and EU AI Act constraints (Kumar et al., 9 Jul 2026).

The architecture consists of a frozen WavLM-Large encoder, a speech projector with two linear layers and ReLU that downsamples the acoustic sequence by $5\times$, and a frozen Llama-3.2-1B-Instruct decoder. Only the projector and LoRA adapters are trained, with LoRA rank $r=16$ and scaling $\alpha=32$ (Kumar et al., 9 Jul 2026). The SFT stage uses teacher forcing with

$$L_{\mathrm{SFT}}(\theta)=- \mathbb{E}_{(x,y^\star)} \left[\sum_{t=1}^{T} \log \pi_\theta(y^\star_t\mid x,y^\star_{<t}) \right],$$

optimized by AdamW with learning rate $\{o_i\}_{i=1}^G$0, batch size $\{o_i\}_{i=1}^G$1, $\{o_i\}_{i=1}^G$2 warmup steps, and $\{o_i\}_{i=1}^G$3 epochs. GRPO then starts from the SFT checkpoint, fixes that checkpoint as $\{o_i\}_{i=1}^G$4, and uses AdamW at learning rate $\{o_i\}_{i=1}^G$5 with group size $\{o_i\}_{i=1}^G$6, temperature $\{o_i\}_{i=1}^G$7, PPO clip $\{o_i\}_{i=1}^G$8, KL coefficient $\{o_i\}_{i=1}^G$9, gradient norm clip $$A_i = \frac{r_i - \mathrm{mean}(\{r_1,\ldots,r_G\})}{\mathrm{std}(\{r_1,\ldots,r_G\}) + \epsilon_{\mathrm{std}}},$$0, rollout limit $$A_i = \frac{r_i - \mathrm{mean}(\{r_1,\ldots,r_G\})}{\mathrm{std}(\{r_1,\ldots,r_G\}) + \epsilon_{\mathrm{std}}},$$1 new tokens, and $$A_i = \frac{r_i - \mathrm{mean}(\{r_1,\ldots,r_G\})}{\mathrm{std}(\{r_1,\ldots,r_G\}) + \epsilon_{\mathrm{std}}},$$2 epochs on a single NVIDIA H100 80 GB GPU in BF16 (Kumar et al., 9 Jul 2026).

The reward directly targets transcription quality:

$$A_i = \frac{r_i - \mathrm{mean}(\{r_1,\ldots,r_G\})}{\mathrm{std}(\{r_1,\ldots,r_G\}) + \epsilon_{\mathrm{std}}},$$3

Alternative rewards based on CER and length consistency are evaluated, but “the best WER outcome comes from $$A_i = \frac{r_i - \mathrm{mean}(\{r_1,\ldots,r_G\})}{\mathrm{std}(\{r_1,\ldots,r_G\}) + \epsilon_{\mathrm{std}}},$$4 alone” (Kumar et al., 9 Jul 2026).

Quantitatively, on the 6.55 h banking-domain test split, synthetic-only SFT yields WER 36.71% and CER 25.06%, while GRPO on the same synthetic data yields WER 22.09% and CER 15.89%. The combined SFT→GRPO pipeline, labeled SofT-GRPO, reaches WER 20.21% and CER 14.68%, corresponding to “≈45% relative improvement over synthetic-only SFT” (Kumar et al., 9 Jul 2026). On real-data adaptation with 54 h of speech, SFT gives WER 10.27%, GRPO from LS960 gives 10.78%, and SFT→GRPO gives 9.49% (Kumar et al., 9 Jul 2026).

The paper attributes the gain primarily to behavioral correction rather than representational rewriting. Synthetic-only SFT exhibits a heavy insertion tail, including short utterances followed by many spurious tokens, which the authors interpret as EOS failure and language-model-driven hallucination. GRPO largely removes this tail, lowers Expected Calibration Error across all length bins, sharpens attention to audio tokens, preserves cleaner monotonic alignments, and leaves early-to-middle representations close to the base model under layer-wise linear CKA. SFT→GRPO remains almost identical to SFT at the representation level, suggesting that GRPO acts as “a targeted policy correction—improving behavior (stopping, insertions) without substantially rewriting representations” (Kumar et al., 9 Jul 2026).

4. Soft-gated GRPO: SAPO, gate functions, and sequence coherence

In a second, widely used sense, SofT-GRPO refers to replacing GRPO’s hard clipping with a smooth gate. "Soft Adaptive Policy Optimization" (Gao et al., 25 Nov 2025) formalizes this as SAPO. The objective keeps GRPO’s group-normalized advantages but replaces the hard clipped term with a continuously differentiable gate:

$$A_i = \frac{r_i - \mathrm{mean}(\{r_1,\ldots,r_G\})}{\mathrm{std}(\{r_1,\ldots,r_G\}) + \epsilon_{\mathrm{std}}},$$5

Its gradient can be written in SofT-GRPO form,

$$A_i = \frac{r_i - \mathrm{mean}(\{r_1,\ldots,r_G\})}{\mathrm{std}(\{r_1,\ldots,r_G\}) + \epsilon_{\mathrm{std}}},$$6

where the weight peaks at $$A_i = \frac{r_i - \mathrm{mean}(\{r_1,\ldots,r_G\})}{\mathrm{std}(\{r_1,\ldots,r_G\}) + \epsilon_{\mathrm{std}}},$$7 and decays smoothly as the ratio moves off-policy (Gao et al., 25 Nov 2025). This produces a “continuous trust region” rather than a hard band.

SAPO is presented as both token-adaptive and sequence-coherent. Under the assumptions of small policy steps and low intra-sequence dispersion of log-ratios, the average token gate concentrates to the sequence-level soft gate

$$A_i = \frac{r_i - \mathrm{mean}(\{r_1,\ldots,r_G\})}{\mathrm{std}(\{r_1,\ldots,r_G\}) + \epsilon_{\mathrm{std}}},$$8

with $$A_i = \frac{r_i - \mathrm{mean}(\{r_1,\ldots,r_G\})}{\mathrm{std}(\{r_1,\ldots,r_G\}) + \epsilon_{\mathrm{std}}},$$9 the length-normalized geometric mean of token ratios (Gao et al., 25 Nov 2025). When those assumptions fail, especially when a sequence contains a few highly off-policy tokens, SAPO suppresses only the outliers rather than the whole sequence. This is the primary contrast drawn against GSPO, which can suppress an entire sequence when the sequence-level weight leaves the clipping band (Gao et al., 25 Nov 2025).

The same line of work also studies admissible smooth gates more generally. "Smooth Gate Functions for Soft Advantage Policy Optimization" (Denisov et al., 22 Feb 2026) requires continuous differentiability, unit derivative at $$\mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}\left[ \frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \min\!\left( r_{i,t}(\theta)\,\widehat{A}_i,\; \mathrm{clip}(r_{i,t}(\theta),1-\varepsilon,1+\varepsilon)\,\widehat{A}_i \right) \right],$$0, monotone decay of the derivative away from $$\mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}\left[ \frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \min\!\left( r_{i,t}(\theta)\,\widehat{A}_i,\; \mathrm{clip}(r_{i,t}(\theta),1-\varepsilon,1+\varepsilon)\,\widehat{A}_i \right) \right],$$1, and vanishing influence of extreme ratios in the sense that $$\mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}\left[ \frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \min\!\left( r_{i,t}(\theta)\,\widehat{A}_i,\; \mathrm{clip}(r_{i,t}(\theta),1-\varepsilon,1+\varepsilon)\,\widehat{A}_i \right) \right],$$2. The paper evaluates sigmoid, error-function, arctangent, and softsign gate families, associating Gaussian attenuation with more localized updates and stronger suppression of heavy-tailed outliers, and polynomial tails with broader exploration (Denisov et al., 22 Feb 2026).

A related formulation, "Soft Sequence Policy Optimization: Bridging GMPO and SAPO" (Glazyrina et al., 22 Feb 2026), moves the soft gate into a sequence-coherent geometric aggregation:

$$\mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}\left[ \frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \min\!\left( r_{i,t}(\theta)\,\widehat{A}_i,\; \mathrm{clip}(r_{i,t}(\theta),1-\varepsilon,1+\varepsilon)\,\widehat{A}_i \right) \right],$$3

and then optimizes

$$\mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}\left[ \frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \min\!\left( r_{i,t}(\theta)\,\widehat{A}_i,\; \mathrm{clip}(r_{i,t}(\theta),1-\varepsilon,1+\varepsilon)\,\widehat{A}_i \right) \right],$$4

SSPO is explicitly described as a “SofT-GRPO” variant because it is group-relative, off-policy, soft, and sequence-coherent (Glazyrina et al., 22 Feb 2026). Taken together, these papers establish a technical family in which “soft” means smooth attenuation of importance ratios rather than a change in the task domain.

5. Soft-thinking and pairwise reinterpretations

A third usage appears in "SofT-GRPO: Surpassing Discrete-Token LLM Reinforcement Learning via Gumbel-Reparameterized Soft-Thinking Policy Optimization" (Zheng et al., 9 Nov 2025). Here the defining change is not the clipping rule but the reasoning representation. Soft-thinking replaces each discrete reasoning token with a continuous vector

$$\mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}\left[ \frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \min\!\left( r_{i,t}(\theta)\,\widehat{A}_i,\; \mathrm{clip}(r_{i,t}(\theta),1-\varepsilon,1+\varepsilon)\,\widehat{A}_i \right) \right],$$5

where $$\mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}\left[ \frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \min\!\left( r_{i,t}(\theta)\,\widehat{A}_i,\; \mathrm{clip}(r_{i,t}(\theta),1-\varepsilon,1+\varepsilon)\,\widehat{A}_i \right) \right],$$6 is a Gumbel-Softmax probability vector over the vocabulary. Gumbel noise is injected into logits, and the policy is optimized with GRPO-style clipped ratios plus KL regularization, but for soft-thinking steps the ratios are computed from reparameterized log-probabilities of the continuous soft token under the current and old policies (Zheng et al., 9 Nov 2025). The method uses decoding temperature $$\mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}\left[ \frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \min\!\left( r_{i,t}(\theta)\,\widehat{A}_i,\; \mathrm{clip}(r_{i,t}(\theta),1-\varepsilon,1+\varepsilon)\,\widehat{A}_i \right) \right],$$7, Gumbel-Softmax temperature $$\mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}\left[ \frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \min\!\left( r_{i,t}(\theta)\,\widehat{A}_i,\; \mathrm{clip}(r_{i,t}(\theta),1-\varepsilon,1+\varepsilon)\,\widehat{A}_i \right) \right],$$8, top-$$\mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}\left[ \frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \min\!\left( r_{i,t}(\theta)\,\widehat{A}_i,\; \mathrm{clip}(r_{i,t}(\theta),1-\varepsilon,1+\varepsilon)\,\widehat{A}_i \right) \right],$$9, and top-$$r_{i,t}(\theta)= \frac{\pi_\theta(y_{i,t}\mid q,y_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(y_{i,t}\mid q,y_{i,<t})}.$$0, and reports average improvements over discrete-token GRPO of +0.13% on Pass@1, +1.80% on Pass@16, and +2.19% on Pass@32 across 1.5B–7B models (Zheng et al., 9 Nov 2025).

A fourth usage is Soft-Pair-GRPO in "A Unified Pair-GRPO Family" (Yu, 7 May 2026). This method retains GRPO’s clipped surrogate and KL regularization but replaces group-normalized scalar rewards with binary pairwise preference rewards:

$$r_{i,t}(\theta)= \frac{\pi_\theta(y_{i,t}\mid q,y_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(y_{i,t}\mid q,y_{i,<t})}.$$1

The paper proves a gradient equivalence theorem stating that, under first-order Taylor expansion around the current policy, the Soft-Pair-GRPO gradient is a positive scalar multiple of the GRPO gradient. This is used to explain why the method can discard continuous reward magnitudes while preserving the local descent direction (Yu, 7 May 2026).

These reinterpretations are connected by theory. "What is the Alignment Objective of GRPO?" (Vojnovic et al., 25 Feb 2025) shows that GRPO’s shift-and-scale normalization makes advantages invariant to affine reward transformations and that, for groups of size two, the normalized rewards reduce to pairwise signs. That result does not itself define SofT-GRPO, but it provides a theoretical bridge to pairwise and temperature-scaled interpretations. The paper further notes that “the paper does not explicitly mention ‘SofT-GRPO’,” and treats soft variants as possible modifications built within the same stationary framework (Vojnovic et al., 25 Feb 2025).

6. Limitations, caveats, and unresolved questions

The literature surrounding SofT-GRPO is heterogeneous, and the limits of each usage are correspondingly different. In the ASR setting, results are specific to banking-domain conversational telephony. Transfer to other domains “may require re-tuning $$r_{i,t}(\theta)= \frac{\pi_\theta(y_{i,t}\mid q,y_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(y_{i,t}\mid q,y_{i,<t})}.$$2, $$r_{i,t}(\theta)= \frac{\pi_\theta(y_{i,t}\mid q,y_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(y_{i,t}\mid q,y_{i,<t})}.$$3, $$r_{i,t}(\theta)= \frac{\pi_\theta(y_{i,t}\mid q,y_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(y_{i,t}\mid q,y_{i,<t})}.$$4, and the mix of synthetic/real,” and robustness across TTS voices, noise conditions, and richer room or channel simulations remains “promising but not fully explored” (Kumar et al., 9 Jul 2026). The same paper also reports non-monotonic scaling with synthetic data: with fixed 1 h real speech, performance is best around 25 h synthetic and degrades at 54 h, indicating that “too much synthetic can drown out limited real supervision” (Kumar et al., 9 Jul 2026).

For soft-gated objectives, the central trade-off is bias versus variance. SAPO and SSPO replace hard truncation with smooth attenuation, which improves stability but still introduces bias under large policy drift; both papers recommend the usual tuning of KL regularization, batch structure, and temperature parameters, and SSPO explicitly notes sensitivity to $$r_{i,t}(\theta)= \frac{\pi_\theta(y_{i,t}\mid q,y_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(y_{i,t}\mid q,y_{i,<t})}.$$5, $$r_{i,t}(\theta)= \frac{\pi_\theta(y_{i,t}\mid q,y_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(y_{i,t}\mid q,y_{i,<t})}.$$6, and $$r_{i,t}(\theta)= \frac{\pi_\theta(y_{i,t}\mid q,y_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(y_{i,t}\mid q,y_{i,<t})}.$$7 (Gao et al., 25 Nov 2025, Glazyrina et al., 22 Feb 2026). The smooth-gate study emphasizes that its current results are structurally focused and that “a comprehensive quantitative comparison is deferred to a future revision” (Denisov et al., 22 Feb 2026).

Soft-thinking SofT-GRPO introduces additional sensitivity. The paper identifies $$r_{i,t}(\theta)= \frac{\pi_\theta(y_{i,t}\mid q,y_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(y_{i,t}\mid q,y_{i,<t})}.$$8 and top-$$r_{i,t}(\theta)= \frac{\pi_\theta(y_{i,t}\mid q,y_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(y_{i,t}\mid q,y_{i,<t})}.$$9 as critical; increasing $\pi_{\mathrm{ref}}$0 to 0.25 or top-$\pi_{\mathrm{ref}}$1 to 1.0 causes training collapse, indicated by elevated KL divergence to the reference model (Zheng et al., 9 Nov 2025). It also notes extra storage and reconstruction overhead for $\pi_{\mathrm{ref}}$2 and $\pi_{\mathrm{ref}}$3, and the possibility of mismatch between soft-thinking states and final discrete outputs (Zheng et al., 9 Nov 2025).

Pairwise formulations improve interpretability but alter the supervision geometry. Soft-Pair-GRPO is presented as a minimal change that improves stability, whereas Hard-Pair-GRPO adds target-distribution construction and constrained KL fitting at “the cost of slightly higher complexity” (Yu, 7 May 2026). More broadly, the current literature does not support a single operational definition of SofT-GRPO. The most precise reading is therefore contextual: in ASR it is an SFT→GRPO pipeline; in policy-optimization theory it usually denotes smooth gating of GRPO-style ratios; in reasoning it may denote soft-thinking reparameterization; and in preference learning it may denote a binary pairwise reward reformulation.