Supergroup Relative Policy Optimization (SGRPO)

• SGRPO is a reinforcement learning framework that optimizes both rollout utility and set-level diversity by sampling multiple candidate groups under the same condition.
• It employs a supergroup structure with leave-one-out credit assignment to redistribute diversity rewards, ensuring effective utility–diversity trade-offs.
• Empirical results show that SGRPO extends the utility–diversity Pareto frontier, outperforming standard GRPO in tasks like de novo molecule and protein design.

Searching arXiv for the cited papers to ground the article in published work. Supergroup Relative Policy Optimization (SGRPO) most specifically denotes a GRPO-style reinforcement learning framework for biomolecular generation in which multiple candidate sets are sampled under the same condition, their set-level diversity is compared, and the resulting diversity signal is redistributed to individual rollouts before a PPO-style policy update. Its purpose is to optimize rollout-level utility and set-level diversity jointly, thereby expanding the utility–diversity Pareto frontier rather than improving utility at the cost of mode concentration (Ye et al., 9 May 2026). The acronym is nevertheless non-unique in recent literature: one reasoning paper uses “SGRPO” only as shorthand for Scaffolded Group Relative Policy Optimization (Scaf-GRPO), while a theoretical analysis of GRPO uses “Supergroup/Supervised GRPO” as a natural variant within a U-statistic framework rather than as the biomolecular method itself (Zhang et al., 22 Oct 2025, Zhou et al., 1 Mar 2026).

1. Terminology and conceptual scope

Recent usage makes “SGRPO” a polysemous acronym. In biomolecular post-training, it refers to Supergroup Relative Policy Optimization, whose defining operation is same-condition comparison among multiple candidate sets and direct optimization of set-level diversity (Ye et al., 9 May 2026). In reasoning-oriented RL from verifiable rewards, “SGRPO” may instead denote Scaffolded Group Relative Policy Optimization, abbreviated in the corresponding paper as Scaf-GRPO, where the central mechanism is tiered hint injection to recover gradients on persistently zero-reward prompts (Zhang et al., 22 Oct 2025). A separate theoretical paper does not present a standalone algorithm under that name, but formalizes a natural “Supervised/ Supergroup GRPO” variant by replacing terminal rewards with bounded supervised scores or by computing baselines over a larger pool, showing that the same U-statistic machinery continues to apply under suitable assumptions (Zhou et al., 1 Mar 2026).

Usage of “SGRPO”	Expansion	Context
SGRPO	Supergroup Relative Policy Optimization	Biomolecular utility–diversity optimization
SGRPO / Scaf-GRPO	Scaffolded Group Relative Policy Optimization	RLVR for hard reasoning problems
SGRPO	Supervised/Supergroup GRPO	Theoretical GRPO variant in U-statistic analysis

For the biomolecular method, the term “supergroup” is literal: for each condition, the policy samples multiple groups, not merely a single enlarged group. Diversity is then computed at the set level within each group, compared across groups under the same condition, and converted into rollout-level learning signals. This distinguishes the method from ordinary large-group GRPO, where group size primarily affects normalization or estimator variance rather than introducing a new reward object (Ye et al., 9 May 2026).

2. Formal framework of Supergroup Relative Policy Optimization

Let $\pi_\theta(x \mid C)$ be a conditional biomolecular generator that produces candidate $x$ under condition $C$. Each rollout receives a scalar utility $u(x, C)$, while diversity is defined over a set of candidates generated under the same condition. For a group $G = \{x_1, \dots, x_K\}$, set-level diversity is denoted $D(G)$. The method evaluates a generator through the utility–diversity frontier

$$\mathcal{P}(\theta) = \{(U(\theta,a), V(\theta,a)) : a \in \mathcal{A}\},$$

with decoding setting $a$ varying over a permissible sweep. The training objective is to push this frontier outward rather than to maximize utility alone (Ye et al., 9 May 2026).

For each condition $C$, SGRPO samples a supergroup

$$\mathcal{S}(C) = \{G_1, \dots, G_M\}, \quad G_m = \{x_{m,1}, \dots, x_{m,K}\}, \quad x_{m,i} \sim \pi_{\theta_{\rm old}}(\cdot \mid C),$$

so the same condition yields $x$0 groups and $x$1 total rollouts. Diversity is then assessed per group. With similarity function $x$2 and dissimilarity $x$3, the general normalized pairwise diversity is

$x$4

For molecules, $x$5 is Tanimoto similarity over Morgan fingerprints; for proteins, it is normalized Levenshtein similarity (Ye et al., 9 May 2026).

Each group obtains a diversity reward $x$6. Within the supergroup, this is centered by a same-condition leave-one-out comparison,

$x$7

This construction is explicitly same-condition: the paper motivates it as avoiding confounding condition difficulty with policy quality (Ye et al., 9 May 2026).

Credit assignment to individual rollouts is handled by leave-one-out diversity contributions,

$x$8

which are standardized within group,

$x$9

The paper then defines sign-aware softmax weights

$C$0

and uses them to redistribute the group diversity reward to rollouts:

$C$1

If a group is more diverse than average, extra positive signal is concentrated on high-contribution rollouts; if less diverse, negative signal is concentrated on low-contribution rollouts. The redistribution preserves the group reward average, since $C$2 (Ye et al., 9 May 2026).

The final composed reward mixes rollout utility and redistributed diversity:

$C$3

where $C$4 sets the utility–diversity trade-off. SGRPO then forms a supergroup-relative advantage over all rollouts in the same-condition supergroup,

$C$5

and optimizes a PPO-style clipped objective with KL regularization to a reference policy $C$6:

$C$7

The method is therefore “GRPO-style” in optimization form, but its reward construction is fundamentally set-relative rather than purely rollout-relative (Ye et al., 9 May 2026).

3. Instantiations, tasks, and implementation regime

SGRPO is explicitly model-agnostic. For autoregressive generators such as ProGen2-small, the policy ratio factorizes over tokens, and SGRPO plugs its composed reward and supergroup-relative advantage into a standard GRPO training loop. For discrete diffusion generators such as GenMol and GenMol-P, the method is instantiated with Coupled-GRPO, which scores completed samples with a coupled-mask log-probability proxy $C$8 built from timestep pairs $C$9 satisfying $u(x, C)$0 and complementary completion masks. In both cases, the diversity machinery is unchanged; only the ratio computation differs (Ye et al., 9 May 2026).

The algorithmic structure is straightforward. At each training iteration, the current parameters are copied to $u(x, C)$1. For each condition in the batch, the policy samples $u(x, C)$2 groups of size $u(x, C)$3, computes rollout utilities, computes diversity for each group, centers diversity within the supergroup, derives leave-one-out contributions, redistributes diversity to rollouts, mixes utility and diversity using $u(x, C)$4, forms the supergroup-relative advantage, and then performs a clipped PPO update with KL regularization. The paper emphasizes that same-condition grouping is essential because diversity comparisons across heterogeneous conditions would entangle reward quality with prompt or conditioning difficulty (Ye et al., 9 May 2026).

The framework is instantiated on three task families. In de novo small-molecule design, GenMol is trained as a discrete diffusion LLM over SAFE strings, with utility

$u(x, C)$5

and internal Tanimoto-based diversity. In pocket-based small-molecule design, GenMol-P adds a pocket prefix from a frozen ESM-IF1 plus MLP projector and uses

$u(x, C)$6

In de novo protein design, ProGen2-small is optimized with

$u(x, C)$7

where foldability uses ESMFold mean pLDDT/100, naturalness uses ESM2-based per-token log-likelihood calibration, stability uses TemBERTure, and developability combines Protein-Sol solubility with liability filters (Ye et al., 9 May 2026).

The reported training settings are matched to baseline rollout budgets. For de novo molecules, GRPO uses 512 rollouts per prompt and SGRPO uses $u(x, C)$8 groups with $u(x, C)$9 rollouts per group, with $G = \{x_1, \dots, x_K\}$0 and $G = \{x_1, \dots, x_K\}$1. For pocket-based molecules, SGRPO uses $G = \{x_1, \dots, x_K\}$2 and $G = \{x_1, \dots, x_K\}$3 with $G = \{x_1, \dots, x_K\}$4. For proteins, SGRPO uses $G = \{x_1, \dots, x_K\}$5 and $G = \{x_1, \dots, x_K\}$6 with $G = \{x_1, \dots, x_K\}$7. Across these setups, the configurable primitives are the supergroup structure $G = \{x_1, \dots, x_K\}$8, the diversity metric $G = \{x_1, \dots, x_K\}$9, the diversity weight $D(G)$0, the contribution temperature $D(G)$1, the LOO stabilizer $D(G)$2, and standard PPO/GRPO controls such as clipping and KL regularization (Ye et al., 9 May 2026).

4. Empirical performance on utility–diversity frontiers

The evaluation protocol constructs utility–diversity frontiers by sweeping decoding settings and extracting the non-dominated set. Frontier quality is quantified by hypervolume (HV, higher is better), distance to the ideal point (DIP, lower is better), and the $D(G)$3 indicator (lower is better). Reported values are mean $D(G)$4 95% CI over five sweeps (Ye et al., 9 May 2026).

Task	SGRPO frontier metrics	Comparison stated in the paper
De novo small molecules	HV $D(G)$5, DIP $D(G)$6, R2 $D(G)$7	Best vs Original, GRPO, Mem-GRPO
Pocket-based small molecules	HV $D(G)$8, DIP $D(G)$9, R2 $$\mathcal{P}(\theta) = \{(U(\theta,a), V(\theta,a)) : a \in \mathcal{A}\},$$0	Best vs Original and GRPO
De novo proteins	HV $$\mathcal{P}(\theta) = \{(U(\theta,a), V(\theta,a)) : a \in \mathcal{A}\},$$1, DIP $$\mathcal{P}(\theta) = \{(U(\theta,a), V(\theta,a)) : a \in \mathcal{A}\},$$2, R2 $$\mathcal{P}(\theta) = \{(U(\theta,a), V(\theta,a)) : a \in \mathcal{A}\},$$3	Best vs Original, GRPO, Mem-GRPO

In de novo small-molecule design, the baseline pretrained generator attains HV $$\mathcal{P}(\theta) = \{(U(\theta,a), V(\theta,a)) : a \in \mathcal{A}\},$$4, GRPO attains $$\mathcal{P}(\theta) = \{(U(\theta,a), V(\theta,a)) : a \in \mathcal{A}\},$$5, memory-assisted GRPO attains $$\mathcal{P}(\theta) = \{(U(\theta,a), V(\theta,a)) : a \in \mathcal{A}\},$$6, and SGRPO attains $$\mathcal{P}(\theta) = \{(U(\theta,a), V(\theta,a)) : a \in \mathcal{A}\},$$7. The corresponding DIP values are $$\mathcal{P}(\theta) = \{(U(\theta,a), V(\theta,a)) : a \in \mathcal{A}\},$$8, $$\mathcal{P}(\theta) = \{(U(\theta,a), V(\theta,a)) : a \in \mathcal{A}\},$$9, $a$0, and $a$1. The paper’s qualitative interpretation is that Coupled-SGRPO produces a slower diversity drop as decoding pushes utility higher, extending the frontier further right without severe downward bend (Ye et al., 9 May 2026).

In pocket-conditioned molecular design, the contrast is stronger. The pretrained model records HV $a$2, but GRPO falls to $a$3, whereas SGRPO rises to $a$4. DIP changes from $a$5 for the pretrained generator to $a$6 for GRPO and $a$7 for SGRPO. The paper attributes this to the tendency of docking-driven utility optimization to collapse generation to a few chemotypes; SGRPO preserves markedly higher within-pocket diversity at comparable utility, especially in high-utility regimes (Ye et al., 9 May 2026).

In de novo protein design, SGRPO likewise dominates frontier-level metrics. HV increases to $a$8, compared with $a$9 for the pretrained model, $C$0 for GRPO, and $C$1 for memory-assisted GRPO. DIP becomes $C$2, compared with $C$3, $C$4, and $C$5. Distribution analyses using UMAP of normalized Levenshtein distances show GRPO and memory-assisted GRPO concentrating into single or distant narrow clusters, whereas SGRPO maintains multiple clusters over training, which the paper uses to explain better high-utility performance with preserved diversity (Ye et al., 9 May 2026).

Ablation results isolate the mechanism. Removing diversity and reverting to coupled-GRPO gives the innermost utility–diversity curve. Adding supergroup diversity without leave-one-out credit pushes the curve outward, and full SGRPO further dominates, indicating that LOO redistribution is important for correct credit assignment. On GenMol, with fixed total supergroup size and group size $C$6, $C$7 recovers GRPO, SGRPO already outperforms GRPO at $C$8, and larger $C$9 improves HV further. Varying $$\mathcal{S}(C) = \{G_1, \dots, G_M\}, \quad G_m = \{x_{m,1}, \dots, x_{m,K}\}, \quad x_{m,i} \sim \pi_{\theta_{\rm old}}(\cdot \mid C),$$0 shows that all nonzero values improve HV over GRPO, with a peak at $$\mathcal{S}(C) = \{G_1, \dots, G_M\}, \quad G_m = \{x_{m,1}, \dots, x_{m,K}\}, \quad x_{m,i} \sim \pi_{\theta_{\rm old}}(\cdot \mid C),$$1 (Ye et al., 9 May 2026).

5. Relation to GRPO theory and group-size debates

SGRPO inherits the optimization lineage of GRPO, but its use of groups serves a different purpose from the large-group normalization commonly discussed in reasoning-oriented RL. In standard GRPO for RLVR, a prompt-specific group is used to compute a centered or standardized advantage. A 2025 analysis reframes GRPO as a contrastive method and shows that, under binary verifiable rewards and ignoring clipping, the learning signal can be written as a positive-versus-negative separation objective. On that basis, the paper studies the minimal pairwise case and reports that 2-GRPO matches 16-GRPO across five benchmarks for all base models, despite using only 1/8 of total rollouts (0.15M vs. 1.2M) and reducing wall-clock training time by over 70% (Wu et al., 1 Oct 2025).

That result directly challenges the assumption that larger groups are intrinsically necessary for effective GRPO training in RLVR. The same paper argues that 2-GRPO produces unbiased advantage estimates up to a constant scaling factor and that, under fixed rollout budget, gradient variance is controlled by total batch size $$\mathcal{S}(C) = \{G_1, \dots, G_M\}, \quad G_m = \{x_{m,1}, \dots, x_{m,K}\}, \quad x_{m,i} \sim \pi_{\theta_{\rm old}}(\cdot \mid C),$$2, so reducing $$\mathcal{S}(C) = \{G_1, \dots, G_M\}, \quad G_m = \{x_{m,1}, \dots, x_{m,K}\}, \quad x_{m,i} \sim \pi_{\theta_{\rm old}}(\cdot \mid C),$$3 need not increase variance if the number of prompts $$\mathcal{S}(C) = \{G_1, \dots, G_M\}, \quad G_m = \{x_{m,1}, \dots, x_{m,K}\}, \quad x_{m,i} \sim \pi_{\theta_{\rm old}}(\cdot \mid C),$$4 is increased accordingly. This suggests that in ordinary GRPO, large groups mainly change the estimator and per-prompt weighting rather than the fundamental contrastive structure (Wu et al., 1 Oct 2025).

A separate theoretical treatment formalizes this intuition differently by proving that the GRPO policy gradient is a second-order U-statistic. For prompt-conditional groups of size $$\mathcal{S}(C) = \{G_1, \dots, G_M\}, \quad G_m = \{x_{m,1}, \dots, x_{m,K}\}, \quad x_{m,i} \sim \pi_{\theta_{\rm old}}(\cdot \mid C),$$5, the gradient estimator can be rewritten as

$$\mathcal{S}(C) = \{G_1, \dots, G_M\}, \quad G_m = \{x_{m,1}, \dots, x_{m,K}\}, \quad x_{m,i} \sim \pi_{\theta_{\rm old}}(\cdot \mid C),$$6

with symmetric kernel

$$\mathcal{S}(C) = \{G_1, \dots, G_M\}, \quad G_m = \{x_{m,1}, \dots, x_{m,K}\}, \quad x_{m,i} \sim \pi_{\theta_{\rm old}}(\cdot \mid C),$$7

From this, the paper derives MSE expressions, finite-sample suboptimality bounds, and a universal scaling law

$$\mathcal{S}(C) = \{G_1, \dots, G_M\}, \quad G_m = \{x_{m,1}, \dots, x_{m,K}\}, \quad x_{m,i} \sim \pi_{\theta_{\rm old}}(\cdot \mid C),$$8

where the optimal group size is independent of training budget and schedule and depends instead on task distribution and policy geometry (Zhou et al., 1 Mar 2026).

For SGRPO in the biomolecular sense, these results do not eliminate the need for grouped same-condition sampling, because the target being optimized is itself a set-level diversity quantity. The supergroup is therefore not justified merely as a better estimator of a rollout-level reward; it is the object required to define and compare same-condition diversity. At the same time, the biomolecular paper’s ablation that SGRPO already outperforms GRPO at $$\mathcal{S}(C) = \{G_1, \dots, G_M\}, \quad G_m = \{x_{m,1}, \dots, x_{m,K}\}, \quad x_{m,i} \sim \pi_{\theta_{\rm old}}(\cdot \mid C),$$9 is consistent with the broader GRPO literature’s resistance to the simplistic claim that “bigger groups are always better” (Ye et al., 9 May 2026, Wu et al., 1 Oct 2025, Zhou et al., 1 Mar 2026).

6. Variants, misconceptions, and limitations

A common misconception is that all uses of “SGRPO” refer to the biomolecular supergroup method. In fact, the reasoning paper titled Scaf-GRPO explicitly states that its “SGRPO” shorthand means Scaffolded Group Relative Policy Optimization, not “Supergroup GRPO.” That method targets a different failure mode: when all sampled attempts for a hard prompt receive binary reward $x$00, GRPO’s group-relative advantage collapses to zero and the prompt becomes invisible to the gradient. Scaf-GRPO addresses this by diagnosing post-exemption stagnation and injecting minimal tiered hints—knowledge, planning, then solution—to create a successful on-policy trajectory, restore non-zero reward variance in the group, and thereby recover non-zero advantages without changing the GRPO loss (Zhang et al., 22 Oct 2025).

This distinction matters because the two methods solve different problems. Supergroup Relative Policy Optimization optimizes a set-level utility–diversity trade-off in biomolecular generation. Scaf-GRPO is a progressive scaffolding framework for RL from verifiable rewards in reasoning tasks. The shared acronym reflects nomenclature overlap rather than methodological identity (Zhang et al., 22 Oct 2025, Ye et al., 9 May 2026).

Theoretical literature introduces a third, more abstract usage. The U-statistic analysis of GRPO presents “Supervised/ Supergroup GRPO” as a natural extension in which bounded supervised scores replace terminal rewards or baselines are computed over a larger leave-one-out pool. Under boundedness and i.i.d. or exchangeability assumptions, the same second-order U-statistic structure and leading-rate conclusions are stated to persist. This is a formal generalization, not the biomolecular SGRPO algorithm with set-level diversity redistribution (Zhou et al., 1 Mar 2026).

The biomolecular SGRPO method has several stated limitations. It optimizes whichever diversity metric is supplied, so poorly chosen metrics can misalign optimization with downstream utility. It requires grouped same-condition sampling and therefore adds rollout overhead relative to utility-only post-training, even though the paper reports that direct set-level diversity remains effective with small groups. As with any reward-based post-training, inappropriate scalarization of utility can induce reward hacking or drift; while SGRPO avoids the history-relative drift attributed to memory-based novelty baselines, extreme $x$01 values or unsuitable diversity metrics can still misdirect learning. The authors identify adaptive group construction, condition-aware chemical series constraints, and multi-axis frontier optimization as directions for extension (Ye et al., 9 May 2026).