---
title: 'MD-GRPO: Group-Relative Policy Optimization'
url: https://www.emergentmind.com/topics/md-grpo
type: topic
---

# MD-GRPO: Group-Relative Policy Optimization

MD-GRPO is a label applied in recent arXiv literature to several domain-specific uses of Group Relative Policy Optimization (GRPO): goal-directed molecular design in GRXForm, clinically aligned medical report generation in MRG-R1, and trajectory-level optimization of masked diffusion models in Co-GRPO. Across these settings, a policy generates a group of candidates for the same conditioning instance, rewards are compared within that group, and the resulting relative advantages are used for policy-gradient updates; the precise normalization rule, surrogate objective, and regularization differ by domain [2602.12162][2512.16145][2512.22288].

## 1. Core formulation

The common GRPO mechanism is a within-context normalization of rewards. In the molecular-design formulation, one samples \(B\) starting scaffolds \(\{S_i\}_{i=1}^B\), generates \(G\) completions \(\{G_{i,1},\dots,G_{i,G}\}\) for each scaffold, computes rewards \(r_{i,j}=R(G_{i,j})\), and forms a group mean
\[
\mu_i = \frac{1}{G}\sum_{j=1}^G r_{i,j},
\]
with an optional group standard deviation
\[
\sigma_i = \sqrt{\frac{1}{G}\sum_{j=1}^G (r_{i,j}-\mu_i)^2}.
\]
The general group-normalized reward is
\[
\hat r_{i,j}=\frac{r_{i,j}-\mu_i}{\sigma_i},
\]
whereas the Dr. GRPO variant used in GRXForm drops the \(\sigma_i\) denominator and uses centered rewards
\[
A_{i,j}=r_{i,j}-\mu_i.
\]
The gradient estimator then becomes
\[
\nabla_\theta \mathcal{J}_{\rm GRPO}(\theta)\approx \frac{1}{BG}\sum_{i=1}^B\sum_{j=1}^G A_{i,j}\sum_{t=1}^{T_{i,j}}\nabla_\theta \log \pi_\theta(a_{i,j,t}\mid s_{i,j,<t}).
\]

In MRG-R1, the GRPO objective is a PPO-like clipped surrogate with a KL penalty toward a frozen reference policy:
\[
J_{\rm GRPO}(\theta)=
E_{q\sim P(Q), \{o_i\}\sim \pi_{\theta_{\rm old}}(\cdot|q)}
\left[
\frac{1}{G}\sum_{i=1}^G
\min\{
\rho_i(\theta)\cdot A_i,\,
\mathrm{clip}(\rho_i(\theta),1-\epsilon,1+\epsilon)\cdot A_i
\}
-\beta\cdot D_{KL}(\pi_{\theta_{\rm new}}(\cdot|q)\,\|\,\pi_{\theta_{\rm ref}}(\cdot|q))
\right],
\]
where
\[
A_i=\frac{r_i-\bar r}{\sigma_r}.
\]

In Co-GRPO for masked diffusion models, the trajectory-level clipped surrogate is extended to a joint policy over denoising actions and schedule actions:
\[
L(\theta,\phi) = -\frac1{G\,T}\sum_{g=1}^G \sum_{t=0}^{T-1}
\left[
\min\bigl(r^g_t(\theta,\phi)A^g_t,\;\mathrm{clip}(r^g_t(\theta,\phi),1-\epsilon,1+\epsilon)A^g_t\bigr)
+ \beta\,D_{KL}(\pi_{\theta,\phi}\|\pi_{\rm ref})
\right].
\]

These formulations share a single structural idea: candidates are compared against other candidates generated for the same scaffold, study, or prompt rather than against a single batch-global baseline. This suggests a common emphasis on conditioning-specific credit assignment under heterogeneous reward scales [2602.12162][2512.16145][2512.22288].

## 2. Goal-directed molecular design

In "Amortized Molecular Optimization via Group Relative Policy Optimization" [2602.12162], goal-directed molecular optimization under structural constraints is stated as learning, for any given starting molecular subgraph \(S\), a policy
\[
\pi_\theta(G\mid S)
\]
that sequentially elaborates \(S\) into a full molecule \(G\) in order to maximize an oracle reward
\[
R(G)\in\mathbb{R}.
\]
The paper contrasts this with instance-optimizers such as genetic algorithms and discrete diffusion with fragment-remasking, which treat each \((S,R)\) pair as a fresh combinatorial search problem and require thousands of expensive oracle calls per input.

GRXForm parametrizes \(\pi_\theta\) as a decoder-only Graph Transformer in a step-wise MDP. The state \(s_t\) is a partial molecular graph with node set \(V_t\) and edge set \(E_t\). Its hierarchical action space comprises: operation selection among “Stop,” “Add Atom,” or “Modify Existing Atom”; target selection of an existing atom \(v\in V_t\); and bond specification with bond order \(b\in\{1,2,3,4,5,6\}\). Valence masking \(\mathcal{M}(s_t)\) masks invalid actions that would exceed an atom’s valence. Input embeddings combine atom type, current degree, and dynamic action-state flags; a virtual super-node connects to all atoms; multi-head self-attention uses ReZero normalization with bond-order attention biases \(\phi(e_{ij})\); and separate MLP heads emit logits for each action level. The model is pre-trained on ChEMBL by supervised teacher-forcing on ground-truth atom-by-atom trajectories.

Fine-tuning uses batch of scaffolds \(B=10\), group size \(G=16\) completions per scaffold, beam width \(=16\), learning rate \(\alpha=10^{-4}\) with Adam, weight decay \(=0\), gradient clipping norm \(=1.0\), maximum fine-tuning epochs \(\approx 500\), no entropy regularization in the scaffold-conditioned setting \((\beta=0)\), and a maximum of 50 atoms per molecule. The optimization loop samples scaffolds, generates completions via stochastic beam search, computes rewards, centers them by scaffold-specific means, accumulates the policy gradient, and updates \(\theta\).

Empirically, the paper reports three settings. In kinase scaffold decoration with 500 held-out Murcko scaffolds cluster-split by Tanimoto \(<0.4\), using a 4-component MPO reward over GSK3\(\beta\), JNK3, QED, and SA', GRXForm-GRPO reaches an objective score of \(0.618\pm0.004\) and a strict success rate of \(17.8\%\pm9.3\%\), compared with \(0.475\pm0.120\) and \(9.1\%\pm15.7\%\) for GRXForm-REINFORCE, \(0.397\pm0.005\) and \(0.1\%\) for GRXForm-DeNovo, and scores \(\approx 0.44\) with \(0\%\) success for Mol GA and GenMol. In prodrug transfer, fine-tuning on 4 parent drugs and testing on 5 unseen drugs, mean scores are 8.65 for GRXForm-REINFORCE and 10.69 for GRXForm-GRPO. On the PMO benchmark with 10 k oracle calls, GRXForm uses standard REINFORCE rather than grouping because all starts are empty, and reports aggregate sum AUC \(=16.433\), second overall.

The paper attributes the gain to heterogeneous task difficulty across starting scaffolds: some scaffolds admit easy high-reward elaborations, while others are chemically constrained. Centering rewards by scaffold-specific group means makes completions compete only among themselves, and Figure 4 is reported to show a mean advantage signal near zero with low variance under GRPO, in contrast to large swings under a global baseline.

## 3. Clinically aligned medical report generation

In "MRG-R1: Reinforcement Learning for Clinically Aligned Medical Report Generation" [2512.16145], MD-GRPO is the application of GRPO to medical report generation with a Med-LVLM policy \(\pi_\theta\) that, given an input study \(q\), autoregressively generates a report \(o\). For each study, the method samples a group of \(G=4\) candidate reports under the old policy, computes a report-level reward, normalizes rewards within the group, and applies a clipped policy-gradient update under a KL penalty toward a frozen reference policy.

The principal clinical reward is Margin-based Cosine Similarity (MCCS), derived from CheXbert’s 14-label chest-X-ray extraction. Multi-class labels are mapped into signed scalars
\[
f(\mathrm{pos})=+1,\quad f(\mathrm{uncertain})=+1,\quad f(\mathrm{neg})=-1,\quad f(\mathrm{blank})=0,
\]
the “No Finding” dimension is discarded, and 13-dimensional signed label vectors \(z(y), z(y^*)\in\mathbb{R}^{13}\) are compared using
\[
CCS(y,y^*)=
\frac{\langle z(y),z(y^*)\rangle}
{(\|z(y)\|_2+\epsilon)(\|z(y^*)\|_2+\epsilon)},\qquad \epsilon=10^{-8}.
\]
A margin \(m\in(-1,1)\) produces
\[
MCCS(y,y^*;m)=\max\!\left(\frac{CCS(y,y^*)-m}{1-m},0\right).
\]
The total reward is
\[
R(q,o)=w_C\cdot MCCS(y,y^*;m)+w_F\cdot R_{\rm format}(o),
\]
with \(w_C=0.75\) and \(w_F=0.25\). The format term lightly rewards compliance with the prescribed `"<think>…</think> → <report>…</report>"` structure.

The base model is HuatuoGPT-Vision-7B-Qwen2.5VL, combining a ViT backbone with object-attention heads and a Qwen2.5 decoder. Fine-tuning uses LoRA adapters with rank \(=128\), \(\alpha=256\), and dropout \(=0.05\) on attention and MLP projections; the bulk of parameters remain frozen, and in SRL the LM head’s parameters are frozen to avoid catastrophic drift. Training uses 8-bit AdamW, learning rate \(5\times10^{-6}\), \(\beta_1=0.9\), \(\beta_2=0.99\), weight decay \(0.1\), cosine decay with 10% warm-up, gradient clipping max\_norm \(=0.1\), effective batch size \(B=16\), clipping threshold \(\epsilon=0.2\), KL penalty \(\beta\approx0.01\), 1 epoch of supervised warm-up, and 1 epoch of GRPO.

The primary metric is CheXbert-based clinical efficacy over the 14 standard chest X-ray observations. MRG-R1 reports CE-F1 \(=51.88\) on IU X-Ray and \(40.39\) on MIMIC-CXR, compared with \(50.53\) and \(27.80\) for R2GenCMN and \(51.15\) and \(31.95\) for CheXagent. Ablations report CE-F1 \(\approx 23/13\) for NLG rewards, \(\approx 45/30\) for CE-F1 only, \(\approx 50/39\) for MCCS only, and \(51.88/40.39\) for MCCS plus format. The discussion attributes the improvement to direct optimization of clinical content agreement, polarity consistency, and semantic completeness rather than token overlap.

## 4. Masked diffusion and joint schedule optimization

In "Co-GRPO: Co-Optimized Group Relative Policy Optimization for Masked Diffusion Model" [2512.22288], MD-GRPO refers to a GRPO-based formulation of masked diffusion model generation as a finite-horizon MDP \((S,A,P,\rho_0,r,\gamma=1)\) with horizon \(T\). The state is
\[
s_t=(V^{(t)},c),
\]
where \(V^{(t)}\in\{1\ldots V\}^N\) is the current discrete token canvas and \(c\) is the text prompt. The action in Co-GRPO is
\[
a_t=(V^{(t+1)},\mathcal{A}_{t+1}),
\]
where \(\mathcal{A}\) collects sampling temperature \(\tau_s\), guidance scale \(s\), re-mask temperature \(\tau_r\), and re-mask ratio \(r\). Transitions are deterministic given the chosen next canvas and prompt, and the reward is zero at intermediate steps and issued only at the final step:
\[
r(s_t,a_t)=0\quad (t<T-1),\qquad
r(s_{T-1},a_{T-1})=R_{\rm quality}(V^{(T)},c).
\]

The joint policy is
\[
\pi_{\theta,\phi,t}(a_t\mid s_t)
=
p_{\theta,t}(V^{(t+1)}\mid \mathcal{A}_{t+1};V^{(t)},c)\cdot
p_{\phi,t}(\mathcal{A}_{t+1}\mid V^{(t)},c),
\]
so that both the denoiser parameters \(\theta\) and schedule parameters \(\phi\) are optimized under a shared scalar reward. In practice, \(p_{\phi,t}(\cdot)\) is modeled as a small Gaussian whose mean \(\eta_{(\phi)}(s_t)\) predicts each continuous schedule component. The shared reward can be written as a human-preference model score minus a small cost for overly aggressive schedules,
\[
R = R_{\rm pref}(V^{(T)},c) - \lambda \sum_t \|\mathcal{A}_{t+1}-\bar{\mathcal{A}}(t)\|^2.
\]

A central technical property is that the method avoids back-propagation through the \(T\)-step generation process. By using the likelihood-ratio trick, it only requires evaluation of \(\log \pi_\theta(a_t\mid s_t)\) and collection of the terminal reward. The paper presents this as a memory and compute saving relative to differentiating through all intermediate activations of the denoising network.

The empirical evaluation fine-tunes a 1B-parameter Meissonic MDM with an approximately 9M-parameter scheduling network for 48 steps. Reported scores are: ImageReward, baseline \(0.942\) and Co-GRPO \(1.122\); HPSv2, baseline \(28.83\) and Co-GRPO \(29.37\); GenEval, baseline \(0.47\) and Co-GRPO \(0.55\); and DPG-Bench, baseline \(64.57\) and Co-GRPO \(70.10\). The paper emphasizes that the schedule network adds less than 1% extra parameters.

## 5. Variance reduction, instability, and related stabilized variants

The supplied papers motivate MD-GRPO largely through variance control, but they do so in different regimes. In molecular design, the stated failure mode is high variance from the heterogeneous difficulty of distinct starting structures: a global baseline causes gradients to be dominated by “easy” tasks, whereas per-scaffold group means ensure that even if all rewards in a group are low, the best completion still receives positive advantage. In MRG-R1, group-relative advantages are presented as lower-variance than vanilla REINFORCE or actor-critic and as eliminating the need for a learned value network. In Co-GRPO, the group-relative objective is combined with PPO-style clipping and KL regularization while jointly optimizing both model and schedule parameters [2602.12162][2512.16145][2512.22288].

A closely related but distinct development is "MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following" [2606.06058]. That paper identifies three pathologies of z-score group normalization under discrete, low-dispersion rewards: low-variance amplification, mean-centering blindness, and zero-variance collapse. Its remedies are multi-temperature sampling, dual-anchor advantages, prospect-theoretic shaping, and asymmetric KL regularization. The mixed advantage is
\[
A_i=(1-\alpha)\tilde z_i+\alpha\tilde\delta_i,
\]
where \(z_i\) is the group-relative anchor, \(\delta_i\) is a goal-aware anchor, and both can be shaped by a bounded loss-averse transform. Reported gains include up to \(+5.0\) percentage points in Hard Success Rate on FollowBench, IFEval, and a curated multi-constraint dataset, with general-capability benchmarks remaining unchanged within \(\pm 0.2\) percentage points.

The juxtaposition is significant because it shows that group-relative normalization is not uniformly stable across reward regimes. In heterogeneous continuous-reward settings it is used to reduce variance; in discrete low-dispersion settings, additional mechanisms may be required to avoid vanishing or distorted learning signals.

## 6. Terminological scope and comparative profile

The term “MD-GRPO” is not used in a single uniform sense across the supplied literature. In GRXForm it denotes GRPO for amortized molecular optimization; in MRG-R1 it denotes GRPO for medical report generation; and in Co-GRPO it is used for masked diffusion models. By contrast, MDP-GRPO is a separate stabilized variant for multi-constraint instruction following rather than another use of the same label [2602.12162][2512.16145][2512.22288][2606.06058].

The algorithmic differences are substantial. GRXForm uses a decoder-only Graph Transformer with chemically valid action masking and, in the reported scaffold-conditioned setting, centered rewards \(A_{i,j}=r_{i,j}-\mu_i\) without entropy regularization. MRG-R1 uses a Med-LVLM fine-tuned with LoRA, a clipped surrogate, KL regularization toward a frozen reference policy, and an MCCS-based clinical reward augmented by a lightweight format reward. Co-GRPO treats masked diffusion inference itself as an MDP and jointly optimizes denoiser parameters and schedule parameters. A plausible implication is that “MD-GRPO” functions more as a family resemblance around group-relative policy optimization than as a single canonical algorithmic specification.

| Setting | Mechanism | Reported outcome |
|---|---|---|
| GRXForm molecular optimization | Per-scaffold centered rewards in scaffold-conditioned fine-tuning | Obj. Score \(0.618\pm0.004\), Success Rate \(17.8\%\pm9.3\%\) |
| MRG-R1 medical report generation | Clipped GRPO with MCCS reward and format reward | CE-F1 \(51.88\) on IU X-Ray, \(40.39\) on MIMIC-CXR |
| Co-GRPO masked diffusion | Joint \(\theta/\phi\) optimization of model and schedule | ImageReward \(1.122\), HPSv2 \(29.37\), GenEval \(0.55\), DPG-Bench \(70.10\) |

One recurrent misconception is that GRPO implies a fixed normalization rule. The supplied literature shows otherwise: some formulations use z-score normalization, some use centered rewards only, and some add PPO-style clipping and KL penalties. Another is that grouping is always required. In the PMO de-novo benchmark, GRXForm uses standard REINFORCE rather than grouping because all starts are empty. The unifying point is therefore not a single implementation detail, but the use of within-context relative rewards to shape policy updates under trajectory-level objectives.

Source: https://www.emergentmind.com/topics/md-grpo