MaskGRPO: Masked Generation with Group RL

• MaskGRPO is a family of Group Relative Policy Optimization adaptations designed for masked or discrete diffusion models in multimodal reinforcement learning.
• It overcomes autoregressive limitations by introducing masking-aware importance sampling and surrogate ELBO estimators for tractable likelihood-ratio estimation.
• The method incorporates modality-specific reverse operators and schedule optimization to enhance stability and efficiency across language and vision tasks.

Searching arXiv for papers on MaskGRPO and closely related variants to ground the article in the cited literature. Using the arXiv search tool to retrieve the cited papers and confirm bibliographic details. MaskGRPO denotes a family of Group Relative Policy Optimization (GRPO) adaptations for masked or discrete-diffusion generative models, introduced to make reinforcement learning feasible in settings where generation is iterative, non-autoregressive, and often multimodal. In this literature, the central problem is that standard GRPO assumes an autoregressive factorization for tractable likelihood-ratio estimation, whereas discrete diffusion models (DDMs), masked generative models (MGMs), and masked diffusion models (MDMs) generate by repeated masking and unmasking steps rather than by left-to-right token emission. The name is used most directly for a multimodal DDM method that provides “the first viable approach to enable scalable multimodal reinforcement learning in discrete diffusion with effective importance sampling and modality-specific adaptations” (Ma et al., 3 Oct 2025), and also for a text-to-image MGM method that formulates unmasking as a multi-step decision-making problem (Luo et al., 15 Oct 2025). A subsequent line of work extends the same GRPO perspective to jointly optimize both masked diffusion model parameters and inference schedules under a unified Markov Decision Process (Zhou et al., 25 Dec 2025).

1. Conceptual scope and problem setting

MaskGRPO arises from the mismatch between GRPO’s usual autoregressive setting and the structure of masked or discrete diffusion generation. In the autoregressive case, GRPO samples a group of full rollouts, computes group-relative advantages, and performs a clipped, KL-regularized policy-gradient update. The key quantity is a tractable per-step likelihood ratio defined over causally factorized token probabilities. For DDMs, this factorization is unavailable: “We no longer have a natural factorization $\pi_\theta(o)=\prod_k\pi_\theta(o^k\mid o^{<k})$. Instead we must ‘reverse’ a completed sequence $o$ by re-masking to varying strengths $t$ and estimate how the joint log-likelihood changes” (Ma et al., 3 Oct 2025).

This obstacle appears in multiple masked-generation paradigms. In multimodal discrete diffusion, the difficulty is framed as intractable importance sampling and complex rollout construction under non-autoregressive generation (Ma et al., 3 Oct 2025). In text-to-image masked generative models, the issue is posed differently: existing reinforcement learning approaches are tailored to either diffusion models or autoregressive models, while “masked generative models” constitute an overlooked alternative whose iterative unmasking must itself be cast as an RL problem (Luo et al., 15 Oct 2025). In masked diffusion models, the discrepancy is broader: inference is a multi-step trajectory governed both by the denoising model and by schedules such as mask ratio and temperature, but training often uses a simplified one-step objective that leaves the trajectory-level schedule unoptimized (Zhou et al., 25 Dec 2025).

Taken together, these works define MaskGRPO not as a single invariant algorithm, but as a methodological class: GRPO-style reinforcement learning reformulated for masked, non-autoregressive generation. This suggests a unifying interpretation in which the primary innovation is not GRPO itself, but the construction of a tractable surrogate for trajectory-level policy optimization when generation is mediated by masking dynamics rather than causal token prefixes.

2. Theoretical foundation in discrete diffusion

The most explicit theoretical derivation appears in the DDM formulation of MaskGRPO (Ma et al., 3 Oct 2025). There, the forward corruption process is written as a continuous-time Markov chain that maps a clean sequence $x_0$ to an absorbing mask token $\mathbf m$. At time $t\in[0,1]$,

$$q(x_t\mid x_0,t)=\mathrm{Cat}\bigl(x_t;\,\alpha_t\,x_0+(1-\alpha_t)\,\mathbf m\bigr),$$

with $\alpha_0=1$ and $\alpha_1=0$. The training objective is expressed in ELBO form as

$\mathcal L_{\mathrm{DDM} =-\,\mathbb E_{t\sim U(0,1),\,x_0,\,x_t}\,\ell_{\pi_\theta}(x_t,x_0)$

with

$o$0

The GRPO side of the construction begins from the usual group-relative advantage,

$o$1

and a clipped, KL-regularized objective over rollout groups. The DDM difficulty is then reformulated as an estimation problem over token fluctuations between adjacent masking levels. Specifically, for two policies $o$2 and $o$3, MaskGRPO seeks to approximate

$o$4

where $o$5 denotes tokens unmasked between $o$6 and $o$7. Because $o$8 is intractable, the method substitutes an ELBO estimator,

$o$9

and further approximates the difference by evaluating terms at the fully masked state. Using the fact that $t$0 sums only over masked positions, the paper derives the approximation

$t$1

This leads to a computable importance weight,

$t$2

and an analogous KL estimator,

$t$3

The resulting GRPO-style objective is discretized over masking strengths $t$4:

$t$5

In this formulation, MaskGRPO’s theoretical contribution is a practical surrogate for importance sampling in non-autoregressive discrete diffusion. The paper explicitly presents this as the basis for “a systematic policy optimization approach” for discretized visual diffusion (Ma et al., 3 Oct 2025).

3. Algorithmic structure and modality-aware design

The DDM variant organizes MaskGRPO into four stages: sample, reverse, estimate, and update (Ma et al., 3 Oct 2025). The algorithm initializes $t$6, samples prompts from $t$7, generates $t$8 full completions from $t$9, computes rewards and advantages, then iterates over $x_0$0 masking strengths

$x_0$1

At each strength, it samples a re-masked version $x_0$2, computes $x_0$3 and $x_0$4, accumulates the GRPO-style objective, and updates parameters with optional clipping at $x_0$5 (Ma et al., 3 Oct 2025).

A distinctive feature is its modality-specific treatment of rollouts and reverse operators.

For text, MaskGRPO adopts a semi-autoregressive rollout strategy. In each block of length $x_0$6, the model predicts logits on masked positions, fills the top-$x_0$7 most confident tokens, and re-masks the remaining positions by lowest confidence, iterating until the sequence is fully unmasked. The reverse operator is “AR-like”: a linear decay weight $x_0$8 is computed over non-prompt positions and mask probabilities $x_0$9 are allocated so that later tokens are more likely to be re-masked (Ma et al., 3 Oct 2025).

For vision, the rollout procedure differs substantially. The paper reports that MaskGIT’s low-confidence sampler degrades on large-vocabulary visual tokenizers such as 8192-token vocabularies. Instead, it uses the discrete diffusion reverse kernel in fully stochastic form:

$\mathbf m$0

then samples $\mathbf m$1 only at currently masked positions. The vision reverse operator is random reversing at a constant ratio $\mathbf m$2, with carefully managed RNG seeds for low variance (Ma et al., 3 Oct 2025).

The paper also specifies three modality-aware stabilizers. First, timestep truncation $\mathbf m$3 clamps $\mathbf m$4 to avoid low-mask regimes with negligible signal in language or exploding variance in vision; ablations indicate $\mathbf m$5 for text and $\mathbf m$6 for images. Second, estimator bias control exploits “AR-ness” in language by concentrating importance estimation on high-entropy later tokens, whereas vision retains high mask levels and random reversal because of strong global correlations. Third, KL clipping differs by modality: vision benefits from $\mathbf m$7, while text uses $\mathbf m$8 (Ma et al., 3 Oct 2025).

These design choices are presented not as generic heuristics but as structural accommodations to the differing stochastic geometry of text and vision generation in discrete diffusion.

4. Mask-GRPO for masked generative text-to-image models

A distinct but related method, titled “Mask-GRPO,” adapts GRPO to text-to-image generation with masked generative models rather than discrete diffusion (Luo et al., 15 Oct 2025). Here the iterative unmasking process is directly formulated as a finite-horizon Markov Decision Process with horizon $\mathbf m$9. At step $t\in[0,1]$0, the state is

$t\in[0,1]$1

where $t\in[0,1]$2 is the current token sequence and $t\in[0,1]$3 is the fixed text prompt. The action is the next full sequence,

$t\in[0,1]$4

The crucial modeling choice is the transition probability. The paper contrasts an “AR-style (naïve)” definition,

$t\in[0,1]$5

with MGM-aware definitions based on the subset $t\in[0,1]$6 of newly unmasked tokens. The full-trick definition is

$t\in[0,1]$7

and the simplified version is

$t\in[0,1]$8

Rewards are sparse: only the final step receives a nonzero reward, based on CLIP or an alternative perceptual model applied to the rendered image from $t\in[0,1]$9 (Luo et al., 15 Oct 2025).

The GRPO objective follows the standard clipped-surrogate pattern:

$$q(x_t\mid x_0,t)=\mathrm{Cat}\bigl(x_t;\,\alpha_t\,x_0+(1-\alpha_t)\,\mathbf m\bigr),$$0

where

$$q(x_t\mid x_0,t)=\mathrm{Cat}\bigl(x_t;\,\alpha_t\,x_0+(1-\alpha_t)\,\mathbf m\bigr),$$1

and $$q(x_t\mid x_0,t)=\mathrm{Cat}\bigl(x_t;\,\alpha_t\,x_0+(1-\alpha_t)\,\mathbf m\bigr),$$2 is the group-normalized advantage (Luo et al., 15 Oct 2025).

The implementation is notable for three auxiliary strategies. The first is a reduction strategy, with both computational reduction—computing the surrogate only on a subset of steps—and unmasking reduction, where training uses $$q(x_t\mid x_0,t)=\mathrm{Cat}\bigl(x_t;\,\alpha_t\,x_0+(1-\alpha_t)\,\mathbf m\bigr),$$3. The second is sample filtering, termed “Vanishing Samples”: the method tracks the empirical standard deviation $$q(x_t\mid x_0,t)=\mathrm{Cat}\bigl(x_t;\,\alpha_t\,x_0+(1-\alpha_t)\,\mathbf m\bigr),$$4 of group rewards over recent batches and rejects a new group if its reward standard deviation falls below the dynamic 10th percentile historical threshold. The third is removal of the KL penalty by setting $$q(x_t\mid x_0,t)=\mathrm{Cat}\bigl(x_t;\,\alpha_t\,x_0+(1-\alpha_t)\,\mathbf m\bigr),$$5, reported as crucial for the 1.3B-parameter Show-o model (Luo et al., 15 Oct 2025).

This variant therefore differs from the DDM MaskGRPO in both formalism and optimization practice. The DDM paper emphasizes an ELBO-based importance estimator and modality-aware reverse operators (Ma et al., 3 Oct 2025), whereas the MGM paper centers on redefining the transition probability of iterative unmasking and finds that “Mask-GRPO w/o KL” performs best for Show-o (Luo et al., 15 Oct 2025).

5. Empirical performance across language and vision

The multimodal DDM paper reports results on both language and vision benchmarks (Ma et al., 3 Oct 2025). For language, using LLaDA-8B-Instruct on GSM8K, MATH500, and MBPP in zero-shot pass@1 evaluation, MaskGRPO yields $$q(x_t\mid x_0,t)=\mathrm{Cat}\bigl(x_t;\,\alpha_t\,x_0+(1-\alpha_t)\,\mathbf m\bigr),$$6 versus $$q(x_t\mid x_0,t)=\mathrm{Cat}\bigl(x_t;\,\alpha_t\,x_0+(1-\alpha_t)\,\mathbf m\bigr),$$7 on GSM8K, $$q(x_t\mid x_0,t)=\mathrm{Cat}\bigl(x_t;\,\alpha_t\,x_0+(1-\alpha_t)\,\mathbf m\bigr),$$8 versus $$q(x_t\mid x_0,t)=\mathrm{Cat}\bigl(x_t;\,\alpha_t\,x_0+(1-\alpha_t)\,\mathbf m\bigr),$$9 on MATH500, and $\alpha_0=1$0 versus $\alpha_0=1$1 on MBPP. These gains are achieved in $\alpha_0=1$2k steps versus $\alpha_0=1$3k+ for prior diffu-GRPO, with $\alpha_0=1$4 fewer iterations. For vision, using MMaDA-8B-Base, the reported metrics are GenEval, DPG-Bench, and human preferences measured with DeQA, ImageReward, and HPSv3. GenEval overall improves from $\alpha_0=1$5 to $\alpha_0=1$6 without SFT and from $\alpha_0=1$7 to $\alpha_0=1$8 with SFT; DPG-Bench overall improves from $\alpha_0=1$9 to $\alpha_1=0$0 without SFT and to $\alpha_1=0$1 with SFT; HPSv3 improves from $\alpha_1=0$2 to $\alpha_1=0$3 and then $\alpha_1=0$4 (Ma et al., 3 Oct 2025).

The Show-o text-to-image paper reports results on standard T2I benchmarks and preference alignment (Luo et al., 15 Oct 2025). On GenEval zero-shot, Show-o improves from $\alpha_1=0$5 to $\alpha_1=0$6, tying the reported SOTA level. On MSCOCO-30K FID, the model improves from $\alpha_1=0$7 to $\alpha_1=0$8. On ImageReward, preference alignment improves from $\alpha_1=0$9 to $\mathcal L_{\mathrm{DDM} =-\,\mathbb E_{t\sim U(0,1),\,x_0,\,x_t}\,\ell_{\pi_\theta}(x_t,x_0)$0. The paper further reports ablations in which the full recipe reaches $\mathcal L_{\mathrm{DDM} =-\,\mathbb E_{t\sim U(0,1),\,x_0,\,x_t}\,\ell_{\pi_\theta}(x_t,x_0)$1 GenEval, compared with $\mathcal L_{\mathrm{DDM} =-\,\mathbb E_{t\sim U(0,1),\,x_0,\,x_t}\,\ell_{\pi_\theta}(x_t,x_0)$2 for a KL-regularized version, $\mathcal L_{\mathrm{DDM} =-\,\mathbb E_{t\sim U(0,1),\,x_0,\,x_t}\,\ell_{\pi_\theta}(x_t,x_0)$3 without sample filtering, $\mathcal L_{\mathrm{DDM} =-\,\mathbb E_{t\sim U(0,1),\,x_0,\,x_t}\,\ell_{\pi_\theta}(x_t,x_0)$4 with computational reduction, and $\mathcal L_{\mathrm{DDM} =-\,\mathbb E_{t\sim U(0,1),\,x_0,\,x_t}\,\ell_{\pi_\theta}(x_t,x_0)$5 with unmasking reduction (Luo et al., 15 Oct 2025).

The later Co-GRPO paper reports gains over a Meissonic baseline plus Naive-GRPO on reward-model and zero-shot benchmarks (Zhou et al., 25 Dec 2025). With Meissonic at $\mathcal L_{\mathrm{DDM} =-\,\mathbb E_{t\sim U(0,1),\,x_0,\,x_t}\,\ell_{\pi_\theta}(x_t,x_0)$6B parameters and $\mathcal L_{\mathrm{DDM} =-\,\mathbb E_{t\sim U(0,1),\,x_0,\,x_t}\,\ell_{\pi_\theta}(x_t,x_0)$7 inference steps, the method reports ImageReward improvement of $\mathcal L_{\mathrm{DDM} =-\,\mathbb E_{t\sim U(0,1),\,x_0,\,x_t}\,\ell_{\pi_\theta}(x_t,x_0)$8 from $\mathcal L_{\mathrm{DDM} =-\,\mathbb E_{t\sim U(0,1),\,x_0,\,x_t}\,\ell_{\pi_\theta}(x_t,x_0)$9 to $o$00, HPSv2 improvement of $o$01 from $o$02 to $o$03, plus zero-shot improvements of $o$04 on GenEval and $o$05 on DPG-Bench (Zhou et al., 25 Dec 2025).

Method	Domain / base model	Reported outcomes
MaskGRPO (Ma et al., 3 Oct 2025)	Multimodal discrete diffusion; LLaDA-8B-Instruct, MMaDA-8B-Base	GSM8K $o$06 vs. $o$07; MATH500 $o$08 vs. $o$09; MBPP $o$10 vs. $o$11; GenEval $o$12 or $o$13; DPG-Bench $o$14 or $o$15
Mask-GRPO (Luo et al., 15 Oct 2025)	Text-to-image MGM; Show-o 1.3B	GenEval $o$16; FID-30K $o$17; ImageReward $o$18
Co-GRPO (Zhou et al., 25 Dec 2025)	Text-to-image MDM; Meissonic 1.0B	ImageReward $o$19; HPSv2 $o$20; GenEval $o$21; DPG-Bench $o$22

Across these works, the recurring empirical theme is that GRPO can be made effective outside autoregressive generation, provided that trajectory likelihood ratios, reverse operations, or schedule actions are defined in a way compatible with masked generation.

6. Relationship to schedule optimization and later extensions

Co-GRPO broadens the MaskGRPO idea by explicitly incorporating inference schedules into the action space (Zhou et al., 25 Dec 2025). The state is

$o$23

and the action is a tuple containing both sampled tokens and the next-step schedule,

$o$24

where $o$25 may include mask ratio $o$26, sampling temperature $o$27, re-mask temperature $o$28, and guidance scale $o$29. The joint policy factorizes as

$o$30

with the schedule policy modeled as a multivariate Gaussian whose mean is predicted by a small neural net and covariance fixed to $o$31 (Zhou et al., 25 Dec 2025).

The clipped surrogate objective is extended to the joint policy:

$o$32

where the trajectory ratio decomposes into model and schedule factors,

$o$33

The optimization is performed by an alternating strategy: in model-update epochs only $o$34 is updated via $o$35, and in schedule-update epochs only $o$36 is updated via $o$37 (Zhou et al., 25 Dec 2025).

This extension is significant because it reframes a limitation shared by earlier masked-generation RL methods: even if the token predictor is optimized, inference schedules still remain hand-designed. Co-GRPO’s central claim is that trajectory-level RL should treat both denoising and scheduling as policy components. A plausible implication is that MaskGRPO-style methods may progressively shift from model-only fine-tuning toward joint optimization of all trajectory-defining control variables.

The ablations in Co-GRPO support this interpretation. Making schedule elements trainable yields monotonic gains; training only $o$38 improves ImageReward by approximately $o$39, and adding $o$40, $o$41, and $o$42 accumulates to the full $o$43 gain. A naive joint-gradient update is reported as less stable than the alternating strategy, with ImageReward $o$44 versus $o$45 (Zhou et al., 25 Dec 2025). These results suggest that schedule optimization is not merely an implementation detail but a first-order factor in masked diffusion RL.

7. Stability, limitations, and points of interpretation

Several stability findings recur across the literature. In the DDM setting, ablation on timestep truncation $o$46 shows that values that are too small or too large lead respectively to instability or slow learning; the AR-like reverse method outperforms TraceRL across $o$47; and after RL, the emerge sampler converges to higher-quality images than MaskGIT even though before RL it scores lower on detector metrics (Ma et al., 3 Oct 2025). In the MGM T2I setting, removing KL regularization improves GenEval from $o$48 to $o$49, and sample filtering improves it from $o$50 to $o$51; unmasking reduction to $o$52 trades approximately $o$53 GenEval points for $o$54–$o$55 speedup (Luo et al., 15 Oct 2025). In Co-GRPO, the scheduling policy adds only approximately $o$56 latency, but the method still inherits the overhead of RL fine-tuning and dependence on learned reward models (Zhou et al., 25 Dec 2025).

A common misconception would be to treat MaskGRPO as a single algorithm with a fixed objective and fixed implementation. The literature does not support that reading. The DDM paper defines MaskGRPO through an ELBO-derived importance estimator and modality-specific reversing in multimodal discrete diffusion (Ma et al., 3 Oct 2025). The Show-o paper uses the name “Mask-GRPO” for an MGM-specific MDP formulation whose key move is transition-probability redefinition and whose best variant removes the KL term entirely (Luo et al., 15 Oct 2025). The later Co-GRPO work is not named MaskGRPO, but it clearly occupies the same conceptual lineage by extending GRPO to masked diffusion trajectories with joint schedule control (Zhou et al., 25 Dec 2025).

Another important interpretive point concerns “first” claims. The DDM paper identifies MaskGRPO as “the first viable approach to enable scalable multimodal reinforcement learning in discrete diffusion” and “the first practical way for discretized visual diffusion” (Ma et al., 3 Oct 2025). The MGM paper identifies Mask-GRPO as “the first method to incorporate Group Relative Policy Optimization (GRPO)-based RL” into masked generative models for T2I generation (Luo et al., 15 Oct 2025). These claims are not contradictory because they concern different model classes: discrete diffusion in one case, masked generative models in the other.

Overall, MaskGRPO designates a developing research direction in which GRPO is reformulated for masked-sequence generation by replacing autoregressive likelihood-ratio assumptions with masking-aware estimators, MDP formulations, and trajectory-level control mechanisms. The available evidence indicates that such reformulations can support stable reinforcement learning in language, vision, and text-to-image settings, while the subsequent extension to schedule co-optimization suggests that the full trajectory—not only token prediction—has become the operative unit of policy design (Ma et al., 3 Oct 2025, Luo et al., 15 Oct 2025, Zhou et al., 25 Dec 2025).