Papers
Topics
Authors
Recent
Search
2000 character limit reached

UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generation

Published 24 Mar 2026 in cs.CV | (2603.23500v1)

Abstract: Unified models capable of interleaved generation have emerged as a promising paradigm, with the community increasingly converging on autoregressive modeling for text and flow matching for image generation. To advance this direction, we propose a unified reinforcement learning framework tailored for interleaved generation. We validate our approach on its fundamental unit: a single round of reasoning-driven image generation, where the model first expands the user prompt through reasoning, followed by image synthesis. Formulating this multimodal generation process as a Markov Decision Process with sparse terminal rewards, we introduce UniGRPO to jointly optimize text and image generation policies using GRPO. Adopting a minimalist methodology to avoid over-design, we leverage established training recipes for both modalities by seamlessly integrating standard GRPO for reasoning and FlowGRPO for visual synthesis. To ensure scalability to multi-round interleaved generation, we introduce two critical modifications to the original FlowGRPO: (1) eliminating classifier-free guidance to maintain linear, unbranched rollouts, which is essential for scaling to complex scenarios involving multi-turn interactions and multi-condition generation (e.g., editing); and (2) replacing the standard latent KL penalty with an MSE penalty directly on the velocity fields, providing a more robust and direct regularization signal to mitigate reward hacking effectively. Our experiments demonstrate that this unified training recipe significantly enhances image generation quality through reasoning, providing a robust and scalable baseline for the future post-training of fully interleaved models.

Summary

  • The paper introduces a unified RL framework that jointly optimizes text reasoning and visual synthesis within a single MDP.
  • It uses novel techniques such as CFG elimination and velocity-based MSE regularization to enhance training stability and prevent reward hacking.
  • Empirical benchmarks demonstrate improved text-image compositionality and photorealism, outperforming existing RL methods on TA and GenEval tasks.

Unified Policy Optimization for Reasoning-Driven Visual Generation: An Expert Analysis

Problem Formulation and Motivation

The paper introduces UniGRPO, a unified reinforcement learning (RL) framework for interleaved multimodal generation—specifically reasoning-driven text-to-image generation (2603.23500). Unlike previous paradigms, which typically optimize either the text (reasoning) or the image synthesis policy in isolation, UniGRPO jointly aligns both modalities within a single Markov Decision Process (MDP), leveraging Group Relative Policy Optimization (GRPO) for text generation and FlowGRPO for visual synthesis. The motivation stems from current trends toward unified, interleaved multimodal models, as seen in architectures such as Bagel and Mogao, which seek to couple autoregressive LLMs with flow-matching generators, aiming for iterative reasoning and improved text-image coherence.

Methodological Contributions

MDP Formulation and Policy Design

UniGRPO formulates the text–image generation pipeline as an MDP with discrete actions for text (autoregressive token prediction) and continuous actions for image (velocity fields in flow-matching models). The reward is sparse and terminal, based on VLM-based alignment scoring of the generated image against the prompt. The framework samples G parallel trajectories, computes group-relative advantages, and updates both policies using a unified objective: J=JText+JFlowJ = J_\text{Text} + J_\text{Flow}, with balanced weighting.

Modifications for Scalability and Regularization

Two architectural modifications are pivotal:

  • CFG Elimination: Classifier-Free Guidance (CFG) is omitted during RL training (present only at inference). This enforces linear, unbranched rollouts, mitigating the exponential complexity and gradient instability inherent to multi-condition, multi-round generation scenarios.
  • Velocity-Based MSE Regularization: Standard KL penalties, which depend on timestep-specific noise variance, are replaced by direct MSE penalization on velocity fields. This uniform constraint is empirically more robust against reward hacking, preserving generative quality across the stochastic trajectory and preventing over-optimization exploits typical with KL-based regularization.

Training Protocol

The backbone is Bagel, further optimized via Supervised Fine-Tuning (SFT) as initialization. GRPO and FlowGRPO are adopted, with SDE-ODE hybrid sampling for computational efficiency. The reward model is a differentiable VLM-like verifier (InternVL-aligned), ensuring compatibility with gradient-based baselines.

Empirical Results

Benchmarks and Performance

UniGRPO is evaluated on two established tasks:

  • Text Alignment (TA) Benchmark: Diverse prompts, with four generated images per prompt, scored by a VLM on granular exam points.
  • GenEval: Object-centric compositional evaluation (counting, spatial relations, attribute binding).

UniGRPO achieves:

  • TA Score: 0.8381
  • GenEval Score: 0.90

These results surpass alternative RL approaches, including ReFL, FPO, FlowGRPO, TextGRPO, and hybrid methods. Ablations show that both CFG elimination and velocity-based MSE regularization are essential for preventing training collapse and reward hacking, and for maintaining photorealistic texture fidelity.

Reasoning Chain Optimization

Joint optimization yields reasoning traces that are tightly coupled with image synthesis, enhancing text-image compositionality. The intermediate reasoning is more task-focused compared to pre-optimized modules, resulting directly from the RL objective tied to the final multimodal reward.

Qualitative Observations

UniGRPO significantly mitigates artifacts and blurriness seen in baseline and SFT-only models, achieving detailed, coherent, and photorealistic outputs. The alignment between prompt, reasoning, and generated image is demonstrably improved.

Theoretical Implications

UniGRPO bridges a gap in unified multimodal RL by eliminating the artificial separation between language and visual generation policies. The minimalist MDP formulation and unified policy optimization clarify the theoretical framework for interleaved generation, suggesting that future multimodal alignment can be approached as joint RL in a high-dimensional, sparse-reward space. The velocity-based regularization advances RL stability theory for ODE/SDE-driven generators.

Practical Implications and Future Directions

Practically, UniGRPO provides a scalable baseline for post-training unified models, enabling efficient interactive generation tasks (multi-condition editing, storytelling, multi-turn dialogue) without the computational penalty of branched CFG-based rollouts.

Future research directions include:

  • Scaling to Multi-Round Generation: Extending the single-round MDP to multi-turn, contextual interaction, leveraging the linear rollout strategy for tractable optimization and memory management.
  • Dense Multimodal Process Reward Modeling: Introduction of process reward models for intermediate reasoning steps could substantially improve credit assignment and interpretability, facilitating integration of black-box and step-wise feedback mechanisms.

Conclusion

UniGRPO provides a principled, RL-based framework for unified, reasoning-driven text–image generation. Its dual-modal policy optimization, scalability enablers (CFG elimination, velocity MSE regularization), and empirical superiority establish it as a robust methodology for future interleaved multimodal models. The framework forms a foundation for interactive, reasoning-intensive visual generation systems, with clear avenues for extension in credit assignment and scalable multimodal RL.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

1) What is this paper about?

This paper shows a new way to train one AI model to both think in words and create pictures, so it can first “reason” about a user’s request and then draw a better image. The method is called UniGRPO. It uses reinforcement learning (a way to learn from rewards) to improve the text-thinking part and the image-making part together, at the same time, so the final pictures match the prompt more accurately and look better.

2) What questions are the researchers trying to answer?

  • Can we train a single model to handle “prompt → reasoning text → image” as one connected process instead of treating text and image as separate tasks?
  • If we do that, does the model produce better images that follow the prompt more closely?
  • How can we make this training stable and efficient so it can later scale to more complex, multi-step interactions (like editing an image over several rounds)?

3) How did they do it? (Methods explained simply)

Think of the model as a team with two roles:

  • A planner (the text part) that thinks through the prompt in words.
  • An artist (the image part) that turns those thoughts into a picture.

They train both roles together using reinforcement learning (RL), which is like giving the team a score only after the final picture is finished. If the score is high, the team learns to repeat what worked; if it’s low, they adjust.

Here’s the basic approach:

  • They view the whole process as a sequence of small decisions (a “game” of many moves): first choosing the next word in the reasoning, then taking small steps to refine the image.
  • At the end, a reward model grades the final image for how well it matches the prompt and its overall quality. Importantly, there’s no reward in the middle—only at the end—so the model must learn which earlier choices led to a good outcome.
  • They use GRPO (Group Relative Policy Optimization), a type of RL that compares several attempts for the same prompt. It’s like asking a group to try different approaches, scoring them, and learning more from the better ones.
    • For text: standard GRPO improves the planner’s “next word” choices.
    • For images: FlowGRPO adapts GRPO to image generation that gradually transforms noise into a picture (a process called flow matching). To allow exploration, they add a bit of randomness during training so the model can try variations and learn which ones are better.

To make this stable and scalable, they make two key tweaks to the image-training part:

  • They remove “classifier-free guidance” (CFG). In simple terms, CFG is like asking for a second opinion every step of the way, which doubles the work and branches the process. Removing it keeps the process as one straight path, which is much simpler and cheaper—important for future, longer conversations or multi-condition tasks like editing.
  • They prevent “reward hacking” (when a model finds sneaky shortcuts to look good to the scorer but actually produces worse images). Instead of a standard penalty that can be uneven across steps, they use a direct, simple check: keep the “direction and speed” of the model’s image updates (its “velocity field”) close to a trusted reference model using a basic MSE (mean squared error). You can think of this like asking the artist to stay within a safe range of brush strokes compared to a skilled guide, so it can improve without going off the rails.

4) What did they find, and why does it matter?

  • Training text reasoning and image creation together works better than training them separately. The model learns to write more helpful thoughts that directly guide the image, and the image improves too.
  • Their method beats strong baselines on two benchmarks:
    • An internal Text Alignment set (checks if images truly match the prompt).
    • GenEval (a public test that measures things like counting objects, placing them correctly, and matching colors and attributes).
  • The pictures are sharper, more realistic, and more faithful to complex prompts. The reasoning text also becomes more focused and useful, not just long.
  • Removing CFG during training did not hurt results; it actually helps make the whole process more efficient and easier to scale.
  • The new “velocity MSE” rule helps stop the model from gaming the reward and keeps training stable.

Why this matters: It shows a clear path to building smarter, unified models that can use their “thinking” to make better visuals—and do so efficiently, which is crucial for future applications.

5) What could this change in the future?

  • Multi-round creativity: Because the process is simple and unbranched, it should scale to longer interactions, like asking the model to revise an image several times, do story-like sequences, or handle multiple conditions (e.g., “keep the background, change the outfit”).
  • Better, more honest training: The anti-cheating guardrails (like the velocity MSE) make it safer to train models with rewards, so they get better without breaking quality.
  • More flexible rewards: While this paper used a reward model that scores the final image, the same training can work with all kinds of judges, including non-differentiable ones (like human feedback or external checkers). In the future, adding “process rewards” that grade the reasoning steps themselves could make learning even faster and clearer.

In short, UniGRPO is a practical, stable way to teach one model to think and draw together—and to do it in a way that sets the stage for longer, smarter, more interactive visual creation.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

Below is a concise, actionable list of unresolved issues and open questions that future work could address:

  • Multi-round interleaved generation: The method is only validated on a single round (Prompt → Reasoning → Image). It remains unknown how UniGRPO performs in multi-turn settings (e.g., iterative refinement, dialogues, visual storytelling) in terms of stability, compute cost, and long-horizon credit assignment.
  • Editing and multi-condition generation: Although the design is motivated by scalability to editing and multi-condition inputs, no experiments verify performance on image editing, inpainting, or combinations of conditions (e.g., image+text+mask).
  • Inference without CFG: Training omits CFG to enforce linear rollouts, but evaluation in ablations still applies CFG. It is unclear whether the approach maintains alignment and quality at inference when CFG is also removed, especially for multi-condition setups.
  • Train–test mismatch (SDE vs. ODE): The image policy is optimized with SDE-based exploration but inference typically uses an ODE sampler. The impact of this mismatch on sample quality, alignment, and stability is not studied.
  • Sparse terminal rewards: The framework relies on a single terminal image-level reward; credit assignment across text tokens and denoising steps remains underexplored. There is no investigation of step-wise or modality-wise credit shaping, nor of process reward models actually implemented.
  • Reward model dependence and Goodhart risk: Experiments use a single differentiable reward model (InternVL-based) similar in flavor to the VLM-based evaluator. Generalization across unseen, independent rewards and susceptibility to over-optimization (reward hacking against the evaluator) are not assessed.
  • Lack of human evaluation: No human preference studies are reported, making it unclear whether gains on VLM-based metrics translate to human judgments of quality, faithfulness, and aesthetics.
  • External benchmark breadth: Evaluation is limited to an internal TA set and GenEval. Performance on broader public benchmarks (e.g., T2I-CompBench, DrawBench, Pick-a-Pic) and in diverse styles (photorealistic, artistic, diagrams) is not reported.
  • Diversity and mode collapse: The impact of reinforcement learning on sample diversity is not measured (e.g., precision–recall, LPIPS diversity), leaving open whether improvements trade off against variety.
  • Safety, bias, and content moderation: There is no analysis of harmful content generation, demographic or aesthetic biases introduced by the reward, or the effect of RL on safety filters.
  • Joint credit allocation between modalities: The unified objective uses a fixed λ=1. There is no sensitivity analysis of the text–image weighting, nor exploration of adaptive or learned weighting strategies.
  • Text-side regularization: TextGRPO’s KL weight is set to 0. The risk of language drift, verbosity, repetition, or degraded instruction-following in text is not assessed; no intrinsic language metrics are reported.
  • Reasoning quality and veracity: The approach optimizes for image-level rewards without verifying the factual or logical correctness of “thinking” tokens. There is no evaluator for reasoning quality or its causal impact on image quality.
  • Velocity MSE regularization: While MSE on the velocity fields stabilizes training, there is no theoretical analysis of why it outperforms latent KL, nor a systematic study of its sensitivity to the noise schedule, timestep distribution, or model architecture.
  • RatioNorm and clipping: The interplay between RatioNorm, importance-ratio clipping, and the new MSE penalty is not ablated. It remains unclear which components are necessary/sufficient for stability across different schedulers and timesteps.
  • SDE window design: The choice of SDE window size, placement, and noise level is fixed; there is no study of how these affect bias/variance of gradients, sample efficiency, or final image quality.
  • Reward hacking diagnostics: Evidence of reward hacking is shown qualitatively, but no quantitative, systematic diagnostics (e.g., cross-reward evaluation, adversarial prompts, evaluator-agnostic metrics) are provided.
  • Generalization to other base models: Results are tied to an internally fine-tuned Bagel model. It is unknown how UniGRPO transfers to other unified architectures or to mainstream T2I backbones (e.g., SDXL/Flux/PixArt).
  • Reproducibility constraints: Key assets (training data, reward model, SFT recipe, code) are not released, limiting independent verification and raising questions about sensitivity to dataset curation and preprocessing.
  • Compute and efficiency: The paper does not quantify training/inference compute, memory footprint, or throughput, especially compared to CFG-based pipelines in multi-round scenarios.
  • Multi-modal extension beyond images: Although framed as “unified,” the work only covers text and images. Extending UniGRPO to video, audio, or 3D remains untested.
  • Robustness to OOD prompts: Performance under long, ambiguous, or adversarial prompts (e.g., conflicting attributes, rare concepts, multilingual instructions) is not evaluated.
  • Baseline coverage: Comparisons exclude modern preference optimization baselines for diffusion (e.g., DPO variants tailored to flow models) or recent RL methods beyond FPO/FlowGRPO, leaving open how UniGRPO stacks up against the latest alternatives.
  • Catastrophic forgetting: There is no evaluation of whether RL fine-tuning harms other capabilities (e.g., general language utility, zero-shot image skills). Retention of base model priors is asserted but not measured across tasks.
  • Process Reward Models (PRMs): Although proposed as future work, no concrete design, annotation strategy, or verifier architecture is offered for multimodal PRMs that assess reasoning steps and their alignment with visual outcomes.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

The following applications can be deployed now by leveraging the unified GRPO training recipe, its CFG-free rollout, and velocity-based regularization to improve reasoning-driven image generation.

  • Unified post-training to boost text-image alignment in production T2I systems
    • Sectors: advertising/marketing, e-commerce, media/entertainment, design tooling, software platforms (APIs, creative SaaS)
    • What to build: a post-training pipeline that jointly optimizes the LLM’s “thinking” and the flow model’s synthesis using UniGRPO; integrate differentiable (or black-box) reward models that score prompt adherence, aesthetic quality, and safety
    • Dependencies/assumptions: access to a unified base model (e.g., Bagel-like) that supports interleaved reasoning and flow-based image generation; reliable domain-specific reward models or verifier ensembles; GPU budget for group sampling and hybrid SDE-ODE training
  • Cost- and latency-reduction via CFG-free inference
    • Sectors: cloud inference providers, consumer creative apps, edge/embedded deployment
    • What to build: switch inference defaults to CFG-free, backed by UniGRPO post-training to internalize prompt adherence; use linear, unbranched rollouts to reduce per-step evaluations and memory
    • Dependencies/assumptions: sufficient reward coverage during RL so that removing CFG does not degrade prompt adherence; regression tests on key alignment metrics
  • Brand/style and policy compliance with reward-driven alignment
    • Sectors: enterprise creative suites, marketplaces, regulated industries (e.g., finance for brand guidelines; public sector)
    • What to build: reward models and/or rule-based verifiers that enforce brand palettes, compositions, safety guidelines, and logo usage; plug them into UniGRPO for post-training; deploy compliance “scorers” at generation-time for audit trails
    • Dependencies/assumptions: curated preference data reflecting brand or policy standards; ongoing calibration of verifiers to avoid false positives/negatives
  • User-facing “thinking-to-image” workflows for prompt debugging and control
    • Sectors: prosumer creative tools, design agencies, education platforms
    • What to build: UI that surfaces and optionally edits the model’s reasoning chain before synthesis; presets that steer thoughts (e.g., composition-first, color-first)
    • Dependencies/assumptions: safe handling of reasoning visibility (e.g., avoid leaking sensitive internal prompts); minimal friction in UX to keep non-experts engaged
  • Safer RL fine-tuning through velocity-based regularization and RatioNorm
    • Sectors: AI safety/alignment teams across industry and academia
    • What to build: training guards that replace latent KL with velocity MSE penalties and normalize importance ratios (RatioNorm) to mitigate reward hacking; include canary prompts and adversarial reward checks
    • Dependencies/assumptions: careful hyperparameter tuning for MSE weighting; monitoring to detect regressions (e.g., oversaturation, grid artifacts)
  • High-fidelity synthetic data generation with strong compositional control
    • Sectors: computer vision (data augmentation), robotics simulation, retail/e-commerce (catalog variants)
    • What to build: controlled data factories that use exam-point-like reward checks (counts, attributes, spatial relations) to produce labeled images aligned to task specs
    • Dependencies/assumptions: task-specific reward designs (e.g., object detectors for counting, VLMs for attribute checks); license and IP policies for synthetic data use
  • Benchmarking and internal evaluation with “exam-point” scoring
    • Sectors: AI research/engineering orgs, product evaluation teams
    • What to build: TA-style evaluation where each prompt has explicit exam points (binary checks) scored by a VLM or verifier ensemble; integrate into CI for A/B testing of models and rewards
    • Dependencies/assumptions: trustworthy evaluators (VLMs/verifiers) that generalize; prompt pools representative of production use
  • Research baselines and teaching materials for unified, interleaved RL
    • Sectors: academia and industrial research labs
    • What to build: reproducible training recipes combining Text GRPO and FlowGRPO-Fast with hybrid SDE sampling; ablation kits on CFG-free training and velocity MSE; open benchmarks mapping reasoning quality to final images
    • Dependencies/assumptions: access to a unified backbone (or modular equivalent), and permissive datasets for SFT and RL

Long-Term Applications

These applications require further research, scaling, or ecosystem development, particularly around multi-round interleaving and process reward models.

  • Multi-round creative agents for iterative image editing and visual planning
    • Sectors: creative suites, AR/VR content creation, game/film previsualization, e-commerce product staging
    • What to build: agents that reason → generate → reflect → edit across multiple turns; support multi-condition inputs (masks, reference images, style prompts) without CFG branching; session memory and credit assignment across turns
    • Dependencies/assumptions: high-quality multi-turn datasets; stable credit assignment across long horizons; scalable training and memory mechanisms
  • Multimodal Process Reward Models (PRMs) that critique reasoning steps
    • Sectors: AI safety/alignment, education/assistive writing tools, enterprise content governance
    • What to build: PRMs that score intermediate thoughts for logical soundness, faithfulness to the visual intent, and safety before synthesis; integrate as dense rewards for sample-efficient RL
    • Dependencies/assumptions: annotated process-level datasets; robust PRM generalization and calibration; safeguards against PRM gaming
  • Unified RL post-training “as-a-service” platforms
    • Sectors: cloud AI providers, MLOps vendors
    • What to build: managed services where customers plug in reward functions (aesthetic, safety, brand), datasets, and base models to obtain tuned, interleaved generators; dashboards for reward diagnostics and failure modes
    • Dependencies/assumptions: standard reward APIs and verifiers; customer-controlled data governance; cost-optimized training/inference stacks
  • Extension to video, 3D, and scene-graph conditioned generation
    • Sectors: media/entertainment, simulation/robotics, digital twins, architecture/industrial design
    • What to build: UniGRPO-like training for spatiotemporal or 3D flows; reasoning that plans temporal beats or object layouts; rewards for temporal coherence, physical plausibility, and camera control
    • Dependencies/assumptions: scalable flow-based backbones for video/3D; multi-aspect reward models; significant compute for long horizons
  • On-device or low-resource creative assistants leveraging linear rollouts
    • Sectors: mobile apps, XR devices, embedded systems
    • What to build: compact unified models trained to internalize alignment (no CFG) and operate with few steps; optional server-offload for heavy edits
    • Dependencies/assumptions: model compression/distillation compatible with flow matching; efficient schedulers and memory use
  • Domain-specific controlled synthesis (e.g., healthcare, scientific visualization)
    • Sectors: healthcare, life sciences, manufacturing QA
    • What to build: tightly constrained generators for data augmentation or didactic visuals (e.g., cell morphologies, instrument configurations) governed by domain verifiers and safety rules
    • Dependencies/assumptions: stringent ethical, regulatory, and validation requirements; expert-crafted rewards; synthetic-to-real gap studies
  • Education content generation with precise compositional constraints
    • Sectors: EdTech, publishing
    • What to build: pipelines that turn curricular “exam points” into reward functions, ensuring exact counts, attributes, and spatial relations in illustrations, worksheets, and interactive content
    • Dependencies/assumptions: curriculum-aligned prompt libraries; verifiers tuned for age and subject; watermarking/provenance
  • Standards and policy for verifier-based alignment and auditability
    • Sectors: public policy, standards bodies, enterprise governance
    • What to build: frameworks to certify reward models/verifiers (bias, robustness, failure modes); logging and audit trails linking generated images to thought traces and reward decisions
    • Dependencies/assumptions: consensus on metrics and reporting; privacy-preserving logging; legal clarity on chain-of-thought exposure
  • Synthetic data factories with controllable diversity and guarantees
    • Sectors: computer vision, autonomy, retail, robotics
    • What to build: pipelines that parameterize “exam points” at scale (attributes, lighting, layouts) and automatically generate balanced datasets with documented reward scores
    • Dependencies/assumptions: coverage metrics for diversity; contracts on data usage/IP; monitoring for distribution drift
  • Tooling for multi-condition generation workflows at scale
    • Sectors: design systems, e-commerce, UGC moderation
    • What to build: editors and APIs that support masks, sketch/reference images, and textual plans in one linear rollout; templating for repeatable brand-safe compositions
    • Dependencies/assumptions: robust conditioning interfaces in the base model; dataset curation for multi-condition tasks; guardrails to prevent misuse

Notes on feasibility across applications:

  • The strongest immediate gains occur when reliable reward models exist for the target domain and when a unified backbone (AR text + flow image) is available or can be approximated.
  • Removing CFG in production is viable after alignment training; skipping this step risks prompt-adherence regressions.
  • For multi-round settings, credit assignment and PRMs are critical dependencies; without them, scaling beyond single-round use cases will be fragile.
  • Safety, IP, and regulatory considerations can be integrated as rewards and verifiers, but require continuous monitoring to avoid reward hacking or overfitting to narrow evaluators.
View Paper Prompt  View All Prompts 

Glossary

  • Advantage Weighted Regression (AWR): An offline RL-style method that weights supervised updates by advantages to align policies with high-reward behavior. "FPO / AWR serves as an alternative to FlowGRPO."
  • Autoregressive (AR) models: Models that generate sequences by predicting the next token conditioned on previous tokens, commonly used for text. "Autoregressive (AR) models for text generation paired with Flow Matching for visual synthesis"
  • Chain-of-Thought (CoT): A prompting/training technique where models generate intermediate reasoning steps to improve problem-solving. "using Chain-of-Thought (CoT)"
  • Classifier-Free Guidance (CFG): A generation technique that blends conditional and unconditional model outputs to improve prompt adherence. "First, we eliminate Classifier-Free Guidance (CFG) during training."
  • credit assignment: The RL challenge of attributing a final reward to earlier decisions or actions in a trajectory. "This can lead to inefficient credit assignment"
  • Direct Preference Optimization (DPO): A preference-learning method that directly optimizes a model to prefer preferred outputs without explicit reward models. "Currently, Direct Preference Optimization (DPO)~\cite{rafailov2024direct, wallace2024diffusion, videoalign, yang2024using, liang2024step, yuan2024self, liu2024videodpo, zhang2024onlinevpo, furuta2024improving,liang2025aesthetic} and PPO-style policy gradients~\cite{schulman2017proximal, black2023training, fan2024reinforcement, gupta2025simple, miao2024training, zhao2025score} have become standard frameworks for fine-tuning diffusion models, alongside various training-free guidance methods~\cite{yeh2024training, tang2024tuning, song2023loss}."
  • Evidence Lower Bound (ELBO): A variational objective that lower-bounds the log-likelihood, often used as a training surrogate in latent-variable models. "uses the Evidence Lower Bound (ELBO) of the denoising process as a surrogate for log⁡pθ(x0∣c)\log p_\theta(x_0|c) to compute importance sampling weights."
  • Flow Matching: A generative modeling approach that learns a continuous-time velocity field to transform noise into data. "the community is increasingly gravitating toward a robust architectural synergy: Autoregressive (AR) models for text generation paired with Flow Matching for visual synthesis"
  • FlowGRPO: A GRPO-based reinforcement learning method adapted to flow matching models via stochastic reformulation for exploration. "Combining the hybrid SDE sampling strategy with the RatioNorm mechanism, the final FlowGRPO objective is computed exclusively over the SDE timestep subset"
  • FlowGRPO-Fast: A computationally efficient variant of FlowGRPO that applies SDE-based training on a subset of steps and ODE sampling elsewhere. "For training efficiency, we adopt the FlowGRPO-Fast variant~\cite{liu2025flow}, which employs a hybrid sampling strategy."
  • FPO: A flow-based policy optimization approach that leverages the forward process and ELBO-derived weights instead of SDE-based exploration. "FPO utilizes the forward process to obtain xtx_t and uses the Evidence Lower Bound (ELBO) of the denoising process as a surrogate for log⁡pθ(x0∣c)\log p_\theta(x_0|c) to compute importance sampling weights."
  • GenEval: A benchmark evaluating compositional text-to-image generation capabilities like counting and spatial relations. "GenEval~\cite{geneval}: A standard benchmark assessing Text-to-Image models on complex compositional capabilities, including object counting, spatial relations, and attribute binding."
  • Group Relative Policy Optimization (GRPO): A PPO-like algorithm that replaces a value model with group-relative baselines to compute advantages efficiently. "the highly efficient Group Relative Policy Optimization (GRPO)~\cite{grpo} eliminates the value model by using group-relative baselines."
  • importance sampling clipping: A stabilization technique that clips importance weights in policy gradient updates to reduce variance and prevent large policy shifts. "The optimization objective maximizes the expected reward while constraining the policy update via importance sampling clipping."
  • Kullback–Leibler (KL) divergence: A measure of divergence between two probability distributions, commonly used as a regularizer to constrain policy updates. "For KL divergence on the latents, the significant drop in training reward indicates that a sufficiently large KL coefficient has been used, yet grid-like artifacts still emerge as early as step 250, prompting us to terminate this run early."
  • Markov Decision Process (MDP): A formal framework for sequential decision-making defined by states, actions, transitions, and rewards. "we propose UniGRPO, a unified RL framework formulating the entire "Prompt →\rightarrow Thinking →\rightarrow Image" sequence as a single Markov Decision Process (MDP)"
  • Ordinary Differential Equation (ODE): A deterministic differential equation used here to describe the standard flow-based generative trajectory. "converting the deterministic Ordinary Differential Equation (ODE) into a Stochastic Differential Equation (SDE) to enable exploration."
  • Process Reward Models (PRMs): Reward models that evaluate intermediate steps (e.g., reasoning tokens) to provide dense process-level feedback. "Multimodal Process Reward Models (PRMs)"
  • Proximal Policy Optimization (PPO): A popular RL algorithm that performs clipped policy updates to maintain stability. "While PPO~\cite{schulman2017proximal} is a standard approach,"
  • Ratio Normalization (RatioNorm): A technique to normalize log-importance ratios to stabilize clipping bounds and prevent reward hacking. "we adopt the Ratio Normalization (RatioNorm) proposed in GRPO-Guard~\cite{wang2025grpo}."
  • ReFL: A reward-model-based fine-tuning approach that backpropagates differentiable reward signals into diffusion/flow models. "ReFL directly fine-tunes diffusion models by viewing reward model scores as human preference losses and back-propagating gradients to a randomly-picked late timestep tt."
  • Reward Weighted Regression (RWR): A supervised-learning formulation of RL that weights regression targets by rewards or advantages. "and Reward Weighted Regression (RWR)~\cite{peng2019advantage, fan2025online, lee2023aligning,dong2023raft}."
  • reward hacking: Undesired exploitation of the reward function that increases scores while degrading true quality or alignment. "providing a more robust and direct regularization signal to mitigate reward hacking effectively."
  • sparse terminal rewards: A reward scheme where only the final outcome is rewarded, with zero intermediate rewards. "Formulating this multimodal generation process as a Markov Decision Process with sparse terminal rewards"
  • Stochastic Differential Equation (SDE): A differential equation with stochastic terms used to inject exploration noise into the generation process. "converting the deterministic Ordinary Differential Equation (ODE) into a Stochastic Differential Equation (SDE) to enable exploration."
  • Text-to-Image (T2I): The task of generating images conditioned on textual prompts. "T2I qualitative comparison."
  • unbranched rollouts: Linear generation trajectories without branching contexts, simplifying computation and gradient estimation. "eliminating Classifier-Free Guidance (CFG) to ensure unbranched rollouts"
  • UniGRPO: The paper’s unified GRPO-based framework that jointly optimizes reasoning (text) and synthesis (image) within one MDP. "we propose UniGRPO, a unified RL framework formulating the entire "Prompt →\rightarrow Thinking →\rightarrow Image" sequence as a single Markov Decision Process (MDP)"
  • velocity fields: The learned vector fields that define how samples move through time in flow matching generative models. "replacing the standard latent KL penalty with an MSE penalty directly on the velocity fields"
  • Visual LLM (VLM): A multimodal model that can evaluate or reason over images and text jointly. "Evaluation is performed by a VLM, which assesses the outputs against multiple specific exam points defined for each prompt."
View Paper Prompt  View All Prompts 

Open Problems

We found no open problems mentioned in this paper.

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 7 tweets with 99 likes about this paper.

Sign Up for Free