UniRL-Zero: Unified Multimodal RL

• The paper introduces UniRL-Zero as a unified reinforcement learning framework that couples a multimodal language model with a diffusion model to handle mixed discrete and continuous actions.
• It employs Group Relative Policy Optimization across six scenarios, enabling end-to-end training for tasks like text-to-image generation and reflective image refinement.
• Empirical results show enhanced GenEval scores and multimodal benchmarks, demonstrating the framework’s potential for robust language reasoning and image synthesis.

Searching arXiv for papers directly relevant to "UniRL-Zero" and closely related usages. UniRL-Zero denotes, in its explicit named usage, a unified reinforcement learning framework in which a multimodal LLM and a diffusion model are optimized within a single end-to-end policy. The framework allows one agent to act in a mixed action space—discrete text tokens for understanding and reasoning, and continuous denoising steps for multimedia generation—and defines six reinforcement learning scenarios spanning pure text reasoning, multimodal reasoning, text-to-image generation, instructional image editing, chain-of-thought-enhanced generation, and reflective image generation (Wang et al., 20 Oct 2025). In adjacent literature, the same label has also been used more broadly for unified zero-RL paradigms that combine verifiable and non-verifiable rewards, self-generated curricula, or language priors with planning systems (Zeng et al., 29 Oct 2025).

1. Nomenclature and conceptual scope

The primary, formal use of the name appears in "UniRL-Zero: Reinforcement Learning on Unified Models with Joint LLM and Diffusion Model Experts" (Wang et al., 20 Oct 2025). There, UniRL-Zero refers specifically to a multimodal RL framework that couples a multimodal LLM (LM) and a diffusion model (DM) inside a single policy, with the stated goal of improving multimodal understanding, diffusion-based multimedia generation, and their interaction capabilities.

The term also has a secondary, looser usage in nearby work. "Zero Reinforcement Learning Towards General Domains" describes a framework called General Zero-RL and maps it directly to UniRL-Zero because it unifies zero-RL training across verifiable and non-verifiable domains under a single policy, reward, and training loop (Zeng et al., 29 Oct 2025). "Absolute Zero: Reinforced Self-play Reasoning with Zero Data" presents Absolute Zero Reasoner as a unified, zero-data RLVR system with one policy, two roles, one environment, and one verifiable reward source, and explicitly states that this maps onto what one might call UniRL-Zero (Zhao et al., 6 May 2025). "SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild" likewise uses UniRL-Zero as a shorthand for a unified zero-RL recipe for open base models (Zeng et al., 24 Mar 2025).

By contrast, some adjacent papers reject the name as an official designation while still resembling parts of the same conceptual space. "UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learning" states that it does not introduce a variant named UniRL-Zero, though it emphasizes zero reliance on external image data during post-training (Mao et al., 29 May 2025). "RLZero: Direct Policy Inference from Language Without In-Domain Supervision" similarly notes that UniRL-Zero is not an official alias, even though RLZero can be viewed as a unified zero-shot RL paradigm mapping language to behavior without in-domain supervision (Sikchi et al., 2024). A plausible implication is that the phrase now denotes both a specific multimodal RL system and a broader family resemblance among unified RL methods that minimize supervised task-specific scaffolding.

2. Unified architecture and mixed action spaces

In the named UniRL-Zero framework, the base unified model integrates a frozen multimodal LM, a diffusion expert, trainable meta-query tokens, and a bidirectional connector transformer (Wang et al., 20 Oct 2025). The LM is Qwen2.5-VL-Instruct and provides multimodal understanding, token-level reasoning, and prompt or feedback generation. The DM is SANA-1.6B, a linear transformer trained in the latent space of a DC-VAE encoder for image synthesis and editing. Two lightweight sets of query vectors are specialized for text-to-image and image editing.

The architectural coupling is asymmetric but explicit. Meta-query tokens extract task-specific features from LM hidden states, and the connector transformer feeds those features to the DM via cross-attention. For editing, the DM input layer is expanded to accept reference image latents; for text-to-image, those channels are zeroed. This yields a unified token-latent interface in which the LM produces structured text, the connector extracts conditioning features, and the DM consumes those features to guide latent-space sampling.

The action space is mixed by design. The LM acts through discrete text tokens $a_t$ from a vocabulary, while the DM acts through continuous per-step denoising transitions. The diffusion process uses a linear parameterization

$\mathbf{x}_t = (1 - t)\mathbf{x}_0 + t\,\boldsymbol{\epsilon}, \qquad \boldsymbol{v} = \boldsymbol{\epsilon} - \mathbf{x}_0,$

with conditioning supplied by LM-derived context through cross-attention (Wang et al., 20 Oct 2025). The paper’s core claim is not merely modular coexistence, but a single policy over both modalities.

3. Joint reinforcement learning formulation

UniRL-Zero defines a joint policy

$\pi_\theta = \{\pi_{\theta_{\text{LM}}}, \pi_{\theta_{\text{DM}}}\},$

and interleaved multimodal trajectories

$\tau = (Q, \tau_{\text{LM}}, \tau_{\text{DM}}, \tau'_{\text{LM}}, \tau'_{\text{DM}}, \ldots),$

where $Q$ is the input query, $\tau_{\text{LM}}$ is the token sequence, and $\tau_{\text{DM}}$ is the denoising trajectory (Wang et al., 20 Oct 2025). The objective is the standard expected return,

$J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\left[\sum_t r_t\right].$

Optimization is performed with Group Relative Policy Optimization (GRPO). For each query, the framework samples a group of trajectories, computes per-trajectory rewards, normalizes them to advantages, and assigns the same normalized advantage to all actions in that rollout, including both LM tokens and DM steps. The LM and DM are updated through separate clipped surrogates, averaged over their respective actions, with KL regularization to a reference policy:

$\mathcal{L}_{\text{total}} = \mathcal{J}_{\text{clip-MLLM}} + \mathcal{J}_{\text{clip-Diffusion}} + \beta \,\mathbb{E}\!\left[D_{\text{KL}}(\pi_\theta \,\|\, \pi_{\text{ref}})\right].$

The paper contrasts this with PPO variants that would require a learned value function and generalized advantage estimation, and states that UniRL-Zero instead relies on GRPO’s group normalization for stability in the unified LM+DM setting (Wang et al., 20 Oct 2025).

A central technical point is that the DM’s continuous denoising process is treated as a policy with per-step likelihood ratios. The framework uses a Gaussian transition view of diffusion steps so that score-function estimators can be applied over denoising actions. This allows the LM branch and DM branch to participate in a single RL optimization despite their different action geometries.

4. Six reinforcement learning scenarios

The framework defines six scenarios for unified model reinforcement learning, presented as a systematic baseline suite rather than a single task family (Wang et al., 20 Oct 2025).

Text understanding and reasoning uses the LM alone on text-to-text episodes. Rewards measure correctness, helpfulness, and reasoning depth, and RL acts only on discrete token sequences.

Multimodal reasoning uses image-plus-text inputs and text outputs. The DM is inactive, and rewards measure performance on multimodal benchmarks such as MME-P, MMB, SEED, MMMU, and MM-Vet.

Text-to-image generation conditions the DM on LM-derived context. UniRL-Zero demonstrates reward shaping with JPEG compressibility or incompressibility and GenEval scores, while RL optimizes the DM’s denoising policy and, where applicable, the LM-side context generation.

Instructional image editing introduces a two-stage forward-edit and reverse-edit episode structure. The reported reward includes edit alignment, cycle consistency, and visual quality:

$R(\tau) = \lambda_1 R_{\text{edit}}(\mathbf{x}_{\text{edit}}, I_{\text{edit}}) + \lambda_2 R_{\text{cycle}}(\mathbf{x}_{\text{ref}}, \mathbf{x}_{\text{cycle}}) + \lambda_3 R_{\text{quality}}(\mathbf{x}_{\text{edit}}),$

where $\mathbf{x}_t = (1 - t)\mathbf{x}_0 + t\,\boldsymbol{\epsilon}, \qquad \boldsymbol{v} = \boldsymbol{\epsilon} - \mathbf{x}_0,$0 is based on CLIP similarity between the reference and cycle-reconstructed images (Wang et al., 20 Oct 2025).

CoT-enhanced text-to-image generation inserts an explicit reasoning stage: text $\mathbf{x}_t = (1 - t)\mathbf{x}_0 + t\,\boldsymbol{\epsilon}, \qquad \boldsymbol{v} = \boldsymbol{\epsilon} - \mathbf{x}_0,$1 text reasoning $\mathbf{x}_t = (1 - t)\mathbf{x}_0 + t\,\boldsymbol{\epsilon}, \qquad \boldsymbol{v} = \boldsymbol{\epsilon} - \mathbf{x}_0,$2 image. The LM first generates structured reasoning and refined prompts, meta-query tokens extract context from LM hidden states, and the DM generates images conditioned on that reasoning. The paper reports a cold-start fine-tuning stage of the DM on 10k CoT-augmented pairs with the LM frozen, followed by RL.

Reflective image generation extends this into an iterative loop: text $\mathbf{x}_t = (1 - t)\mathbf{x}_0 + t\,\boldsymbol{\epsilon}, \qquad \boldsymbol{v} = \boldsymbol{\epsilon} - \mathbf{x}_0,$3 image $\mathbf{x}_t = (1 - t)\mathbf{x}_0 + t\,\boldsymbol{\epsilon}, \qquad \boldsymbol{v} = \boldsymbol{\epsilon} - \mathbf{x}_0,$4 text reflection $\mathbf{x}_t = (1 - t)\mathbf{x}_0 + t\,\boldsymbol{\epsilon}, \qquad \boldsymbol{v} = \boldsymbol{\epsilon} - \mathbf{x}_0,$5 image. The LM judges image errors, produces corrective feedback, and the DM refines the image. Rewards include correctness of LM judgments, required output format, and improved GenEval scores of corrected images (Wang et al., 20 Oct 2025).

Taken together, these scenarios make UniRL-Zero a benchmarked framework for coupling understanding and generation under one RL formalism rather than a narrowly task-specific training recipe.

5. Diffusion policy mechanics, cross-modal coupling, and training pipeline

UniRL-Zero’s diffusion branch is based on a flow-matching-style linear diffusion parameterization with $\mathbf{x}_t = (1 - t)\mathbf{x}_0 + t\,\boldsymbol{\epsilon}, \qquad \boldsymbol{v} = \boldsymbol{\epsilon} - \mathbf{x}_0,$6-prediction. Timesteps are drawn via

$\mathbf{x}_t = (1 - t)\mathbf{x}_0 + t\,\boldsymbol{\epsilon}, \qquad \boldsymbol{v} = \boldsymbol{\epsilon} - \mathbf{x}_0,$7

and the rollout sampler uses an SDE-based step that first adds forward noise and then denoises with an Euler-like update (Wang et al., 20 Oct 2025). The paper gives the Gaussian transition

$\mathbf{x}_t = (1 - t)\mathbf{x}_0 + t\,\boldsymbol{\epsilon}, \qquad \boldsymbol{v} = \boldsymbol{\epsilon} - \mathbf{x}_0,$8

with

$\mathbf{x}_t = (1 - t)\mathbf{x}_0 + t\,\boldsymbol{\epsilon}, \qquad \boldsymbol{v} = \boldsymbol{\epsilon} - \mathbf{x}_0,$9

This formulation is used to obtain per-step log probabilities and thus the DM likelihood ratios required by GRPO.

Cross-modal interaction is mediated by meta-query tokens and cross-attention connectors. LM-generated prompts, CoT traces, or reflection outputs are converted into DM conditioning vectors; image outputs can then be re-ingested by the LM for critique or reasoning. Rewards combine CLIP-based similarities, GenEval compositional metrics, cycle consistency terms, and simple quality measures such as JPEG compressibility. The same normalized rollout advantage is assigned to LM tokens and DM denoising steps, which is the mechanism by which improvements in one modality can propagate to the other (Wang et al., 20 Oct 2025).

The training pipeline begins with large-scale supervised pretraining concentrated on image generation and editing while keeping the LM frozen. The reported corpora include text-to-image-2M and flux_generated, amounting to approximately 3.5M pairs before top-50% PickScore filtering, plus around 50k Flux-generated images from GenEval prompts filtered by GenEval criteria; the paper later summarizes total pretraining as approximately 1.8M image-text pairs and approximately 1.3M editing pairs, trained for 10 epochs (Wang et al., 20 Oct 2025). Editing pretraining uses OmniEdit, augmented with 100k high-PickScore flux_generated images, Claude-generated creative editing instructions, and edited outputs by Flux-Kontext. Two targeted cold starts follow: 10k CoT-augmented pairs for CoT-enhanced text-to-image generation, and roughly 10k reflection-augmented pairs for reflective generation.

Stabilization relies on KL control to a reference model, advantage normalization across grouped rollouts, clipped surrogates for each module, and reward normalization. The paper characterizes these as practical rather than theoretically exhaustive design choices, emphasizing reproducibility and simplicity over exhaustive hyperparameter exploration (Wang et al., 20 Oct 2025).

6. Reported results, limitations, and broader significance

The base unified model reports an overall GenEval score of 0.69 at $\pi_\theta = \{\pi_{\theta_{\text{LM}}}, \pi_{\theta_{\text{DM}}}\},$0 resolution, outperforming PixArt-$\pi_\theta = \{\pi_{\theta_{\text{LM}}}, \pi_{\theta_{\text{DM}}}\},$1 at 0.48, PixArt-$\pi_\theta = \{\pi_{\theta_{\text{LM}}}, \pi_{\theta_{\text{DM}}}\},$2 at 0.52, LUMINA-Next at 0.46, and SDXL at 0.55, while being competitive with the FLUX family (Wang et al., 20 Oct 2025). On multimodal reasoning with a Qwen2.5-VL 3B backbone, the same model reports MM-Vet 63.2, MME-P 1574.3, and SEED 73.8.

For RL on text-to-image generation, the paper reports a progression on 50k random GenEval prompts from Base 0.69 to T2I-RL 0.80 and then to CoT-enhanced T2I-RL 0.85 overall (Wang et al., 20 Oct 2025). The gains are described as particularly strong in counting, two-object composition, and color attribution, with CoT-enhanced RL further improving positional accuracy. Instructional image editing is evaluated with a curated set of 200 instruction and reverse-instruction pairs and is reported to show a favorable trade-off between instruction following and reference similarity. Reflection-based generation uses approximately 1.5k data points and is reported to improve LM judge accuracy and correction rates, alongside qualitative improvements in iterative refinement.

The paper identifies several limitations. Reward bias remains a primary issue because CLIP alignment and GenEval metrics do not capture all aspects of geometry, long-range coherence, or fine-grained attributes. Scale constraints are explicit: the work uses modest data volume, model size, and training duration relative to proprietary systems. Safety mechanisms are also limited; the framework does not include explicit content safety filters or harmful-content avoidance policies, and reflection loops may amplify LM judgment errors or biases if the reward design is weak (Wang et al., 20 Oct 2025).

In the broader literature, the significance of UniRL-Zero lies partly in how it intersects with other unified RL trends. General Zero-RL extends zero-RL beyond verifiable domains by combining binary rewards with a generative reward model and a smooth length penalty under a single multi-task policy (Zeng et al., 29 Oct 2025). Absolute Zero demonstrates a zero-data regime in which one policy both proposes and solves tasks using only an executable verifier (Zhao et al., 6 May 2025). PriorZero, although not named UniRL-Zero, is explicitly described as a UniRL-Zero-style framework that unifies LLM semantic priors with Zero-family world-model planning by injecting language priors only at the MCTS root and separating rollout from training (Xiong et al., 12 May 2026). This suggests that UniRL-Zero has become both a concrete multimodal RL system and a useful umbrella term for unified reinforcement learning systems that merge heterogeneous experts, reward sources, or planning components while minimizing task-specific supervision.