Omni-R1-Zero: RL for Multimodal Reasoning

• Omni-R1-Zero is a reinforcement learning-based paradigm that leverages synthetic supervision and group-based policy optimization for robust multimodal reasoning.
• It unifies vision, text, and audio modalities using flexible architectures and cost-efficient, text-only training protocols.
• Empirical results demonstrate that Omni-R1-Zero outperforms supervised benchmarks through innovative RL fine-tuning and synthetic data construction.

Omni-R1-Zero is a family of reinforcement learning (RL)–driven LLM training paradigms that eliminate the need for costly multimodal annotation while achieving state-of-the-art performance across diverse reasoning tasks. Encompassing both generative multimodal models (vision, text) and multimodal LLMs (MLLMs) with additional modalities such as audio, Omni-R1-Zero unifies previously domain-specific “R1-Zero” optimization with architectural flexibility, data efficiency, and adaptability across modalities. Central to these systems are group-based policy optimization algorithms (notably Group Relative Policy Optimization, GRPO, and refinements) and a data regime that allows fully synthetic or text-only supervision to suffice for strong cross-modal generalization (Cheng et al., 14 Jan 2026, Rouditchenko et al., 14 May 2025, Liu et al., 26 Mar 2025).

1. Unified Foundations and Motivation

Omni-R1-Zero addresses two key bottlenecks in scaling RL for LLMs: (1) the annotation and computational cost of multimodal (e.g., image–text or audio–text) traces, and (2) the need for generalizable reasoning skills beyond single-task or modality-constrained settings. Building upon the R1-Zero framework, which demonstrated that RL at scale can directly enhance reasoning in LLMs without supervised multimodal fine-tuning, Omni-R1-Zero extends this paradigm into a generalized, architecture-agnostic setting (Liu et al., 26 Mar 2025).

Across instantiations, Omni-R1-Zero reuses the core “omni” model backbone (such as Anole or Qwen2.5-Omni architectures) and applies two-stage training: (1) a supervised pretraining or alignment phase with synthetic or text-only data, followed by (2) group-based RL, typically leveraging group-level comparison among sampled model outputs for unbiased advantage estimation and robust reward shaping (Cheng et al., 14 Jan 2026, Rouditchenko et al., 14 May 2025).

2. Architectural Paradigms and Trajectory Generation

Multimodal Generative Reasoning (Vision-Text)

The canonical vision-text instantiation uses an autoregressive transformer backbone interfacing with a frozen quantized VQVAE codebook $E \in \mathbb{R}^{K \times D}$ for image segments. The model emits a stream of alternated text and image tokens; actions use a unified template (e.g., ZOOM-in, BBOX, MARK, LINE, PRED), each associated with numeric arguments and control tokens. A renderer $R$ processes each (image-token, control token) pair to produce the next visual state, enabling “functional” image reasoning (e.g., zooming or annotating regions) (Cheng et al., 14 Jan 2026).

Each multimodal trajectory for input $(x_M, x_T)$ takes the form:

$$(\text{rat}_1^T, a_1, \text{rat}_1^M), ..., (\text{rat}_L^T, a_L, \text{rat}_L^M), \text{Ans}.$$

This sequence is autoregressively generated, supporting complex step-wise reasoning over both visual and linguistic information.

Audio–Text and Modular Multimodality

In the audio–text domain, the core model (e.g., Qwen2.5-Omni-7B) augments a text transformer with modality-specific encoders (for audio, vision), aligning all token sequences into a shared embedding space. For audio, convolutional feature extractors transform waveforms or spectrograms into token embeddings concatenated with textual sequences, permitting flexible inclusion or omission of modalities at inference (Rouditchenko et al., 14 May 2025).

3. Optimization Algorithms: GRPO and Variants

The underlying RL method is Group Relative Policy Optimization (GRPO), which eschews a value head or learned critic and instead operates by grouping sampled trajectories per question. For question $q$, $G$ completions $\{\tau_i\}_{i=1}^G$ are generated under the old policy $\pi_{\theta_\text{old}}$. Returns $R_i$ are assigned by rule-based reward mechanisms (e.g., 0/1 for correctness, or composite weighted sums in vision settings). The (per-token) advantage is computed as:

$$\hat{A}_{i,t} = \frac{R_i - \mu_G}{\sigma_G}, \quad \text{where } \mu_G = \mathrm{mean}_G,\, \sigma_G = \mathrm{std}_G.$$

This forms the surrogate PPO loss (plus KL penalty when used):

$R$0

with $R$1 (Rouditchenko et al., 14 May 2025, Liu et al., 26 Mar 2025).

Critical perspectives highlighted biases in vanilla GRPO—namely, response-length and difficulty biases—leading to the refinement Dr. GRPO, which removes length/std normalization for unbiased advantages:

$R$2

applied identically for all tokens per trajectory (Liu et al., 26 Mar 2025). This approach forms the recommendation for robust R1-Zero-style RL, including Omni-R1-Zero.

4. Training Protocols and Synthetic Data Construction

Omni-R1-Zero is characterized by eliminating the need for human-annotated stepwise multimodal traces. Instead, for vision–text, a small corpus of text-only chain-of-thought (CoT) exemplars (e.g., from M3CoT) suffices. Each reasoning step in the CoT is “visualized” by prompting the base omni-model to synthesize an image (crop, annotation, etc.), forming stepwise interleaved (text, image, action) triples that are formatted into token sequences. This yields $R$3791 synthetic samples in the vision-text study (Cheng et al., 14 Jan 2026).

Audio–text instantiations use a large, high-quality text-only QA dataset (e.g., ARC-Easy) sharing the same multiple-choice format as audio benchmarks (MMAU), allowing the model to learn reasoning and answer selection in a protocol nearly isomorphic to target multimodal tasks—without ever activating the audio encoder during RL (Rouditchenko et al., 14 May 2025).

5. Reward Shaping, Losses, and Domain-Specific Extensions

Reward shaping is domain- and task-specific but adheres to a modular design. In multimodal generative settings, each trajectory's total reward is a weighted sum:

$R$4

where $R$5 is final-answer accuracy, $R$6 penalizes format errors, and $R$7 enforces perception-calibrated image smoothness (quantified by 2D total variation over VQVAE embeddings) (Cheng et al., 14 Jan 2026). This composite reward promotes both correct answers and functional intermediate generations.

Supervised fine-tuning (Stage 1) optimizes the hybrid cross-entropy and perception-alignment loss over synthetic stepwise trajectories:

$R$8

In audio–text, reward is binary (correct/incorrect), with formats for sampled answer selection. RL fine-tuning maximizes expected reward under the GRPO surrogate (Rouditchenko et al., 14 May 2025).

Omni-R1-Zero's extensibility is supported by proposals for modular reward heads (e.g., for style, safety, factuality), adaptive baseline estimation (learned value functions), and dynamic KL annealing for domain adaptation (Liu et al., 26 Mar 2025).

6. Empirical Results and Benchmarks

On Omni-Bench (covering natural scenes, structured images, diagrammatic math, vision-operations), Omni-R1-Zero (vision–text) achieves mean accuracy 0.159, outperforming fully supervised Zebra-CoT (avg 0.129) and even outpacing the supervised Omni-R1-L variant (avg 0.152). On general multimodal benchmarks (MME-P, MM-Vet, etc.), Omni-R1-Zero-S yields a scaled average of 50.19, compared to 38.12 (Zebra-CoT) and 48.29 (Omni-R1-L). Ablation demonstrates that omission of PeRPO or perception reward severely degrades performance (down –29% and –18%, respectively), confirming the necessity of both synthetic visualizations and functional RL (Cheng et al., 14 Jan 2026).

In audio–text benchmarks (MMAU), Omni-R1-Zero, trained only on text QA, achieves 68.2% (w/audio) and 51.7% (w/o audio) on the MMAU Test-mini, nearly matching the multimodal-trained Omni-R1 (68.6% and 51.7%). Analysis confirms that text-driven reasoning can suffice when tasks hinge on world knowledge or distractor elimination rather than perceptual cues (Rouditchenko et al., 14 May 2025).

Further studies of R1-Zero-like training report Oat-Zero-7B achieving 43.3% accuracy on AIME 2024 and 51.4% across five math tasks (no RL: 0.2%); the minimalist protocol uses Dr. GRPO with Qwen2.5-Math-7B and rule-based math verification (Liu et al., 26 Mar 2025).

7. Generalization, Practical Applications, and Limitations

Omni-R1-Zero's core insight is the decoupling of multimodal reasoning competency from direct exposure to multimodal annotated traces during training. In vision and audio settings, synthetic or text-only supervision, coupled with RL fine-tuning, suffices to match or surpass models trained on expensive real multimodal data. The approach generalizes across domains via: base model modularity, reward shaping extensions, curriculum learning, and domain adaptation through continued pretraining or data concatenation (Liu et al., 26 Mar 2025).

A central implication is that expensive annotation can be circumvented, provided architectures and data protocols permit synthetic or protocol-aligned fine-tuning, and that RL (particularly group-based PPO/GRPO with unbiased advantage estimation) is sufficiently powerful to extract general reasoning improvements.

However, the analysis also highlights potential biases (architectural and optimization-induced), such as GRPO response length inflation and sensitivity to pretraining domain. While RL-driven text QA can transfer in many benchmarks, domains requiring fine-grained cross-modal alignment or perception may resist this shortcut. Careful reward design and task curation are essential to maintain stable and aligned learning.

Omni-R1-Zero thus constitutes a unified, modular RL fine-tuning protocol that supports strong multimodal reasoning across vision, text, audio, and arbitrary future modalities, while minimizing annotation cost and maintaining empirical competitiveness against fully supervised alternatives (Cheng et al., 14 Jan 2026, Rouditchenko et al., 14 May 2025, Liu et al., 26 Mar 2025).