Image-Only Training for UMMs

• IOMM is a data- and compute-efficient framework for UMM visual generation that leverages 11M unlabeled images via a two-stage, masked modeling process.
• It first employs image-only masked visual modeling to learn visual priors, then fine-tunes with a mix of paired and unlabeled data to improve text-to-image alignment.
• The approach achieves state-of-the-art text-to-image results with high efficiency, using techniques like a Residual Query Adapter to optimize training convergence and resource use.

Image-Only Training for UMMs (IOMM) is a data- and compute-efficient framework for pre-training unified multimodal model (UMM) visual generation components. IOMM departs from conventional UMM pipelines, which are bottlenecked by the need for large quantities of high-quality, paired text–image data, by leveraging abundant unlabeled images and a two-stage masked modeling methodology. Its empirical benchmarks demonstrate state-of-the-art text-to-image generation, advanced instruction alignment, and high computational efficiency (Sun et al., 17 Mar 2026).

1. Two-Stage IOMM Paradigm

IOMM implements a two-phase pre-training and fine-tuning scheme:

1. Stage I: Image-Only Masked Visual Modeling The visual generative model is pretrained using $|\mathcal D_{\rm unlabeled}| \approx 11$ million unlabeled images—specifically, Megalith-10M (10M public-domain photos) and text-to-image-2M (2M high-quality images ignoring captions). No text conditioning occurs in this stage.

• Conditioning:
  • Auxiliary prompt embeddings $\mathbf c_{\rm aux}\in \mathbb R^{T \times D}$ encode a phrase such as “Generate an image identical to the reference image.”
  • Vision Transformer (ViT) patch embeddings $$\mathbf c_{\rm img}=v(\mathbf x)\in \mathbb R^{P^2\times D}$$ encode the input image.
  • A random binary mask $\mathbf M$ (mask ratio $r$) sparsifies $\mathbf c_{\rm img}$: $$\widetilde{\mathbf c}_{\rm img} = \mathbf M\odot \mathbf c_{\rm img}$$.
  • A Residual Query Adapter (RQA) $q_\theta(\cdot)$ augments the condition.
  • The full condition vector: $$\mathbf c=[\mathbf c_{\rm aux};\;\widetilde{\mathbf c}_{\rm img};\;q_\theta(\mathbf c_{\rm aux},\widetilde{\mathbf c}_{\rm img})]$$.
• Semantic Encoding:
  • Frozen MLLM encoder maps $\mathbf c$ to $\mathbf c_{\rm aux}\in \mathbb R^{T \times D}$0.
• Training Target:
  • The generative model, a flow-matching network $\mathbf c_{\rm aux}\in \mathbb R^{T \times D}$1, predicts the velocity field $\mathbf c_{\rm aux}\in \mathbb R^{T \times D}$2, where $\mathbf c_{\rm aux}\in \mathbb R^{T \times D}$3, $\mathbf c_{\rm aux}\in \mathbb R^{T \times D}$4, $\mathbf c_{\rm aux}\in \mathbb R^{T \times D}$5.
  • The loss is a sparse-to-dense reconstruction:

  $\mathbf c_{\rm aux}\in \mathbb R^{T \times D}$6

• Hyperparameters:

  • Mask ratio $\mathbf c_{\rm aux}\in \mathbb R^{T \times D}$7 (IOMM-B/L), $\mathbf c_{\rm aux}\in \mathbb R^{T \times D}$8 (IOMM-XL); batch size up to 4096; AdamW or Muon optimizer.
• Resource Use:
  • IOMM-B: Stage I consumes $\mathbf c_{\rm aux}\in \mathbb R^{T \times D}$9 H800 GPU hours.

2. Stage II: Mixed Unlabeled and Paired Fine-Tuning The model is fine-tuned on a mixed batch of previously used unlabeled images and $$\mathbf c_{\rm img}=v(\mathbf x)\in \mathbb R^{P^2\times D}$$0k high-quality paired text–image samples: BLIP3-o-60K, Echo-4o-Image, ShareGPT-4o-Image.

• Batch Composition:
  • Each batch uses a fraction $$\mathbf c_{\rm img}=v(\mathbf x)\in \mathbb R^{P^2\times D}$$1 paired, $$\mathbf c_{\rm img}=v(\mathbf x)\in \mathbb R^{P^2\times D}$$2 unlabeled.
• Conditioning:
  • For paired data, text prompt embeddings $$\mathbf c_{\rm img}=v(\mathbf x)\in \mathbb R^{P^2\times D}$$3 (via MLLM) replace $$\mathbf c_{\rm img}=v(\mathbf x)\in \mathbb R^{P^2\times D}$$4.
• Objective:
  • The same flow-matching loss applies, with $$\mathbf c_{\rm img}=v(\mathbf x)\in \mathbb R^{P^2\times D}$$5 from either image-only or text-conditioned input.
• Resource Use:
  • IOMM-B: Stage II requires $$\mathbf c_{\rm img}=v(\mathbf x)\in \mathbb R^{P^2\times D}$$6 H800 GPU hours.

2. Visual Generation Architecture

The IOMM generative model employs the Multi-Modal Diffusion Transformer (MM-DiT) backbone, as used in FLUX.1 and SD3.

• Input:
  • Noised image $$\mathbf c_{\rm img}=v(\mathbf x)\in \mathbb R^{P^2\times D}$$7, flow index $$\mathbf c_{\rm img}=v(\mathbf x)\in \mathbb R^{P^2\times D}$$8, and semantic token sequence $$\mathbf c_{\rm img}=v(\mathbf x)\in \mathbb R^{P^2\times D}$$9 (from frozen MLLM + RQA).
• Output:
  • Vector field $\mathbf M$0. Sampling is solved via

  $\mathbf M$1

  with $\mathbf M$2 as the synthesized image.

• Probabilistic Interpretation:

  • Defines $\mathbf M$3 via a probability-flow ODE.

3. Data Efficiency and Dataset Composition

IOMM’s staged regime sharply reduces dependence on expensive paired data.

• Stage I:
  • $\mathbf M$4M unlabeled images; no text data needed.
• Stage II:
  • $\mathbf M$5K paired images plus $\mathbf M$6M unlabeled images.
• Resolution:
  • $\mathbf M$7 or $\mathbf M$8 pixels, centrally cropped.
• Mask Ratios:
  • 0.85 (IOMM-B/L), 0.45 (IOMM-XL).
• Paired Data Fraction:
  • IOMM fine-tunes on only $\mathbf M$9M paired images (vs. $r$0M in prior work).

This approach eliminates the need for proprietary large corpora during the heaviest pretraining phase.

4. Resource Utilization and Efficiency Mechanisms

The IOMM-B model (1.6B parameters) achieves total training in approximately $r$1 H800 GPU hours:

• Stage I:
  • $r$2 hours (image-only pretraining)
• Stage II:
  • $r$3 hours (mixed-data fine-tuning)

In comparison, previous UMMs (e.g., BLIP3-o-8B) employ several thousand GPU hours with 30M proprietary text–image pairs.

Contributing factors to this efficiency:

• Reliance on cheap self-conditioning with unlabeled images
• Masked modeling objective, which converges faster than full-image reconstruction
• Lightweight RQA module (29M parameters), avoiding costly fine-tuning of $r$4B-parameter MLLM

5. Text-to-Image Generation Results

IOMM establishes state-of-the-art results in text-to-image fidelity and instruction alignment, as summarized below:

Model	GenEval (↑)	WISE (↑)	Notes
BLIP3-o-4B	0.81	0.50	open paired
BAGEL-7B	0.88	0.52	scratch
IOMM-B (512px)	0.89	0.55	$r$5kh total GPU

The IOMM-B model achieves strong improvement over baselines on GenEval (0.89), equalling or surpassing top models on WISE (0.55).

Fine-grained results across benchmarks such as SDXL and DALL·E 3 appear in supplementary materials (Sun et al., 17 Mar 2026).

6. Ablation Studies

Systematic ablations clarify architectural and hyperparameter choices:

A) Residual Query Adapter (RQA)

• Replacing RQA with MetaQuery (same 256 queries) substantially slows GenEval convergence (MetaQuery peaks at $r$6 after +8K steps vs. RQA at $r$7).

B) Mask Ratio ($r$8) in IOMM-XL

• Performance on GenEval and DPGBench peaks at mask ratio $r$9. Lower ratios underutilize masking; higher ratios impair image reconstruction:
  • $\mathbf c_{\rm img}$0: GenEval$\mathbf c_{\rm img}$1, DPGBench$\mathbf c_{\rm img}$2
  • $\mathbf c_{\rm img}$3: GenEval$\mathbf c_{\rm img}$4, DPGBench$\mathbf c_{\rm img}$5 (max)
  • $\mathbf c_{\rm img}$6: GenEval$\mathbf c_{\rm img}$7, DPGBench$\mathbf c_{\rm img}$8

C) Fine-Tuning Mix Ratio ($\mathbf c_{\rm img}$9) in IOMM-XL

• Stability and quality are optimal at $$\widetilde{\mathbf c}_{\rm img} = \mathbf M\odot \mathbf c_{\rm img}$$0 (half paired, half image-only):
  • $$\widetilde{\mathbf c}_{\rm img} = \mathbf M\odot \mathbf c_{\rm img}$$1: GenEval$$\widetilde{\mathbf c}_{\rm img} = \mathbf M\odot \mathbf c_{\rm img}$$2
  • $$\widetilde{\mathbf c}_{\rm img} = \mathbf M\odot \mathbf c_{\rm img}$$3: GenEval$$\widetilde{\mathbf c}_{\rm img} = \mathbf M\odot \mathbf c_{\rm img}$$4 (optimal)
  • $$\widetilde{\mathbf c}_{\rm img} = \mathbf M\odot \mathbf c_{\rm img}$$5 (pure images): GenEval drops sharply ($$\widetilde{\mathbf c}_{\rm img} = \mathbf M\odot \mathbf c_{\rm img}$$6)

7. Implications and Research Directions

IOMM establishes that large-scale, image-only masked modeling is sufficient to form strong visual priors for UMM generative components. The staged approach, featuring (i) substantial masked-modeling on unlabeled images, (ii) minimal curated paired data for instruction alignment, and (iii) efficient adapters (RQA), consistently outperforms naive mixing or single-phase recipes.

The framework dispenses with the expensive reliance on vast paired corpora, preserves frozen MLLM encoders, and bestows efficient training routines suitable for scaling. A notable observation is that zero-shot image editing is competitive (ImgEdit-Bench overall 2.82 vs. UltraEdit 2.70).

A plausible implication is that future UMM backbones may increase pre-training on unlabeled image collections, decrease paired-data requirements, and standardize parameter-efficient adapters. The codebase is publicly available at https://github.com/LINs-lab/IOMM (Sun et al., 17 Mar 2026).