---
title: 'Skywork UniPic 3.0: Unified Multimodal Framework'
url: https://www.emergentmind.com/topics/skywork-unipic-3-0
type: topic
---

# Skywork UniPic 3.0: Unified Multimodal Framework

Skywork UniPic 3.0 is a unified multimodal generative framework designed for both single-image editing and multi-image composition, emphasizing Human-Object Interaction (HOI)-centric synthesis. It supports an arbitrary number (1–6) and resolution of input images under a shared pixel budget, outputs high-fidelity compositions at target resolutions, and achieves state-of-the-art performance benchmarks by recasting composition as a sequence-modeling task with efficient inference [2601.15664].

## 1. Architecture and Unified Sequence Construction

Skywork UniPic 3.0 integrates various subsystems to address the complexity of compositional image synthesis across modalities. The framework consists of:

- **Vision–Language Encoder**: Utilizes Qwen2.5-VL to encode user instructions alongside metadata (e.g., capture resolution, shape descriptors).
- **Image Tokenizer**: Employs a VAE encoder \(f_{\mathrm{vae\text{-}enc}}\) mapping each image \(I\) to a latent tensor \(z \in \mathbb{R}^{1 \times C \times H' \times W'}\).
- **Patch-Wise Packing**: Latents are transformed into sequences of patch tokens by \(s = \mathrm{pack}(z) \in \mathbb{R}^{N \times D}\), with \(N = H' W'/4\), \(D = 4C\).
- **Backbone Diffusion Model**: An MMDiT (latent diffusion transformer) that operates over a unified sequence of tokens.

For processing, \(K\) reference images \(\{I_k\}_{k=1}^K\) and the output image \(O\) are packed as a concatenated sequence:
\[
S = [s_O \,\|\, s_1 \,\|\, s_2 \,\|\cdots\|\, s_K] \in \mathbb{R}^{(N_O + \sum_{k=1}^K N_k) \times D},
\]
with a parallel sequence of shape descriptors \(\mathcal{H} = \{h_O, h_1, ..., h_K\}\), where each \(h_i = (H'_i,W'_i)\).

To preserve computational tractability, the model enforces a global pixel budget:
\[
\sum_{i = O,1}^K H_i\,W_i\leq 1024^2
\]
ensuring that the combined input-output resolution remains bounded.

## 2. Sequence-Modeling Paradigm for Unified Tasks

Both single-image editing and multi-image composition are formulated uniformly as sequence-to-sequence conditional synthesis. Context tokens \(\mathbf{X} = \{s_1, ..., s_K\}\) (representing references) condition the generation of output tokens \(\mathbf{Y} = s_O\):
\[
p_\theta(\mathbf{Y}\mid\mathbf{X}) = \prod_{t=1}^T p_\theta(y_t\mid y_{< t}, \mathbf{X})
\]
with the cross-entropy loss objective
\[
\mathcal{L}_{\mathrm{CE}} = -\mathbb{E}_{(\mathbf{X},\mathbf{Y})} \sum_{t=1}^{T} \log p_\theta(y_t\mid y_{< t}, \mathbf{X})
\]
Variable-length reference handling is inherently supported by concatenation, subject to pixel budget and accompanied by explicit shape descriptors to maintain token-to-patch correspondence during generation and reconstruction.

## 3. Data Curation, Filtering, and Synthesis Pipeline

A stringent and automated three-stage data pipeline is central to Skywork UniPic 3.0’s empirical performance, with an explicit focus on HOI-centric compositions:

1. **Data Collection**:
   - Person images: 18,000 from CC12M, captioned with InternVL3.5-38B for fine-grained pose and apparel labeling.
   - Object categories: 300 human-interactive classes generated with GPT-4o; 5,000 text prompts per class rendered via Qwen-Image, yielding 150,000 images.

2. **Data Filtering**:
   - InternVL3.5-38B: holistic quality scoring in \([0,100]\); retain instances with scores \(\geq 75\).
   - Face/body detectors: threshold face visibility (\(\geq 90\%\)) and subject occupancy (\(\geq 60\%\)).
   - CLIPScore and resolution: prompt-image consistency ≥ threshold, object images ≥ \(768^2\).

3. **Synthesis**:
   - For \(K\) in \([2,6]\), consider all valid image combinations, subject to a conflict matrix (e.g., one person cannot hold multiple guitars).
   - Compose prompts using InternVL3.5; for \(K\leq 3\), targets are synthesized by NanoBanana; for \(K>3\), by Seedream4.0.
   - Retain only samples passing stringent aesthetic and identity checks.

This procedure results in 215,000 HOI triplets, with the full dataset comprising approximately 700,000 high-quality sample pairs.

## 4. Training Regimen and Loss Formulations

Skywork UniPic 3.0 adopts a multi-stage, diffusion-based optimization pipeline:

- **Pre-Training (Multi-Task Diffusion)**: Optimizes the MMDiT backbone with the flow-matching loss
  \[
  \mathcal{L}_{\mathrm{FM}}(\theta) = \mathbb{E}_{z_0 \sim p_{\mathrm{data}}, t \sim [0,1]} \left\| F_\theta(z_t, t) - \frac{dz_t}{dt} \right\|_2^2,
  \]
  where \(z_t = (1-t)z_0 + t\epsilon\), \(\epsilon \sim \mathcal{N}(0,I)\).
- **Continuous-Time Consistency Tuning**: Applies a consistency loss for trajectory mapping
  \[
  \mathcal{L}_{\mathrm{CM}}(\theta) = \mathbb{E}_{z_t,t} \left\| F_\theta(z_t,t) + z_t + F_{\theta^-}(z_t,t) + t\partial_t F_{\theta^-}(z_t,t) \right\|_2^2,
  \]
  facilitating direct mapping from \(z_t\) to \(z_0\) for efficient sampling.
- **Distribution Matching Distillation**: Distills the student model to match a multi-step teacher, using a reverse-KL-based loss:
  \[
  \mathcal{L}_{\mathrm{DMD}}(\theta) = \frac{1}{2} \mathbb{E}_{z_t, t} \left\| F_\theta(z_t, t) - [F_{\theta^-}(z_t, t) - \frac{t}{1-t}(\nabla_{z_t}\log p_{\mathrm{teacher}}(z_t) - \nabla_{z_t}\log p_\phi(z_t))] \right\|_2^2,
  \]
  with \(\phi\) as a LoRA-adapter network derived from the teacher parameters.

## 5. Accelerated Inference Techniques

After consistency tuning and distribution-matching distillation, inference is substantially accelerated:
- The trajectory mapping mechanism enables the model to move efficiently from the noise prior (\(t=1\)) to sample (\(t=0\)) space.
- Post-distillation, high-quality generations are achievable in only 8 function evaluations, representing a measured \(12.5\times\) sampling speedup compared to typical ~100-step diffusion pipelines.
- Distribution matching ensures fidelity of the accelerated (student) sampler relative to the full-step (teacher) output distribution, maintaining output quality.

## 6. Quantitative and Qualitative Performance

Skywork UniPic 3.0 achieves leading results across multiple benchmarks:

| Model                 | ImgEdit-Bench | GEdit-Bench |
|-----------------------|--------------|-------------|
| Qwen-Image-Edit       | 4.25         | 7.88        |
| Nano-Banana           | 4.22         | 7.59        |
| Seedream 4.0          | 4.11         | 7.92        |
| UniPic 3.0            | 4.35         | 7.79        |

| Model                 | 2–3 Inputs | 4–6 Inputs | Overall  |
|-----------------------|------------|------------|----------|
| Qwen-Image-Edit       | 0.7705     | 0.4793     | 0.6249   |
| Nano-Banana           | 0.7982     | 0.6466     | 0.7224   |
| Seedream 4.0          | 0.7997     | 0.6197     | 0.7088   |
| UniPic 3.0            | 0.8214     | 0.6296     | 0.7255   |

Qualitative analysis highlights advances in natural occlusion, facial consistency, and instruction fidelity, particularly for complex HOI tasks.

## 7. Dataset and Code Availability

The full codebase, trained models, and MultiCom-Bench dataset are publicly released at https://skywork-unipic-v3.github.io. The dataset includes curated HOI triplet compositions, open-domain multi-image, and single-image editing data, supporting further research in unified multimodal image synthesis and compositional generative modeling [2601.15664].

Source: https://www.emergentmind.com/topics/skywork-unipic-3-0