UniPic2-Metaquery: Unified Multimodal Model

• UniPic2-Metaquery integrates a frozen multimodal LLM, a diffusion transformer generator, and a learnable connector for unified understanding, generation, and editing.
• It employs a MetaQuery training paradigm with online reinforcement learning to synergistically enhance text-to-image generation and editing performance.
• Ablations demonstrate that dual conditioning and progressive RL yield improved metrics and overcome fragmentation in multimodal pipelines.

UniPic2-Metaquery is the final unified multimodal model introduced within Skywork UniPic 2.0. It couples a frozen multimodal LLM, Qwen2.5-VL-7B, with a 2B-parameter diffusion transformer generator, UniPic2-SD3.5M-Kontext, through a 24-layer transformer connector with learnable queries, and is trained under a MetaQuery paradigm further strengthened by online reinforcement learning. In this formulation, the system is intended to integrate visual understanding, text-to-image generation, and instruction-based image editing inside a single coherent model rather than as separate pipelines (Wei et al., 4 Sep 2025).

1. Position within Skywork UniPic 2.0

Skywork UniPic 2.0 is presented as the second generation of Skywork’s unified multimodal framework. The distinction from UniPic 1.0 is explicit: UniPic 1.0 is described as “an autoregressive unified model trained from scratch for understanding and generation,” whereas UniPic 2.0 is a “modular, MetaQuery-style” system that combines a frozen multimodal LLM, a pretrained diffusion transformer, and a learnable high-capacity connector. Within this framework, UniPic2-Metaquery is the “unified” face of the system, while UniPic2-SD3.5M-Kontext is the underlying generation-and-editing backbone (Wei et al., 4 Sep 2025).

The motivating problem is also stated in architectural rather than purely scaling terms. The work argues that many open-source unified models emphasize parameter count, including generators with 7B or 12B parameters, instead of optimizing training strategy. The claimed consequences are prohibitive compute, slow inference, and uneven instruction following or editing consistency. A second stated problem is fragmentation between understanding and generation: one model performs multimodal reasoning, another performs generation or editing, and there is no principled transfer of the former into the latter. UniPic2-Metaquery is therefore positioned as a simple and scalable unification strategy: a frozen MLLM for perception and reasoning, a compact but heavily optimized DiT for generation and editing, and a connector that translates multimodal hidden states into generative conditioning (Wei et al., 4 Sep 2025).

This positioning also clarifies a common misunderstanding. UniPic2-Metaquery is not merely UniPic2-SD3.5M-Kontext with broader prompting. Kontext alone is described as “only the DiT generator/editor”; Metaquery adds the MLLM and connector, enabling arbitrary multimodal instructions, text responses, image responses, and mixed workflows such as analyzing an image and then editing it accordingly (Wei et al., 4 Sep 2025).

2. Architectural composition

Formally, UniPic2-Metaquery is a three-part network: Qwen2.5-VL-7B encodes text and images into multimodal hidden states; a MetaQuery connector transforms those hidden states into conditioning tokens; and UniPic2-SD3.5M-Kontext performs generation or editing conditioned on those tokens and, for editing, optionally on VAE latents of a reference image (Wei et al., 4 Sep 2025).

Component	Size or status	Function
Qwen2.5-VL-7B	7B, frozen	Multimodal understanding
MetaQuery connector	24 layers, ~1B	Translates hidden states into conditioning tokens
UniPic2-SD3.5M-Kontext	2B DiT	Image generation and editing

The DiT backbone is based on SD3.5-Medium, described as a rectified flow-based diffusion transformer for latent image generation. UniPic 2.0 modifies this model into a Kontext backbone that natively handles both text-to-image generation and image editing. Images are encoded with a VAE, and the DiT operates on latent tokens corresponding to the target image. For editing, the reference image is encoded by the same VAE, projected into context tokens, and concatenated with the noisy target tokens in a single sequence. Positional encodings distinguish reference-image tokens from target-image tokens, and the DiT self-attention layers are modified so that reference latents are injected directly to preserve structure and texture. Training alternates text-to-image and editing batches within a single model, while progressive resolution training and sampling across aspect ratios such as $1{:}1$, $4{:}3$, $3{:}2$, and $16{:}9$ are used to support dynamic-resolution inference (Wei et al., 4 Sep 2025).

The connector follows the MetaQuery pattern more directly. It receives hidden states from Qwen2.5-VL together with a small set of learnable query tokens and outputs conditioning tokens for the DiT, replacing or complementing the original SD3.5 text encoder. The paper reports that “the DiT module prompted by our learnable queries and connector exhibits even better text-to-image generation performance compared to using its original text encoders, e.g., T5 in SD3.5.” Qwen is kept frozen so that its understanding benchmarks are preserved; the model’s MMBench, MMMU, and MM-Vet performance is reported as matching Qwen2.5-VL (Wei et al., 4 Sep 2025).

3. Training paradigm: MetaQuery pre-training, Kontext optimization, and PDTR

The training pipeline is staged. First, the MetaQuery connector is pre-trained with Qwen2.5-VL and SD3.5-Medium frozen. This stage uses 150M image-text pairs: 30M from open datasets including CC12M, Megalith10M, RedCaps, and LAION-Aesthetics, and 120M internal synthetic images. The connector and learnable queries are trained for 500k steps with global batch size 1024 and learning rate $1\times10^{-4}$ (Wei et al., 4 Sep 2025).

Second, UniPic2-SD3.5M-Kontext is trained as a unified generation-and-editing backbone. The reported data are approximately 5M image-editing pairs, drawn from UniWorld-V1, OmniGen2, NHR-Edit, ShareGPT4o-Image, GPT-Image-Edit-1.5M, and 450k internal samples, together with approximately 6M text-to-image samples consisting of 5.6M internal high-quality real images and 400k from BLIP3-o and ShareGPT4o-Image. The backbone is trained for 200k steps with global batch size 128, learning rate $4\times10^{-5}$, and AdamW with $\beta_1=0.9$, $\beta_2=0.95$, $\epsilon=10^{-8}$, and weight decay $0.05$ (Wei et al., 4 Sep 2025).

The third stage replaces base SD3.5-Medium in the MetaQuery pipeline with the PDTR-enhanced UniPic2-SD3.5M-Kontext, freezes Qwen2.5-VL, and jointly fine-tunes the connector, learnable queries, and Kontext on the same high-quality generation and editing data. This is the step that yields UniPic2-Metaquery as a unified multimodal system rather than a connector-pretrained T2I stack (Wei et al., 4 Sep 2025).

A central element of the training strategy is Progressive Dual-Task Reinforcement, or PDTR. The reinforcement procedure is based on Flow-GRPO, an online RL method for flow-matching models. For a text hidden state $4{:}3$0, the model samples a group of $4{:}3$1 images and trajectories, and the advantage of the $4{:}3$2-th sample is defined as

$4{:}3$3

The work states that GRPO uses a clipped objective with a KL penalty to a reference pre-RL policy. “Online RL” means that images are generated on the fly by the current policy, scored online by automatic metrics, VLM evaluators, or a learned reward model, and then used for iterative policy updates through LoRA. The reported hyperparameters for all RL stages are $4{:}3$4, $4{:}3$5, noise level $4{:}3$6, resolution $4{:}3$7, KL ratio $4{:}3$8, LoRA rank $4{:}3$9, $3{:}2$0, learning rate $3{:}2$1, global batch size 256, and Adam with $3{:}2$2 (Wei et al., 4 Sep 2025).

PDTR itself proceeds sequentially rather than jointly. Stage 1 performs image-editing reinforcement using reference image plus textual edit instruction, with rewards from Skywork-EditReward or online GPT-4.1 evaluation. Skywork-EditReward is trained by generating 333k editing samples with UniPic2-SD3.5M-Kontext, using GPT-4.1 with carefully designed templates to rate instruction following, image quality, and aesthetic consistency, and then fine-tuning Qwen2.5-VL-7B with regression loss to predict those scores. Stage 2 performs text-to-image reinforcement starting from the editing-reinforced model, using GenEval-style compositional metrics and detector-based checks from MMDetection and Mask2Former. The stated claim is that editing RL and T2I RL are mutually beneficial and do not induce negative interference (Wei et al., 4 Sep 2025).

4. Inference semantics and supported task classes

At inference time, UniPic2-Metaquery is described as a single interface model. A user provides a text prompt and optionally one or more reference images. Qwen2.5-VL reads the entire context and produces hidden states; if the request is for analysis, it can respond directly in text, while if the request is to generate or edit, the hidden states are routed through the connector to condition the DiT. The paper does not prescribe a rigid prompt format, but this routing pattern is the stated operational logic (Wei et al., 4 Sep 2025).

The supported capability classes are broad but explicitly enumerated. For multimodal understanding, the model supports VQA, captioning, detailed scene description, OCR, fine-grained recognition, and structured outputs such as localization with JSON bounding boxes. For text-to-image generation, it is described as supporting high-fidelity images with accurate object counts, attributes, and spatial relations, together with multiple aspect ratios and resolutions. For instruction-based editing, the reported operations include adding, removing, and replacing objects, as well as local attribute or style modifications while preserving layout. The paper also emphasizes mixed multimodal tasks such as “understand then generate,” “understand then edit,” and structured outputs combined with generation (Wei et al., 4 Sep 2025).

A further inference detail is crucial for editing. Reference images are used twice: they are fed into Qwen2.5-VL for semantic understanding, and they are independently encoded as VAE latents and injected into DiT self-attention for structural preservation. The authors state that this dual conditioning is crucial in ablations. This corrects another common misconception: in UniPic2-Metaquery, editing quality is not obtained by MLLM-side reasoning alone, but by jointly combining semantic conditioning and direct latent-level structural guidance (Wei et al., 4 Sep 2025).

5. Empirical performance and ablation evidence

The main comparison table reports results across generation, editing, and understanding. UniPic2-SD3.5M-Kontext, with a 2B generator, achieves GenEval $3{:}2$3, DPG $3{:}2$4, GEdit-EN $3{:}2$5, and ImgEdit $3{:}2$6. UniPic2-Metaquery improves these to GenEval $3{:}2$7, DPG $3{:}2$8, GEdit-EN $3{:}2$9, and ImgEdit $16{:}9$0, while also reporting MMBench $16{:}9$1, MMMU $16{:}9$2, and MM-Vet $16{:}9$3. In the same table, BAGEL is listed with a 7B generator and scores $16{:}9$4, $16{:}9$5, $16{:}9$6, $16{:}9$7, $16{:}9$8, $16{:}9$9, and $1\times10^{-4}$0; FLUX.1-Kontext is listed with a 12B generator and editing scores $1\times10^{-4}$1 and $1\times10^{-4}$2; Ovis-U1 is listed with a 1.2B generator and scores $1\times10^{-4}$3, $1\times10^{-4}$4, $1\times10^{-4}$5, $1\times10^{-4}$6, $1\times10^{-4}$7, $1\times10^{-4}$8, and $1\times10^{-4}$9. GPT-4o is reported at $4\times10^{-5}$0, $4\times10^{-5}$1, $4\times10^{-5}$2, $4\times10^{-5}$3, $4\times10^{-5}$4, $4\times10^{-5}$5, and $4\times10^{-5}$6 (Wei et al., 4 Sep 2025).

The evaluation therefore supports two narrower claims. First, a 2B DiT can match or exceed substantially larger open unified generators on key generation and editing metrics. Second, attaching Qwen2.5-VL through the MetaQuery connector yields a unified model whose understanding scores remain comparable to the underlying MLLM while generation and editing improve slightly over Kontext alone, especially on editing (Wei et al., 4 Sep 2025).

The ablations are unusually explicit. For image conditioning during editing, feeding the reference image only to the MLLM yields GEdit-EN $4\times10^{-5}$7; feeding it only to the DiT yields $4\times10^{-5}$8; feeding it to both yields $4\times10^{-5}$9, compared with Kontext-only $\beta_1=0.9$0. For fine-tuning policy, freezing the connector and fine-tuning the DiT gives GenEval $\beta_1=0.9$1 and GEdit-EN $\beta_1=0.9$2; fine-tuning the connector while freezing the DiT gives $\beta_1=0.9$3 and $\beta_1=0.9$4; jointly fine-tuning both gives $\beta_1=0.9$5 and $\beta_1=0.9$6. For PDTR reward ablations, the baseline without RL is GenEval $\beta_1=0.9$7, DPG $\beta_1=0.9$8, GEdit-EN $\beta_1=0.9$9, ImgEdit $\beta_2=0.95$0; editing-only RL yields $\beta_2=0.95$1, $\beta_2=0.95$2, $\beta_2=0.95$3, $\beta_2=0.95$4; T2I-only RL yields $\beta_2=0.95$5, $\beta_2=0.95$6, $\beta_2=0.95$7, $\beta_2=0.95$8; and both editing plus T2I RL yield $\beta_2=0.95$9, $\epsilon=10^{-8}$0, $\epsilon=10^{-8}$1, $\epsilon=10^{-8}$2. The stated conclusion is that the two tasks can be iteratively improved in a synergistic manner without negative interference (Wei et al., 4 Sep 2025).

6. Terminological ambiguity, limitations, and broader implications

In the primary sense established by Skywork UniPic 2.0, UniPic2-Metaquery is a concrete multimodal model artifact. However, the expression “metaquery” is used more broadly in adjacent literature, and this creates a terminological ambiguity that warrants separation. In metaproteomics, the provided Unipept tutorial uses “UniPic2-Metaquery” as an umbrella label for peptide-centric, large-scale metaqueries over taxonomic and functional indices, including Lowest Common Ancestor determination, advanced missed cleavage handling, interactive taxonomic and functional profiling, and custom reference databases for increased species- or strain-level specificity. In that domain, the term refers to a style of biodiversity analysis rather than to a diffusion-plus-MLLM architecture (Bossche et al., 2023).

A second analogical use appears in work on heterogeneous LLM-based semantic query systems. “Meta Engine” is described as a “query system on query systems” with a natural-language query parser, operator generator, query router, adapters, and result aggregator. The supplied exposition explicitly presents it as realizing the kind of unified “metaquery” layer one might want for a UniPic2-Metaquery system. In that usage, the relevant abstraction is not image synthesis but orchestration across specialized semantic backends for text, tables, and images (Li et al., 2 Feb 2026).

Within Skywork UniPic 2.0 itself, the limitations are more conventional. The authors identify failure modes in complex text rendering, character-level fidelity in book-cover or clock-face edits, exact object counting, intricate semantics in long prompts, special object substitution, object extraction, and precise high-resolution text modification. They suggest that further data scaling and more targeted RL rewards could mitigate many of these issues. They also imply future directions such as swapping in larger DiTs or MLLMs, extending PDTR to tasks such as video generation, 3D, inpainting, or layout control, enriching reward models, and possibly fine-tuning the frozen MLLM in the loop (Wei et al., 4 Sep 2025).

Taken together, these usages suggest a broader pattern rather than a single universal definition. In the strictest and most direct sense, UniPic2-Metaquery denotes the Skywork unified multimodal model that integrates understanding, generation, and editing through a frozen MLLM, a compact DiT backbone, a learnable connector, and staged online RL. A plausible implication is that the wider appeal of the term comes from the same organizing principle across otherwise unrelated domains: a metaquery layer that translates rich upstream semantics into downstream task execution, whether the downstream substrate is a diffusion transformer, a peptide-centric biodiversity index, or a heterogeneous ensemble of semantic query systems (Wei et al., 4 Sep 2025, Bossche et al., 2023, Li et al., 2 Feb 2026).