---
title: 'Janus-Pro: Unified Multimodal Model'
url: https://www.emergentmind.com/topics/janus-pro
type: topic
---

# Janus-Pro: Unified Multimodal Model

Janus-Pro is a unified multimodal model family that performs both multimodal understanding and text-to-image generation with a single autoregressive transformer, while using decoupled visual encoding for the two tasks. In its core formulation, it is an improved successor to Janus, with changes in training strategy, training data, and model scaling; in later work, Janus-Pro-1B is also fine-tuned into a prompt-layout parsing module that converts free-form prompts into structured object layouts for downstream diffusion-based multi-instance synthesis [2501.17811] [2503.21069].

## 1. Unified multimodal scope and design objective

Janus-Pro is defined as a **unified multimodal model** that can both understand images and text and generate images from text instructions. The central architectural claim is that these tasks are handled by **one transformer** rather than by separate understanding and generation backbones. In the base system, “unified multimodal understanding and generation” means that the same autoregressive language backbone answers questions about images, describes or reasons about them, and predicts image tokens to synthesize images from text prompts [2501.17811].

The model inherits Janus’s **decoupled visual encoding** design. For understanding, it uses a vision encoder specialized for semantic alignment; for generation, it uses a tokenizer specialized for reconstructive image tokenization. Both streams are projected into the same language-model embedding space through separate adapters. This arrangement is explicitly presented as a way to mitigate the objective conflict between semantic visual understanding and low-level image token prediction [2501.17811].

Janus-Pro is released in two principal variants:

| Variant | Backbone and scale | Core transformer configuration |
|---|---|---|
| Janus-Pro-1B | DeepSeek-LLM 1.5B, named as the “1B series” | embedding size 2048, 24 layers, 16 heads, context length 4096 |
| Janus-Pro-7B | DeepSeek-LLM 7B | embedding size 4096, 30 layers, 32 heads, context length 4096 |

This design places Janus-Pro in the family of unified multimodal systems such as Show-o, VILA-U, Chameleon, Transfusion, Monoformer, and Emu3. The distinctive point emphasized in the Janus-Pro work is not merely joint training, but joint training with separated visual front ends and a shared autoregressive core [2501.17811].

## 2. Architecture, tokenization, and sequence formulation

Janus-Pro retains four principal components: a language backbone, an understanding encoder, a generation tokenizer, and autoregressive heads. The understanding side uses **SigLIP-Large-Patch16-384**, with input resized or padded to \(384 \times 384\), followed by a two-layer MLP understanding adapter. The generation side uses a **VQ tokenizer** with codebook size 16,384 and downsampling factor 16, so that a \(384 \times 384\) image becomes a discrete sequence on the order of \(24 \times 24 = 576\) tokens, followed by a two-layer MLP generation adapter and a separate image prediction head over the codebook [2501.17811].

At the sequence level, all tasks are cast as autoregressive next-token prediction. Text-only tasks use ordinary text-token sequences. Image-to-text tasks insert visual tokens from the understanding encoder into the sequence and predict text tokens. Text-to-image tasks begin with prompt tokens and then predict image VQ tokens. The image generation factorization is written as
\[
p(\mathbf{y}\mid \mathbf{x})=\prod_{t=1}^{T} p(y_t\mid y_{<t},\mathbf{x}),
\]
with \(\mathbf{x}\) the prompt tokens and \(\mathbf{y}\) the image-token sequence [2501.17811].

The same paper also makes the training objectives explicit in standard autoregressive form. For multimodal understanding,
\[
\mathcal{L}_{\text{understand}}=
-\mathbb{E}_{(\mathbf{v},\mathbf{x},\mathbf{y})}
\left[
\sum_{t=1}^{|\mathbf{y}|}
\log p_{\theta}(y_t\mid \mathbf{v},\mathbf{x},y_{<t})
\right],
\]
and for text-to-image generation,
\[
\mathcal{L}_{\text{gen}}=
-\mathbb{E}_{(\mathbf{x},\mathbf{z})}
\left[
\sum_{t=1}^{|\mathbf{z}|}
\log p_{\theta}(z_t\mid \mathbf{x},z_{<t})
\right].
\]
In practice, Janus-Pro mixes understanding, pure text, and generation batches according to task ratios rather than through a separately weighted multi-objective optimizer [2501.17811].

A notable architectural boundary is that Janus-Pro’s native generation path is fully autoregressive rather than diffusion-based. This matters later, because the downstream layout-planning use of Janus-Pro in multi-instance generation does not reuse Janus-Pro as an image generator; it reuses it as a multimodal planner [2503.21069].

## 3. Training curriculum and data scaling

Janus-Pro uses a refined **three-stage curriculum**. **Stage I** is a warm-up phase for the understanding adapter, generation adapter, and image prediction head, with the LLM frozen. Its data ratio is \(1:0:3\) for understanding:text:generation, it runs for 20K steps with batch size 256, and it is intended in part to let the generation stack learn pixel dependence on ImageNet class-label prompts [2501.17811].

**Stage II** is unified pretraining. Here Janus-Pro departs from Janus by dropping ImageNet from the main text-to-image phase after Stage I, and using only normal text-to-image data with dense prompts. The stated purpose is to improve both efficiency and semantic alignment. Stage II uses learning rate \(1.0\times10^{-4}\), constant scheduling with 5000 warm-up steps, AdamW with \(\beta_1=0.9\), \(\beta_2=0.95\), weight decay 0, gradient clipping 1.0, batch size 512, planned 360K steps with early stopping at 270K, and data ratio \(2:3:5\) [2501.17811].

**Stage III** is supervised fine-tuning. SigLIP is unlocked in this stage. Janus-Pro changes the Janus Stage III ratio from \(7:3:10\) to \(5:1:4\), explicitly reducing the text-to-image proportion relative to Janus in order to preserve or improve generation while strengthening multimodal understanding. Stage III uses learning rate \(4.0\times10^{-5}\), no warm-up, batch size 128, and 80K steps for the 1B series or 40K for the 7B series [2501.17811].

Data scaling is a central part of the Janus-Pro revision. On the understanding side, Stage II adds about **90M multimodal samples**, including image-caption data and document-oriented sources such as Docmatix; Stage III adds instruction-style and conversational data, including MEME understanding, Chinese conversational data, and dialogue-oriented mixtures. On the generation side, the paper reports adding **72M synthetic aesthetic samples**, roughly balancing real and synthetic data \(1:1\) in Stage II. The stated effect is faster convergence, more stable outputs, and significantly higher aesthetic quality [2501.17811].

Training is implemented with **HAI-LLM**, a lightweight distributed framework built on PyTorch. The 1.5B model is reported to train for about 9 days on 16 A100 nodes with 8 GPUs each, and the 7B model for about 14 days on 32 such nodes [2501.17811].

## 4. Empirical performance and scaling behavior

On multimodal understanding benchmarks, Janus-Pro improves substantially over Janus. Janus-Pro-1B is reported at POPE 86.2, MME-P 1444.0, MMB 75.5, SEED 68.3, GQA 59.3, MMMU 36.3, and MM-Vet 39.8, versus Janus at POPE 87.0, MME-P 1338.0, MMB 69.4, SEED 63.7, GQA 59.1, MMMU 30.5, and MM-Vet 34.3. Janus-Pro-7B further reaches POPE 87.4, MME-P 1567.1, MMB 79.2, SEED 72.1, GQA 62.0, MMMU 41.0, and MM-Vet 50.0 [2501.17811].

The generation results are likewise strong on instruction-following benchmarks. On **GenEval**, Janus-Pro-1B reaches an overall score of 0.73 and Janus-Pro-7B reaches 0.80; the same table lists SD3-Medium at 0.74, DALL-E 3 at 0.67, Janus at 0.61, and Emu3-Gen at 0.54. On **DPG-Bench**, Janus-Pro-1B reaches 82.63 and Janus-Pro-7B reaches 84.19, slightly above SD3-Medium at 84.08 and above DALL-E 3 at 83.50, while Janus is reported at 79.68 [2501.17811].

The 7B model’s GenEval sub-scores are also specified: Single Object 0.99, Two Objects 0.89, Counting 0.59, Colors 0.90, Position 0.79, and Color Attributes 0.66. The scaling discussion in the same work states that the 7B model converges faster in both multimodal understanding loss and text-to-image generation loss, and that the decoupled visual encoding plus unified autoregressive transformer architecture scales effectively with LLM size [2501.17811].

Qualitative comparisons in the paper focus especially on short prompts. Janus-Pro is reported to produce cleaner and more detailed images than Janus for prompts such as “a cute cat” and “a red car on the street,” with fewer blurry or artifact-laden failures. The paper interprets this as improved text-to-image stability, particularly for short prompts [2501.17811].

## 5. Janus-Pro as a prompt-layout planner for multi-instance generation

A later use of Janus-Pro recasts **Janus-Pro-1B** as a compact multimodal planning model rather than as a direct image generator. In “Efficient Multi-Instance Generation with Janus-Pro-Driven Prompt Parsing,” Janus-Pro-1B is fine-tuned into a **prompt-layout parsing module** that maps prompts, and optionally images during training, into layouts wrapped by `<layout> ... </layout>`, with per-instance `<scap>` sub-captions and `<bbox>` coordinates \((x_1,y_1,x_2,y_2)\). The paper describes this module as “the language-layout ‘brain’ that sits in front of the diffusion models” [2503.21069].

The parser is trained in a two-way manner. **Layout Understanding** takes image plus textual sub-prompts and predicts bounding boxes and spatial relationships. **Draft Generation** uses an inverse training objective so that, conditioned on learned object-layout correspondences, Janus-Pro extracts semantic cues from global prompts to synthesize layout-consistent object descriptors. This makes the same Janus-Pro backbone serve as both image+text\(\rightarrow\)layout and text\(\rightarrow\)layout model [2503.21069].

The layout representation is not only textual. Bounding boxes are converted to binary masks and then to embeddings through three convolutional layers:
\[
E_{\text{bbox}}=
f_{\text{conv}_3}\circ f_{\text{conv}_2}\circ f_{\text{conv}_1}(M_{\text{binary}}),
\]
with channels \(16 \rightarrow 32 \rightarrow 64\) and stride 2 in each layer. The masks are bilinearly interpolated to match image-token dimensions and concatenated with visual tokens, so that Janus-Pro learns the alignment between textual layout annotations and visual features [2503.21069].

Stage I training for this layout parser samples 1M instances from **GRIT-20M**, filters them to **600K samples**, and runs with batch size 128 for **94K iterations on an A800 GPU**. In downstream synthesis, Janus-Pro’s layouts are consumed by **MIGLoRA**, a LoRA-based plug-in for SD1.5, SDXL, and SD3 in which the base diffusion weights are frozen and about **2M trainable parameters** are added, described as **86% fewer parameters** than ControlNet-style baselines [2503.21069].

The layout-planning evaluation reports Janus-Pro-1B at **Acc 90.65**, **Quality 72.10**, and **ReAcc 93.90**, compared with Qwen2.5-VL-7B at 92.58, 89.22, and 83.21, and MiniCPM3-4B at 77.25, 60.11, and 75.34. In this setting, Janus-Pro-1B is emphasized as a compact alternative with strong self-correction behavior. System-level results also improve when Janus-Pro parsing is used: on COCO \(512\times512\), MIGLoRA(SD1.5) to MIGLoRA\(^\text{JP(SD1.5)}\) changes FID from 16.0 to 15.7, LPIPS from 0.71 to 0.65, AP from 39.5 to 40.1, AP\(_{50}\) from 57.8 to 58.3, AR from 52.1 to 53.6, and IoU from 64.0 to 64.5 [2503.21069].

This downstream use clarifies a practical distinction. In the Janus-Pro paper, Janus-Pro is a unified autoregressive generator at \(384\times384\). In the multi-instance synthesis paper, Janus-Pro is not used to directly generate images; it is used as a structured planner whose outputs condition diffusion backbones up to \(1024\times1024\) [2501.17811] [2503.21069].

## 6. Limitations, failure modes, and research position

The base Janus-Pro paper identifies several limitations. Both understanding and generation operate at **\(384\times384\)** resolution. For understanding, this is said to be limiting for fine-grained tasks such as OCR and small-text reading. For generation, the resolution limit and the VQ tokenizer introduce reconstruction artifacts and can leave small regions, such as facial features, under-detailed. The paper also notes that it does not go deeply into safety mechanisms, while acknowledging that curated aesthetic data and synthetic prompts can bias the model toward certain styles and subjects [2501.17811].

The prompt-layout parsing paper states separately why Janus-Pro is not simply used alone for end-to-end multi-instance image generation. Two limits are given explicitly: a **resolution bottleneck**, since Janus-Pro can only generate at \(384\times384\), and **attention collapse with many masks**, because as the number of masks or objects exceeds 4, self-attention faces quadratic token growth,
\[
N_{\text{tokens}} \propto N_{\text{masks}}^2,
\]
which weakens interactions between text tokens and bounding-box tokens in dense scenes [2503.21069].

In broader methodological context, Janus-Pro is positioned against both unified multimodal systems and diffusion systems. Relative to unified models, the papers emphasize decoupled visual encoding and large-scale data/model scaling. Relative to diffusion systems such as SDXL, SD3, and DALL-E 3, Janus-Pro uses discrete autoregressive generation rather than diffusion, yet remains competitive on instruction-following metrics at its native resolution. In prompt-to-layout work, Janus-Pro-1B is presented as a compact alternative to larger planners such as Qwen2.5-VL-7B and MiniCPM3-4B, and as a more specialized layout-planning front end than generic LLM-based systems such as LayoutGPT, RPG, Ranni, and CreatiLayout [2501.17811] [2503.21069].

The future directions named across the two papers are consistent: higher image resolution, better visual tokenizers, possible end-to-end integration of layout generation and image synthesis, further evaluation on more complex benchmarks, and application of the layout-planning approach to larger diffusion backbones such as SD3.5 and FLUX.dev. This suggests that Janus-Pro occupies two related but distinct roles in current multimodal research: as a unified autoregressive model for understanding and image generation, and as a compact multimodal planner that can structure prompts for higher-resolution diffusion systems [2501.17811] [2503.21069].

Source: https://www.emergentmind.com/topics/janus-pro