Janus-Pro: Unified Multimodal Model

• Janus-Pro is a unified multimodal model that integrates image-text understanding and text-to-image generation within a single autoregressive transformer.
• It employs decoupled visual encoding with separate pathways for semantic alignment and reconstructive tokenization to balance high-level interpretation and detailed image synthesis.
• The model utilizes a three-stage training curriculum and fine-tuning for prompt-layout parsing to enhance performance on both multimodal benchmarks and image generation tasks.

Janus-Pro is a unified multimodal model family that performs both multimodal understanding and text-to-image generation with a single autoregressive transformer, while using decoupled visual encoding for the two tasks. In its core formulation, it is an improved successor to Janus, with changes in training strategy, training data, and model scaling; in later work, Janus-Pro-1B is also fine-tuned into a prompt-layout parsing module that converts free-form prompts into structured object layouts for downstream diffusion-based multi-instance synthesis (Chen et al., 29 Jan 2025, Qi et al., 27 Mar 2025).

1. Unified multimodal scope and design objective

Janus-Pro is defined as a unified multimodal model that can both understand images and text and generate images from text instructions. The central architectural claim is that these tasks are handled by one transformer rather than by separate understanding and generation backbones. In the base system, “unified multimodal understanding and generation” means that the same autoregressive language backbone answers questions about images, describes or reasons about them, and predicts image tokens to synthesize images from text prompts (Chen et al., 29 Jan 2025).

The model inherits Janus’s decoupled visual encoding design. For understanding, it uses a vision encoder specialized for semantic alignment; for generation, it uses a tokenizer specialized for reconstructive image tokenization. Both streams are projected into the same language-model embedding space through separate adapters. This arrangement is explicitly presented as a way to mitigate the objective conflict between semantic visual understanding and low-level image token prediction (Chen et al., 29 Jan 2025).

Janus-Pro is released in two principal variants:

Variant	Backbone and scale	Core transformer configuration
Janus-Pro-1B	DeepSeek-LLM 1.5B, named as the “1B series”	embedding size 2048, 24 layers, 16 heads, context length 4096
Janus-Pro-7B	DeepSeek-LLM 7B	embedding size 4096, 30 layers, 32 heads, context length 4096

This design places Janus-Pro in the family of unified multimodal systems such as Show-o, VILA-U, Chameleon, Transfusion, Monoformer, and Emu3. The distinctive point emphasized in the Janus-Pro work is not merely joint training, but joint training with separated visual front ends and a shared autoregressive core (Chen et al., 29 Jan 2025).

2. Architecture, tokenization, and sequence formulation

Janus-Pro retains four principal components: a language backbone, an understanding encoder, a generation tokenizer, and autoregressive heads. The understanding side uses SigLIP-Large-Patch16-384, with input resized or padded to $384 \times 384$, followed by a two-layer MLP understanding adapter. The generation side uses a VQ tokenizer with codebook size 16,384 and downsampling factor 16, so that a $384 \times 384$ image becomes a discrete sequence on the order of $24 \times 24 = 576$ tokens, followed by a two-layer MLP generation adapter and a separate image prediction head over the codebook (Chen et al., 29 Jan 2025).

At the sequence level, all tasks are cast as autoregressive next-token prediction. Text-only tasks use ordinary text-token sequences. Image-to-text tasks insert visual tokens from the understanding encoder into the sequence and predict text tokens. Text-to-image tasks begin with prompt tokens and then predict image VQ tokens. The image generation factorization is written as

$$p(\mathbf{y}\mid \mathbf{x})=\prod_{t=1}^{T} p(y_t\mid y_{<t},\mathbf{x}),$$

with $\mathbf{x}$ the prompt tokens and $\mathbf{y}$ the image-token sequence (Chen et al., 29 Jan 2025).

The same paper also makes the training objectives explicit in standard autoregressive form. For multimodal understanding,

$$\mathcal{L}_{\text{understand}}= -\mathbb{E}_{(\mathbf{v},\mathbf{x},\mathbf{y})} \left[ \sum_{t=1}^{|\mathbf{y}|} \log p_{\theta}(y_t\mid \mathbf{v},\mathbf{x},y_{<t}) \right],$$

and for text-to-image generation,

$$\mathcal{L}_{\text{gen}}= -\mathbb{E}_{(\mathbf{x},\mathbf{z})} \left[ \sum_{t=1}^{|\mathbf{z}|} \log p_{\theta}(z_t\mid \mathbf{x},z_{<t}) \right].$$

In practice, Janus-Pro mixes understanding, pure text, and generation batches according to task ratios rather than through a separately weighted multi-objective optimizer (Chen et al., 29 Jan 2025).

A notable architectural boundary is that Janus-Pro’s native generation path is fully autoregressive rather than diffusion-based. This matters later, because the downstream layout-planning use of Janus-Pro in multi-instance generation does not reuse Janus-Pro as an image generator; it reuses it as a multimodal planner (Qi et al., 27 Mar 2025).

3. Training curriculum and data scaling

Janus-Pro uses a refined three-stage curriculum. Stage I is a warm-up phase for the understanding adapter, generation adapter, and image prediction head, with the LLM frozen. Its data ratio is $1:0:3$ for understanding:text:generation, it runs for 20K steps with batch size 256, and it is intended in part to let the generation stack learn pixel dependence on ImageNet class-label prompts (Chen et al., 29 Jan 2025).

Stage II is unified pretraining. Here Janus-Pro departs from Janus by dropping ImageNet from the main text-to-image phase after Stage I, and using only normal text-to-image data with dense prompts. The stated purpose is to improve both efficiency and semantic alignment. Stage II uses learning rate $1.0\times10^{-4}$, constant scheduling with 5000 warm-up steps, AdamW with $384 \times 384$0, $384 \times 384$1, weight decay 0, gradient clipping 1.0, batch size 512, planned 360K steps with early stopping at 270K, and data ratio $384 \times 384$2 (Chen et al., 29 Jan 2025).

Stage III is supervised fine-tuning. SigLIP is unlocked in this stage. Janus-Pro changes the Janus Stage III ratio from $384 \times 384$3 to $384 \times 384$4, explicitly reducing the text-to-image proportion relative to Janus in order to preserve or improve generation while strengthening multimodal understanding. Stage III uses learning rate $384 \times 384$5, no warm-up, batch size 128, and 80K steps for the 1B series or 40K for the 7B series (Chen et al., 29 Jan 2025).

Data scaling is a central part of the Janus-Pro revision. On the understanding side, Stage II adds about 90M multimodal samples, including image-caption data and document-oriented sources such as Docmatix; Stage III adds instruction-style and conversational data, including MEME understanding, Chinese conversational data, and dialogue-oriented mixtures. On the generation side, the paper reports adding 72M synthetic aesthetic samples, roughly balancing real and synthetic data $384 \times 384$6 in Stage II. The stated effect is faster convergence, more stable outputs, and significantly higher aesthetic quality (Chen et al., 29 Jan 2025).

Training is implemented with HAI-LLM, a lightweight distributed framework built on PyTorch. The 1.5B model is reported to train for about 9 days on 16 A100 nodes with 8 GPUs each, and the 7B model for about 14 days on 32 such nodes (Chen et al., 29 Jan 2025).

4. Empirical performance and scaling behavior

On multimodal understanding benchmarks, Janus-Pro improves substantially over Janus. Janus-Pro-1B is reported at POPE 86.2, MME-P 1444.0, MMB 75.5, SEED 68.3, GQA 59.3, MMMU 36.3, and MM-Vet 39.8, versus Janus at POPE 87.0, MME-P 1338.0, MMB 69.4, SEED 63.7, GQA 59.1, MMMU 30.5, and MM-Vet 34.3. Janus-Pro-7B further reaches POPE 87.4, MME-P 1567.1, MMB 79.2, SEED 72.1, GQA 62.0, MMMU 41.0, and MM-Vet 50.0 (Chen et al., 29 Jan 2025).

The generation results are likewise strong on instruction-following benchmarks. On GenEval, Janus-Pro-1B reaches an overall score of 0.73 and Janus-Pro-7B reaches 0.80; the same table lists SD3-Medium at 0.74, DALL-E 3 at 0.67, Janus at 0.61, and Emu3-Gen at 0.54. On DPG-Bench, Janus-Pro-1B reaches 82.63 and Janus-Pro-7B reaches 84.19, slightly above SD3-Medium at 84.08 and above DALL-E 3 at 83.50, while Janus is reported at 79.68 (Chen et al., 29 Jan 2025).

The 7B model’s GenEval sub-scores are also specified: Single Object 0.99, Two Objects 0.89, Counting 0.59, Colors 0.90, Position 0.79, and Color Attributes 0.66. The scaling discussion in the same work states that the 7B model converges faster in both multimodal understanding loss and text-to-image generation loss, and that the decoupled visual encoding plus unified autoregressive transformer architecture scales effectively with LLM size (Chen et al., 29 Jan 2025).

Qualitative comparisons in the paper focus especially on short prompts. Janus-Pro is reported to produce cleaner and more detailed images than Janus for prompts such as “a cute cat” and “a red car on the street,” with fewer blurry or artifact-laden failures. The paper interprets this as improved text-to-image stability, particularly for short prompts (Chen et al., 29 Jan 2025).

5. Janus-Pro as a prompt-layout planner for multi-instance generation

A later use of Janus-Pro recasts Janus-Pro-1B as a compact multimodal planning model rather than as a direct image generator. In “Efficient Multi-Instance Generation with Janus-Pro-Driven Prompt Parsing,” Janus-Pro-1B is fine-tuned into a prompt-layout parsing module that maps prompts, and optionally images during training, into layouts wrapped by <layout> ... </layout>, with per-instance <scap> sub-captions and <bbox> coordinates $384 \times 384$7. The paper describes this module as “the language-layout ‘brain’ that sits in front of the diffusion models” (Qi et al., 27 Mar 2025).

The parser is trained in a two-way manner. Layout Understanding takes image plus textual sub-prompts and predicts bounding boxes and spatial relationships. Draft Generation uses an inverse training objective so that, conditioned on learned object-layout correspondences, Janus-Pro extracts semantic cues from global prompts to synthesize layout-consistent object descriptors. This makes the same Janus-Pro backbone serve as both image+text$384 \times 384$8layout and text$384 \times 384$9layout model (Qi et al., 27 Mar 2025).

The layout representation is not only textual. Bounding boxes are converted to binary masks and then to embeddings through three convolutional layers: $24 \times 24 = 576$0 with channels $24 \times 24 = 576$1 and stride 2 in each layer. The masks are bilinearly interpolated to match image-token dimensions and concatenated with visual tokens, so that Janus-Pro learns the alignment between textual layout annotations and visual features (Qi et al., 27 Mar 2025).

Stage I training for this layout parser samples 1M instances from GRIT-20M, filters them to 600K samples, and runs with batch size 128 for 94K iterations on an A800 GPU. In downstream synthesis, Janus-Pro’s layouts are consumed by MIGLoRA, a LoRA-based plug-in for SD1.5, SDXL, and SD3 in which the base diffusion weights are frozen and about 2M trainable parameters are added, described as 86% fewer parameters than ControlNet-style baselines (Qi et al., 27 Mar 2025).

The layout-planning evaluation reports Janus-Pro-1B at Acc 90.65, Quality 72.10, and ReAcc 93.90, compared with Qwen2.5-VL-7B at 92.58, 89.22, and 83.21, and MiniCPM3-4B at 77.25, 60.11, and 75.34. In this setting, Janus-Pro-1B is emphasized as a compact alternative with strong self-correction behavior. System-level results also improve when Janus-Pro parsing is used: on COCO $24 \times 24 = 576$2, MIGLoRA(SD1.5) to MIGLoRA$24 \times 24 = 576$3 changes FID from 16.0 to 15.7, LPIPS from 0.71 to 0.65, AP from 39.5 to 40.1, AP$24 \times 24 = 576$4 from 57.8 to 58.3, AR from 52.1 to 53.6, and IoU from 64.0 to 64.5 (Qi et al., 27 Mar 2025).

This downstream use clarifies a practical distinction. In the Janus-Pro paper, Janus-Pro is a unified autoregressive generator at $24 \times 24 = 576$5. In the multi-instance synthesis paper, Janus-Pro is not used to directly generate images; it is used as a structured planner whose outputs condition diffusion backbones up to $24 \times 24 = 576$6 (Chen et al., 29 Jan 2025, Qi et al., 27 Mar 2025).

6. Limitations, failure modes, and research position

The base Janus-Pro paper identifies several limitations. Both understanding and generation operate at $24 \times 24 = 576$7 resolution. For understanding, this is said to be limiting for fine-grained tasks such as OCR and small-text reading. For generation, the resolution limit and the VQ tokenizer introduce reconstruction artifacts and can leave small regions, such as facial features, under-detailed. The paper also notes that it does not go deeply into safety mechanisms, while acknowledging that curated aesthetic data and synthetic prompts can bias the model toward certain styles and subjects (Chen et al., 29 Jan 2025).

The prompt-layout parsing paper states separately why Janus-Pro is not simply used alone for end-to-end multi-instance image generation. Two limits are given explicitly: a resolution bottleneck, since Janus-Pro can only generate at $24 \times 24 = 576$8, and attention collapse with many masks, because as the number of masks or objects exceeds 4, self-attention faces quadratic token growth,

$24 \times 24 = 576$9

which weakens interactions between text tokens and bounding-box tokens in dense scenes (Qi et al., 27 Mar 2025).

In broader methodological context, Janus-Pro is positioned against both unified multimodal systems and diffusion systems. Relative to unified models, the papers emphasize decoupled visual encoding and large-scale data/model scaling. Relative to diffusion systems such as SDXL, SD3, and DALL-E 3, Janus-Pro uses discrete autoregressive generation rather than diffusion, yet remains competitive on instruction-following metrics at its native resolution. In prompt-to-layout work, Janus-Pro-1B is presented as a compact alternative to larger planners such as Qwen2.5-VL-7B and MiniCPM3-4B, and as a more specialized layout-planning front end than generic LLM-based systems such as LayoutGPT, RPG, Ranni, and CreatiLayout (Chen et al., 29 Jan 2025, Qi et al., 27 Mar 2025).

The future directions named across the two papers are consistent: higher image resolution, better visual tokenizers, possible end-to-end integration of layout generation and image synthesis, further evaluation on more complex benchmarks, and application of the layout-planning approach to larger diffusion backbones such as SD3.5 and FLUX.dev. This suggests that Janus-Pro occupies two related but distinct roles in current multimodal research: as a unified autoregressive model for understanding and image generation, and as a compact multimodal planner that can structure prompts for higher-resolution diffusion systems (Chen et al., 29 Jan 2025, Qi et al., 27 Mar 2025).