UniCanvas: A Diffusion-base Unified Model for Text-in-Image Joint Generation

Abstract: Recent years have seen remarkable progress in unified vision-LLMs handling both multimodal understanding and generation within a single architecture. While autoregressive VLMs can reason across modalities, they fail to generate high-quality images. In contrast, diffusion models produce photorealistic visuals yet struggle to generate coherent text, making it challenging to develop a single unified model that can seamlessly handle both visual and text generation. Recent advances suggest that language can be effectively embedded within visual representations, allowing models to reason about textual semantics directly from images. To this end, we propose UniCanvas, a first attempt that unifies diffusion models to generate interleaved multimodal contents through text-in-image generation. Diffusion models naturally capture transformations on a shared pixel canvas, which can be viewed as world models of visual change. Instead of producing discrete text tokens, the model learns to represent language as visual patterns inside images, leveraging its inherent multimodal embedding space. This design allows the model to "draw" text naturally within a single pixel canvas during image synthesis, achieving seamless multimodal generation. Experiments demonstrate that UniCanvas improves performance over previous unified models, positioning text-in-image generation with diffusion models as a promising unified multimodal generation paradigm.

• The paper introduces a two-stage, pixel-native diffusion model that renders reasoning as optical text on a shared canvas before generating the next image state.
• UniCanvas achieves a 0.77 average success rate on VSP maze navigation, outperforming all interleaved unified baselines below 0.25, while reaching 0.84 with oracle states.
• Ablations show that text-in-image pre-training is essential, while long-horizon performance remains limited by state drift, hallucinated text, and cumulative visual synthesis errors.

Motivation and problem statement

UniCanvas addresses the persistent split in unified multimodal systems between text generation and image synthesis. Autoregressive VLMs reason well but synthesize poor images; diffusion models synthesize photorealistic images but generate text poorly. The authors argue that most unified architectures inherit three structural gaps: an objective gap (next-token prediction for text versus denoising for images), a representation gap (symbolic token space versus pixels), and an inference gap (reasoning confined to a language head decoupled from the visual generator). Their proposed remedy is to eliminate text tokens entirely: language is rendered as optical text on a shared, pixel-native canvas that also carries the image, so that both modalities are produced by the same diffusion mechanism under a single denoising-style objective. The paper positions the canvas as a world model of visual change, in which each generation step is a transformation-conditioned state update.

Method

The model is built on Qwen-Image-Edit-2509 (a DiT backbone), fine-tuned with LoRA rank 32 via DiffSynth. Generation is formulated as flow matching between latent states: a velocity network $v_\theta(x_t, t, c)$ is trained to predict $v_t = x_1 - x_0$ along linear interpolations between source and target latents.

The core mechanism is a two-stage generation procedure sharing a single backbone. In Stage 1 (visual reasoning), the model transforms the current image $x$ into a text-annotated latent $x^{\text{text}}_0$, embedding the inferred action or instruction as optical text, conditioned on a reasoning prompt. In Stage 2 (visual synthesis), the model maps the reasoning-enhanced canvas to the next scene state $x^{\text{state}}_0$, conditioned on an execution prompt. For multi-step tasks the predicted state is fed back as the new input, producing a trajectory of canvases that externalizes task progress.

Two auxiliary components address weaknesses the authors observed empirically. First, because flow-matching objectives only weakly constrain small text regions (the model can ignore them without affecting the global loss), a frozen-CLIP contrastive alignment loss ($\lambda_{\text{CLIP}} = 0.1$) ties rendered optical text to the intended semantics. Second, the authors found that existing open-source image-editing checkpoints cannot render legible in-image text at all; they therefore introduce a pre-training stage on VQA data (LLaVA-Instruct-150K mixed with OmniEdit) that formulates answer rendering as an image-editing task, regularized by an LPIPS perceptual loss on the text region. This pre-training stage proved decisive in the ablations.

Experimental results

Evaluation covers VSP maze navigation (grids 3–6), a curated 2,000-episode cooking Recipe dataset, and 2,000 RLBench robot trajectories annotated with GPT-4o. All baselines (Anole, MVoT, MMaDA, BAGEL) are fine-tuned on the same interleaved data.

Method	Grid 3	Grid 4	Grid 5	Grid 6	Avg.
Qwen2.5-VL-3B (finetuned)	0.88	0.82	0.73	0.47	0.72
GPT-4o (zero-shot)	0.68	0.58	0.35	0.24	0.46
Anole	0.13	0.08	0.03	0.01	0.06
MVoT	0.24	0.13	0.09	0.03	0.12
MMaDA	0.18	0.09	0.02	0.02	0.08
BAGEL-Text	0.72	0.60	0.52	0.34	0.55
BAGEL-Interleave	0.32	0.23	0.26	0.13	0.24
UniCanvas	0.92	0.81	0.79	0.57	0.77

The headline result is a 0.77 average success rate versus below 0.25 for all interleaved unified baselines. Notably, BAGEL-Text outperforms BAGEL-Interleave by a wide margin (0.55 vs 0.24), which the authors interpret as evidence that existing unified models reason better when visual synthesis is removed — a strong, somewhat contradictory claim about the current state of unified architectures. The authors attribute UniCanvas's advantage to the elimination of repeated cross-modality transitions that amplify representation mismatch and error accumulation over long horizons.

Qualitative comparisons against Nano-Banana (zero-shot, one- and two-stage prompting) show that the baseline hallucinates duplicated or missing agents and produces next-state frames inconsistent with its own generated text, whereas UniCanvas maintains coherent action–state alignment. Quantitative DINO and CLIP scores on Recipe and RLBench (e.g., DINO 80.72 and CLIP 89.79 under image+text conditioning on RLBench) confirm high perceptual fidelity, with the lower Recipe scores attributed to the intrinsic one-to-many nature of the task rather than model failure.

Analysis and ablations

Ablations on VSP quantify each component's contribution. Removing text-in-image pre-training is the most damaging, dropping average success from 0.77 to 0.23; removing the CLIP loss yields 0.55, and removing the two-stage decomposition yields 0.43. This confirms the authors' observation that pretrained editing checkpoints lack text-rendering capability and that optical text is otherwise under-constrained by the denoising objective.

An oracle experiment replacing model-generated states with ground-truth simulator states isolates decision quality: UniCanvas reaches 0.84 average (0.97/0.87/0.83/0.69), surpassing Qwen2.5-VL-3B (0.82) and approaching Qwen2.5-VL-7B (0.89). This indicates that residual performance loss stems primarily from compounding visual synthesis error rather than weak decision-making — a clear implication that improving state-update fidelity would translate directly into planning gains.

On general reasoning (COCO-QA 0.58, Visual7W 0.78 versus 0.72 and 0.86 for Qwen2.5-VL-3B), the model lags VLMs but remains competitive given that its textual supervision is limited to LLaVA data. Inference is slower than autoregressive VLMs (4.6× Qwen2.5-VL-7B per sample on VSP) but faster than other diffusion-based sequential baselines (MVoT at 5.1×, MindJourney at 9.4×), owing to conditioning only on the most recent frame.

Limitations and open questions

The paper is candid about failure modes. Long-horizon rollouts degrade through two distinct mechanisms: (i) correct optical text but drifting next-state predictions, and (ii) corrupted or hallucinated text (e.g., "LIIHT") that locks the model into an incorrect state for the remainder of the rollout. Both stem from cumulative error propagation in a pipeline where each step is conditioned only on the previous generated state. Long-form, multi-sentence reasoning is unreliable, though the authors show that chunking answers into sequential canvas updates with OCR-based context improves stability. Representative VQA failures include counting errors, character hallucination despite correct semantic intent, prompt-bias copying of question words, and reasoning mismatches. An emergent self-correction behavior is documented but not systematically characterized. Open questions left by the paper include how to suppress error accumulation over long rollouts, whether the paradigm can scale to rich free-form text beyond localized rendering, and how much additional textual supervision is needed to close the gap with instruction-tuned VLMs.

Conclusion

UniCanvas demonstrates that a single diffusion backbone, operating entirely in pixel space, can perform interleaved text-and-image generation with multi-step reasoning, achieving a 0.77 average success rate on VSP against sub-0.25 results for token-based unified baselines. The results support the paper's central claim — that keeping reasoning within a unified visual modality mitigates cross-modal error accumulation — while the oracle experiments show that rendering fidelity, not decision quality, is the current bottleneck. The text-in-image paradigm remains constrained by long-horizon drift and weak long-form text generation, and the paper leaves open whether these can be resolved by scale or require architectural changes to the canvas-update mechanism.