Exploring MLLM-Diffusion Information Transfer with MetaCanvas (2512.11464v1)

Abstract: Multimodal learning has rapidly advanced visual understanding, largely via multimodal LLMs (MLLMs) that use powerful LLMs as cognitive cores. In visual generation, however, these powerful core models are typically reduced to global text encoders for diffusion models, leaving most of their reasoning and planning ability unused. This creates a gap: current multimodal LLMs can parse complex layouts, attributes, and knowledge-intensive scenes, yet struggle to generate images or videos with equally precise and structured control. We propose MetaCanvas, a lightweight framework that lets MLLMs reason and plan directly in spatial and spatiotemporal latent spaces and interface tightly with diffusion generators. We empirically implement MetaCanvas on three different diffusion backbones and evaluate it across six tasks, including text-to-image generation, text/image-to-video generation, image/video editing, and in-context video generation, each requiring precise layouts, robust attribute binding, and reasoning-intensive control. MetaCanvas consistently outperforms global-conditioning baselines, suggesting that treating MLLMs as latent-space planners is a promising direction for narrowing the gap between multimodal understanding and generation.

• The paper introduces MetaCanvas, which directly transfers MLLM reasoning into diffusion models by using learnable latent canvas tokens at patch-level granularity.
• The methodology integrates multimodal RoPE, lightweight connector modules, and dynamic conditioning to achieve precise spatial, temporal, and compositional control.
• Experimental evaluations demonstrate that MetaCanvas outperforms baselines in text-to-image, video editing, and in-context video generation tasks with improved convergence and fidelity.

MetaCanvas: Enabling Explicit MLLM-to-Diffusion Information Transfer via Learnable Latent Canvas

Motivation and Problem Setting

A fundamental challenge in multimodal generation systems is the underutilization of Multimodal LLMs (MLLMs) when interfaced with diffusion models. While MLLMs possess strong reasoning and planning capabilities, their interaction with diffusion backbones is typically mediated via global conditioning—compressing complex multimodal knowledge into a single embedding that is used as a coarse prompt. This paradigm limits spatial, compositional, and temporal control, leading to generation outputs that lack precise layout, attribute binding, or fine-grained edits. The "Exploring MLLM-Diffusion Information Transfer with MetaCanvas" (2512.11464) paper introduces an architecture to overcome these constraints, proposing MetaCanvas as a lightweight, extensible bridge that enables MLLMs to directly plan and inject information at patch-level granularity in the latent space of diffusion models.

Architecture Overview

MetaCanvas augments the MLLM input sequence with a set of learnable, multidimensional canvas tokens that are processed using multimodal RoPE. These tokens, appended after the end-of-sentence marker, serve as implicit visual/spatiotemporal sketches that the MLLM can utilize for explicit layout or temporal planning. After encoding by the MLLM, both the context tokens (from text and multimodal input) and the canvas tokens are processed by dedicated lightweight connectors and injected into the diffusion model (DiT) at a patch-wise level.

Figure 1: MetaCanvas architecture, illustrating tokenization, multimodal encoding, MLLM context and canvas connectors, and patchwise fusion with diffusion noisy latents.

The connector module comprises two key blocks: a vanilla transformer block for canvas token alignment with the DiT latent space, and a DiT block with Adaptive LayerNorm, implementing a ControlNet-style schema to allow dynamic modulation of the injected priors. Zero-initialization is employed to ensure initial training stability.

Figure 2: Detailed MetaCanvas connector for alignment and fusion of canvas-token embeddings with diffusion latents.

For video tasks, a sparse keyframe canvas mechanism is introduced, where learnable keyframes are linearly interpolated to cover the full latent-frame space, maintaining temporal coherence while minimizing interface bandwidth.

Figure 3: Injection strategy for MetaCanvas video keyframes and reference/condition frames fusion in DiT patch embedding.

Experimental Evaluation

MetaCanvas is instantiated over three diffusion backbones and benchmarked on a diverse suite of six tasks, including text-to-image generation, text/image-to-video generation, image/video editing, and in-context video generation. Its efficacy is established via extensive qualitative and quantitative studies across GenEval, GEdit, ImgEdit, VBench, GPT-4o evaluations, and custom human assessment protocols.

Exploratory Text-to-Image Generation

Canvas tokens yield strong improvements in both convergence speed and attribute alignment on GenEval. MetaCanvas outperforms both vanilla text conditioning and 1D query-embedding baselines (MetaQuery, BLIP-3o), demonstrating that explicit spatial priors are essential for precise reasoning-intensive synthesis.

Figure 4: GenEval score convergence, contrasting MetaCanvas with global text and query-token baselines.

The latent canvas, even without text signals, is capable of producing meaningful visual plans that the DiT can render with semantic fidelity.

Figure 5: Canvas token PCA feature visualizations (row 1) and corresponding generated samples (row 2) using only canvas tokens.

Image Editing

MetaCanvas-augmented FLUX.1-Kontext [Dev] consistently achieves higher scores on both GEdit (semantic edit accuracy, overall quality) and ImgEdit (edit operations: Add, Remove, Replace, etc.) benchmarks compared to text-only or query-only variants. These gains are achieved with minimal additional parameter and compute overhead.

Figure 6: Training loss and GEdit score comparison between baseline and MetaCanvas-enhanced models.

Video Generation and Editing

On video benchmarks (VBench-I2V, VBench-Q), MetaCanvas matches or exceeds state-of-the-art open-source models in overall video quality, while displaying notable superiority in prompt-following accuracy and spatiotemporal consistency on video editing tasks. Human preference assessments on local/global edits and background/style manipulations strongly favor MetaCanvas over Ditto and Lucy-Edit architectures.

Figure 7: Video editing visualizations showing precise edit localization, spatial alignment, and detail preservation by MetaCanvas.

Sparse keyframe ablation studies indicate three canvas keyframes provide an optimal trade-off between temporal coherence and computational efficiency, outperforming dense or single-frame alternatives in both automated scores and observed artifact minimization.

In-context Video Generation

On OmniContext-Video, MetaCanvas demonstrates robust multi-reference composition and prompt grounding, performing competitively or better than Wan-VACE baselines, especially for multi-ID scenarios. The latent canvas efficiently enables the synthesis of videos that align multiple reference images under complex, instruction-driven specifications.

Figure 8: Visualization of in-context video generation where reference images are accurately transformed into the target scene per textual instructions.

Comparative Architecture Analysis

MetaCanvas bridges the gap between high-level MLLM cognition and low-level generative control, outperforming prior approaches that rely solely on global sequence conditioning.

Figure 9: Schematic visualizations of competing architectures, highlighting MetaCanvas’ explicit spatial-temporal interfacing.

Key ablation results demonstrate the necessity of patch-wise injection, transformer-driven canvas alignment, and dynamic conditioning for maximal structural fidelity and editability.

Figure 10: Connector architecture ablation visualizations showing the effect of architectural variants on canvas-aligned synthesis.

Implications and Future Directions

Practically, MetaCanvas introduces a scalable and cost-effective blueprint for synthesizing high-fidelity, compositionally controlled media across domains. It enables frozen or LoRA-augmented MLLMs to serve as decomposable planners for both images and videos, leveraging their native capabilities without extensive retraining. These results substantiate the hypothesis that latent-space spatial conditioning is critical for bridging generative and understanding tasks, and have direct implications for improving interactive creative workflows, controllable editing, and procedurally grounded content synthesis. The framework is extensible to more modalities, higher resolutions, and longer temporal horizons.

Theoretically, MetaCanvas demonstrates that explicit latent canvases allow for richer transfer of multimodal reasoning abilities and are more effective than single-sequence embeddings for structured scene generation. This insight can inform future research in model architectures, multimodal alignment, and general-purpose generator interfaces.

Further work may explore fully unified routing strategies, advanced temporal planning mechanisms, and more intricate interfacing for 3D or embodied environments. Improving data curation and scaling of reference-based video synthesis tasks are also highlighted as important avenues.

Conclusion

MetaCanvas defines a principled mechanism for interfacing MLLMs with diffusion models, architected around learnable multi-dimensional latent canvases and lightweight connectors. It achieves high performance across six diverse multimodal generative and editing tasks while maintaining efficiency and extensibility. The findings support the assertion that explicit, patch-level spatial-temporal grounding is essential for unlocking the full representational and reasoning potential of MLLMs in structured visual and video generation.

(2512.11464)