Multi-Task Canvas Training

• Multi-Task Canvas Training Strategy is a unified approach that encodes spatial, pose, and layout controls into a single composite canvas for text-to-image synthesis.
• The method leverages diverse canvas types to enhance cross-modal reasoning, enabling robust generalization to novel combinations of user-specified controls.
• By integrating a diffusion transformer and LoRA-based fine-tuning in a shared training loop, this strategy overcomes limitations of fragmented, modular pipelines.

Multi-Task Canvas Training Strategy is a unified paradigm for training text-to-image diffusion models to perform compositional image generation under heterogeneous, multimodal control regimes. Developed in the context of the Canvas-to-Image framework, this approach encodes various user-specified control signals—such as subject references, spatial arrangements, pose constraints, and layout annotations—into a single composite “canvas” input. By jointly training a diffusion backbone on multiple control modalities within the same learning loop, Multi-Task Canvas Training enables integrated visual-spatial reasoning and robust generalization to novel combinations of controls not seen during training (Dalva et al., 26 Nov 2025).

1. Motivation and Conceptual Foundation

Prevailing latent diffusion models are highly performant for unconstrained image synthesis but demonstrate clear failure modes when tasked with concurrent, fine-grained controls (e.g., specifying both layout and pose for multiple referenced identities). Prior methods have typically resorted to modular pipelines, stacking independently trained networks—for example, combining ControlNet for pose constraints with IP-Adapter for subject reference—resulting in fragmented, brittle implementations that lack cross-modal consistency and fail to generalize to compositional queries.

Multi-Task Canvas Training addresses these limitations by encoding all user controls into a “Visual Canvas”: a single RGB image that integrates reference cutouts, pose skeletons, bounding boxes, and textual tags. This enables a single pretrained diffusion model to learn fully compositional, cross-modal interpretation, as opposed to relying on task-specific heuristics or sequential control modules. The model samples and trains on different canvas types (spatial, pose, box) within a shared training loop, using task-indicator tokens to condition and decouple representations while maintaining compatibility across modalities (Dalva et al., 26 Nov 2025).

2. Multi-Task Dataset Construction

The strategy leverages three principal canvas variants curated from two data sources:

• Spatial Canvas: Composed from approximately six million cross-frame, human-centric images covering one million unique identities. Subject cutouts from one frame are pasted into desired positions on a masked background from a different frame, forming a composite canvas specifying spatial arrangement and subject identity. The target supervision is the original unmasked frame.
• Pose Canvas: Derived from the same human-centric dataset. Here, a semi-transparent 2D pose skeleton (such as from OpenPose) is overlaid on a spatial canvas. The training protocol may randomly omit subject cutouts, forcing the model to learn plausible generation given pose alone or in combination with references. The target must preserve both identity and pose articulation.
• Box Canvas: Sourced from an augmented CreatiDesign layout dataset, with bounding boxes and textual tags rendered directly onto the canvas. No reference subject cutouts are included; instead, the layout (bounding boxes) and associated text (entity names) serve as sole conditioning cues. The target requires the specified entities to be accurately placed in the designated regions.

Each training sample consists of a tuple (canvasImage, textPrompt with taskIndicator token, targetImage), with tasks uniformly sampled to ensure balanced representation. This provides comprehensive multi-modal supervision and exposes the model to all task variants without segmentation into isolated training regimes (Dalva et al., 26 Nov 2025).

Canvas Type	Control Modality	Dataset Source
Spatial	Subject reference, location	Human-centric “cross-frame” dataset
Pose	Reference, pose constraints	Human-centric dataset + pose labels
Box	Layout, text tags	CreatiDesign + annotations

3. Model Architecture and Input Encoding

Multi-Task Canvas Training builds on a DiT-style diffusion transformer (Qwen-Image-Edit), extended as follows:

• Vision-LLM (VLM) Encoder: Receives the 3-channel visual canvas and text prompt (prepended with “[Spatial]”, “[Pose]”, or “[Box]”) to compute cross-modal embeddings $h_{\mathrm{VLM}}$.
• VAE Encoder: Maps the canvas image to a latent code $z_{\mathrm{canvas}}$, facilitating direct latent-space conditioning.
• Conditioning Pipeline: At each diffusion denoising step, the model processes a noisy latent $x_t$, the concatenated conditioning vector $$[\text{textEmbedding}; h_{\mathrm{VLM}}; z_{\mathrm{canvas}}]$$, and the task-indicator token $c$.
• Fusion Mechanism: Attention and cross-attention in the transformer backbone jointly process these components in all layers.
• Fine-Tuning Regime: Only attention, image-modulation, and text-modulation layers are fine-tuned using LoRA (rank=128); feed-forward layers are frozen, preserving pretrained generative priors.

This holistic input encoding ensures that the model operates on all compositional signals simultaneously and facilitates task-specific disambiguation via the indicator token (Dalva et al., 26 Nov 2025).

4. Training Objectives and Optimization

Training employs the flow-matching (velocity) objective for denoising diffusion probabilistic models. With $x_0$ as the VAE-encoded target image, $x_1$ as a latent of a random background, and $x_t = (1-\alpha_t)x_0 + \alpha_t x_1$ at time step $t$, the model predicts $v^* = x_0 - x_1$, optimizing:

$$\mathcal{L}_{\mathrm{flow}} = \mathbb{E}_{x_0,x_1,t,h,c}\left[ \left\| v_\theta(x_t, t, [h; c]) - (x_0 - x_1) \right\|_2^2 \right]$$

A single objective is applied for all canvas types; no auxiliary decoders or custom loss terms are introduced per modality. Task disambiguation is handled by prepending the task-indicator token $c$ to the text prompt—not by explicit loss penalization (the auxiliary loss term $\mathcal{L}_{\text{taskIndicator}}$ is effectively infinite, as performance collapses without the token). The overall loss is:

$$\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{flow}}\ \text{across all canvas types} \ +\ \lambda_{\text{task}} \mathcal{L}_{\text{taskIndicator}}$$

where nonzero $\mathcal{L}_{\text{taskIndicator}}$ occurs only when the model confuses task modes, though in practice no explicit auxiliary loss is implemented (Dalva et al., 26 Nov 2025).

5. Training Protocol and Hyperparameters

The uniform sampling strategy is adopted: at each iteration, one of {Spatial, Pose, Box} canvas types is selected with $p=1/3$. The protocol specifies:

• Batch Size: 32, with a task-uniform mix.
• Optimizer: AdamW, learning rate $5 \times 10^{-5}$, weight decay $1 \times 10^{-2}$.
• Parameter Updates: LoRA rank 128 for attention, text-, and image-modulation layers; feed-forward layers remain frozen.
• Training Steps: 200,000 iterations on 32 A100 GPUs, totaling approximately 3 days.
• Curriculum: No explicit scheduling; uniform multi-task sampling throughout. Early validation indicates control fidelity saturates after approximately 50k steps, but visual quality continues improving until the end of training (Dalva et al., 26 Nov 2025).

6. Experimental Results and Ablations

Ablative experiments using the “Multi-Control” benchmark (simultaneous pose, layout, and identity controls) establish the contribution of each canvas component:

Trained Task(s)	ArcFace	HPSv3	Control-QA	VQAScore
Spatial only	0.389	10.79	4.16	—
+ Pose	0.389	11.44	4.19	—
+ Box	0.389	12.04	4.28	0.906

Findings:

• Spatial-only training yields suboptimal performance; the model disregards poses or bounding boxes not seen during training.
• Addition of Pose Canvas improves pose adherence; only then does the model align outputs with pose skeletons.
• Addition of Box Canvas further increases control fidelity (HPSv3, Control-QA, VQAScore), demonstrating effective layout compliance.

Qualitative inspection corroborates that single-task models are unable to respect controls they were not exposed to in composition, whereas multi-task trained models exhibit robust generalization and compositional reasoning—correctly synthesizing images under novel combinations such as simultaneous pose, layout, and identity constraints (Dalva et al., 26 Nov 2025).

7. Significance and Implications

Multi-Task Canvas Training provides a unified backbone for compositional control, eliminating the need for modular model stacks and specialized heuristics. By exposing the diffusion model to all control modalities within a shared and balanced training curriculum, the approach supports integrated, user-friendly control interfaces and enables practitioners to specify complex, multimodal requirements in a natural canvas format.

A plausible implication is broader applicability to nonhuman-centric domains and to additional control modalities, provided suitable canvas encodings and data are constructible. The demonstrated gains in identity preservation and control adherence across a suite of compositional benchmarks (notably in multi-person, pose-constrained, and layout-aware settings) establish this strategy as a baseline for future multimodal diffusion model research (Dalva et al., 26 Nov 2025).