Sketch-Guided Diffusion Transformer

• The paper introduces transformer mechanisms that incorporate sketch-derived signals to constrain the denoising process, enhancing spatial layout and structural control.
• It leverages hybrid architectures that fuse latent diffusion models with self- and cross-attention, integrating freehand sketches with textual and image cues.
• Empirical results demonstrate improved performance in image and video synthesis tasks, highlighting enhanced realism, compositionality, and conditional controllability.

Sketch-Guided Diffusion Transformer denotes a family of diffusion-based generative systems in which sketch information—freehand line art, edge maps, layered concept sketches, or full sketch sequences—constrains denoising inside a transformer or transformer-like attention backbone. In the current literature, the term covers both explicit Diffusion Transformer (DiT) backbones and latent diffusion U-Nets whose self-attention and cross-attention mechanisms are treated as directly transferable to DiT-style implementations. The common objective is to combine the structural specificity of sketches with the realism, compositionality, and conditional controllability of modern diffusion priors, whether for image synthesis, editing, morphing, reconstruction from memory, video colourization, or even colored point cloud generation (Chen et al., 30 Jun 2025, Sadihin et al., 2 Jul 2025, Chen et al., 2024).

1. Conceptual scope

Sketch guidance addresses a persistent asymmetry in generative modeling. Text prompts are rich in semantics and appearance cues but are ambiguous about layout, relative placement, and exact geometry; sketches provide strong layout and shape priors but are rough, abstract, and typically underspecified in color, texture, and lighting. Sketch-guided diffusion systems therefore treat sketches as structural constraints and diffusion priors as realism-inducing distributions. In several works, the sketch signal is not merely auxiliary: it is the primary determinant of spatial organization, while text or reference imagery supplies semantics, appearance, or relational context (Chen et al., 30 Jun 2025, Mitsouras et al., 2024, Roy et al., 19 Feb 2025).

The literature is heterogeneous. Some systems are explicitly transformer-based, such as DiT video colorization and efficient latent diffusion transformers; others retain Stable Diffusion-style latent U-Nets but rely on transformer attention blocks, cross-attention maps, or token-like conditioning strategies that are described as directly portable to DiT formulations. A useful operational definition is therefore architectural rather than taxonomic: a Sketch-Guided Diffusion Transformer is any diffusion system in which sketch-derived signals modulate denoising through attention, token fusion, latent initialization, or latent optimization in a transformer-compatible manner (Sadihin et al., 2 Jul 2025, Chen et al., 2024, Ding et al., 2024).

System	Backbone / conditioning	Distinguishing property
Subjective Camera	Latent diffusion with self-attention and cross-attention	Sequence-aware concept-sequential generation
SketchColour	CogVideoX-5B-I2V DiT with channel concat and LoRA	Reference-based sketch-to-colour video
EDT	Latent diffusion transformer	Sketch-inspired alternating local/global attention
U-Sketch	Stable Diffusion with latent edge predictor	Gradient-based latent edge guidance
d-Sketch	Frozen SD2.1 with LCTN	No retraining of diffusion backbone
DiffMorph	Stable Diffusion plus ConditionFlow	Prompt-less sketch-guided morphing

2. Backbone architectures and tokenization

A major line of work retains latent diffusion as the generative substrate while changing how sketches enter the model. In Subjective Camera, the backbone is a latent diffusion model akin to Stable Diffusion, with a VAE encoder $E_\phi$, decoder $D_\phi$, and a U-Net noise predictor $\epsilon_\theta$ using self-attention and cross-attention with text embeddings. The paper explicitly notes that, although the backbone is not a pure DiT, it is effectively a sketch-guided diffusion with Transformer attention, and its mechanisms transfer directly to a configuration in which the U-Net is replaced by a DiT (Chen et al., 30 Jun 2025). A similar hybrid view appears in DiffMorph, which uses Stable Diffusion together with ConditionFlow, a modified ControlNet, while emphasizing that the underlying U-Net already contains self-attention and cross-attention layers and is therefore a hybrid CNN–Transformer architecture (Chatterjee, 2024).

Explicit DiT instantiations appear most clearly in video and efficiency-oriented work. SketchColour adapts CogVideoX-5B-I2V, a DiT-based image-to-video model, with a frozen 3D VAE and spatio-temporal latents

$$Z \in \mathbb{R}^{\frac{T}{4} \times \frac{H}{4} \times \frac{W}{4} \times 16}.$$

The model tokenizes the entire video volume into spatio-temporal patches and performs global self-attention over all frames jointly, so temporal coherence is not added as a separate module but emerges from 3D latent structure and transformer attention over the full clip (Sadihin et al., 2 Jul 2025). EDT, by contrast, is a latent diffusion transformer organized as a lightweight encoder–decoder in token space, with down-sampling and up-sampling stages, long skip connections, AdaLN conditioning, and an alternation of global and locally modulated attention in decoder stages (Chen et al., 2024).

Transformer-specific guidance has also been introduced directly at the denoiser level. Internal Guidance adds an auxiliary prediction head to an intermediate transformer layer and, at sampling time, extrapolates between intermediate and final predictions: $$D_w(\mathbf{x}_t; \mathbf{c}) = D_i(\mathbf{x}_t; \mathbf{c}) + w\left(D_f(\mathbf{x}_t; \mathbf{c}) - D_i(\mathbf{x}_t; \mathbf{c})\right).$$ The proposal is class-agnostic and conditioning-agnostic, and the paper explicitly frames it as transferable to sketch-conditioned Diffusion Transformers by attaching the auxiliary head after a sketch-aware early–mid layer (Zhou et al., 30 Dec 2025).

3. Conditioning pathways and latent control

The simplest transformer-compatible conditioning strategy is direct latent fusion. SketchColour encodes the colored first frame, the noisy target video, and the sketch sequence with the same frozen 3D VAE and concatenates them along the channel dimension: $$Z_{\text{concat}} = \text{concat}\big(Z_{I_{\text{start}}}, Z_{V_{\text{GT}}}, Z_{\text{sketch}}\big) \in \mathbb{R}^{T' \times H' \times W' \times 48}.$$ The patch embedding layer is then extended from 32 to 48 input channels, with the newly added sketch-channel weights initialized to zero, so the pretrained model initially ignores sketch channels and only gradually learns to use them during LoRA fine-tuning (Sadihin et al., 2 Jul 2025). DiffMorph adopts a more explicit conditional branch: ConditionFlow modifies ControlNet by removing the skip connections of the base input image from encoder to decoder, so the decoder is less influenced by original image spatial features and more dependent on the sketch condition (Chatterjee, 2024). DiffFaceSketch uses yet another pathway: a Multi-Auto-Encoder encodes five facial regions into sketch features, a conditioning decoder expands them to an 8-channel feature map, and that map is concatenated with the latent diffusion input to yield an 11-channel U-Net input (Peng et al., 2023).

A second line of work does not inject sketches as fixed features at every layer, but instead constructs sketch-aware initial conditions or updates latents during sampling. d-Sketch uses a lightweight Latent Code Translation Network (LCTN) that maps sketch-conditioned intermediate U-Net features to an initial latent $z_0$ and then applies partial noising and denoising through a frozen SD2.1 prior. The only training objective is

$\mathcal{L}_{LCTN} = \| z_0 - \overline{x} \|^2,$

with the diffusion backbone frozen throughout (Roy et al., 19 Feb 2025). U-Sketch trains a U-Net latent edge predictor on intermediate Stable Diffusion features and applies inference-time latent updates using gradients of an edge consistency loss between predicted latent edges and encoded sketch edges; guidance is applied only in the first half of DDIM steps, where global layout is decided (Mitsouras et al., 2024).

A third pathway uses attention maps as the structural interface. In the training-free latent-optimization framework, a sketch is first inverted with DDIM under a prompt of the form “a sketch of a CLS”, cross-attention maps for the class token are stored at every step, and generation then begins from fresh Gaussian noise under “a photo of a CLS”. At each guided step, the current latent is optimized so that its cross-attention maps match the stored sketch-derived maps using a symmetric KL loss, yielding structure control without any retraining (Ding et al., 2024). Subjective Camera extends this latent-optimization view by encoding each sketch $s_i$ with the shared VAE,

$z_i = E_\phi(s_i) + \xi_i,$

and then refining $D_\phi$0 through a diffusion-driven optimization loop so that sketch latents become more compatible with the model’s learned 3D-aware photographic prior (Chen et al., 30 Jun 2025). DiffSketching approaches alignment differently: the sketch is not fed directly to the diffusion backbone, but instead enters through a cross-domain perceptual loss between the input sketch and a sketchified version of the generated image, combined with classifier guidance for semantic control (Wang et al., 2023).

4. Structural decomposition, sequencing, and multimodal fusion

One of the most distinctive developments is sequence-aware sketch guidance. Subjective Camera treats the user’s drawing order as a prior over conceptual importance. The framework defines a mapping

$D_\phi$1

where $D_\phi$2 is a text prompt, $D_\phi$3 is a sequence of rough sketches, $D_\phi$4 is an appearance prior extracted from text, and $D_\phi$5 is a geometric prior inferred from sketch order. Each concept sketch $D_\phi$6 is associated with an optimized concept latent $D_\phi$7 and an attention-derived spatial mask

$D_\phi$8

Generation proceeds concept by concept in drawing order, with a global latent update

$D_\phi$9

guided by an energy that balances integration of the current concept against preservation of earlier ones. This is not holistic conditioning on a final sketch; it is explicitly concept-sequential and treats earlier sketches as more central to the final image (Chen et al., 30 Jun 2025).

Other systems decompose structure in different ways. DiffMorph is prompt-less at inference and derives semantic classes for the primary image and sketches through a CLIP classifier, then queries ConceptNet for a “topmost relation” such as “wearing”, “holding”, or “on”. A relation embedding conditions the fine-tuned denoiser so that morphing is not merely additive but relational; a sketch of a hat is not just synthesized, but placed on a head if the selected relation is “wear” (Chatterjee, 2024). In the 3D setting, the sketch-and-text guided point-cloud model uses a sparse sketch encoder with capsule attention, a BERT text encoder, and a two-stage attention fusion in which the sketch queries the text to extract relevant semantics before a second attention block forms geometry and appearance conditioning vectors $\epsilon_\theta$0 and $\epsilon_\theta$1. Geometry and color are then generated by separate staged diffusion models, rather than by a joint 6D diffusion, because the paper reports that joint diffusion degrades both geometry and color through feature interference (Wu et al., 2023).

This body of work collectively suggests that “sketch guidance” is not limited to contour matching. It can encode temporal priority, object relations, localized concept insertion, staged geometry–appearance factorization, or even human memory reconstruction. A plausible implication is that the most effective Sketch-Guided Diffusion Transformer systems are those that treat sketches as structured, typed signals rather than as undifferentiated raster constraints.

5. Training regimes and empirical behavior

Training regimes range from full backbone adaptation to entirely training-free control. Subjective Camera is explicitly training-free: the diffusion model, CLIP model, and VAE are all pre-trained, and the contributions are optimization and guidance strategies rather than parameter learning (Chen et al., 30 Jun 2025). The training-free latent-optimization method likewise manipulates only the latent trajectory of Stable Diffusion v1.4 during sampling (Ding et al., 2024). U-Sketch freezes Stable Diffusion v1.5 and trains only the auxiliary latent edge predictor (Mitsouras et al., 2024). d-Sketch freezes SD2.1 completely and learns only the lightweight LCTN (Roy et al., 19 Feb 2025). DiffFaceSketch adopts a two-stage regime in which the Multi-AE sketch encoder is pre-trained and then frozen before conditional latent diffusion training, motivated by stability and smoother sketch–image latent alignment (Peng et al., 2023). SketchColour occupies the lightweight adaptation end of explicit DiT training: it keeps the 3D VAE and most CogVideoX weights frozen, fine-tunes LoRA modules of rank 192 on attention and MLP layers, and reports “fine-tuning a small LoRA of only 10 million parameters” on 2× NVIDIA A40 with batch size 2 for 40K steps (Sadihin et al., 2 Jul 2025).

Quantitative behavior differs by task, but several recurring empirical patterns appear. In DiffMorph, ConditionFlow improves sketch-to-image conversion over ControlNet on TU-Berlin sketches, with FID 19.96 vs 26.76, CLIP classification accuracy 91.14% vs 82.21%, and CLIP aesthetic score 4.97 vs 4.58; on the multi-concept customization task, DiffMorph reports FID 16.35, Class Alignment 81.3%, and Image Alignment 79.7%, outperforming DreamBooth, Textual Inversion, and Custom Diffusion in FID and class alignment (Chatterjee, 2024). U-Sketch reports Recall 0.645 versus 0.595 for the MLP latent edge predictor, and in user rating studies obtains $\epsilon_\theta$2 for realism, $\epsilon_\theta$3 for edge fidelity, and $\epsilon_\theta$4 for structural coherence, all above the MLP baseline (Mitsouras et al., 2024). DiffSketching reports FID 6.46 and IS 89.91 on Sketchy, outperforming GAN baselines and maintaining competitiveness even on QuickDraw-style sketches (Wang et al., 2023).

For explicit DiTs, the strongest evidence comes from video and generic image synthesis. SketchColour reports that on 14-frame evaluation, it achieves MSCE 2214.18, PSNR 20.23, SSIM 0.79, LPIPS 0.24, and FVD 829.27, outperforming AniDoc and LVCD on all listed metrics; it also outperforms ToonCrafter on all 16-frame metrics despite using only half the training data of competing methods (Sadihin et al., 2 Jul 2025). EDT reports speed-ups of 3.93x, 2.84x, and 1.92x in training, and 2.29x, 2.29x, and 2.22x in inference for EDT-S, EDT-B, and EDT-XL relative to corresponding MDTv2 sizes, while also improving FID (Chen et al., 2024). Internal Guidance shows that auxiliary intermediate supervision can materially improve transformer diffusion quality, with LightningDiT-XL/1+IG reaching FID 1.34 and, combined with CFG, FID 1.19 on ImageNet 256×256 (Zhou et al., 30 Dec 2025). These results do not evaluate the same task, but together they indicate that sketch guidance, efficient transformer design, and internal guidance are not mutually exclusive design axes.

6. Limitations, misconceptions, and future directions

A common misconception is that Sketch-Guided Diffusion Transformer refers to a single standardized backbone. The literature does not support that interpretation. Some systems are explicit DiTs, some are U-Nets with transformer attention, and some are training-free control schemes whose main contribution is to treat attention maps or intermediate latents as the sketch interface rather than to modify the denoiser architecture itself (Chen et al., 30 Jun 2025, Ding et al., 2024). Another misconception is that better sketch control necessarily requires training on large sketch–image paired datasets. Several methods contradict this directly: Subjective Camera is train-free; d-Sketch avoids retraining the diffusion backbone entirely; U-Sketch trains only a guidance head; and DiffSketching relies on ImageNet pretraining plus limited Sketchy fine-tuning rather than massive paired corpora (Roy et al., 19 Feb 2025, Mitsouras et al., 2024, Wang et al., 2023).

Reported limitations are varied but structurally consistent. Subjective Camera notes computation overhead from text-reward optimization and per-concept latent optimization, imperfect CLIP reward signals, nontrivial sketch decomposition into concept-wise $\epsilon_\theta$5, and the absence of an explicit recurrent or transformer time-series model for sketch trajectories (Chen et al., 30 Jun 2025). DiffMorph reports semantic relation mismatch when ConceptNet selects an unintended relation, lack of explicit spatial control, difficulty with large structural edits, and limited generalization due to ConditionFlow training on a relatively small ImageNet subset (Chatterjee, 2024). SketchColour is constrained to 17-frame clips by CogVideoX and GPU limits, depends on Anime2Sketch-generated training sketches, and does not explicitly address palette changes beyond the first frame reference (Sadihin et al., 2 Jul 2025). EDT acknowledges that AMM placement is heuristic and model-dependent, and that its experiments are limited to class-conditional ImageNet rather than explicit sketch conditioning (Chen et al., 2024). U-Sketch observes seed dependence, a realism-versus-perfect-edge trade-off, and limited out-of-domain robustness because the edge predictor is trained on only 6000 ImageNet images (Mitsouras et al., 2024). DiffFaceSketch emphasizes over-strong sketch adherence, lack of color control, and domain restriction to faces (Peng et al., 2023).

Future directions are correspondingly convergent. Subjective Camera explicitly proposes replacing the U-Net with a DiT, representing sketch patches or vector strokes as tokens with temporal positional encoding, learning richer reward models beyond CLIP, and introducing online personalization through user embeddings (Chen et al., 30 Jun 2025). Internal Guidance suggests that sketch-aware auxiliary heads at early–mid transformer layers could improve convergence and structure fidelity in sketch-conditioned DiTs (Zhou et al., 30 Dec 2025). EDT suggests a path toward genuine sketch-conditional transformers by adding sketch encoders and replacing Euclidean AMM locality with sketch-defined locality or geodesic distance along strokes (Chen et al., 2024). Across the literature, a plausible synthesis is that future Sketch-Guided Diffusion Transformer systems will combine native multimodal tokenization of text, sketch, and image/video latents with selective training-free optimization, staged geometry–appearance control, and stronger interfaces for user-specific sequencing, region editing, and preference adaptation.