Structurally-Guided Complex Diffusion

• Structural knowledge-guided complex diffusion is a framework that steers the denoising process using explicit multi-tier edge priors to achieve photorealistic design rendering.
• It employs a multi-level edge representation—double, single, and soft edges—fused through a multi-frequency lines module to preserve geometric fidelity and texture details.
• The framework leverages training-free components like Stable Diffusion, ControlNet, and IP-Adapter to inject structural and appearance cues, ensuring robust control over lighting, material, and composition.

A structural knowledge-guided complex diffusion model is a diffusion-based generative framework in which the denoising trajectory is steered by explicit structural priors rather than relying solely on prompt semantics or end-to-end retraining. In the domain of professional design drawing rendering, this formulation is concretely instantiated by LineArt, a training-free framework for appearance transfer from line drawings that uses a pre-trained Stable Diffusion, ControlNet, and IP-Adapter stack while injecting domain-specific structural knowledge and painting heuristics into sampling (Wang et al., 2024). The objective is to generate a photorealistic rendering that preserves the structural geometry of a design drawing, faithfully reproduces complex appearance from a reference image, and provides fine-grained control over structure, lighting, and material without precise $3\text{D}$ modeling, physical property specification, or network training.

1. Conceptual scope and problem formulation

In LineArt, the input consists of a design drawing $L_{\text{original}}$, defined as a clean, object-centered line drawing with fine structure, and a reference appearance image $I_{\text{appearance}}$, whose material, texture, lighting, and color are to be transferred (Wang et al., 2024). The target output is a photorealistic rendering that preserves edge fidelity, thin lines, and fine details while reproducing materials such as metal, cloth, wood, and bronze, together with lighting, shading, texture, high-frequency texels, and color distribution.

This formulation is positioned against two limitations. Text-to-image diffusion struggles to encode complex material and texture requirements precisely via text and cannot reliably guarantee adherence to given line geometry and small details. Image-to-image translation and stylization, by contrast, often entangle structure and style, distort geometry when style is changed, suffer style degradation, and commonly depend on paired training data that are scarce for professional design drawings (Wang et al., 2024).

The phrase “structural knowledge-guided” denotes that the conditioning signal is not a single binary edge map but a deliberately designed representation of drawing structure derived from theories of visual cognition and from artistic practice. The phrase “complex diffusion” denotes that the denoising process is tasked with handling multi-material surfaces, realistic shading, intricate texels, and multiple interacting controls within a single diffusion framework. In LineArt, these controls are injected at inference time rather than learned through additional training (Wang et al., 2024).

A closely related but independently developed idea appears in "A Structure-Guided Diffusion Model for Large-Hole Image Completion" (Horita et al., 2022), where edge structure is generated first and then used to guide texture generation. That parallel underscores that explicit structural intermediates have become a general design pattern in diffusion systems when texture realism must remain subordinate to geometric plausibility.

2. Structural knowledge representation

LineArt defines structural knowledge through a three-level edge representation: double lines, single lines, and soft edges (Wang et al., 2024). Double lines emphasize key local details and pattern thickness. Single lines provide continuous closed contours for region division and geometric boundaries. Soft edges are represented as a discrete point or edge set encoding implicit spatial gradients and high-frequency texture cues.

This decomposition is operationalized through the multi-frequency lines fusion module. Given $L_{\text{original}}$ and a foreground mask $\text{Mask}$, the single-line representation is formed by erosion,

$L_{\text{single}} = \text{erode}(\text{Mask}),$

while the double-line representation is produced by morphological closing on the original lines,

$$L_{\text{double}} = \text{dilate}(\text{erode}(L_{\text{original}})).$$

Soft edges are obtained indirectly: an initial appearance-guided image is first produced using ControlNet-depth,

$$G_{\text{initial}} = \text{SD}(\text{ControlNet\_depth}(L_{\text{original}}, I_{\text{appearance}})),$$

and a Haar wavelet transform extracts high-frequency components,

$S_{\text{soft}} = \text{Haar}(G_{\text{initial}}).$

These three components are then fused into a unified structural condition,

$$\text{Condition}_{\text{geometry}} = \text{Fusion}(L_{\text{double}}, L_{\text{single}}, S_{\text{soft}}).$$

The role of this fused condition is central. It is fed into ControlNet as the geometric guide for diffusion, thereby replacing a flat contour prior with a multi-frequency structural prior (Wang et al., 2024). In the standard latent-diffusion notation,

$L_{\text{original}}$0

the reverse denoiser is conditioned not only on text and appearance but also on geometry: $L_{\text{original}}$1 The paper treats the fusion operator heuristically, essentially as layered composition of the three maps, but the conceptual point is precise: denoising is bent toward line-preserving solutions by a structural condition that spans local importance, region boundaries, and high-frequency spatial gradients (Wang et al., 2024).

This multi-tier structural representation is also cognition-inspired. The paper explicitly relates it to low-level perception of edges and local contrast, higher-level shape understanding and segmentation, and material interpretation through later stages of the pipeline. This suggests a hierarchical reading of line drawings rather than a purely pixelwise one.

3. Two-stage diffusion pipeline and painting logic

LineArt organizes generation into two stages called Base Layer Shaping and Surface Layer Coloring, drawing on the classical oil-painting technique Imprimatura, understood as underpainting followed by glazing (Wang et al., 2024). The first stage shapes global illumination and structure; the second adds color, material texture, and texels.

In Base Layer Shaping, the appearance image is processed with multi-scale Retinex to compute an illumination map $L_{\text{original}}$2, followed by brightness analysis,

$L_{\text{original}}$3

This brightness statistic modifies the initial latent sample,

$L_{\text{original}}$4

and the noisy latent at diffusion time $L_{\text{original}}$5 becomes

$L_{\text{original}}$6

The effect is a soft constraint that pulls global brightness toward the reference appearance while preserving geometry through ControlNet conditioned on $L_{\text{original}}$7 (Wang et al., 2024).

In Surface Layer Coloring, the reference appearance is first stripped of background and converted into a pure material texture image by patch shuffling: $L_{\text{original}}$8 This texture image is fed to IP-Adapter so that the appearance encoder focuses on material rather than scene background. The Stable Diffusion U-Net then uses decoupled cross-attention to integrate text features, structural features from ControlNet, and appearance features from the IP-Adapter embedding of $L_{\text{original}}$9 (Wang et al., 2024).

The material embedding is injected at specific layers. The layer up_blocks.0.attentions.1 controls color, material, and atmosphere, whereas down_blocks.2.attentions.1 controls structure and spatial composition (Wang et al., 2024). The paper describes this as a glazing-like operation in which color and texture are laid over a structurally consistent base, thereby minimizing conflicts between texels and line structure.

Taken end to end, the pipeline proceeds as follows: input drawing and appearance image are preprocessed; single lines, double lines, and soft edges are fused into $I_{\text{appearance}}$0; the appearance image is background-filtered and patch-synthesized into $I_{\text{appearance}}$1; Retinex statistics reshape the initial latent for Base Layer Shaping; and Surface Layer Coloring continues diffusion with text, structural control, and layer-specific appearance injection (Wang et al., 2024).

4. Guidance mechanisms, training-free operation, and controllability

LineArt is explicitly training-free. It uses off-the-shelf Stable Diffusion, ControlNet, and IP-Adapter, with no additional network parameters trained or fine-tuned (Wang et al., 2024). The gain comes from structural preprocessing, appearance preprocessing, latent initialization, and carefully routed conditioning signals.

Three guidance channels are distinguished. Structural guidance is supplied by ControlNet using $I_{\text{appearance}}$2, where soft edges additionally function as a high-frequency guide to texture placement. Brightness and illumination guidance come from the Retinex-based modification of the initial latent statistics. Material and appearance guidance come from the IP-Adapter embedding of the patch-synthesized material texture $I_{\text{appearance}}$3, injected into selected cross-attention layers (Wang et al., 2024).

Fine-grained control follows directly from these separate channels. Altering or suppressing single and double lines changes line thickness and emphasis. Changing soft edges modulates texture density and high-frequency detail. Replacing the reference image or changing the patch synthesis modifies material appearance. Adjusting $I_{\text{appearance}}$4 and the scaling in $I_{\text{appearance}}$5 changes exposure, contrast, and mood (Wang et al., 2024). This suggests a stronger separation between geometry, lighting, and material than in typical image-to-image stylization pipelines, although the paper frames that separation procedurally rather than as a formally disentangled latent space.

A broader adapter-based perspective appears in "Efficient Text-Guided Convolutional Adapter for the Diffusion Model" (Das et al., 16 Feb 2026), where prompt-aware structural conditioning is embedded directly in lightweight adapters for structure-preserving conditional generation. The relation is methodological rather than identical: LineArt keeps the backbone fixed and engineers conditioning and sampling, whereas Nexus Adapters train prompt-aware structural modules. Both, however, treat structure as an explicit control stream rather than a side effect of prompt semantics.

5. Dataset, evaluation protocol, and empirical findings

For evaluation, the paper introduces a new design drawing dataset called ProLines and describes a curated collection of 5101 professional design line drawings gathered from four sources: Bronze (380 images), DifferSketching (362), ImageNet-Sketch (1756), and DeepPatent (2603) (Wang et al., 2024). Images are scored by ICNet in $I_{\text{appearance}}$6, and only drawings in mid-range image-complexity intervals are selected to avoid overly simple or noisy sketches. Preprocessing includes automated mask extraction, background removal, cropping, and multiple rounds of manual verification by experts.

Evaluation is organized around three groups of metrics. Edge fidelity is measured by SSIM and Chamfer Distance. Appearance transfer quality is measured by PSNR, Color Histogram loss, and Gray-Level Co-occurrence Matrix texture similarity. Overall perceptual quality is assessed by FID, LPIPS, and CLIP image similarity (Wang et al., 2024).

Across the four subsets, LineArt is reported as typically achieving best or second-best performance in nearly all metrics. On Bronze, the reported values are FID $I_{\text{appearance}}$7, LPIPS $I_{\text{appearance}}$8, CLIP$I_{\text{appearance}}$9 $L_{\text{original}}$0, SSIM $L_{\text{original}}$1, and CD $L_{\text{original}}$2, all designated best in the detailed summary. On ImageNet-Sketch, the corresponding reported best values include FID $L_{\text{original}}$3, LPIPS $L_{\text{original}}$4, CLIP$L_{\text{original}}$5 $L_{\text{original}}$6, and SSIM $L_{\text{original}}$7 (Wang et al., 2024).

Qualitative comparison attributes LineArt’s advantage to better preservation of thin, detailed structures, such as bronze vessel patterns, together with more believable material and lighting transfer than ZeST, StyleID, AesPA-Net, DreamBooth, or ControlNet+IP-Adapter baselines (Wang et al., 2024). A user study with 20 participants ranks it highest in edge fidelity, appearance perception, and overall perception.

These results are consistent with the intended division of labor in the model: multi-frequency structure stabilizes geometry, while Retinex and layer-specific appearance injection improve photometric realism. The paper does not present this as a theorem, but the metric profile strongly aligns with that interpretation.

6. Broader diffusion context and limitations

Within the broader diffusion literature, LineArt belongs to a family of systems that make structure explicit and then use it to regulate appearance generation. "A Structure-Guided Diffusion Model for Large-Hole Image Completion" introduces two cascaded diffusion probabilistic models, one for structure and one for texture, and trains them jointly through optimal Bayesian denoising so that edges become an editable structural latent for image completion (Horita et al., 2022). "Knowledge-guided Complex Diffusion Model for PolSAR Image Classification in Contourlet Domain" uses low-frequency complex coefficients for diffusion and high-frequency contourlet coefficients as structural guidance, again separating statistical content from boundary information (Shi et al., 8 Jul 2025). "Structured Diffusion Models with Mixture of Gaussians as Prior Distribution" shifts structural knowledge into the prior itself by replacing the standard Gaussian with a mixture prior chosen to encode clusters or regimes (Jia et al., 2024).

These comparisons clarify that “structural knowledge-guided complex diffusion model” is not a single architecture but a design principle. In LineArt, the structural prior is a multi-frequency decomposition of line drawings combined with human painting heuristics. In SGDM for inpainting, it is an explicit edge latent. In the PolSAR model, it is the high-frequency contourlet domain. In mixture-prior diffusion, it is encoded in the prior distribution. What unifies them is that structure is externalized, represented in an interpretable form, and injected into denoising rather than left implicit in a monolithic backbone.

LineArt also states clear limitations. Sketches with heavy shading or complex textures inside the lines can confuse the structural decomposition and degrade results. Tasks requiring precise semantic alignment, such as swapping species while keeping structure, may fail or yield unnatural mappings. The method is designed mainly for object-centered design drawings, and performance on scene-level sketches or very abstract doodles may be limited (Wang et al., 2024).

The paper lists several future directions consistent with the same concept: extending structural conditioning to $L_{\text{original}}$8 geometry, normal maps, CAD, semantic maps, or segmentation; introducing trainable fusion of multi-frequency cues or learned weighting of line types; and improving robustness for sketches with mixed shading or complex hand-drawn styles (Wang et al., 2024). A plausible implication is that the structural knowledge-guided paradigm is especially attractive when domain expertise can be encoded at inference time, allowing strong control without the cost or rigidity of retraining large diffusion backbones.