DealMaTe: Multi-Dimensional Material Transfer via Diffusion Transformer

Abstract: Recently, diffusion-based material transfer methods rely on image fine-tuning or complex architectures with auxiliary networks but face challenges such as text dependency, additional computational costs, and feature misalignment. To address these limitations, we propose \textbf{DealMaTe}, using \underline{\textbf{de}}pth, norm\underline{\textbf{a}}l, and \underline{\textbf{l}}ighting images for \underline{\textbf{ma}}terial \underline{\textbf{t}}ransf\underline{\textbf{e}}r. DealMaTe is a simplified diffusion framework that eliminates text guidance and reference networks. We design a lightweight 3D information injection method, Multi-Dim 3D Shader LoRA, which, without modifying the base model weights, enables compatible control conditions and achieves harmonious and stable results. Additionally, we optimize the attention mechanism with Shader Causal Mutual Attention and key-value (KV) caching to reduce inference latency caused by multiple conditions, improve computational efficiency, and achieve high-quality material transfer results with low architectural complexity. Extensive experiments covering a wide variety of objects and lighting conditions consistently demonstrate that DealMaTe achieves remarkable high-fidelity material transfer under arbitrary input materials. The code is available at https://github.com/haha-lisa/DealMaTe.

• The paper presents a unified visual-conditional diffusion model that eliminates text guidance, achieving robust and structurally consistent material transfer.
• It introduces Multi-Dim 3D Shader LoRA and Shader Causal Mutual Attention to maintain feature integrity and reduce inference time by up to 2.61×.
• Empirical results show superior performance with SSIM of 0.8906 and LPIPS of 0.1285, outperforming previous state-of-the-art pipelines.

DealMaTe: Multi-Dimensional Material Transfer via Diffusion Transformer

Overview and Motivation

DealMaTe addresses the persistent limitations in material transfer methods grounded in diffusion architectures, specifically targeting computational inefficiency, over-reliance on textual guidance, and insufficient structural consistency. Existing pipelines either depend on fine-tuning sample sets with text identifiers or utilize additional image encoders (e.g., IP-Adapters), which introduce feature misalignment, increased inference latency, and poor 3D property preservation. DealMaTe introduces a unified visual-conditional diffusion paradigm, utilizing depth, normal, and lighting images as multidimensional control signals, and architectures for their efficient fusion and acceleration, marking a shift towards fully visual, text-free, and zero-shot-capable material transfer.

Figure 1: Comparison of structural paradigms across material transfer methods, highlighting DealMaTe’s elimination of text prompts and external image encoders via direct visual control.

Methodological Innovations

Multi-Dim 3D Shader LoRA

At the core of DealMaTe is the Multi-Dimensional 3D Shader LoRA, which introduces dedicated branches for depth, normals, and lighting conditions through lightweight Low-Rank Adaptation (LoRA), preserving the frozen backbone of the diffusion transformer. Each branch is functionally isolated such that low-rank projections are strictly applied to the conditional pathway only. This prevents the common issue of latent space corruption observed in concurrent LoRA or control module designs, while permitting highly granular control over spatial, geometric, and photometric characteristics.

Figure 2: High-fidelity material transfer is achieved by injecting depth, normal, and lighting into respective LoRA branches, maintaining feature space consistency throughout the diffusion process.

Shader Causal Mutual Attention and KV Caching

To further optimize both fidelity and inference speed, DealMaTe introduces Shader Causal Mutual Attention (SCMA), a novel attention scheme enforcing strict causal dependencies and isolating information flow across conditional tokens. The masking design ensures condition-to-content flow while blocking cross-conditional interference—a critical factor in disentangling depth, normal, and lighting contributions. This is coupled with KV caching, allowing key-value pairs for all conditional tokens to be computed and stored once per sample, amortizing the cost across all denoising iterations and reducing latency by up to 2.61× without impacting generative quality.

Figure 3: Shader Causal Mutual Attention precisely manages token-wise information flow, enabling efficient direct key–value caching for significant inference acceleration.

Empirical Validation

Qualitative and Quantitative Results

DealMaTe demonstrates consistent superiority on the MTB benchmark and across qualitative evaluations. Key metrics—SSIM, LPIPS, CLIP, PSNR, DreamSim—consistently favor DealMaTe over all SOTA competitors, including the authors' prior MaTe system and publicly available multi-modal agents (Gemini 2.5, IP-Adapter pipelines).

• SSIM 0.8906 (second-best: 0.8825)
• LPIPS 0.1285 (second-best: 0.1317)
• CLIP 0.8927 (second-best: 0.8834)

Improved material-to-structure alignment and photorealistic consistency are evident in side-by-side visualizations.

Figure 4: Comparative qualitative results; DealMaTe outperforms prior art in structural consistency and visual realism.

Ablation Studies

Systematic ablations show that removing the 3D-SCB (conditional branch) or replacing SCMA with standard MMA results in artifacts, texture distortions, and loss of condition independence. Eliminating any single control signal (depth, normal, lighting) empirically degrades the output along complementary axes, underscoring the necessity of each dimension. KV caching maintains output quality, as reflected in constant CLIP scores, while reducing inference time from 73s to 28s on 1024×1024 images.

Figure 5: Removal of the 3D-SCB induces artifacts and structural inconsistencies, emphasizing its necessity.

Figure 6: Disabling SCMA corrupts conditional disentanglement, resulting in structure loss and blurred details.

Figure 7: Unimodal ablation of depth, lighting, or normal clearly demonstrates their complementarity and necessity for robust control.

Applications and Limitations

DealMaTe is shown to generalize robustly across application domains—product packaging, garment design, and virtual furniture fabrication—where high-fidelity synthetic material transfer is key for rapid prototyping.

Figure 8: Representative downstream applications demonstrate generalization to diverse object categories and real-world constraints.

Observed limitations center on cases where the accuracy of pre-computed geometric conditions (particularly depth and normal estimation) breaks down, as well as when input materials possess physical textural characteristics at odds with the geometric constraints of the target object. Extreme material mixtures or colored illuminants in the reference images also induce undesired transfer of unwanted features.

Figure 9: Failure cases arise when geometric-material mismatch occurs or material maps are ambiguous.

Figure 10: Inferior geometric condition estimation yields overly smooth results, missing fine structural details.

Theoretical and Practical Implications

DealMaTe stands as the first material transfer system to unify dense visual controls (depth, normal, lighting) in a pure LoRA architecture with transformer-based diffusion, eschewing both text supervision and reference encoders. The architectural modularity—frozen backbone, parallel LoRA branches—facilitates plug-and-play extensibility for future conditional controls (e.g., semantic segmentation, region-aware transfer). On the theoretical side, the SCMA formulation and its masking constraints provide a template for token-wise independence in other dense conditional generative tasks. Practically, inference-time acceleration via KV caching will likely become a fundamental component in real-time or interactive diffusion-based visual synthesis.

Future Directions

Challenges remain in robustly estimating geometric and photometric controls for highly complex or non-Lambertian surfaces. Extension toward full 3D-aware transfer (volumetric or multi-view) and region-aware/material-category-conditioned blending are natural next steps. The cross-fertilization of LoRA-based modularity and high-resolution diffusion transformers (e.g., FLUX, DiT-family) may open further avenues for scalable, controllable generative graphics pipelines.

Conclusion

DealMaTe fundamentally redefines the architectural and algorithmic landscape for image-guided material transfer by merging dense geometric and lighting control into a single, text-independent transformer diffusion model. Empirical evidence—both quantitative and subjective—shows that this design preserves structural consistency and material fidelity beyond what is attainable with current SOTA approaches. Its modularity, efficiency, and visual quality substantially broaden the potential for AI-driven material design, synthesis, and editing tasks in industry and research (2605.15681).