TextureExchangeUnit: Precision Retexturing

• TextureExchangeUnit is a module that precisely transfers local textures from a reference while preserving the original geometry, pose, and motion in images or videos.
• It employs a dedicated Texture Remover, jigsaw permutation augmentation, and a conditional diffusion backbone to disentangle texture from structure.
• The architecture ensures high-fidelity retexturing by rigorously maintaining spatial-temporal consistency and reducing inference steps through DMD2 distillation.

The TextureExchangeUnit is a module designed to achieve precise and controllable transfer of local surface textures from a reference image to a target object in either images or videos while strictly preserving the geometric structure, pose, and motion of the source. Introduced as a core component of the Refaçade system (Huang et al., 4 Dec 2025), the TextureExchangeUnit addresses limitations of prior diffusion-based approaches by disentangling texture from structure through architectural decomposition and targeted data augmentation. It combines a dedicated Texture Remover network, a learned jigsaw-permutation augmentation, and a multi-conditional diffusion backbone to facilitate geometry-constrained retexturing at scale.

1. Formal Problem Statement and Conceptual Overview

The TextureExchangeUnit learns the mapping

$$(X_{\text{src}}, M, I_{\text{ref}}) \rightarrow X_{\text{out}}$$

where $X_{\text{src}}$ denotes the source image or video, $M$ is its foreground mask, and $I_{\text{ref}}$ is the reference image providing the desired surface texture. The output $X_{\text{out}}$ retains the original geometry, pose, and motion from $X_{\text{src}}$, but replaces its local texture with that of $I_{\text{ref}}$.

The architecture introduces two critical decompositions:

• A Texture Remover, producing a geometry-only (texture-free) latent $X_{\text{unt}}$ from $X_{\text{src}}$.
• A Jigsaw Permutation module, constructing a patched version $I_{\text{jig}}$ of $X_{\text{src}}$0 in which global shape is disrupted, but local texture statistics are preserved.

Both outputs are processed by a conditional diffusion backbone, realized as a ControlNet/VACE-style U-Net, yielding $X_{\text{src}}$1 with transferred texture and preserved structure.

2. Component Architectures and Dataflow

2.1 Texture Remover

• Network: Video diffusion U-Net based on VACE, comprising 24 transformer blocks (hidden size 1536), with control blocks at layers $X_{\text{src}}$2.
• Inputs: VAE-encoded latents of $X_{\text{src}}$3 (post-background removal) and per-frame mask $X_{\text{src}}$4.
• Outputs: VAE-latents corresponding to a uniform gray (albedo-only) rendering, matching the original camera trajectory and object geometry.
• Training Data: For 72K textured meshes, paired videos with and without texture are rendered under identical conditions, yielding approximately 576K pairs.
• Training Methodology: The main network branch is frozen; only control blocks are updated to learn the mapping from appearance to geometry. Speedup at inference is achieved by DMD2 distillation, reducing sampling from 50 to 3 steps.

2.2 Jigsaw Permutation

• Procedure:

1. From $X_{\text{src}}$5, extract $X_{\text{src}}$6 square patches (default: $X_{\text{src}}$7 of image side), excluding patches with $X_{\text{src}}$8 background.
2. Randomly shuffle and optionally flip these patches.
3. Tile patches into a novel canvas, preserving width but changing height.

• Output: $X_{\text{src}}$9, which contains only the local texture statistics from the reference.

2.3 Conditional Diffusion Backbone

• Control Branch: For each control block, concatenate latents of background video $M$0, texture-free video $M$1, mask $M$2, and jigsaw reference $M$3. Each passes through a condition layer, then channels are summed as additive bias per U-Net block.
• Main Branch: At denoising timestep $M$4, the latent $M$5 (for frame 0) is prepended with reference latent tokens. Control block hidden states are injected at each resolution.
• Outcome: Geometry and motion are enforced from $M$6 while local texture derives exclusively from $M$7.

3. Mathematical Foundations

3.1 Texture Removal via Flow-Matching

Let $M$8, $M$9. Sampling $I_{\text{ref}}$0, $I_{\text{ref}}$1, define $I_{\text{ref}}$2, $I_{\text{ref}}$3. The velocity network $I_{\text{ref}}$4 is trained to minimize

$I_{\text{ref}}$5

with $I_{\text{ref}}$6 the VAE-coded source and mask. After DMD2 distillation, the same objective holds with a 3-step sampler.

3.2 Jigsaw Data Augmentation

No loss is imposed on the Jigsaw module beyond standard diffusion training. The transformation $I_{\text{ref}}$7 forces the network to ignore global reference structure. While a perceptual regularizer could be used,

$I_{\text{ref}}$8

(with $I_{\text{ref}}$9 e.g. VGG/CLIP), this is not utilized in the implementation.

3.3 Diffusion Loss for Texture Exchange

Overall flow-matching is used for the primary objective: $X_{\text{out}}$0 where $X_{\text{out}}$1 comprises latents for $X_{\text{out}}$2, $X_{\text{out}}$3, $X_{\text{out}}$4, and $X_{\text{out}}$5.

4. Geometry and Motion Preservation

• The Texture Remover is explicitly trained on video pairs differing only in albedo, guaranteeing geometry, pose, and motion consistency between input and texture-free latents.
• In the diffusion backbone, guidance from the frozen $X_{\text{out}}$6 and $X_{\text{out}}$7 ensures that geometry and fine structural details are enforced in each step.
• Empirical assessment of spatio-temporal consistency employs EWarp (optical-flow-based warping error) without introducing any explicit motion losses into the training regime.

5. Controllability and Structure-Texture Disentanglement

• The Texture Remover eliminates all appearance features from $X_{\text{out}}$8, preventing leakage of source textures or colors.
• The Jigsaw module ensures that global spatial structure of the reference is destroyed, so only local appearance statistics influence the output object’s surface.
• The conditional architecture processes input structure and reference through independent condition layers, delaying any cross-modal fusion until late-stage attention mechanisms.
• Classifier-free guidance, enabled by randomly omitting conditioning with $X_{\text{out}}$9, permits tuning of the fidelity–flexibility trade-off at inference, modulating influence between strict texture application and environmental realism.

6. Training Protocols and Implementation Details

6.1 Texture Remover Training

Dataset: 576K paired textured/gray videos from 72K mesh identities, each rendered under eight camera/light/motion settings. The Remover is trained with AdamW (learning rate $X_{\text{src}}$0, batch size 32) for two epochs, typically totaling 38 hours on 32 NVIDIA A800 GPUs. DMD2 distillation further reduces the inference sampler to three steps (300 steps, $X_{\text{src}}$1 LR, batch 8).

6.2 Main Diffusion Model Training

• Stage 1 (Pretraining): 1.8M WebVid-10M videos + 0.9M synthetic SelfForcing videos + 0.8M SD3.5 images with masks; 96×A800 GPUs; global batch 96; $X_{\text{src}}$2 LR; 18K steps (~120 hours).
• Stage 2 (Finetuning): 180K Pexels videos; same learning parameters, 2.8K steps on 32 GPUs (~28 hours).
• All training utilizes mixed precision and gradient checkpointing.

6.3 Data Augmentation and Hyperparameters

• Random frame resizing/downsampling.
• Random dropout of conditioning (replace $X_{\text{src}}$3 with white, $X_{\text{src}}$4 with black) at $X_{\text{src}}$5.
• Default Jigsaw patch size: 10% of side; ablations for 5–20% show no major effect.
• Inference resolution: $X_{\text{src}}$6 for video (81 frames); arbitrary $X_{\text{src}}$7 for single images.

6.4 Best Practices (as recommended by Huang et al. (Huang et al., 4 Dec 2025))

• Train the Texture Remover on strictly aligned paired data.
• Apply aggressive jigsaw shuffling.
• Freeze the diffusion backbone except small control/extraction blocks.
• Use classifier-free guidance at inference to dial the texture-to-background fidelity balance.
• Distill the remover for efficient end-to-end training and inference.
• Evaluate with both per-frame metrics (MSE, PSNR, SSIM, LPIPS), temporal metrics (EWarp), and perceptual metrics (CLIPScore, DINOScore, DreamSim, LLM scoring).

7. Significance and Implementation Implications

The TextureExchangeUnit enables high-fidelity, controllable object retexturing for both images and videos while ensuring geometric and spatio-temporal structure remain unaltered. This is accomplished through architectural and training choices that rigorously disentangle appearance from structure at every stage. The method, as described and evaluated by Huang et al. (Huang et al., 4 Dec 2025), provides a framework for object retexturing tasks that outperforms previous diffusion-based methods in controllability, visual quality, and empirical consistency, without reliance on adversarial or explicit perceptual losses.