MatPedia: Unified Generative Material Synthesis

• MatPedia is a unified framework that integrates natural image appearance and PBR material properties through a joint latent representation to enable comprehensive material synthesis.
• It employs a five-frame sequence formulation with a spatio-temporal VAE and video diffusion transformer, capturing key intra-material dependencies and ensuring high-quality outputs.
• The model supports versatile tasks including text-to-material, image-to-material synthesis, and intrinsic decomposition, with applications in photorealistic rendering and digital content creation.

MatPedia is a foundational model and framework designed to bridge natural image appearance and physically-based rendering (PBR) material properties within a unified generative architecture. Its primary goal is to overcome longstanding limitations of fragmented material synthesis pipelines by enabling high-fidelity, large-scale material generation, decomposition, and editing directly from textual or image descriptions, leveraging a novel joint RGB–PBR representation and video diffusion architectures. Trained on the comprehensive MatHybrid-410K corpus, MatPedia achieves native $1024\times1024$ output, setting new standards for diversity and quality in PBR material synthesis (Luo et al., 21 Nov 2025).

1. Motivation and Problem Statement

The creation of PBR materials—a cornerstone of photorealistic computer graphics—requires the hand-crafting of four or more texture maps, including basecolor, normal, roughness, and metallic channels. This process is labor-intensive and demands advanced expertise. Prior generative pipelines are typically task-specific (e.g., intrinsic decomposition or unconditional generation), cannot transfer knowledge across tasks, and are often restricted by the limited scale of available PBR datasets (commonly $\sim$6$$K materials). This leads to material synthesis capabilities that lag behind RGB image generation in both <a href="https://www.emergentmind.com/topics/fidelity-alpha-precision" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">fidelity</a> and diversity. MatPedia was developed to unify all appearance and physics information of materials in a single latent representation, support multiple downstream generative and analytic tasks, and exploit the diversity of large-scale RGB-only image collections without explicit PBR labels (<a href="/papers/2511.16957" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Luo et al., 21 Nov 2025</a>).</p> <h2 class='paper-heading' id='joint-rgb-pbr-latent-representation'>2. Joint RGB–PBR Latent Representation</h2> <p>MatPedia encodes material data into two closely coupled latent spaces:$$z_\mathrm{rgb} \in\mathbb{R}^{d_\mathrm{rgb}}$models shaded RGB image appearance, and$z_\mathrm{pbr} \in\mathbb{R}^{d_\mathrm{pbr}}$encodes the four PBR maps (basecolor$a$, normal$n$, roughness$r$, metallic$m$), with$z_\mathrm{pbr}$$conditioned on intermediary features of the <a href="https://www.emergentmind.com/topics/rgb-branch-convnext-tiny" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">RGB branch</a>. Formally, the stacked$$[I_\mathrm{rgb}; a; n; r; m]$ sequence is processed as a 5-frame &quot;video&quot; by a <a href="https://www.emergentmind.com/topics/spatio-temporal-variational-autoencoder-wan-vae" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">spatio-temporal variational autoencoder</a> (VAE), such as Wan2.2-VAE. The encoder network produces: $z_\mathrm{rgb} = \mathcal{E}_\mathrm{rgb}(I_\mathrm{rgb}),\quad z_\mathrm{pbr} = \mathcal{E}_\mathrm{pbr}([\mathcal{F}_\mathrm{enc}(z_\mathrm{rgb}), a, n, r, m])$where$\mathcal{F}_\mathrm{enc}(\cdot)$$are cached features from the RGB encoder. Decoding mirrors this process, ensuring$$z_\mathrm{pbr}$$only needs to encode PBR features complementary to those already represented in$$z_\mathrm{rgb}$$. The reconstruction loss incorporates both pixelwise and VGG-based perceptual distances:$$\mathcal{L}_\mathrm{VAE} = \lambda_1\|\hat{x} - x\|_1 + \lambda_2\|\phi(\hat{x}) - \phi(x)\|_2^2$where$x$and$\hat{x}$$are ground truth and reconstructed frame sequences, and$$\phi(\cdot)$$denotes VGG perceptual features (<a href="/papers/2511.16957" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Luo et al., 21 Nov 2025</a>).</p> <h2 class='paper-heading' id='5-frame-sequence-formulation'>3. 5-Frame Sequence Formulation</h2> <p>The input to the model consists of five sequential frames:</p> <ul> <li>Frame 0: shaded RGB image</li> <li>Frames 1–4: PBR maps (basecolor$$a$, normal$n$, roughness$r$, metallic$m$)

Treating the multi-channel material data as a video enables the application of 3D and temporal convolution modules, as well as temporal self-attention, capturing intra-material dependencies (e.g., the coupling between surface details and shading response) naturally within the latent space (Luo et al., 21 Nov 2025).

4. Video Diffusion Architecture and Training Paradigm

MatPedia utilizes a Video Diffusion Transformer (DiT) backbone, structurally based on a 3D U-Net with interleaved self-attention and cross-attention mechanisms, as established in contemporary video diffusion models (e.g., Wan et al.). Cross-attention supports multiple condition modalities, such as text embeddings for text-to-material tasks or encoded image features for image-guided tasks. LoRA adapters (rank 128) are used for efficient task-specific fine-tuning across all attention and feedforward network projections.

The method adopts the rectified flow framework for diffusion: for $t\in[0,1]$, noisy latents $x_t = (1-t)x_0 + t x_1$, $x_1 \sim \mathcal{N}(0,I)$ are denoised by training a velocity network $v_\theta$ to predict $(x_0 - x_1)$: $$\mathcal{L}_\mathrm{RF} = \mathbb{E}_{x_0, x_1, t} \|\, v_\theta(x_t, t, c) - (x_0 - x_1)\|_2^2$$ Conditioning signal $c$ is provided by text tokens or image latents, as appropriate for downstream tasks. Inference uses typically 50 denoising steps with uniform $t$-scheduling, and outputs are optionally upsampled to $4$K resolution using RealESRGAN (Luo et al., 21 Nov 2025).

5. Training Corpus and Preprocessing

MatPedia is trained on the MatHybrid-410K corpus, which unifies:

• $\,\sim$50K RGB-only planar material views obtained from advanced image synthesis tools (e.g., Gemini 2.5 Flash Image) and web sources, with captions generated by Qwen2.5-VL.
• $\,\sim$360K planar and $168$K distorted pairs from $\sim$6K unique SVBRDFs (e.g., Matsynth, OpenSVBRDF), rendered under 32 HDR lighting conditions across multiple geometric primitives with random cropping for coverage diversity.

Standard augmentations include flips, crops, color jittering, and environment map rotation; all images are normalized to $[-1, 1]$ and resized to $1024 \times 1024$ (Luo et al., 21 Nov 2025).

6. Unified Task Capabilities

MatPedia supports three major generative and analytic tasks within a single model and latent space:

• Text-to-Material: Text prompts are tokenized and used to condition the DiT, jointly generating RGB appearance and PBR maps.
• Image-to-Material: Input photographs or distorted renderings are encoded to a latent, conditioning the DiT to regenerate planar RGB + PBR maps.
• Intrinsic Decomposition: Given a planar RGB image, the encoder produces $z_\mathrm{rgb}$ and the DiT infers $z_\mathrm{pbr}$, from which all physical maps are recovered; the RGB branch is clamped in this process.

All tasks leverage the same DiT weights, with LoRA adapters providing task specialization when necessary. Task performance is evaluated using CLIP Score, DINO-FID, MSE on basecolor, and LPIPS; MatPedia demonstrates substantial improvements over MatFuse and Material Palette in both quality and diversity (Luo et al., 21 Nov 2025).

7. Applications, Integration, and Future Directions

MatPedia-generated materials can be exported directly into mainstream graphics engines (Unreal, Unity) and digital content creation tools (Substance Painter). Decoded latents can be cached for real-time previews. The unified latent space and DiT framework facilitate text/image-driven material search, editing, and transfer. Potential future extensions include adding channels for height/displacement, opacity, or subsurface scattering; generating tileable textures via latent noise manipulations; supporting interactive local editing; and modeling higher-order statistics such as anisotropy and clearcoat properties (Luo et al., 21 Nov 2025).

A plausible implication is that MatPedia’s modular architecture, scalable training paradigm, and joint RGB–PBR latent approach establish a new baseline for foundation models in material synthesis, supporting universal, high-throughput content generation for photorealistic rendering.