Joint RGB-PBR Representation for Material Synthesis

• Joint RGB–PBR representation is a unified latent encoding framework that captures both visually rich RGB images and physically-based rendering maps.
• It leverages a coupled VAE and diffusion architecture to jointly synthesize and decompose material properties across tasks like text-to-material and image-to-material.
• Hybrid training on large-scale RGB and specialized PBR datasets yields significant improvements in metrics, enhancing material synthesis quality and realism.

A joint RGB-PBR representation is a foundational concept in photorealistic graphics, physically based rendering (PBR), and material synthesis, referring to a unified latent encoding or structural framework that captures both the visual appearance (RGB images) and the underlying physical material properties (PBR parameter maps: typically albedo, normal, roughness, and metallic channels). This representation is critical for generating, editing, or understanding material appearances in ways that support realistic relighting, novel view synthesis, and cross-modal generation/intrinsic decomposition—from text, images, or 3D data. Recent models such as MatPedia demonstrate that compact, interdependent RGB–PBR representation enables state-of-the-art synthesis quality and flexibility, bridging large-scale natural image priors with physically grounded material attributes (Luo et al., 21 Nov 2025).

1. Latent Space Design and Encoding Mechanisms

The core principle is to encode a material sample as a multi-channel field, jointly representing both the shaded image (RGB) and the set of PBR maps. In MatPedia, this is realized as a five-frame pseudo-video: frame 0 is the RGB image $\mathbf I_{\mathrm{rgb}}$, frames 1–4 correspond to basecolor ($\mathbf a$), normal ($\mathbf n$), roughness ($\mathbf r$), and metallic ($\mathbf m$) maps, each of shape $\mathbb R^{H\times W\times C}$ (Luo et al., 21 Nov 2025).

A 3D (spatio-temporal) VAE encodes these frames into two interdependent latent tensors:

• $$\mathbf z_{\mathrm{rgb}} = \mathcal E_{\mathrm{rgb}}(\mathbf I_{\mathrm{rgb}})$$ captures RGB appearance features.
• $$\mathbf z_{\mathrm{pbr}} = \mathcal E_{\mathrm{pbr}}(\mathcal F_{\mathrm{enc}}(\mathbf z_{\mathrm{rgb}}),\,\mathbf a,\,\mathbf n,\,\mathbf r,\,\mathbf m)$$ encodes complementary physical information, conditioned on features extracted from the RGB encoding.

Decoding is symmetric: $\mathbf z_{\mathrm{rgb}}$ reconstructs the RGB image, while $\mathbf z_{\mathrm{pbr}}$ (conditioned on decoded RGB features) reconstructs the PBR maps. This results in a highly compact, asymmetric two-latent scheme where the PBR latent stores only the physical detail that is complementary to what is implied by appearance.

2. Joint Sampling and Diffusion Architectures

Embedding the RGB and PBR latents as a five-frame “video” permits unified processing via video-diffusion models. Specifically, MatPedia concatenates these latents in the order $$\{z_{\mathrm{rgb}}, z_{\mathrm{pbr}}^{(1)}, z_{\mathrm{pbr}}^{(2)}, z_{\mathrm{pbr}}^{(3)}, z_{\mathrm{pbr}}^{(4)}\}$$, allowing 3D convolutional layers to capture both spatial and inter-map correlations.

The generative backbone is a Diffusion Transformer (DiT), which applies spatio-temporal self-attention across the entire five-frame latent “video,” directly propagating correlations between appearance and all physical channels. Conditioning (e.g., via text or image latents) is injected using LoRA adapters, and the forward process is formulated via rectified-flow, enabling efficient parameterization. The loss is an $L_2$ velocity-matching objective on latent interpolations (Luo et al., 21 Nov 2025).

This arrangement enables consistent and high-fidelity co-generation of both RGB and PBR, natively supporting $1024 \times 1024$ output resolution per map with upsampling to production resolutions.

3. Unified Multi-Task Material Synthesis and Decomposition

The joint RGB–PBR representation enables a single model and architecture to address multiple canonical tasks in material synthesis:

• Text-to-Material: Text embeddings condition the diffusion process to jointly generate both $\mathbf z_{\mathrm{rgb}}$ and $\mathbf z_{\mathrm{pbr}}$.
• Image-to-Material: By encoding a distorted photograph, the model conditions on its latent and rectifies to generate both clean planar RGB appearance and corresponding PBR maps.
• Intrinsic Decomposition: Starting from a planar RGB image, only $\mathbf z_{\mathrm{pbr}}$ is generated/decoded, yielding disambiguated PBR maps representing intrinsic material properties.

All three tasks share the same DiT weights and VAE decoder, with task-specific LoRA adapters, maximizing cross-task generalization and leveraging shared inductive biases (Luo et al., 21 Nov 2025).

4. Hybrid Data Regimes and Semantic Coverage

MatPedia employs a hybrid dataset (“MatHybrid-410K”) comprising two complementary sources:

• A large-scale RGB-only subset ($\approx50$K planar images with text captions), including both real and synthetically generated images.
• A complete PBR-material subset ($\approx6$K materials, $\approx360$K renderings) from datasets such as Matsynth and OpenSVBRDF, rendered under a range of HDR environments and geometric contexts.

This mixture enables the model to learn rich visual priors from diverse images while grounding PBR map decoding on physically correct, parameterized data. In ablations, removal of RGB-only data degrades text-to-material generation (CLIP 0.283$\rightarrow$0.275, DINO-FID 1.31$\rightarrow$1.62), demonstrating the necessity of leveraging large-scale RGB corpora for generalization (Luo et al., 21 Nov 2025).

5. Training Objectives and Optimization

The training regime consists of two principal objectives:

• 3D-VAE Fine-Tuning: The decoder is trained end-to-end with an $L_1$ reconstruction loss and a VGG-based perceptual loss on the synthesized five-frame stack.
• Diffusion/Rectified-Flow: For joint RGB–PBR latent diffusion, the loss is the expected squared error between predicted and true velocity fields over sampled timepoints and conditions.

No adversarial loss is required due to the high expressivity of the diffusion process combined with perceptual losses. Fine-tuning is performed via LoRA adapters to adapt foundation DiT weights to material-specific tasks (Luo et al., 21 Nov 2025).

6. Quantitative Evaluation and Ablative Analysis

Joint RGB–PBR models achieve significant improvements over prior state-of-the-art:

• Text-to-Material: CLIP score increases from 0.261 to 0.283 and DINO-FID decreases from 1.90 to 1.31 compared to previous best (MatFuse).
• Image-to-Material: Basecolor CLIP and DINO scores surpass prior models.
• Intrinsic Decomposition: Render MSE/LPIPS metrics outperform Material Palette and other decomposition baselines.

Ablative studies show that VAE decoder fine-tuning improves normal map PSNR by 3.55 dB and roughness estimation by 5.20 dB. Hybrid training is critical for task generalization; training on PBR-only data degrades all downstream metrics (Luo et al., 21 Nov 2025).

7. Practical Advantages and Current Limitations

Technical advantages:

• Compact encoding: Conditioning PBR on RGB appearance reduces PBR map latent dimensionality, enabling efficient parameterization and sampling.
• Unified processing: The video diffusion adapts naturally to cross-map interactions, accommodating complex appearance–material dependencies.
• Cross-modal transfer: The framework leverages RGB-based generation priors for physical map synthesis, facilitating robust text/image/material transfer.
• Scalability: Natively supports large-scale, high-resolution output with consistent structure across tasks.

Identified limitations:

• The representation handles only four standard PBR channels; planned extensions (height, subsurface, etc.) remain undeveloped.
• Noise-rolling for tileable materials is nontrivial due to the asymmetric, coupled latent design.
• Generation speed (20 s per map set at $1024^2$, 50 steps) is on par with other diffusion models but slower than regression-based approaches (Luo et al., 21 Nov 2025).

---

In summary, the joint RGB–PBR representation, as exemplified in models such as MatPedia, defines a compact, coupled latent encoding that bridges natural RGB appearance and physically grounded material properties, yielding a flexible and unified framework for material generation, editing, and decomposition across modalities. This framework enables state-of-the-art synthesis quality, cross-task generalization, and meaningful exploitation of both large-scale image and specialized PBR datasets (Luo et al., 21 Nov 2025).