Texture Multi-View Diffusion Techniques

• Texture multi-view diffusion is a class of methods that generates consistent 3D textures by synchronizing outputs across multiple views using diffusion models.
• It employs techniques like UV-space synchronization, feature-space communication, and learned aggregation to enforce cross-view consistency and preserve fine details.
• Recent advancements extend these methods for full PBR material generation by integrating geometry control, explicit conditioning, and illumination invariance for realistic results.

Searching arXiv for papers on multi-view diffusion for 3D texture generation and related PBR/material methods. arXiv search query: "multi-view diffusion 3D texture generation PBR texturing"

Texture multi-view diffusion denotes a class of methods that adapt diffusion models to synthesize, refine, or transfer textures for 3D assets by operating on multiple rendered views, a shared UV texture domain, or both. The central objective is to obtain a single texture representation whose renderings remain coherent across viewpoints while preserving high-frequency detail and, in some systems, physically based material structure. Published work spans synchronized latent-UV aggregation for text-guided texturing (Liu et al., 2023), optimization-based color fusion during DDIM sampling (Zhang et al., 2024), geometry-controlled multi-view RGB generation followed by PBR extraction (Shao et al., 19 May 2025), and end-to-end multi-view PBR diffusion with illumination-invariant training (He et al., 13 Mar 2025).

1. Problem formulation and historical trajectory

Early multi-view texturing pipelines commonly relied on per-view generation followed by projection and inpainting, and several papers identify this design as the source of visible seams, ghosting, local discontinuities, and inconsistent overlapping regions. "Text-Guided Texturing by Synchronized Multi-View Diffusion" explicitly attributes inconsistent artifacts to asynchronous diffusion and insufficient information sharing among views, while TexPainter notes that pre-trained diffusion models are trained in screen space and that converting their outputs into a multi-view-consistent texture image is a major obstacle to quality (Liu et al., 2023, Zhang et al., 2024).

From 2023 onward, the field diversified into several lines. One line emphasizes synchronization during sampling, as in latent-UV aggregation and per-step re-rendering (Liu et al., 2023). A second line treats consistency as a fusion or optimization problem in UV space, exemplified by optimization-based color fusion (Zhang et al., 2024), dynamic UV merging for indoor scenes (Huang et al., 2024), and noise-fusion in atlas space (Yildirim et al., 3 Apr 2025). A third line introduces learned multi-view generation with explicit geometric control, such as joint six-view generation conditioned on normal and depth maps (Shao et al., 19 May 2025) or grid-based simultaneous generation with a reprojection-synchronization loop (Yan et al., 3 Jun 2025). A fourth line extends the objective from RGB texture toward full PBR stacks, including albedo, metallic, roughness, and normal bump representations (Vainer et al., 2024, He et al., 13 Mar 2025).

The recurrent technical difficulty is that texture generation must satisfy several constraints simultaneously: view consistency, geometry-texture alignment, local detail preservation, occlusion handling, and, for material systems, illumination invariance. The literature repeatedly frames these as coupled rather than separable requirements. MVPainter formalizes them as three core dimensions—reference-texture alignment, geometry-texture consistency, and local texture quality—while RoomPainter distinguishes global consistency from local consistency and addresses them in two separate stages (Shao et al., 19 May 2025, Huang et al., 2024).

2. Common diffusion formulations

Most methods retain standard latent diffusion or DDPM/DDIM machinery and modify the conditioning, synchronization, or reconstruction pathway. MaterialMVP states the forward and reverse processes as

$$q(x_t \mid x_{t-1})=\mathcal{N}\bigl(x_t;\sqrt{1-\beta_t}\,x_{t-1},\beta_t I\bigr),$$

$$p_\theta(x_{t-1}\mid x_t)=\mathcal{N}\bigl(x_{t-1};\mu_\theta(x_t,t),\Sigma_\theta(x_t,t)\bigr),$$

with the simplified $\epsilon$-prediction loss

$$\mathcal{L}_{\text{simple}} = \mathbb{E}_{x_0,\epsilon,t}\bigl\|\epsilon-\epsilon_\theta(x_t,t)\bigr\|_2^2.$$

This formulation is representative of end-to-end trainable texture and material generators (He et al., 13 Mar 2025).

A major design choice is whether multi-view consistency is imposed explicitly or implicitly. MaterialMVP introduces a separate consistency term,

$$\mathcal{L}_{\mathrm{cons}} =\mathbb{E}_t\bigl\|\epsilon_\theta(z_t,t,c(I_1))-\epsilon_\theta(z_t,t,c(I_2))\bigr\|_2^2,$$

and combines it with the PBR reconstruction term through

$$\mathcal{L}_{\mathrm{total}}=(1-\lambda)\mathcal{L}_{\mathrm{pbr}}+\lambda\mathcal{L}_{\mathrm{cons}},\quad \lambda=0.1.$$

By contrast, MVPainter states that there is no additional explicit multi-view consistency loss, because all six views are generated jointly in one pass and geometric conditioning back-propagates through the same denoising objective (He et al., 13 Mar 2025, Shao et al., 19 May 2025).

Sampling-time systems often keep the pretrained 2D model frozen and enforce consensus outside the training objective. TexPainter runs DDIM denoising in parallel per camera, decodes the per-view noiseless estimates to color space, fuses them into a single UV texture by weighted averaging, then optimizes the latent predictions so that each decoded view matches the rendering of that common texture (Zhang et al., 2024). MD-ProjTex similarly computes per-view cleaned latents, decodes and projects them into UV space, re-encodes the fused atlas, estimates guided noise, and performs a joint update using fused denoising directions rather than latent optimization (Yildirim et al., 3 Apr 2025). FlexPainter replaces DDPM with a latent flow-matching formulation over a 2×2 grid of four views, then inserts a UV reprojection-synchronization loop at every denoising step (Yan et al., 3 Jun 2025).

3. Mechanisms for enforcing cross-view consistency

The literature converges on three recurrent mechanisms. This suggests a practical taxonomy consisting of UV-space synchronization, feature-space communication, and learned aggregation from generated views into texture space.

Mechanism	Core idea	Representative papers
UV-space synchronization	Fuse denoised views into a shared atlas or latent texture at each step	(Liu et al., 2023, Zhang et al., 2024, Huang et al., 2024, Yildirim et al., 3 Apr 2025)
Feature-space communication	Exchange information across views inside the denoiser via attention or shared keys/values	(Shao et al., 19 May 2025, He et al., 13 Mar 2025, Xiong et al., 2 Sep 2025, Vainer et al., 2024)
Learned backprojection	Replace heuristic baking/averaging with a trained geometric aggregation module	(Georgiou et al., 19 Feb 2025)

The UV-space family treats a shared atlas as the synchronization variable. The synchronized multi-view diffusion method maintains a single latent texture $W_t$ in UV space, renders it into all views, projects per-view denoised estimates back into UV, aggregates them as

$$\hat W_{0|t}= \frac{\sum_{i=1}^N [M_i\odot w_{0|t}^{(v_i)}]} {\sum_{i=1}^N M_i+\gamma},$$

and performs the reverse update in UV rather than independently per view (Liu et al., 2023). TexGen similarly maintains an explicit RGB texture map parameterized by the denoising step, combines multi-view DDIM sampling with attention-guided sampling, and re-estimates noise through its text-texture-guided noise resampling formulation (Huo et al., 2024). RoomPainter generalizes the same principle to indoor scenes through dynamic UV-space merging in Stage I and a repaint procedure in Stage II, while MVPaint inserts a synchronization loss after decoding, UV fusion, and re-encoding to align per-view latents to a single fused texture (Huang et al., 2024, Cheng et al., 2024).

Feature-space methods instead move consistency into the denoiser. MaterialMVP introduces synchronized attention across views,

$$z_i^{\text{new}} =\sum_{j=1}^n \mathrm{Softmax}\!\Bigl(\tfrac{Q_iK_j^T}{\sqrt d}\Bigr)V_j,$$

to fuse geometric cues across all views, and adds Multi-Channel Aligned Attention so that metallic-roughness latents are updated by residual aligned attention from the albedo stream (He et al., 13 Mar 2025). MVPainter conditions a UNet on a fused embedding built from depth, normal, and reference-image features using a Union ControlNet side branch, and because the six views are generated jointly as a 3×2 grid, view coherence is obtained in one pass (Shao et al., 19 May 2025). In C33D’s Texture Multi-View Diffusion, all non-front views replace self-attention keys and values with those projected from a fixed novel front-view latent, forcing every view to query the same texture prototype (Xiong et al., 2 Sep 2025). In collaborative-control PBR texturing, the multi-view communication block comprises point-wise correspondence attention, cross-attention to reference-view hidden states, and cross-attention to DINOv2 features (Vainer et al., 2024).

Learned aggregation methods focus on the projection stage. Im2SurfTex replaces "most front-facing" or uniform averaging heuristics with a neural backprojection module that gathers local pixel neighborhoods across views and uses relative 3D displacement, normal differences, view angle, and geodesic distance within a three-block cross-attention network to predict each texel (Georgiou et al., 19 Feb 2025). This does not jointly denoise views, but it addresses the same multi-view consistency problem at the texture assembly stage.

4. Geometry control, conditioning, and view layout

A defining characteristic of texture multi-view diffusion is the use of explicit geometry priors to align appearance with the underlying surface. MVPainter’s conditioning uses per-view normals and depths plus a single reference image, with a condition encoder and a lightweight transformer that fuse these inputs into a single conditioning embedding injected through ControlNet at multiple resolutions (Shao et al., 19 May 2025). RoomPainter renders depth maps and similarity masks, converts them to UV-space weighting maps, and uses SDXL latent U-Net plus ControlNet-depth as the base model (Huang et al., 2024). FlexPainter appends depth channels to a 2×2 four-view grid and conditions a latent flow-matching model on a shared multimodal embedding that can combine text and reference-image inputs (Yan et al., 3 Jun 2025).

Several systems embed camera geometry directly into the denoiser. Collaborative Control adds Plücker-coordinate ray embeddings to the multi-view blocks’ queries, keys, and values, and appends VAE-encoded surface normals to each UNet input (Vainer et al., 2024). 3DEnhancer’s pose-aware encoder forms a 9-channel input from RGB and per-pixel Plücker coordinates, then injects pose-aware features into a DiT-based denoiser via a ControlNet-style copy mechanism; its transformer blocks add multi-view row attention and explicit near-view epipolar aggregation (Luo et al., 2024). Although 3DEnhancer addresses enhancement rather than UV texturing, its modules illustrate how multi-view consistency can be enforced at the feature level under varying camera configurations.

View layout is itself treated as an architectural variable. TexPainter fixes $N=8$ cameras on a sphere (Zhang et al., 2024). TexGen uses $$p_\theta(x_{t-1}\mid x_t)=\mathcal{N}\bigl(x_{t-1};\mu_\theta(x_t,t),\Sigma_\theta(x_t,t)\bigr),$$0 views, specifically eight equatorial azimuths plus a top-down view (Huo et al., 2024). MaterialMVP is mesh-conditioned through concatenated normal and position maps, while MVPainter generates six RGB views arranged as a 3×2 grid (He et al., 13 Mar 2025, Shao et al., 19 May 2025). These design choices imply different trade-offs between coverage, redundancy, and computational cost. A plausible implication is that view-parallel methods simplify synchronization but place greater pressure on the denoiser’s cross-view communication, whereas atlas-centric methods simplify consistency but must still solve occlusion and backprojection quality.

5. PBR and material-aware extensions

Texture multi-view diffusion increasingly targets physically based material generation rather than only RGB texture. MaterialMVP is an end-to-end, one-stage multi-view PBR diffusion model that generates albedo and metallic-roughness textures from 3D meshes and image prompts. Its core contributions are Reference Attention, Consistency-Regularized Training, Dual-Channel Material Generation, Multi-Channel Aligned Attention, and learnable material embeddings for the albedo and MR channels (He et al., 13 Mar 2025). The paper reports CLIP-FID $$p_\theta(x_{t-1}\mid x_t)=\mathcal{N}\bigl(x_{t-1};\mu_\theta(x_t,t),\Sigma_\theta(x_t,t)\bigr),$$1, FID $$p_\theta(x_{t-1}\mid x_t)=\mathcal{N}\bigl(x_{t-1};\mu_\theta(x_t,t),\Sigma_\theta(x_t,t)\bigr),$$2, CMMD $$p_\theta(x_{t-1}\mid x_t)=\mathcal{N}\bigl(x_{t-1};\mu_\theta(x_t,t),\Sigma_\theta(x_t,t)\bigr),$$3, CLIP-I $$p_\theta(x_{t-1}\mid x_t)=\mathcal{N}\bigl(x_{t-1};\mu_\theta(x_t,t),\Sigma_\theta(x_t,t)\bigr),$$4, and LPIPS $$p_\theta(x_{t-1}\mid x_t)=\mathcal{N}\bigl(x_{t-1};\mu_\theta(x_t,t),\Sigma_\theta(x_t,t)\bigr),$$5 in the image-to-texture setting, and qualitatively emphasizes illumination-invariant albedo, precise multi-view consistency, and aligned metallic/roughness details (He et al., 13 Mar 2025).

MVPainter approaches PBR differently. It first generates six RGB views with geometric control, then applies a dedicated network $$p_\theta(x_{t-1}\mid x_t)=\mathcal{N}\bigl(x_{t-1};\mu_\theta(x_t,t),\Sigma_\theta(x_t,t)\bigr),$$6, based on IDArb and modified with parallel attention blocks and higher output resolution, to predict $$p_\theta(x_{t-1}\mid x_t)=\mathcal{N}\bigl(x_{t-1};\mu_\theta(x_t,t),\Sigma_\theta(x_t,t)\bigr),$$7, $$p_\theta(x_{t-1}\mid x_t)=\mathcal{N}\bigl(x_{t-1};\mu_\theta(x_t,t),\Sigma_\theta(x_t,t)\bigr),$$8, and $$p_\theta(x_{t-1}\mid x_t)=\mathcal{N}\bigl(x_{t-1};\mu_\theta(x_t,t),\Sigma_\theta(x_t,t)\bigr),$$9, which are then UV-baked onto the mesh (Shao et al., 19 May 2025). In this formulation, multi-view diffusion is responsible for geometry-texture consistency and local detail, while PBR attribute extraction is a second learned stage. Collaborative Control goes further by directly modeling PBR image probability distributions and outputting an eight-channel per-view latent containing linear albedo, roughness, metallic, and a tangent-space normal bump map, with cross-domain communication between RGB and PBR branches and multi-view communication after each self-attention block (Vainer et al., 2024).

The emergence of material-aware systems also clarifies a frequent misconception: multi-view consistency alone does not guarantee physically meaningful texture. Several papers explicitly separate view consistency from lighting or material disentanglement. MaterialMVP addresses illumination invariance by training across varying viewpoints and illumination conditions (He et al., 13 Mar 2025), and MVPainter renders reference views under random point or area lights or HDRI specifically to teach the network to factor out lighting before PBR extraction (Shao et al., 19 May 2025). This suggests that consistent RGB baking is insufficient when the target is relightable material.

A related development is the critique of multi-view baking itself. NaTex argues that 2D MVD pipelines have inherent limitations in occluded-region handling, boundary alignment, and cross-view color coherence, and replaces view-based baking with latent color diffusion over a dense color point cloud in 3D (Lai et al., 20 Nov 2025). In that sense, native 3D texture generation is not a rejection of texture multi-view diffusion so much as a response to the constraints identified by its most successful 2D/UV-based systems.

6. Evaluation practices, applications, and limitations

Evaluation protocols vary substantially across papers, reflecting the multi-objective nature of the task. Texture-generation benchmarks commonly report FID, KID, and CLIP-score, as in Im2SurfTex and TexGen (Georgiou et al., 19 Feb 2025, Huo et al., 2024). Material systems add CLIP-FID, CMMD, LPIPS, and CLIP-I, as in MaterialMVP (He et al., 13 Mar 2025). TexPainter reports FID against SD-2-Depth ground-truth renderings, user-study consistency rankings, and PSNR/SSIM gains over TexFusion (Zhang et al., 2024). MVPainter departs from purely image-based metrics by using a human-aligned VLM evaluation with Elo scoring across reference-texture alignment, geometry-texture consistency, and local texture quality, and reports the highest Elo in all three dimensions across TripoSG, Hunyuan3D-2.0, TRELLIS, and Hi3DGen geometry sources (Shao et al., 19 May 2025).

Applications now extend well beyond prompt-only mesh texturing. RoomPainter targets indoor scene texturing with a two-stage global/local strategy (Huang et al., 2024). FlexPainter adds flexible multi-modal conditioning, including text strings and reference images in a shared embedding space, plus image-based CFG for structure-style decomposition (Yan et al., 3 Jun 2025). C33D uses Texture Multi-View Diffusion to refine remaining views after a novel front view is produced by adaptive text-image harmony (Xiong et al., 2 Sep 2025). FROMAT adapts pretrained multiview models for appearance transfer by learning layer-wise self-attention mixing from object and reference streams, enabling transferred materials, textures, or styles while preserving object identity (Kompanowski et al., 10 Dec 2025).

Several limitations recur. Occluded or never-observed texels remain difficult: TexPainter notes that uncovered regions rely entirely on decoder priors (Zhang et al., 2024), and Collaborative Control states that any texel never observed among generated views remains blank (Vainer et al., 2024). Boundary alignment and UV discontinuities remain a structural weakness of view-bake pipelines, leading to texture bleeding or jagged seams (Zhang et al., 2024, Lai et al., 20 Nov 2025). Compute can also be substantial. TexPainter reports $\epsilon$0 minutes per mesh, explicitly attributing the cost to per-step optimization (Zhang et al., 2024), whereas MD-ProjTex emphasizes a faster training-free alternative with $\epsilon$1 minutes on 36 views and 20 DDIM steps on a single Tesla T4 (Yildirim et al., 3 Apr 2025). Data quality and viewpoint bias also persist: synchronized MVD notes bias toward canonical front views, and Collaborative Control notes the consequences of heterogeneous Objaverse quality for real-world PBR styles (Liu et al., 2023, Vainer et al., 2024).

Taken together, these results define texture multi-view diffusion as a research area organized around a single technical imperative: obtaining one texture or material representation from multiple denoising trajectories without sacrificing geometric alignment or fine detail. Existing methods differ in where that consensus is enforced—UV space, latent space, attention layers, learned baking, or native 3D color space—but they share the same central premise: consistency must be built into the diffusion process itself, not delegated to a final projection step.