3D PixBrush: Localized Texture Synthesis

• 3D PixBrush is a method for image-driven, localized texture synthesis on 3D meshes that automatically identifies mesh regions and applies reference styles without user input.
• It utilizes a modified score distillation sampling framework that integrates both text- and image-driven loss terms with an MLP-based mask predictor for precise localization.
• The system achieves high-fidelity texture placement with global coherence and local precision, outperforming baselines in metrics such as CLIP R-Precision and LPIPS.

3D PixBrush is a method for image-driven, localized texture synthesis on 3D meshes. It introduces a framework that predicts both a localization mask and a region-specific color field on the mesh surface, enabling automatic placement and high-fidelity transfer of visually complex objects or styles from a reference image to a specified mesh region, all without user marking or region selection. Its architecture relies on modifications to score distillation sampling (SDS) that handle both text- and image-driven loss terms, culminating in a system that is globally contextualized and locally precise (Decatur et al., 4 Jul 2025).

1. Problem Formulation and Objectives

3D PixBrush is designed to address the challenge of transplanting an object—defined jointly by appearance and geometric extent—from a 2D reference image onto a 3D mesh in a localized manner. The system operates on:

• Inputs
  • A 3D mesh $M$ with UV parameterization; points on the surface are $x \in \mathbb{R}^3$.
  • A reference image $I_{\text{ref}} \in [0,1]^{H \times W \times 3}$ depicting the target object/style.
  • A text prompt $y$ (e.g., “a cow with sunglasses”) specifying coarse semantics.
• Outputs
  • A probability-valued localization mask $p(x) \in [0,1]$ identifying mesh regions for synthesis.
  • A synthesized color field $c(x) \in [0,1]^3$ that matches $I_{\text{ref}}$ within the region $p(x) > 0$.

The compound objective is to autonomously select, localize, and texture mesh regions so that $p(x)$ identifies the reference object’s geometry and $c(x)$ replicates the style/structure of $x \in \mathbb{R}^3$0. No user-provided region hints are required. Optimization is performed jointly over a localization network $x \in \mathbb{R}^3$1 and a texture network $x \in \mathbb{R}^3$2 using mixed text- and image-guided SDS losses (Decatur et al., 4 Jul 2025).

2. Mask Prediction Architecture and Training

The localization mask $x \in \mathbb{R}^3$3 is parameterized by a compact multilayer perceptron (MLP) coupled to a periodic positional encoder $x \in \mathbb{R}^3$4 as used in NeRF, forming $x \in \mathbb{R}^3$5 with $x \in \mathbb{R}^3$6 being the sigmoid nonlinearity.

• Network details
  • Six ($x \in \mathbb{R}^3$7) MLP blocks: (fully connected → BatchNorm → ReLU), each width $x \in \mathbb{R}^3$8.
  • Evaluation per-vertex, per-face, or per-texel as needed.
• Losses
  1. Text-driven SDS Loss ($x \in \mathbb{R}^3$9): Based on rendering the mask via a differentiable renderer $I_{\text{ref}} \in [0,1]^{H \times W \times 3}$0, producing image $I_{\text{ref}} \in [0,1]^{H \times W \times 3}$1, and applying SDS with a pretrained diffusion noise predictor $I_{\text{ref}} \in [0,1]^{H \times W \times 3}$2. The objective iteratively nudges mask renders to match semantic content inferred from the text prompt.
  2. Image-modulated Texture Loss: Propagates gradients through masked $I_{\text{ref}} \in [0,1]^{H \times W \times 3}$3 to penalize mismatches between synthesized texture and reference image in the selected region.
  3. Smoothing Regularization:

  $I_{\text{ref}} \in [0,1]^{H \times W \times 3}$4

  enforcing local spatial coherence in the mask to avoid spurious regions.

3. Localization-Modulated Image Guidance Mechanism

The core innovation is "localization-modulated image guidance" (LMIG), which restricts image-conditioned diffusion guidance to only those surface areas selected by the predicted mask:

• SDS is generalized to support both text ($I_{\text{ref}} \in [0,1]^{H \times W \times 3}$5) and image ($I_{\text{ref}} \in [0,1]^{H \times W \times 3}$6) conditions via a joint-conditioned diffusion backbone with IP-Adapter.
• The mask $I_{\text{ref}} \in [0,1]^{H \times W \times 3}$7 is rendered and binarized to generate $I_{\text{ref}} \in [0,1]^{H \times W \times 3}$8.
• At every decoder layer $I_{\text{ref}} \in [0,1]^{H \times W \times 3}$9, attention from the reference image tokens, $y$0, is multiplied with $y$1 (downsampled mask), ensuring spatial correspondence.
• Formally, the modified gradient is:

$y$2

where only regions inside $y$3 are updated with image guidance; outside, only text conditioning applies.

This approach produces globally coherent positioning and locally accurate segmentation of complex objects without explicit interaction.

4. Texture Field Synthesis and Rendering Pipeline

The color field $y$4 is parameterized by an MLP of architecture identical to $y$5, except for output channels (3 per surface point). The network is trained to match $y$6 within $y$7 using LMIG loss propagated through the rasterized, masked texture field:

• Texture Atlas Evaluation:
  • For UV-mapped surfaces, colors are synthesized at each texel center ($y$8), multiplied by $y$9, and rendered to screen.
• Additional Losses:
  • Total Variation (TV):

  $p(x) \in [0,1]$0

  promoting local patch smoothness. $p(x) \in [0,1]$1. - Perceptual (VGG) Loss:

  $p(x) \in [0,1]$2

  with $p(x) \in [0,1]$3, where $p(x) \in [0,1]$4 denote VGG features.

5. Implementation Protocol

• Data:

  • Meshes include VOC 3D models, real human scans, busts, animals.
  • Reference images feature high-resolution photos of clothing, accessories, and decorative patterns.
  • Prompts composed as, e.g., “a <mesh-class> with <object>.”
• Optimization:
  • Warm-up ($p(x) \in [0,1]$5k iters): $p(x) \in [0,1]$6 only, localizes $p(x) \in [0,1]$7 using text-driven SDS.
  • Joint phase ($p(x) \in [0,1]$8k iters): $p(x) \in [0,1]$9 on both $c(x) \in [0,1]^3$0 and $c(x) \in [0,1]^3$1.
  • Adam optimizer, learning rate decay $c(x) \in [0,1]^3$2, $c(x) \in [0,1]^3$3.
  • Each iteration uses one random view, $c(x) \in [0,1]^3$4 render.
  • Full training: $c(x) \in [0,1]^3$5 hours on an A40 GPU, with reasonable quality achieved in $c(x) \in [0,1]^3$6 hour.

6. Evaluation, Benchmarks, and Ablations

• Quantitative Metrics:
  • CLIP R-Precision (textual and image-based) between $c(x) \in [0,1]^3$7 and mesh renders.
  • Cosine similarity in CLIP embedding space.
  • LPIPS (AlexNet/VGG) for perceptual similarity.
  • User ratings (1–5) on texture and structure quality.
  • Intersection-over-Union (IoU) for predicted mask $c(x) \in [0,1]^3$8 in synthetic settings.
• Performance:
  • Exceeds a text-only editor (3D Paintbrush) by more than $c(x) \in [0,1]^3$9 percentage points in CLIP R-Precision.
  • Achieves $I_{\text{ref}}$0–$I_{\text{ref}}$1 lower LPIPS.
  • User studies report mean scores: $I_{\text{ref}}$2 (3D PixBrush) vs. $I_{\text{ref}}$3/$I_{\text{ref}}$4 (baselines) for structure/texture.
• Qualitative Ablations:
  • Without cross-attention masking, image information leaks outside the mask region.
  • Removing warm-up leads to poor mask separability and over-globalized texture.

7. Limitations and Prospective Extensions

• Failure Modes:
  • Fine detail (e.g., logos, small text) in $I_{\text{ref}}$5 are rendered illegibly (e.g., “adadds”).
  • Semantic bleed: co-texturing in related regions (makeup applied to lips).
  • Janus effect: mirrored/symmetric texture propagation front to back.
• Future Directions:
  • Extension of LMIG to NeRFs, point clouds, volumetric grids, or video.
  • Integration of geometric deformation within active region $I_{\text{ref}}$6.
  • Support for simultaneous multi-object editing, stacking disjoint $I_{\text{ref}}$7 fields.

A plausible implication is that LMIG and its multi-modal guidance framework are generally applicable to broader classes of 3D editing, where geometry-aware, high-fidelity, and user-free local synthesis are sought (Decatur et al., 4 Jul 2025).