Pygmalion Effect: Clay-Guided 3D Reconstruction

• The paper introduces a dual-branch framework that converts reflective images into clay-like representations to enhance 3D geometry recovery.
• It employs an image-to-image diffusion pipeline to generate pseudo–ground truth clay images, ensuring robust convergence under challenging lighting.
• Quantitative evaluations reveal up to 28% improvement in Chamfer-L1 error and notable gains in normal fidelity for highly reflective objects.

The Pygmalion Effect in Vision is a metaphor-driven computational framework for reflection-robust 3D geometry reconstruction, introduced by Lee et al. in 2025. It addresses the persistent challenge of disentangling object geometry from view-dependent specular reflections in multi-view images of highly reflective surfaces. Drawing on the myth of Pygmalion, the approach embeds an internalized "belief"—a learned clay rendering prior—guiding the model to suppress harmful view-dependent radiance and recursively refine geometry recovery. Central to this technique is the translation of real, reflective images into "clay-like" images containing only diffuse shading, which serve as pseudo–ground truth in a dual-branch network. The Pygmalion Effect in Vision demonstrates state-of-the-art improvements in normal fidelity and mesh accuracy, and exposes fundamental principles in leveraging self-generated priors as powerful inductive biases for complex appearance domains (Lee et al., 26 Nov 2025).

1. Definition, Metaphor, and Core Intuition

The Pygmalion Effect in Vision is defined as the recursive loop in which a model’s internal belief—specifically, a learned clay-rendering prior—is projected back onto the observed data to neutralize view-dependent radiance, thereby stabilizing and improving geometry recovery. The "Radiance → Clay" intuition arises from the observation that specular reflections entangle observed color with environmental lighting, complicating geometric inference. By translating each input photograph $I$ into a neutral, matte, clay-like image $I_{clay} = f_{clay}(I)$, the network effectively "un-shines" the object. This transformation isolates geometric cues by suppressing specular highlights, ensuring that any residual brightness variations encode only surface orientation rather than mirrored environmental content.

Metaphorically, this process emulates Pygmalion's act of sculpting an ideal form: the model internalizes a canonical, reflection-free template (white clay) against which all reconstructions are regularized, closing a feedback loop between internally generated priors and observed evidence.

2. Motivation for Specular Suppression

Reflective surfaces fundamentally challenge traditional multi-view stereo and photometric consistency assumptions, as the observed color of a surface point becomes a function of view-dependent environmental illumination. When jointly optimizing for geometry and BRDF, the optimizer can trade off geometry adjustments against changes in specular reflectance, yielding unstable or ambiguous solutions. By early removal or reduction of the view-dependent components, the clay-guided branch imposes a strong geometric prior: $I_{clay}$ is nearly free of environmental "noise," so shape recovery is based on diffuse shading determined by object orientation. This approach enables robust convergence and mitigates the instability endemic to purely photometric or inverse rendering-based methods when confronted with glossy inputs.

3. Dual-Branch Network and Rendering Architecture

The architecture is structured as a dual-branch system sharing common geometric parameters (Gaussian centroids $p_i$, local tangents $t_{u,i}$, $t_{v,i}$, scales $s_{u,i}$, $s_{v,i}$, opacities $\alpha_i$, and material parameters $\lambda_i$, $m_i$, $r_i$, $n_i$). The rendering process bifurcates into (a) a BRDF-based reflective branch and (b) a clay-guided branch:

• Reflective (BRDF) Branch:
  • Inputs: camera output direction $\omega_o$, prefiltered environment map $L_i(\omega_i)$, and per-Gaussian features $\theta_i = [\lambda_i, m_i, r_i, n_i]$.
  • Shading: Diffuse component integrated in closed form; specular via GGX microfacet model
  • $$f_s(\omega_i, \omega_o) = D(n; \omega_h, r) \cdot G(n, \omega_i, \omega_o) \cdot F(\omega_h, n)$$.
  • Outgoing radiance:

  $$L_o(x, \omega_o) = \int_\Omega f(\omega_i, \omega_o) L_i(\omega_i) (n \cdot \omega_i) d\omega_i$$ - Split-sum approximation:

  $$L_s(\omega_o) \approx \left(\int_\Omega f_s (n \cdot \omega_i) d\omega_i\right) \left(\int_\Omega L_i(\omega_i) D...(n \cdot \omega_i) d\omega_i\right)$$ - Output: rendered RGB image $I_{rgb}$.

• Clay-Guided Branch:
  • Inputs: geometry from shared Gaussians, single color code $\hat{c}_i$ per Gaussian.
  • Rendering (strictly view-independent):

  $$\hat{I}_{clay}(x) = \sum_{i=1}^N \hat{c}_i \alpha_i G_i(u(x)) \prod_{j<i} (1 - \alpha_j G_j(u(x)))$$ - Supervision: $I_{clay}$ generated by $f_{clay}$; architecture omits any view-dependent components, enforcing radiance neutrality.

This synergy allows the clay branch to provide geometry stabilization while the reflective branch models complex appearance effects.

4. Training Objective and Loss Functions

The full training loss combines supervisory and regularization terms aligned with network modularity:

• RGB Photometric Loss on reflective branch:

$$L_{rgb} = \|I_{rgb} - I_{real}\|_1 + \lambda_{ssim}(1 - SSIM(I_{rgb}, I_{real}))$$

• Clay Supervision Loss (reflection suppression) on clay branch: $$L_{clay} = \|\hat{I}_{clay} - I_{clay}\|_1 + \lambda_{dssim}(1 - SSIM(\hat{I}_{clay}, I_{clay}))$$ with $\lambda_{dssim} \approx 0.8$.
• Normal Smoothness / Geometry Consistency Loss:

$$L_{smooth} = (1 - \lambda_{smooth})\text{sg}(L_{rgb})(n_i) + \lambda_{smooth}\|\nabla n_i\|_2^2$$

where $\lambda_{smooth} = t/T_{clay}$, and $\text{sg}(·)$ denotes stop-gradient for stability.

The total training loss is

$$L_{total} = L_{rgb} + \lambda_{clay} L_{clay} + \lambda_{smooth} L_{smooth}$$

No adversarial loss is used; the L1+SSIM term suffices for clay-domain reconstruction. Geometry is decoupled from RGB gradients during early iterations to further stabilize optimization.

5. Data Pipeline and Diffusion-Based Clay Generation

Clay-image generation leverages an image-to-image diffusion pipeline. $f_{clay}$ is instantiated as a diffusion transformer (OminiControl) with minimal LoRA fine-tuning, trained on 100,000 rendered pairs from Objaverse (random metalness $m \sim \{0,1\}$, roughness $r \sim U(0.03, 0.3)$) and 5,000 FLUX→Nano-Banana–generated pairs. During 3D reconstruction, the operator $f_{clay}$ is applied per-view to yield clay supervision images $I_{clay}$. The use of domain-specific, high-fidelity clay images as pseudo–ground truth is critical in regularizing geometry against reflection-induced noise.

Dataset	Image Pairs	Metalness Sampling	Roughness Range
Objaverse	100,000	$m \sim \{0,1\}$	$r \sim U(0.03,0.3)$
FLUX→Nano-Banana	5,000	—	—

6. Quantitative and Qualitative Evaluation

Performance is evaluated using mesh completeness/accuracy (Chamfer-L1) and normal accuracy (mean angular error, MAE):

• GlossySynthetic (Chamfer-L1): RGS baseline 0.0085 → Pygmalion 0.0061 (~28% improvement)
• DTU (Chamfer-L1): 0.84 → 0.74
• Shiny Blender (Normal MAE): RGS ~2.94° → Pygmalion ~2.40°, with pronounced improvements on highly specular objects.

Ablation analysis reveals that applying the clay branch for the initial 10,000 iterations further reduces Chamfer-L1 (0.0069 → 0.0061), and detaching geometry from RGB gradients during this period enhances stability.

Qualitative studies demonstrate accurate highlight removal in clay translations, preservation of fine geometric detail, and superior mesh reconstructions for real and synthetic reflective objects. Recovery of smooth surfaces (e.g., car fenders, mugs) and sharper normal maps are visually evident.

7. Broader Implications and Generalization Potential

Harnessing the Pygmalion Effect in Vision provides a new inductive bias for geometry learning in reflective settings. By training a model to align with its own internally synthesized "ideal" clay representations, the framework advances mesh accuracy and normal fidelity beyond prior reflection-handling techniques. A notable implication is the advocacy for neutralizing harmful variability—such as specular highlights—via domain translation techniques, rather than escalating the sophistication of inverse rendering architectures. The approach of "seeing by un-shining" may plausibly extend to domains with translucent or subsurface scattering materials, or inspire new forms of domain translation (e.g., generating "lighting-agnostic" sketches) to further untangle appearance from shape (Lee et al., 26 Nov 2025).