3D Discriminative Autoencoder

• The paper introduces a novel framework that leverages unsupervised adversarial training and differentiable rendering to learn realistic 3D object surfaces without annotations.
• It employs a convolutional generator to produce explicit 3D mesh geometry, vertex-level textures, and background images, integrated with mesh-smoothness regularization.
• Experimental results on ShapeNet and CelebA demonstrate effective 3D reconstruction and pose estimation, while highlighting challenges like the hollow-mask illusion.

A 3D discriminative autoencoder is a framework that learns 3D object surfaces, textures, and viewpoints directly from unannotated image collections. In this architecture, a convolutional neural network generator outputs explicit 3D mesh geometry and corresponding texture maps, alongside a background image, which are rendered into 2D images using a differentiable renderer. The principal innovation is unsupervised learning: the model is trained adversarially such that if the generated image is indistinguishable from real images, the underlying 3D representation must be realistic as well. This is achieved without using annotations such as object pose, landmarks, or masks. The framework can pair the generative model with an encoder to enable direct 3D reconstruction and pose estimation for single images, demonstrating that truly unsupervised 3D mesh learning is feasible from in-the-wild data (Szabó et al., 2018).

1. Architecture and Workflow

The 3D discriminative autoencoder consists of an encoder (E), generator (G), differentiable renderer (R), and discriminator/critic (D).

• Encoder (E): Accepts a single image $x \in \mathbb{R}^{H \times W \times 3}$ and produces a latent code $z_e = E_z(x) \in \mathbb{R}^d$ and viewpoint estimate $v_e = E_v(x)$, where $v_e$ typically models Euler angles (azimuth, elevation, yaw).
• Generator (G): Receives latent code $z \in \mathbb{R}^d$, partitions it into $(z_o,z_b)$, decodes $z_o$ to a fixed-topology mesh $S(z_o)$ (vertex positions $s \in \mathbb{R}^{N_v \times 3}$) and texture $T(z_o)$ (RGB colors $t \in \mathbb{R}^{N_v \times 3}$), and $z_b$ to background $B(z_b) \in \mathbb{R}^{H \times W \times 3}$.
• Random Viewpoint (v): For synthetic samples, $v_f \sim p_v$ (uniform over azimuth and elevation).
• Differentiable Renderer (R): Projects geometry and texture under a perspective camera with smooth silhouette blending, outputting image $\hat x = R(S(z), T(z), B(z); v)$ with exact gradients.
• Training Workflow:
  • GAN-style phase: G synthesizes samples from $z_f \sim \mathcal{N}(0,I)$, $v_f$, and D distinguishes $R(G(z_f),v_f)$ (fake) from $x \sim p_{\rm data}$ (real).
  • Autoencoder phase: E encodes $x_r$, produces $(z_e,v_e)$, forwarded to fixed G and R for $\hat x_e = R(G(z_e),v_e)$, optimization minimizes reconstruction loss $\|x_r-\hat x_e\|$.

2. Mathematical Formulation of Objectives

The training regime leverages both adversarial and reconstruction losses, with mesh regularization.

• Reconstruction Loss:

  $${\cal L}_{\rm rec} = \mathbb{E}_{x_r \sim p_{\rm data}} [\|x_r - \hat x_e\|_1] \quad\text{or}\quad \mathbb{E}_{x_r \sim p_{\rm data}} [\|x_r - \hat x_e\|_2^2]$$

  where $\hat x_e = R(G(z_e),v_e)$ for encoded $(z_e,v_e)$.

• Adversarial (Wasserstein-GAN with Gradient Penalty) Loss:

  $$\min_{D:\|D\|_L\le1} -\mathbb{E}_{x_r}[D(x_r)] + \mathbb{E}_{z_f,v_f}[D(\hat x_f)] + \gamma\mathbb{E}_{\tilde x}[(\|\nabla_{\tilde x}D(\tilde x)\|_2-1)^2]$$

  where fake samples $\hat x_f = R(G(z_f),v_f)$.

  • Generator minimizes:

  $$\min_{G} -\mathbb{E}_{z_f,v_f}[D(R(G(z_f),v_f))] + \lambda_S{\cal L}_{\rm smooth}(G)$$

• Mesh-Smoothness Regularization:

  $${\cal L}_{\rm smooth}(G) = \mathbb{E}_{z_f}\left[\sum_{(i,j)\in\mathcal{N}}(1-n_i(z_f)\cdot n_j(z_f))\right]$$

  penalizing normal flips between adjacent triangles.

• Autoencoder Training (with fixed G):

  $$\min_E {\cal L}_{\rm rec}, \quad \text{subject to } z_e \in D_z,\, v_e \in D_v$$

  with constraints ensuring $z_e$ lies in the generator's support region.

3. Representation: Shape, Texture, Background, Viewpoint

Object geometry is parameterized as a mesh with $N_v$ vertices, where spherical–radial coordinates $\rho(\theta,\phi)$ are predicted and mapped to Cartesian positions. Texture assignment is per-vertex RGB, facilitating color interpolation using barycentric weights over rendered triangles.

Background images $B(z_b)$ are modeled on a distant sphere, so viewpoint shifts induce planar background motion, decoupling object and background parallax.

Viewpoint $v$ is parameterized by azimuth $\alpha$, elevation $\beta$, and optionally yaw $\gamma$. In GAN training, $\alpha$ and $\beta$ are sampled from a known prior $p_v$.

4. Differentiable Rendering Layer

Rendering employs a high-focal-length perspective transformation. Projected meshes are tiled onto the image plane; color within each triangle is interpolated per-vertex via barycentric weights. A soft silhouette model blends foreground at triangle and occlusion edges over Gaussian or linear ramp bands, crucial for ensuring nonzero gradients with respect to mesh vertices and colors. Importantly, the model omits explicit shading or illumination: it assumes purely Lambertian surfaces where vertex colors are emitted without simulated light or shadow.

5. Training Regime and Hyperparameters

Training proceeds in two distinct phases:

• GAN Phase: G and D are trained with random latent vectors and viewpoints over $10^5$–$2\times10^5$ steps. D is updated according to the WGAN-GP objective (gradient penalty $\gamma=10$), G includes the mesh-smoothness term ($\lambda_S$ in $10^{-4}$–$10^{-2}$). Adam optimizer is used with learning rate $10^{-4}$, $\beta_1=0.0$, $\beta_2=0.9$. Batch size: 16–64.

• Autoencoder Phase: With G frozen, E is optimized for $\ell_1$ or $\ell_2$ reconstruction over ~50k steps. Optionally, small regularization is applied to latent code magnitude.

6. Empirical Outcomes and Analysis

Evaluation is performed on synthetic ShapeNet classes and real face datasets (CelebA). On ShapeNet, geometry coverage is assessed via generated surface normals. For CelebA, results are qualitatively compared to conventional supervised 3D morphable model methods, specifically examining multi-azimuth renderings and normal maps.

• Findings:

  • Generator yields plausible 3D facial features (nose, brow, lips) rendered at novel azimuths ($\pm 90^\circ$).
  • Mesh-smoothness regularization eliminates high-frequency geometric spikes but, if excessive ($\lambda_S$ large), causes degenerate ellipsoid-like output.
  • Autoencoder reconstructions preserve large-scale geometry and pose; fine detail is coarsely reproduced.
• Failure Modes:
  • Hollow-mask illusion: for limited viewpoint data, concave facial reconstructions become plausible. Mitigation involves stricter object size constraints or expanded viewing angle priors.
  • Reference ambiguity: the generator's canonical orientation can arbitrarily rotate with respect to Euler axes. Reconstructions are plausible but axis alignment is arbitrary.
  • Deficient texture variation on features outside the mesh scope (mouth interiors, ears).

A plausible implication is that fully unsupervised learning of explicit 3D meshes and textures from uncontrolled images is tractable when adversarial realism constraints are enforced in 2D render space, and inversion via autoencoding enables direct estimation of shape and pose for novel inputs (Szabó et al., 2018).