EscherNet++: 3D Reconstruction & 2D Tiling

• The paper introduces a unified diffusion model that performs amodal completion and novel view synthesis, enhancing 3D reconstruction efficiency.
• It employs dual masking and cross-attention techniques to robustly infer occluded structures, achieving notable improvements in key metrics like PSNR.
• EscherNet++ also features a differentiable, text-guided pipeline for synthesizing periodic, tileable 2D meshes inspired by M.C. Escher.

EscherNet++ denotes two distinct methods introduced in the computer vision and generative modeling literature: (1) a masked fine-tuned diffusion model for unified amodal completion and zero-shot novel view synthesis in 3D reconstruction pipelines (Zhang et al., 10 Jul 2025), and (2) an automatic, text-guided pipeline for generating non-overlapping, periodic, tileable 2D meshes in the style of M.C. Escher (Aigerman et al., 2023). Both approaches leverage modern generative diffusion models and differentiable pipelines to resolve complex geometric and appearance constraints, but their technical objectives and architectures are sharply different.

1. Unified Amodal Completion and View Synthesis via Masked Fine-Tuned Diffusion

EscherNet++ (Zhang et al., 10 Jul 2025) addresses the challenge of simultaneously inferring missing object structure (amodal completion) and synthesizing arbitrary novel views from sparse, potentially occluded visual inputs. Unlike conventional pipelines, which decompose inpainting and view synthesis into distinct modules, EscherNet++ deploys a single end-to-end diffusion network, enabling cross-view consistency and efficient downstream 3D mesh reconstruction.

Architectural Components

The backbone is a latent diffusion model with a U-Net architecture interleaving residual convolutional blocks and transformer modules. Input images are encoded via a ConvNeXt-V2 extractor to obtain feature maps $f_i \in \mathbb{R}^{l \times c}$ per reference, while camera poses are encoded using Camera Positional Encoding (CaPE). During the diffusion process, the noisy latent $z_t$ is denoised through a function $$\epsilon_\theta\left(z_t; \{f_i\}, \{P_i^R\}, P^T\right)$$, leveraging cross-attention to ensure multi-view coherence for both source (reference) and target cameras.

Masked Fine-Tuning

Robustness to occlusion is imparted via two complementary masking regimes during fine-tuning:

• Input-level masking: For each reference view $X_i^R$, a random object occlusion silhouette $S$ is superimposed, yielding $$X_i^{R,\text{mask}} = (1 - S) \odot X_i^R + S \odot B$$, with $B$ the background. Each view is masked independently with probability $p_\text{in} = 0.5$.
• Feature-level masking: Randomly selects 25% of reference feature encodings per batch and zeroes out half of their spatial tokens via a binary mask $M_\text{feat}$, enforcing $F_\text{mask} = F \odot (1-M_\text{feat})$ on the concatenated feature tensor.

The loss objective is a weighted sum of standard diffusion loss over unmasked, input-masked, and feature-masked data streams: $$L = L_0 + \lambda_\text{in} L_\text{in} + \lambda_\text{feat} L_\text{feat}$$ with $\lambda_\text{in} = \lambda_\text{feat} = 1$.

Tightly Coupled Completion and Synthesis

Cross-view self- and cross-attention mechanisms within the U-Net's transformer blocks allow the network to localize occlusions and infer complete geometry jointly, enabling generation of consistent, hallucinated novel views suitable for object-centric reconstruction without explicit inpainting-then-render cascades.

2. Scalable Integration for Near–Real-Time 3D Reconstruction

EscherNet++ is designed for plug-and-play integration with pre-trained, feed-forward image-to-mesh systems such as InstantMesh (Xu et al. 2024), obviating per-object optimization. Synthesized images at arbitrary camera poses are fed directly as multi-view supervision, enabling full mesh reconstruction in as little as 1.3 minutes—a reduction of over 95% in inference time compared to iterative pipelines, with the further benefit of eliminating object-specific checkpoint storage (Zhang et al., 10 Jul 2025).

Pipeline	Time per object	Storage
EscherNet++ + InstantMesh	≈1.3 min	Universal
EscherNet+NeuS (overfit)	27 min	Per-object

This scalable approach supports rapid evaluation across large datasets and deployment in real-world occlusion scenarios.

3. Empirical Evaluation and Performance

Training utilizes 300K objects rendered from Objaverse-1.0 with 3 reference views per object, while evaluation employs the OccNVS benchmark with Google Scanned Objects (GSO), RTMV, and NeRF-Synthetic splits, examining both complete and heavily occluded cases. Key metrics include photometric measures (PSNR, SSIM, LPIPS) and 3D structure quality (Volume IoU, Chamfer distance).

Notable results:

• In 10-view, occluded NVS, EscherNet++ surpasses its precursor by +3.9 dB in PSNR and reduces average LPIPS by more than 0.02.
• When paired with InstantMesh for 3D reconstruction (10 references), Volume IoU increases by +0.28 (e.g., 0.4557→0.5912 on GSO3D), with qualitative improvements in handling complex real-world occluders.
• The model generalizes to heavily cluttered real-world captures, reconstructing invisible or ambiguous structure (e.g., missing limbs), attributed to its unified masking and cross-view reasoning (Zhang et al., 10 Jul 2025).

4. Limitations and Prospective Directions

Known limitations include smoothing of high-frequency structures (notably thin, rod-like elements) at standard $256\times 256$ input resolution and failures on out-of-distribution occluder silhouettes, where the model occasionally generates implausible completions. Planned enhancements involve:

• Conditioning on higher-resolution images ($>512\times 512$) and incorporating multi-scale masking.
• Leveraging additional supervision modalities such as depth and semantic cues for fine-tuning.
• Joint optimization of segmentation, pose estimation, and reconstruction alongside the diffusion core (Zhang et al., 10 Jul 2025).

5. Generative, Text-Guided Tiling of 2D Meshes for Escher-Style Patterns

A second use of the name EscherNet++ (Aigerman et al., 2023) refers to an end-to-end differentiable method for synthesizing periodic, non-overlapping 2D mesh tiles under arbitrary wallpaper group symmetries—addressing a different domain of geometric generative modeling than the 3D view synthesis context above.

Mathematical Parameterization

Tiles are realized as disk-topology triangular meshes $(V, F)$, with $V\in\mathbb{R}^{N\times 2}$ constrained by an Orbifold Tutte Embedding (OTE) linear system: $L(W)V = 0\ C V = b$ where $L(W)$ is the weighted combinatorial Laplacian (weights $w_{ij} > 0$ per directed edge), and $C$ encodes all periodic (wallpaper group-induced) boundary identifications. A key theorem states all valid, tileable, bijective embeddings correspond bijectively to some strictly positive $W$ solution; thus, optimizing $W$ furnishes a differentiable coordinate chart over all valid tilings.

Pipeline and Optimization

The system jointly optimizes:

• Geometry, by varying $W$ (and a global rotation angle).
• Texture, via a learnable $256^2$ or $512^2$ image per mesh.

The mesh is rendered using a differentiable renderer (Nvdiffrast). Text guidance arises by backpropagating a Score Distillation Sampling (SDS) loss (à la DreamFusion) through the rendered image, using a pre-trained text-to-image diffusion model conditioned on the user’s prompt; including randomized backgrounds per step avoids degenerate solutions.

Tiling Patterns and Evaluation

Any two-dimensional wallpaper group is supported. Experiments produce visually rich tilings (“a ballet dancer”, “a dragon”, “a puzzle piece in the shape of Z”) with seamless global structure. Ablation studies highlight the effects of background sampling and color channel representation on geometric refinement, and multi-shape tilings are achievable by mesh subdivision and parallel text prompts.

The correctness of the produced infinite patterns is ensured by the OTE parameterization, though some phenomena—such as loss of thin details or the emergence of nearly convex shapes on simple prompts—reflect both the guidance strength and the generative model’s limitations. Quantitative FID or user studies are not reported; assessment is via qualitative fidelity and geometric guarantees (Aigerman et al., 2023).

6. Relationship and Distinctions Between the Two Methods

The EscherNet++ name refers to two unrelated methodologies, each notable within its respective research context:

• The 3D reconstruction EscherNet++ (Zhang et al., 10 Jul 2025) unifies occlusion-robust amodal completion and scalable novel view synthesis in a diffusion-based image pipeline, substantially accelerating mesh reconstruction workflows.
• The generative tiling EscherNet++ (Aigerman et al., 2023) constructs text-to-pattern pipelines for 2D periodic mesh tile synthesis via differentiable geometric parameterization, fulfilling guarantees of perfect planarity and seamlessness.

Their only unifying themes are the emphasis on generative diffusion models, differentiable end-to-end architectures, and a geometric perspective on model constraints. Each method defines state-of-the-art procedures for its own problem domain, and neither architecture nor data flow is shared.