Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketches (2104.00482v2)

Abstract: Reconstructing 3D shape from 2D sketches has long been an open problem because the sketches only provide very sparse and ambiguous information. In this paper, we use an encoder/decoder architecture for the sketch to mesh translation. When integrated into a user interface that provides camera parameters for the sketches, this enables us to leverage its latent parametrization to represent and refine a 3D mesh so that its projections match the external contours outlined in the sketch. We will show that this approach is easy to deploy, robust to style changes, and effective. Furthermore, it can be used for shape refinement given only single pen strokes. We compare our approach to state-of-the-art methods on sketches -- both hand-drawn and synthesized -- and demonstrate that we outperform them.

• The paper introduces an encoder/decoder model that refines latent shape representations to accurately reconstruct 3D shapes from sparse sketches.
• It employs differentiable rasterization and Chamfer distance optimization to align mesh projections with sketch contours.
• The method demonstrates superior reconstruction quality and robustness across diverse sketch styles, enhancing CAD and interactive design workflows.

Sketch2Mesh: A Novel Approach to Reconstructing and Editing 3D Shapes from Sketches

The paper, "Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketches," addresses the enduring challenge of transforming sparse, 2D sketches into coherent 3D shapes. Utilizing an encoder/decoder architecture, it advances the state of sketch-based 3D reconstruction by leveraging latent parametrization to refine mesh projections, thus aligning them with the external contours delineated in sketches. The methodology not only enhances robustness to stylistic variances in sketches—both synthetic and hand-drawn—but also enables shape refinement from singular pen strokes.

Overview and Contributions

Sketch2Mesh distinguishes itself from extant methodologies that predominantly generate coarse volumetric grids or require cumbersome multi-view sketches for effective mesh reconstruction. The approach circumvents the limitations of local image-plane feature pooling, which impedes single-view reconstruction from sparse sketch data, by nurturing a compact latent shape representation suitable for downstream optimization.

The paper introduces two pivotal mechanisms for aligning 3D meshes with sketches:

1. Image Translation and Differentiable Rasterization: Employing a state-of-the-art image translation model to convert sketches to foreground/background images, followed by its alignment through differentiable rasterization.
2. Chamfer Distance Optimization: Innovatively refining mesh projections by directly minimizing the 2D Chamfer distance between sketch contours and those of projected meshes.

Numerical Comparisons and Results

Empirical validation evidenced Sketch2Mesh’s superior performance vis-à-vis contemporary techniques such as Pix2Vox, MeshRCNN, and DISN. The approach demonstrated resilience in producing semantically accurate reconstructions across varying sketch styles, including Suggestive sketches, SketchFD renderings, and real hand-drawn illustrations. Quantitatively, it achieved lower Chamfer distance metrics and enhanced normal consistency, particularly noticeable in scenarios involving unseen sketch styles.

Practical Implications and Prospective Developments

The application potential of Sketch2Mesh is broad, encompassing domains such as industrial design and CAD systems, where it offers a sophisticated tool for digitizing existing model archives and crafting new designs from intuitive 2D inputs. In this context, the ability to edit and refine meshes interactively—by manipulating partial sketches—represents a significant advancement over previous rigid approaches.

Future research directions may encompass integrating deeper shape priors to seamlessly blend sketch-based refinements with inherent geometric constraints—such as symmetry or functional specifications. Additionally, extending the methodology to incorporate internal contour lines could improve reconstruction fidelity, yet necessitates mechanisms to maintain robustness across diverse sketching styles.

In summary, Sketch2Mesh presents an efficient and adaptable solution to 3D reconstruction from sketches, suggesting promising pathways for enhancing CAD workflows and interactive design paradigms. The paper lays a foundation for subsequent explorations in integrating user-friendly, yet technically rigorous reconstruction tools into practical applications in design and engineering industries.