SketchMetaFace: A Learning-based Sketching Interface for High-fidelity 3D Character Face Modeling (2307.00804v2)

Abstract: Modeling 3D avatars benefits various application scenarios such as AR/VR, gaming, and filming. Character faces contribute significant diversity and vividity as a vital component of avatars. However, building 3D character face models usually requires a heavy workload with commercial tools, even for experienced artists. Various existing sketch-based tools fail to support amateurs in modeling diverse facial shapes and rich geometric details. In this paper, we present SketchMetaFace - a sketching system targeting amateur users to model high-fidelity 3D faces in minutes. We carefully design both the user interface and the underlying algorithm. First, curvature-aware strokes are adopted to better support the controllability of carving facial details. Second, considering the key problem of mapping a 2D sketch map to a 3D model, we develop a novel learning-based method termed "Implicit and Depth Guided Mesh Modeling" (IDGMM). It fuses the advantages of mesh, implicit, and depth representations to achieve high-quality results with high efficiency. In addition, to further support usability, we present a coarse-to-fine 2D sketching interface design and a data-driven stroke suggestion tool. User studies demonstrate the superiority of our system over existing modeling tools in terms of the ease to use and visual quality of results. Experimental analyses also show that IDGMM reaches a better trade-off between accuracy and efficiency. SketchMetaFace is available at https://zhongjinluo.github.io/SketchMetaFace/.

• The paper introduces SketchMetaFace, an innovative sketching interface that simplifies the creation of detailed 3D character faces for non-expert users.
• The methodology employs a coarse-to-fine user interface paired with a novel IDGMM approach that fuses implicit field information and depth cues for accurate mesh deformation.
• Experimental results demonstrate significant improvements in modeling speed and quality over traditional tools, broadening accessibility in 3D design.

Insightful Overview of SketchMetaFace

"SketchMetaFace: A Learning-based Sketching Interface for High-fidelity 3D Character Face Modeling" outlines an innovative approach to simplifying the complex task of 3D facial modeling for non-expert users. The authors, Zhongjin Luo et al., present a sketch-based interface called SketchMetaFace, which allows amateur users to create high-fidelity, detailed 3D character faces efficiently and intuitively. The impetus for this system comes from the recognition that 3D modeling, particularly of intricate facial features, typically requires considerable expertise and time using traditional tools such as ZBrush or MAYA.

System Design and Methodology

The paper distinguishes SketchMetaFace from existing sketch-based 3D modeling tools through two primary innovations: a novel user interface and the underlying algorithmic framework.

• User Interface: The SketchMetaFace interface employs a coarse-to-fine modeling scheme. Users start by sketching the overall shape of the face and attachments (e.g., ears) on a 2D canvas. Part-separated modeling enables easy manipulation of facial features at this stage. Subsequently, users can enhance facial details using curvature-aware strokes—an advancement that provides precise control over geometric features such as ridges and valleys, addressing a common shortcoming in existing sketch-based systems.
• Algorithmic Advance - IDGMM: A key methodological contribution is the Implicit and Depth Guided Mesh Modeling (IDGMM) approach. This leverages implicit field, mesh, and depth representations to map 2D sketches to detailed 3D models efficiently. The IDGMM process involves implicit-guided mesh updating, which uses learned Signed Distance Functions (SDF) for robust mesh deformation, and depth-guided refinement, which utilizes depth maps for transferring fine surface details to the mesh. This hybrid approach achieves a superior balance of accuracy and efficiency compared to existing methods like PIFu and DeepSDF.

Experimental Evaluation and Implications

The paper includes extensive quantitative and qualitative analyses demonstrating that SketchMetaFace significantly outperforms prior systems such as DeepSketch2Face and SimpModeling. User studies underscore its usability, with non-expert participants creating high-quality 3D faces in minutes rather than the extensive durations traditionally required. The SketchMetaFace system thus represents a substantial advancement in accessibility to 3D facial modeling.

Several implications for future developments in AI and 3D modeling arise from this research:

1. Enhancing User Interaction in AI Systems: The integration of intuitive, sketch-based interfaces can broaden user engagement in traditionally complex domains. This suggests potential expansions of similar systems to other areas of 3D modeling, such as full-body avatars or mechanical parts.
2. Leveraging Hybrid Representations for Efficiency: The coupling of different 3D representations (mesh, depth, and implicit fields) exhibits how multiple data forms can be harnessed synergistically to enhance both computational efficiency and output fidelity. This can inspire further research into hybrid methods for various AI applications.
3. Data-Driven User Assistance: Incorporating data-driven assistance—such as the stroke suggestion tool—demonstrates the power of machine learning to simplify user interactions. As machine learning models evolve, we can expect increased integration of automated guidance in user-driven design tasks.

Overall, this research not only advances the field of sketch-based 3D modeling but also opens new avenues for accessible and efficient design paradigms in AI-enhanced creative applications.