MeshUp: Multi-Target Mesh Deformation via Blended Score Distillation (2408.14899v3)

Abstract: We propose MeshUp, a technique that deforms a 3D mesh towards multiple target concepts, and intuitively controls the region where each concept is expressed. Conveniently, the concepts can be defined as either text queries, e.g., "a dog" and "a turtle," or inspirational images, and the local regions can be selected as any number of vertices on the mesh. We can effectively control the influence of the concepts and mix them together using a novel score distillation approach, referred to as the Blended Score Distillation (BSD). BSD operates on each attention layer of the denoising U-Net of a diffusion model as it extracts and injects the per-objective activations into a unified denoising pipeline from which the deformation gradients are calculated. To localize the expression of these activations, we create a probabilistic Region of Interest (ROI) map on the surface of the mesh, and turn it into 3D-consistent masks that we use to control the expression of these activations. We demonstrate the effectiveness of BSD empirically and show that it can deform various meshes towards multiple objectives. Our project page is at https://threedle.github.io/MeshUp.

• The paper introduces MeshUp with Blended Score Distillation to enable multi-target 3D mesh deformations with precise regional control.
• It combines text, image, and mesh inputs to blend feature activations, achieving smooth transitions and preserving detailed textures.
• Empirical results demonstrate MeshUp's robust performance in complex deformations, paving the way for advanced AI-driven 3D modeling.

Overview of MeshUp: Multi-Target Mesh Deformation via Blended Score Distillation

The paper introduces MeshUp, an innovative approach to deform 3D meshes towards multiple target concepts while enabling intuitive regional control of these deformations. The significance of MeshUp lies in its ability to handle complex mesh deformation tasks using high-level controls, compatible with non-experts, thus broadening accessibility to 3D content creation. By leveraging text prompts, images, or mesh inputs, MeshUp allows for a user-friendly experience that fosters creative workflows in 3D shape generation, surpassing current methods lacking intuitive interface or multi-target capabilities.

Methodological Contributions

MeshUp's core methodology involves the Blended Score Distillation (BSD) technique, which operates with a denoising U-Net in a diffusion model to extract and blend feature activations. BSD injects activations corresponding to targeted objectives into a unified channel to compute deformation gradients. This novel blending of score distillation processes differentiates MeshUp, offering users the ability to specify deformation influence through concept weight assignments and localized vertex control.

The foundation of MeshUp’s deformation process is the probabilistic Region of Interest (ROI) mapping, which informs where and how each concept should manifest on the mesh's surface. Through employing score distillation sampling within the diffusion model's denoising pipeline, the paper presents a mechanism for probabilistically controlling feature expression via localized constraints. This approach enhances precision in blending multiple target concepts, leveraging both 3D mesh and viewpoint-consistent masks to achieve seamless transitional deformations and smooth global transformations.

Results and Implications

The empirical results demonstrate MeshUp’s capability to successfully deform complex meshes toward multiple objectives, even facilitating intricate concept mixing (e.g., blending animal features such as a bear's head with a frog's legs). MeshUp excels in preserving mesh attributes like tessellations and textures, crucial for generating production-ready assets. Furthermore, MeshUp's application of Jacobian-based deformation, contrary to traditional vertex optimization, aids in mitigating artifact production, as substantiated by the quantitative evaluation.

MeshUp’s implications extend broadly within the computer graphics domain, as its methodology champions greater artistic control, allowing researchers and developers to integrate disparate 3D concept paradigms seamlessly. Additionally, this approach fosters future developments in AI-driven 3D modeling by encouraging exploration of more fluid inter-concept blending facilitated by AI, fortifying the linkage between artistry and technology.

Reflections and Future Directions

MeshUp's ability to transcend prior limitations in mesh deformation paradigms opens pathways for refining volumetric and surface deformation techniques, achieving robustness in less-constrained 3D environments. This work provocatively indicates that by further enhancing the model’s ability to handle topological changes, its influence could extend to applications in virtual reality and immersive graphics designing, with implications for interactive AI-driven creative processes.

In summary, the paper convincingly delineates MeshUp as a sophisticated tool that extends mesh deformation capabilities through multi-target score distillation. Its contributions present a granular understanding of deformation control, heralding advancements in mesh manipulation, ultimately promising scalability across diverse graphical applications within and beyond computational artistic endeavors.