Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

DragD3D: Realistic Mesh Editing with Rigidity Control Driven by 2D Diffusion Priors (2310.04561v2)

Published 6 Oct 2023 in cs.GR and cs.LG

Abstract: Direct mesh editing and deformation are key components in the geometric modeling and animation pipeline. Mesh editing methods are typically framed as optimization problems combining user-specified vertex constraints with a regularizer that determines the position of the rest of the vertices. The choice of the regularizer is key to the realism and authenticity of the final result. Physics and geometry-based regularizers are not aware of the global context and semantics of the object, and the more recent deep learning priors are limited to a specific class of 3D object deformations. Our main contribution is a vertex-based mesh editing method called DragD3D based on (1) a novel optimization formulation that decouples the rotation and stretch components of the deformation and combines a 3D geometric regularizer with (2) the recently introduced DDS loss which scores the faithfulness of the rendered 2D image to one from a diffusion model. Thus, our deformation method achieves globally realistic shape deformation which is not restricted to any class of objects. Our new formulation optimizes directly the transformation of the neural Jacobian field explicitly separating the rotational and stretching components. The objective function of the optimization combines the approximate gradients of DDS and the gradients from the geometric loss to satisfy the vertex constraints. Additional user control over desired global shape deformation is made possible by allowing explicit per-triangle deformation control as well as explicit separation of rotational and stretching components of the deformation. We show that our deformations can be controlled to yield realistic shape deformations that are aware of the global context of the objects, and provide better results than just using geometric regularizers.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. Neural jacobian fields: Learning intrinsic mappings of arbitrary meshes. arXiv preprint arXiv:2205.02904 (2022).
  2. Re-imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate janus problem and beyond. arXiv preprint arXiv:2304.04968 (2023).
  3. Botsch M., Sorkine O.: On linear variational surface deformation methods. IEEE transactions on visualization and computer graphics 14, 1 (2007), 213–230.
  4. A simple geometric model for elastic deformations. ACM transactions on graphics (TOG) 29, 4 (2010), 1–6.
  5. Local deformation for interactive shape editing. arXiv preprint arXiv:2306.06550 (2023).
  6. Deformed implicit field: Modeling 3d shapes with learned dense correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021), pp. 10286–10296.
  7. Textdeformer: Geometry manipulation using text guidance. In ACM SIGGRAPH 2023 Conference Proceedings (2023), pp. 1–11.
  8. iwires: An analyze-and-edit approach to shape manipulation. In ACM SIGGRAPH 2009 papers. 2009, pp. 1–10.
  9. Get3d: A generative model of high quality 3d textured shapes learned from images. In Advances In Neural Information Processing Systems (2022).
  10. Delta denoising score. arXiv preprint arXiv:2304.07090 (2023).
  11. Local 3d editing via 3d distillation of clip knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023), pp. 12674–12684.
  12. Ho J., Salimans T.: Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022).
  13. libigl: A simple C++ geometry processing library, 2018. https://libigl.github.io/.
  14. Keypointdeformer: Unsupervised 3d keypoint discovery for shape control. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021), pp. 12783–12792.
  15. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2020), pp. 8110–8119.
  16. Non-homogeneous resizing of complex models. ACM Transactions on Graphics (TOG) 27, 5 (2008), 1–9.
  17. Scalable 3d captioning with pretrained models. arXiv preprint arXiv:2306.07279 (2023).
  18. Editing conditional radiance fields. In Proceedings of the IEEE/CVF international conference on computer vision (2021), pp. 5773–5783.
  19. Discrete differential-geometry operators for triangulated 2-manifolds. In Visualization and mathematics III. Springer, 2003, pp. 35–57.
  20. Clip-mesh: Generating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia 2022 conference papers (2022), pp. 1–8.
  21. Sked: Sketch-guided text-based 3d editing. arXiv preprint arXiv:2303.10735 (2023).
  22. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM 65, 1 (2021), 99–106.
  23. Dragondiffusion: Enabling drag-style manipulation on diffusion models. arXiv preprint arXiv:2307.02421 (2023).
  24. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2019), pp. 165–174.
  25. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 (2022).
  26. Interactive and linear material aware deformations. International Journal of Shape modeling 13, 01 (2007), 73–100.
  27. Drag your gan: Interactive point-based manipulation on the generative image manifold. In ACM SIGGRAPH 2023 Conference Proceedings (2023), pp. 1–11.
  28. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2022), pp. 10684–10695.
  29. Learning transferable visual models from natural language supervision. In International conference on machine learning (2021), PMLR, pp. 8748–8763.
  30. Texture: Text-guided texturing of 3d shapes. arXiv preprint arXiv:2302.01721 (2023).
  31. Sorkine O., Alexa M.: As-rigid-as-possible surface modeling. In Symposium on Geometry processing (2007), vol. 4, pp. 109–116.
  32. Clip-sculptor: Zero-shot generation of high-fidelity and diverse shapes from natural language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023), pp. 18339–18348.
  33. Neuralmls: Geometry-aware control point deformation. arXiv preprint arXiv:2201.01873 (2022).
  34. Dragdiffusion: Harnessing diffusion models for interactive point-based image editing. arXiv preprint arXiv:2306.14435 (2023).
  35. Mesh-based inverse kinematics. ACM transactions on graphics (TOG) 24, 3 (2005), 488–495.
  36. Textmesh: Generation of realistic 3d meshes from text prompts. arXiv preprint arXiv:2304.12439 (2023).
  37. Neural shape deformation priors. Advances in Neural Information Processing Systems 35 (2022), 17117–17132.
  38. Umeyama S.: Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions on Pattern Analysis & Machine Intelligence 13, 04 (1991), 376–380.
  39. 3dn: 3d deformation network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019), pp. 1038–1046.
  40. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models. arXiv preprint arXiv:2208.06677 (2022).
  41. Neural cages for detail-preserving 3d deformations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020), pp. 75–83.
  42. Geometry processing with neural fields. Advances in Neural Information Processing Systems 34 (2021), 22483–22497.
  43. A revisit of shape editing techniques: From the geometric to the neural viewpoint. Journal of Computer Science and Technology 36, 3 (2021), 520–554.
  44. Yumer M. E., Mitra N. J.: Learning semantic deformation flows with 3d convolutional networks. In European Conference on Computer Vision (2016), Springer, pp. 294–311.
  45. Nerf-editing: geometry editing of neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022), pp. 18353–18364.
  46. Interactive nerf geometry editing with shape priors. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).
  47. Learning object-compositional neural radiance field for editable scene rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (2021), pp. 13779–13788.
  48. Zhou Q., Jacobson A.: Thingi10k: A dataset of 10,000 3d-printing models. arXiv preprint arXiv:1605.04797 (2016).
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now