Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

As-Plausible-As-Possible: Plausibility-Aware Mesh Deformation Using 2D Diffusion Priors (2311.16739v2)

Published 28 Nov 2023 in cs.CV and cs.GR

Abstract: We present As-Plausible-as-Possible (APAP) mesh deformation technique that leverages 2D diffusion priors to preserve the plausibility of a mesh under user-controlled deformation. Our framework uses per-face Jacobians to represent mesh deformations, where mesh vertex coordinates are computed via a differentiable Poisson Solve. The deformed mesh is rendered, and the resulting 2D image is used in the Score Distillation Sampling (SDS) process, which enables extracting meaningful plausibility priors from a pretrained 2D diffusion model. To better preserve the identity of the edited mesh, we fine-tune our 2D diffusion model with LoRA. Gradients extracted by SDS and a user-prescribed handle displacement are then backpropagated to the per-face Jacobians, and we use iterative gradient descent to compute the final deformation that balances between the user edit and the output plausibility. We evaluate our method with 2D and 3D meshes and demonstrate qualitative and quantitative improvements when using plausibility priors over geometry-preservation or distortion-minimization priors used by previous techniques. Our project page is at: https://as-plausible-aspossible.github.io/

Definition Search Book Streamline Icon: https://streamlinehq.com
References (69)
  1. Luma AI. Genie.
  2. Neural Jacobian Fields: Learning Intrinsic Mappings of Arbitrary Meshes. ACM TOG, 2022.
  3. MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation. In ICML, 2023.
  4. Automatic Rigging and Animation of 3D Characters. ACM TOG, 2007.
  5. MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing. In ICCV, 2023.
  6. ShapeNet: An Information-Rich 3D Model Repository. arXiv preprint arXiv:1512.03012, 2015.
  7. Objaverse-XL: A Universe of 10M+ 3D Objects. In NeurIPS, 2023.
  8. Objaverse: A Universe of Annotated 3D Objects. In CVPR, 2023.
  9. Mesh editing with poisson-based gradient field manipulation. ACM Trans. Graph., 23(3), 2004.
  10. Hugging Face. Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch.
  11. Hugging Face. DreamBooth fine-tuning with LoRA.
  12. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion. In ICLR, 2023.
  13. TextDeformer: Geometry Manipulation using Text Guidance. ACM TOG, 2023.
  14. GIQA: Generated Image Quality Assessment. In ECCV, 2020.
  15. Delta Denoising Score. In ICCV, 2023.
  16. Prompt-to-Prompt Image Editing with Cross-Attention Control. In ICLR, 2023.
  17. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR, 2022.
  18. As-Rigid-as-Possible Shape Manipulation. ACM TOG, 2005.
  19. Word-As-Image for Semantic Typography. ACM TOG, 2023.
  20. Bounded Biharmonic Weights for Real-Time Deformation. ACM TOG, 2011.
  21. libigl: A simple C++ geometry processing library, 2018.
  22. VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Models. In CVPR, 2023.
  23. KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control. In CVPR, 2020.
  24. Jong Chul Ye Jangho Park, Gihyun Kwon. ED-NeRF: Efficient Text-Guided Editing of 3D Scene using Latent Space NeRF. arXiv, 2023.
  25. Harmonic Coordinates for Character Articulation. ACM TOG, 2007.
  26. Mean Value Coordinates for Closed Triangular Meshes. ACM TOG, 2005.
  27. CLIP-Mesh: Generating Textured Meshes from Text Using Pretrained Image-Text Models. SIGGRAPH ASIA, 2022.
  28. OptCtrlPoints: Finding the Optimal Control Points for Biharmonic 3D Shape Deformation. Computer Graphics Forum, 2023.
  29. Collaborative Score Distillation for Consistent Visual Synthesis. In NeurIPS, 2023.
  30. Adam: A method for stochastic optimization. In ICLR, 2015.
  31. Segment Anything. In ICCV, 2023.
  32. SALAD: Part-Level Latent Diffusion for 3D Shape Generation and Manipulation. In ICCV, 2023.
  33. Modular Primitives for High-Performance Differentiable Rendering. ACM TOG, 2020.
  34. SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions. In NeurIPS, 2023.
  35. Green Coordinates. ACM TOG, 2008.
  36. Differential Coordinates for Interactive Mesh Editing. In Proceedings of Shape Modeling International, pages 181–190, 2004.
  37. Linear Rotation-Invariant Coordinates for Meshes. ACM TOG, 2005.
  38. DeepMetaHandles: Learning Deformation Meta-Handles of 3D Meshes with Biharmonic Coordinates. In CVPR, 2021.
  39. SyncDreamer: Learning to Generate Multiview-consistent Images from a Single-view Image. arXiv, 2023.
  40. Text2Mesh: Text-Driven Neural Stylization for Meshes. In CVPR, 2022.
  41. Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold. ACM TOG, 2023.
  42. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. arXiv, 2023.
  43. DreamFusion: Text-to-3D using 2D Diffusion. In ICLR, 2023.
  44. Learning Transferable Visual Models from Natural Language Supervision. In ICML, 2021.
  45. DreamBooth3D: Subject-Driven Text-to-3D Generation. In ICCV, 2023.
  46. Daniel Ritchie. Rudimentary framework for running two-alternative forced choice (2afc) perceptual studies on mechanical turk.
  47. High-Resolution Image Synthesis with Latent Diffusion Models. In CVPR, 2022.
  48. DreamBooth: Fine Tuning Text-to-image Diffusion Models for Subject-Driven Generation. In CVPR, 2023.
  49. LAION-5B: An open large-scale dataset for training next generation image-text models. In NeurIPS, 2022.
  50. Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis. In NeurIPS, 2021.
  51. MVDream: Multi-view Diffusion for 3D Generation. arXiv, 2023.
  52. DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing. arXiv, 2023.
  53. 3D Neural Field Generation using Triplane Diffusion. In CVPR, 2023.
  54. As-Rigid-As-Possible Surface Modeling. Proceedings of EUROGRAPHICS/ACM SIGGRAPH Symposium on Geometry Processing, pages 109–116, 2007.
  55. Laplacian Surface Editing. Proceedings of the EUROGRAPHICS/ACM SIGGRAPH Symposium on Geometry Processing, pages 179–188, 2004.
  56. Neural Shape Deformation Priors. In NeurIPS, 2022.
  57. DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation. arXiv, 2023.
  58. Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation. In CVPR, 2023.
  59. Linear Subspace Design for Real-Time Shape Deformation. ACM TOG, 2015.
  60. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In NeurIPS, 2023.
  61. Complex barycentric coordinates with applications to planar shape deformation. Computer Graphics Forum, 2009.
  62. Context-Aware Skeletal Shape Deformation. Computer Graphics Forum, 2007.
  63. OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation. In CVPR, 2023.
  64. RigNet: Neural Rigging for Articulated Characters. ACM TOG, 2020.
  65. Morig: Motion-Aware Rigging of Character Meshes from Point Clouds. In SIGGRAPH ASIA, 2022.
  66. Neural Cages for Detail-Preserving 3D Deformations. In CVPR, 2020.
  67. Adding Conditional Control to Text-to-Image Diffusion Models. In ICCV, 2023.
  68. Real-World Image Variation by Aligning Diffusion Inversion Chain. In NeurIPS, 2023.
  69. DreamEditor: Text-Driven 3D Scene Editing with Neural Fields. ACM TOG, 2023.
Citations (10)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets