Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Mesh Neural Cellular Automata (2311.02820v2)

Published 6 Nov 2023 in cs.CV, cs.AI, and cs.GR

Abstract: Texture modeling and synthesis are essential for enhancing the realism of virtual environments. Methods that directly synthesize textures in 3D offer distinct advantages to the UV-mapping-based methods as they can create seamless textures and align more closely with the ways textures form in nature. We propose Mesh Neural Cellular Automata (MeshNCA), a method that directly synthesizes dynamic textures on 3D meshes without requiring any UV maps. MeshNCA is a generalized type of cellular automata that can operate on a set of cells arranged on non-grid structures such as the vertices of a 3D mesh. MeshNCA accommodates multi-modal supervision and can be trained using different targets such as images, text prompts, and motion vector fields. Only trained on an Icosphere mesh, MeshNCA shows remarkable test-time generalization and can synthesize textures on unseen meshes in real time. We conduct qualitative and quantitative comparisons to demonstrate that MeshNCA outperforms other 3D texture synthesis methods in terms of generalization and producing high-quality textures. Moreover, we introduce a way of grafting trained MeshNCA instances, enabling interpolation between textures. MeshNCA allows several user interactions including texture density/orientation controls, grafting/regenerate brushes, and motion speed/direction controls. Finally, we implement the forward pass of our MeshNCA model using the WebGL shading language and showcase our trained models in an online interactive demo, which is accessible on personal computers and smartphones and is available at https://meshnca.github.io.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (62)
  1. A database and evaluation methodology for optical flow. International Journal of Computer Vision 92, 1 (2011), 1–31.
  2. Mesh2Tex: Generating Mesh Textures from Image Queries. arXiv preprint arXiv:2304.05868 (2023).
  3. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR]. Stanford University — Princeton University — Toyota Technological Institute at Chicago.
  4. Learning to predict 3d objects with an interpolation-based differentiable renderer. Advances in neural information processing systems 32 (2019).
  5. Joey de Vries. 2017. Learn OpenGL - PBR. https://learnopengl.com/PBR/Lighting1
  6. Single-image svbrdf capture with a rendering-aware deep network. ACM Transactions on Graphics (ToG) 37, 4 (2018), 1–15.
  7. Texture Generation Using a Graph Generative Adversarial Network and Differentiable Rendering. In Image and Vision Computing, Wei Qi Yan, Minh Nguyen, and Martin Stommel (Eds.). Springer Nature Switzerland, Cham, 388–401.
  8. Alexei A Efros and Thomas K Leung. 1999. Texture synthesis by non-parametric sampling. In Proceedings of the Seventh IEEE International Conference on Computer Vision, Vol. 2. IEEE, 1033–1038.
  9. Cellular texture generation. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques. 239–248.
  10. WT Freeman and Ce Liu. 2011. Markov random fields for super-resolution and texture synthesis. Advances in Markov Random Fields for Vision and Image Processing 1, 155-165 (2011), 3.
  11. StyleGAN-NADA: CLIP-guided domain adaptation of image generators. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1–13.
  12. Texture synthesis using convolutional neural networks. Advances in Neural Information Processing Systems 28 (2015).
  13. Stéphane Gobron and Norishige Chiba. 1999. 3D surface cellular automata and their applications. The Journal of Visualization and Computer Animation 10, 3 (1999), 143–158.
  14. Learning graph cellular automata. Advances in Neural Information Processing Systems 34 (2021), 20983–20994.
  15. Robin Green. 2003. Spherical harmonic lighting: The gritty details. In Archives of the game developers conference, Vol. 56. 4.
  16. MatFormer: A Generative Model for Procedural Materials. ACM Transactions on Graphics 41, 4 (2022).
  17. On demand solid texture synthesis using deep 3d networks. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 511–530.
  18. Fast example-based surface texture synthesis via discrete optimization. The Visual Computer 22 (2006), 918–925.
  19. Learning a neural 3d texture space from 2d exemplars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8356–8364.
  20. A novel framework for inverse procedural texture modeling. ACM Transactions on Graphics (ToG) 38, 6 (2019), 1–14.
  21. Generating Procedural Materials from Text or Image Prompts. In ACM SIGGRAPH 2023 Conference Proceedings. 1–11.
  22. OpenCLIP. https://doi.org/10.5281/zenodo.5143773
  23. ReLU Fields: The Little Non-Linearity That Could. In ACM SIGGRAPH 2022 Conference Proceedings (Vancouver, BC, Canada) (SIGGRAPH ’22). Association for Computing Machinery, New York, NY, USA, Article 27, 9 pages. https://doi.org/10.1145/3528233.3530707
  24. Style transfer by relaxed optimal transport and self-similarity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10051–10060.
  25. Solid Texture Synthesis from 2D Exemplars. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2007) 26, 3 (2007), 2:1–2:9.
  26. Sylvain Lefebvre and Hugues Hoppe. 2006. Appearance-Space Texture Synthesis. ACM Trans. Graph. 25, 3 (jul 2006), 541–548. https://doi.org/10.1145/1141911.1141921
  27. Tango: Text-driven photorealistic and robust 3d stylization via lighting decomposition. Advances in Neural Information Processing Systems 35 (2022), 30923–30936.
  28. X-Mesh: Towards Fast and Accurate Text-driven 3D Stylization via Dynamic Textual Guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2749–2760.
  29. Stéphane Mérillou and Djamchid Ghazanfarpour. 2008. A survey of aging and weathering phenomena in computer graphics. Computers & Graphics 32, 2 (2008), 159–174.
  30. Text2mesh: Text-driven neural stylization for meshes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13492–13502.
  31. Alexander Mordvintsev and Eyvind Niklasson. 2021. μ𝜇\muitalic_μNCA: Texture Generation with Ultra-Compact Neural Cellular Automata. arXiv preprint arXiv:2111.13545 (2021).
  32. Differentiable image parameterizations. Distill 3, 7 (2018), e12.
  33. Differentiable Programming of Reaction-Diffusion Patterns. In ALIFE 2022: The 2022 Conference on Artificial Life. MIT Press.
  34. Growing Neural Cellular Automata. Distill (2020). https://doi.org/10.23915/distill.00023 https://distill.pub/2020/growing-ca.
  35. Alessandro Muntoni and Paolo Cignoni. 2021. PyMeshLab. https://doi.org/10.5281/zenodo.4438750
  36. John von Neumann. 1966. Theory of self-reproducing automata. Math. Comp. 21 (1966), 745.
  37. Self-organising textures. Distill 6, 2 (2021), e00027–003.
  38. Texture fields: Learning texture representations in function space. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4531–4540.
  39. DyNCA: Real-time Dynamic Texture Synthesis Using Neural Cellular Automata. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 20742–20751.
  40. Joao Paulo. 2019. Free seamless PBR textures with Diffuse, Normal, Displacement, Occlusion and Roughness Maps. https://3dtextures.me/
  41. Darwyn R Peachey. 1985. Solid texturing of complex surfaces. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques. 279–286.
  42. Ken Perlin. 1985. An image synthesizer. ACM Siggraph Computer Graphics 19, 3 (1985), 287–296.
  43. Gramgan: Deep 3d texture synthesis from 2d exemplars. Advances in Neural Information Processing Systems 33 (2020), 6994–7004.
  44. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748–8763.
  45. Texture: Text-guided texturing of 3d shapes. arXiv preprint arXiv:2302.01721 (2023).
  46. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems 35 (2022), 25278–25294.
  47. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 (2021).
  48. Match: Differentiable material graphs for procedural material capture. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–15.
  49. Texturify: Generating textures on 3d shape surfaces. In European Conference on Computer Vision. Springer, 72–88.
  50. Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1409.1556
  51. Yiren Song. 2022. CLIPTexture: Text-Driven Texture Synthesis. In Proceedings of the 30th ACM International Conference on Multimedia. 5468–5476.
  52. Generating procedural 3D materials from images using neural networks. In Proceedings of the 2022 4th International Conference on Image, Video and Signal Processing. 32–40.
  53. Two-stream convolutional networks for dynamic texture synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6703–6712.
  54. AM Turing. 1952. The Chemical Basis of Morphogenesis. Philosophical Transactions of the Royal Society of London Series B 237, 641 (1952), 37–72.
  55. Greg Turk. 1991. Generating textures on arbitrary surfaces using reaction-diffusion. Acm Siggraph Computer Graphics 25, 4 (1991), 289–298.
  56. Greg Turk. 2001. Texture synthesis on surfaces. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. 347–354.
  57. Bruno Vallet and Bruno Lévy. 2008. Spectral geometry processing with manifold harmonics. In Computer Graphics Forum, Vol. 27. Wiley Online Library, 251–260.
  58. Li-Yi Wei and Marc Levoy. 2001. Texture synthesis over arbitrary manifold surfaces. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. 355–360.
  59. Texture and shape synthesis on surfaces. In Rendering Techniques 2001: Proceedings of the Eurographics Workshop in London, United Kingdom, June 25–27, 2001 12. Springer, 301–312.
  60. Design space for graph neural networks. Advances in Neural Information Processing Systems 33 (2020), 17009–17021.
  61. Rethinking texture mapping. In Computer graphics forum, Vol. 38. Wiley Online Library, 535–551.
  62. Synthesis of progressively-variant textures on arbitrary surfaces. In ACM SIGGRAPH 2003 Papers. 295–302.
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. Mesh Neural Cellular Automata
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets