Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Compositional Neural Textures (2404.12509v2)

Published 18 Apr 2024 in cs.GR, cs.AI, cs.CV, and cs.LG

Abstract: Texture plays a vital role in enhancing visual richness in both real photographs and computer-generated imagery. However, the process of editing textures often involves laborious and repetitive manual adjustments of textons, which are the recurring local patterns that characterize textures. This work introduces a fully unsupervised approach for representing textures using a compositional neural model that captures individual textons. We represent each texton as a 2D Gaussian function whose spatial support approximates its shape, and an associated feature that encodes its detailed appearance. By modeling a texture as a discrete composition of Gaussian textons, the representation offers both expressiveness and ease of editing. Textures can be edited by modifying the compositional Gaussians within the latent space, and new textures can be efficiently synthesized by feeding the modified Gaussians through a generator network in a feed-forward manner. This approach enables a wide range of applications, including transferring appearance from an image texture to another image, diversifying textures,texture interpolation, revealing/modifying texture variations, edit propagation, texture animation, and direct texton manipulation. The proposed approach contributes to advancing texture analysis, modeling, and editing techniques, and opens up new possibilities for creating visually appealing images with controllable textures.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (69)
  1. Time-varying weathering in texture space. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1–11.
  2. Learning texture manifolds with the periodic spatial GAN. arXiv preprint arXiv:1705.06566 (2017).
  3. Stephen Brooks and Neil Dodgson. 2002. Self-similarity based texture editing. ACM Transactions on Graphics (TOG) 21, 3 (2002), 653–656.
  4. Reference-based image super-resolution with deformable attention transformer. In European conference on computer vision. Springer, 325–342.
  5. End-to-end object detection with transformers. In European Conference on Computer Vision. 213–229.
  6. Design of 2d time-varying vector fields. IEEE Transactions on Visualization and Computer Graphics 18, 10 (2011), 1717–1730.
  7. General image-to-image translation with one-shot image guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 22736–22746.
  8. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 1290–1299.
  9. Revealing and Modifying Non-Local Variations in a Single Image. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) (2015).
  10. Alexei A. Efros and William T. Freeman. 2001. Image Quilting for Texture Synthesis and Transfer. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’01). Association for Computing Machinery, New York, NY, USA, 341–346. https://doi.org/10.1145/383259.383296
  11. Savi++: Towards end-to-end object-centric learning from real-world videos. arXiv preprint arXiv:2206.07764 (2022).
  12. BlobGAN: Spatially disentangled scene representations. ECCV (2022).
  13. Example-based super-resolution. IEEE Computer graphics and Applications 22, 2 (2002), 56–65.
  14. Tilegan: synthesis of large-scale non-homogeneous textures. ACM Transactions on Graphics (ToG) 38, 4 (2019), 1–11.
  15. Texture synthesis using convolutional neural networks. In International Conference on Neural Information Processing Systems. 262–270.
  16. MatFormer: A generative model for procedural materials. arXiv preprint arXiv:2207.01044 (2022).
  17. Diffusion-based Holistic Texture Rectification and Synthesis. SIGGRAPH Asia Conference Paper (2023).
  18. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
  19. Autolink: Self-supervised learning of human skeletons and object outlines by linking keypoints. arXiv preprint arXiv:2205.10636 (2022).
  20. Ganseg: Learning to segment by unsupervised hierarchical image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1225–1235.
  21. Learning a neural 3d texture space from 2d exemplars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8356–8364.
  22. Spaghetti: Editing implicit shapes through part aware generation. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1–20.
  23. Image Analogies. In SIGGRAPH ’01. Association for Computing Machinery, New York, NY, USA, 327–340. https://doi.org/10.1145/383259.383295
  24. Generating Procedural Materials from Text or Image Prompts. In ACM SIGGRAPH 2023 Conference Proceedings. 1–11.
  25. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1125–1134.
  26. Categorical Reparameterization with Gumbel-Softmax. In International Conference on Learning Representations.
  27. Object-Centric Slot Diffusion. NeurIPS (2023).
  28. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 8110–8119.
  29. Self Tuning Texture Optimization. Comput. Graph. Forum 34, 2 (May 2015), 349–359. https://doi.org/10.1111/cgf.12565
  30. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
  31. Style transfer by relaxed optimal transport and self-similarity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10051–10060.
  32. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25 (2012).
  33. Harold W Kuhn. 1955. The Hungarian method for the assignment problem. Naval research logistics quarterly 2, 1-2 (1955), 83–97.
  34. Texture Optimization for Example-Based Synthesis. ACM Trans. Graph. 24, 3 (jul 2005), 795–802. https://doi.org/10.1145/1073204.1073263
  35. Graphcut textures: Image and video synthesis using graph cuts. Acm transactions on graphics (tog) 22, 3 (2003), 277–286.
  36. Sylvain Lefebvre and Hugues Hoppe. 2005. Parallel controllable texture synthesis. In ACM SIGGRAPH 2005 Papers. 777–786.
  37. Sylvain Lefebvre and Hugues Hoppe. 2006. Appearance-space Texture Synthesis. In ACM SIGGRAPH 2006 Papers (Boston, Massachusetts) (SIGGRAPH ’06). ACM, New York, NY, USA, 541–548. https://doi.org/10.1145/1179352.1141921
  38. End-to-End Procedural Material Capture with Proxy-Free Mixed-Integer Optimization. ACM Transactions on Graphics (TOG) 42, 4 (2023), 1–15.
  39. Scraping Textures from Natural Images for Synthesis and Editing. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XV. Springer, 391–408.
  40. Near-regular texture analysis and manipulation. ACM Transactions on Graphics (TOG) 23, 3 (2004), 368–376.
  41. Texture splicing. In Computer Graphics Forum, Vol. 28. Wiley Online Library, 1907–1915.
  42. Object-centric learning with slot attention. Advances in Neural Information Processing Systems 33 (2020), 11525–11538.
  43. Dominant texture and diffusion distance manifolds. In Computer Graphics Forum, Vol. 28. Wiley Online Library, 667–676.
  44. Texture Design Using a Simplicial Complex of Morphable Textures. ACM Trans. Graph. 24, 3 (July 2005), 787–794. https://doi.org/10.1145/1073204.1073262
  45. Meta. 2023. Segment Anything. https://segment-anything.com/
  46. Image segmentation using deep learning: A survey. IEEE transactions on pattern analysis and machine intelligence 44, 7 (2021), 3523–3542.
  47. Charlie Nash and Christopher KI Williams. 2017. The shape variational autoencoder: A deep generative model of part-segmented 3D objects. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 1–12.
  48. Michael Niemeyer and Andreas Geiger. 2021. Giraffe: Representing scenes as compositional generative neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11453–11464.
  49. Neural Scene Graphs for Dynamic Scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2856–2865.
  50. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2337–2346.
  51. Swapping autoencoder for deep image manipulation. Advances in Neural Information Processing Systems 33 (2020), 7198–7211.
  52. Layered Shape Synthesis: Automatic Generation of Control Maps for Non-Stationary Textures. ACM Trans. Graph. 28, 5 (Dec. 2009), 1–9. https://doi.org/10.1145/1618452.1618453
  53. Object scene representation transformer. Advances in Neural Information Processing Systems 35 (2022), 9512–9524.
  54. Match: Differentiable material graphs for procedural material capture. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–15.
  55. Splicing vit features for semantic appearance transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10748–10757.
  56. State of the art in example-based texture synthesis. In Eurographics 2009, State of the Art Report, EG-STAR. Eurographics Association, 93–117.
  57. Deep texture manifold for ground terrain recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 558–567.
  58. Superpixel segmentation with fully convolutional networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 13964–13973.
  59. Learning texture transformer network for image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 5791–5800.
  60. Texture Mixer: A Network for Controllable Synthesis and Interpolation of Texture. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  61. Deep structure-revealed network for texture recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11010–11019.
  62. Deep ten: Texture encoding network. In Proceedings of the IEEE conference on computer vision and pattern recognition. 708–717.
  63. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR.
  64. Unsupervised discovery of object landmarks as structural representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2694–2703.
  65. Image super-resolution by neural texture transfer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 7982–7991.
  66. Deep learning-based human pose estimation: A survey. Comput. Surveys 56, 1 (2023), 1–37.
  67. Crossnet: An end-to-end reference-based super resolution network using cross-scale warping. In Proceedings of the European conference on computer vision (ECCV). 88–104.
  68. Neural Texture Synthesis With Guided Correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18095–18104.
  69. Non-stationary texture synthesis by adversarial expansion. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–13.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube