Papers
Topics
Authors
Recent
Search
2000 character limit reached

TexTile: A Differentiable Metric for Texture Tileability

Published 19 Mar 2024 in cs.CV, cs.AI, cs.GR, and cs.LG | (2403.12961v1)

Abstract: We introduce TexTile, a novel differentiable metric to quantify the degree upon which a texture image can be concatenated with itself without introducing repeating artifacts (i.e., the tileability). Existing methods for tileable texture synthesis focus on general texture quality, but lack explicit analysis of the intrinsic repeatability properties of a texture. In contrast, our TexTile metric effectively evaluates the tileable properties of a texture, opening the door to more informed synthesis and analysis of tileable textures. Under the hood, TexTile is formulated as a binary classifier carefully built from a large dataset of textures of different styles, semantics, regularities, and human annotations.Key to our method is a set of architectural modifications to baseline pre-train image classifiers to overcome their shortcomings at measuring tileability, along with a custom data augmentation and training regime aimed at increasing robustness and accuracy. We demonstrate that TexTile can be plugged into different state-of-the-art texture synthesis methods, including diffusion-based strategies, and generate tileable textures while keeping or even improving the overall texture quality. Furthermore, we show that TexTile can objectively evaluate any tileable texture synthesis method, whereas the current mix of existing metrics produces uncorrelated scores which heavily hinders progress in the field.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (75)
  1. Generative escher meshes. arXiv preprint arXiv:2309.14564, 2023.
  2. A survey of exemplar-based texture synthesis methods. Computer Vision and Image Understanding, 172:12–24, 2018.
  3. Understanding and simplifying perceptual distances. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12226–12235, 2021.
  4. Generalized thin-plate spline warps. International Journal of Computer Vision, 88:85–110, 2010.
  5. Learning texture manifolds with the periodic spatial gan. pages 469–477, 2017.
  6. Demystifying mmd gans. In International Conference on Learning Representations, 2018.
  7. Texfusion: Synthesizing 3d textures with text-guided image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4169–4181, 2023.
  8. SMPLitex: A Generative Model and Dataset for 3D Human Texture Estimation from Single Image. In British Machine Vision Conference (BMVC), 2023.
  9. Text2Tex: Text-driven Texture Synthesis via Diffusion Models. 2023.
  10. Describing textures in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2014.
  11. Procedural stochastic textures by tiling and blending. GPU Zen, 2, 2019.
  12. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 248–255. Ieee, 2009.
  13. Single-image svbrdf capture with a rendering-aware deep network. ACM Transactions on Graphics (TOG), 37(4):1–15, 2018.
  14. Image quality assessment: Unifying structure and texture similarity. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(5):2567–2581, 2020.
  15. Timothy Dozat. Incorporating nesterov momentum into adam. 2016.
  16. The kth-tips database. 2004.
  17. Axiom-based grad-cam: Towards accurate visualization and explanation of cnns. arXiv preprint arXiv:2008.02312, 2020.
  18. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. arXiv preprint arXiv:2306.09344, 2023.
  19. Gabor noise by example. ACM Transactions on Graphics (TOG), 31(4):73:1–73:9, 2012.
  20. A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576, 2015.
  21. U-attention to textures: hierarchical hourglass vision transformer for universal texture synthesis. In Proceedings of the ACM SIGGRAPH European Conference on Visual Media Production, pages 1–10, 2022.
  22. Materialgan: reflectance capture using a generative svbrdf model. ACM Transactions on Graphics (TOG), 39(6):1–13, 2020.
  23. A survey on vision transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):87–110, 2022.
  24. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 770–778, 2016.
  25. A sliced wasserstein loss for neural texture synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9412–9420, 2021.
  26. The origins and prevalence of texture bias in convolutional neural networks. Advances in Neural Information Processing Systems, 33:19000–19015, 2020.
  27. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
  28. A novel framework for inverse procedural texture modeling. ACM Transactions on Graphics (TOG), 38(6):1–14, 2019.
  29. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1125–1134, 2017.
  30. Radon transform orientation estimation for rotation invariant texture analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(6):1004–1008, 2005.
  31. Geometry score: A method for comparing generative adversarial networks. In International Conference on Machine Learning, pages 2621–2629. PMLR, 2018.
  32. Gustaf Kylberg. The kylberg texture dataset v. 1.0. External report (Blue series) 35, Centre for Image Analysis, Swedish University of Agricultural Sciences and Uppsala University, Uppsala, Sweden, 2011.
  33. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
  34. A sparse texture representation using local affine regions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(8):1265–1278, 2005.
  35. Repeated pattern detection using cnn activations. In IEEE Winter Conference on Applications of Computer Vision, pages 47–55. IEEE, 2017.
  36. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2475–2484, 2020.
  37. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908.03265, 2019.
  38. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12009–12019, 2022a.
  39. A convnet for the 2020s. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11976–11986, 2022b.
  40. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
  41. The kth-tips 2 database. 2006.
  42. Mixed precision training. In International Conference on Learning Representations, 2018.
  43. No-reference image quality assessment in the spatial domain. IEEE Transactions on Image Processing, 21(12):4695–4708, 2012.
  44. A two-step framework for constructing blind image quality indices. IEEE Signal Processing Letters, 17(5):513–516, 2010.
  45. Texture stationarization: Turning photos into tileable textures. In Computer Graphics Forum, pages 177–188. Wiley Online Library, 2017.
  46. Intriguing properties of vision transformers. Advances in Neural Information Processing Systems, 34:23296–23308, 2021.
  47. Sinfusion: Training diffusion models on a single image or video. In International Conference on Machine Learning. PMLR, 2023.
  48. Self-organising textures. Distill, 2021. https://distill.pub/selforg/2021/textures.
  49. Vcrnet: Visual compensation restoration network for no-reference image quality assessment. IEEE Transactions on Image Processing, 31:1613–1627, 2022.
  50. Automatic differentiation in pytorch. 2017.
  51. Pieapp: Perceptual image-error assessment through pairwise preference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1808–1817, 2018.
  52. Kornia: an open source differentiable computer vision library for pytorch. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3674–3683, 2020.
  53. Seamlessgan: Self-supervised synthesis of tileable texture maps. IEEE Transactions on Visualization and Computer Graphics, 2022.
  54. Automatic extraction and synthesis of regular repeatable patterns. Computers & Graphics, 83:33–41, 2019.
  55. Umat: Uncertainty-aware single image high resolution material capture. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
  56. SinGAN: Learning a Generative Model from a Single Natural Image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
  57. Improved techniques for training gans. Advances in Neural Information Processing Systems, 29, 2016.
  58. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations. Computational and Biological Learning Society, 2015.
  59. Blindly assess image quality in the wild guided by a self-adaptive hyper network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3667–3676, 2020.
  60. Visual perception of texture regularity: Conjoint measurements and a wavelet response-distribution model. PLoS Computational Biology, 17(10):e1008802, 2021.
  61. The shape of data: Intrinsic distance for data distributions. In International Conference on Learning Representations, 2019.
  62. Controlmat: A controlled generative approach to material capture. arXiv preprint arXiv:2309.01700, 2023.
  63. Plan2scene: Converting floorplans to 3d scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10733–10742, 2021.
  64. Exploring clip for assessing the look and feel of images. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2555–2563, 2023.
  65. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020.
  66. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  67. Learning in a single domain for non-stationary multi-texture synthesis. arXiv preprint arXiv:2305.06200, 2023.
  68. Maniqa: Multi-dimension attention network for no-reference image quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1191–1200, 2022.
  69. Fsim: A feature similarity index for image quality assessment. IEEE Transactions on Image Processing, 20(8):2378–2386, 2011.
  70. Lookahead optimizer: k steps forward, 1 step back. Advances in neural information processing systems, 32, 2019.
  71. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 586–595, 2018.
  72. Adversarial single-image svbrdf estimation with hybrid training. Computer Graphics Forum, 2021.
  73. Look-ahead training with learned reflectance loss for single-image svbrdf estimation. ACM Transactions on Graphics (TOG), 41(6), 2022.
  74. Tilegen: Tileable, controllable material generation and capture. In SIGGRAPH Asia 2022 Conference Papers, pages 1–9, 2022.
  75. Non-stationary texture synthesis by adversarial expansion. ACM Transactions on Graphics (TOG), 37(4):1–13, 2018.
Citations (1)
View on Semantic Scholar

Summary

  • The paper presents TexTile, a differentiable metric that quantifies texture tileability through an innovative attention-enhanced convolutional classifier.
  • It employs advanced data augmentation and a custom training regime to robustly detect local discontinuities and global patterns in diverse textures.
  • TexTile enhances existing texture synthesis methods by acting as an integrated loss term, enabling the production of seamless, high-quality tileable textures.

Introduction to TexTile: Assessing Texture Tileability with a Novel Differentiable Metric

Overview

The paper introduces TexTile, a novel differentiable metric designed to quantitatively assess the tileability of textures. Tileability refers to the ability of a texture to be seamlessly repeated without noticeable artifacts. This property is crucial in various applications, notably in graphics and material design, where textures often need to be applied to larger surfaces without visible repetitions or seams. The authors identified a gap in existing texture synthesis evaluation methods, which primarily focus on general texture quality, lacking a mechanism for assessing the intrinsic repeatability properties. TexTile addresses this gap by providing a metric that not only measures tileability effectively but also integrates seamlessly with state-of-the-art texture synthesis methods to ensure or even enhance texture quality.

Key Contributions

  • Differentiable Metric for Texture Tileability: TexTile introduces a data-driven approach to quantify texture repeatability, integrating easily as an additional loss term in learning-based texture synthesis algorithms.
  • Custom Architecture and Training Regime: The development of an attention-enhanced convolutional classifier adapted specifically for evaluating tileability. This includes a suite of custom data augmentation strategies aiming at enhancing the model's robustness and accuracy in diverse texture styles and semantics.
  • Enhancement of Texture Synthesis Methods: Demonstrations of TexTile's utility in extending existing texture synthesis techniques, including diffusion-based methods, to produce tileable textures without sacrificing texture quality.
  • Facilitation of Quantitative Evaluation: By providing an open-source metric with trained models, TexTile paves the way for benchmarking tileability in texture synthesis methods, a task that was previously challenging due to the absence of a dedicated metric.

Technical Insight

Model Design

At its core, TexTile is structured as a binary classifier with a novel architecture that combines convolutional networks with self-attention mechanisms. This design enables the classifier to detect not just local discontinuities—which are indicative of tiling artifacts—but also to understand global image context, crucial for identifying patterns and artifacts relevant to tileability. Training incorporates a diverse dataset encompassing various texture types, along with a comprehensive data augmentation policy tailored to enforce and negate tileability, thus ensuring model robustness and generality.

Differentiability and Application

One of TexTile's significant advantages is its differentiability, allowing it to function as a loss term in the optimization processes of texture synthesis algorithms. This capability is demonstrated through its application to modify existing methods, enabling them to output tileable textures without degradation in overall texture quality. Additionally, the paper showcases the use of TexTile in texture analysis tasks, such as texture alignment and repeating pattern detection, demonstrating its versatility beyond simple evaluation.

Implications and Future Directions

The introduction of TexTile represents a substantial advancement in texture analysis, particularly in the niche area of texture tileability. For researchers and practitioners in fields such as computer graphics, virtual reality, and material science, TexTile offers a powerful tool for both the evaluation and generation of tileable textures. This could lead to improvements in content creation workflows, particularly in scenarios where realism and perceptual quality are paramount.

Looking forward, the authors suggest several potential areas for further exploration, including the integration of perceptual quality metrics with tileability assessments to offer a more holistic evaluation of textures. Additionally, the potential biases arising from the pre-training and curated datasets used to develop TexTile highlight the need for ongoing research into more generalized and unbiased training methodologies, possibly leveraging synthetic datasets.

In summary, TexTile advances the field of texture synthesis by providing a nuanced, quantifiable measure of tileability, opening up new possibilities for both the analysis and creation of seamless textures.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 46 likes about this paper.

Sign Up for Free