Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

TUVF: Learning Generalizable Texture UV Radiance Fields (2305.03040v3)

Published 4 May 2023 in cs.CV

Abstract: Textures are a vital aspect of creating visually appealing and realistic 3D models. In this paper, we study the problem of generating high-fidelity texture given shapes of 3D assets, which has been relatively less explored compared with generic 3D shape modeling. Our goal is to facilitate a controllable texture generation process, such that one texture code can correspond to a particular appearance style independent of any input shapes from a category. We introduce Texture UV Radiance Fields (TUVF) that generate textures in a learnable UV sphere space rather than directly on the 3D shape. This allows the texture to be disentangled from the underlying shape and transferable to other shapes that share the same UV space, i.e., from the same category. We integrate the UV sphere space with the radiance field, which provides a more efficient and accurate representation of textures than traditional texture maps. We perform our experiments on synthetic and real-world object datasets where we achieve not only realistic synthesis but also substantial improvements over state-of-the-arts on texture controlling and editing. Project Page: https://www.anjiecheng.me/TUVF

Definition Search Book Streamline Icon: https://streamlinehq.com
References (83)
  1. Image generators with conditionally-independent pixel synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  14278–14287, 2021.
  2. Neural rgb-d surface reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  6290–6301, 2022.
  3. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In IEEE International Conference on Computer Vision, pp.  5855–5864, 2021.
  4. Zip-nerf: Anti-aliased grid-based neural radiance fields. In IEEE International Conference on Computer Vision, 2023.
  5. Geometric texture synthesis by example. In SGP, pp.  41–44, 2004.
  6. Dynamic faust: Registering human bodies in motion. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  6233–6242, 2017.
  7. Gunilla Borgefors. Hierarchical chamfer matching: A parametric edge matching algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 10(6):849–865, 1988.
  8. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In European Conference on Computer Vision, pp.  608–625. Springer, 2020.
  9. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  5799–5809, 2021.
  10. Efficient geometry-aware 3d generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  16123–16133, 2022.
  11. Shapenet: An information-rich 3d model repository. arXiv preprint, 2015.
  12. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In IEEE International Conference on Computer Vision, pp.  14124–14133, 2021.
  13. Text2tex: Text-driven texture synthesis via diffusion models. arXiv preprint, 2023.
  14. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Neural Information Processing Systems, 29, 2016.
  15. Auv-net: Learning aligned uv maps for texture transfer and synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
  16. Learning 3d dense correspondence via canonical point autoencoder. Neural Information Processing Systems, 34:6608–6620, 2021.
  17. Autoregressive 3d shape generation via canonical mapping. In European Conference on Computer Vision, pp.  89–104. Springer, 2022.
  18. Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  4456–4465, 2023.
  19. Markov random field texture models. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp.  25–39, 1983.
  20. Spsg: Self-supervised photometric scene generation from rgb-d scans. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  1747–1756, 2021.
  21. Volume rendering. ACM SIGGRAPH, 22(4):65–74, 1988.
  22. Adversarially learned inference. arXiv preprint, 2016.
  23. Texture synthesis by non-parametric sampling. In IEEE International Conference on Computer Vision, volume 2, pp.  1033–1038. IEEE, 1999.
  24. 3d shape variational autoencoder latent disentanglement via mini-batch feature swapping for bodies and faces. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  18730–18739, 2022.
  25. Get3d: A generative model of high quality 3d textured shapes learned from images. In Neural Information Processing Systems, 2022.
  26. Generative adversarial networks. In Neural Information Processing Systems, volume 27, 2014.
  27. Efficientnerf efficient neural radiance fields. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  12902–12911, 2022.
  28. Multimodal unsupervised image-to-image translation. In European Conference on Computer Vision, pp.  172–189, 2018.
  29. Disentangling factors of variation with cycle-consistent variational auto-encoders. In European Conference on Computer Vision, pp.  805–820, 2018.
  30. A style-based generator architecture for generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  4401–4410, 2019.
  31. Training generative adversarial networks with limited data. Neural Information Processing Systems, 33:12104–12114, 2020a.
  32. Analyzing and improving the image quality of stylegan. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  8110–8119, 2020b.
  33. Auto-encoding variational bayes. arXiv preprint, 2013.
  34. Segment anything. In IEEE International Conference on Computer Vision, 2023.
  35. Solid texture synthesis from 2d exemplars. In ACM SIGGRAPH, 2007.
  36. Deep convolutional inverse graphics network. Neural Information Processing Systems, 28, 2015.
  37. Mine: Towards continuous depth mpi with nerf for novel view synthesis. In IEEE International Conference on Computer Vision, pp.  12578–12588, 2021a.
  38. Sp-gan: Sphere-guided 3d shape generation and manipulation. ACM Transactions on Graphics, 40(4):1–12, 2021b.
  39. Learning to factorize and relight a city. In European Conference on Computer Vision, pp.  544–561. Springer, 2020.
  40. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. IEEE International Conference on Computer Vision, Oct 2019.
  41. Disentangling disentanglement in variational autoencoders. In ICML, pp.  4402–4412. PMLR, 2019.
  42. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, 2020.
  43. Coordgan: Self-supervised dense correspondences emerge from gans. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  10011–10020, 2022.
  44. Hologan: Unsupervised learning of 3d representations from natural images. In IEEE International Conference on Computer Vision, pp.  7588–7597, 2019.
  45. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  3504–3515, 2020.
  46. Texture fields: Learning texture representations in function space. In IEEE International Conference on Computer Vision, pp.  4531–4540, 2019.
  47. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In IEEE International Conference on Computer Vision, pp.  5589–5599, 2021.
  48. Rotational symmetry field design on surfaces. ACM Transactions on Graphics, 26(3):55–es, 2007.
  49. Photoshape: Photorealistic materials for large-scale shape collections. ACM Transactions on Graphics, 37(6), November 2018a.
  50. Photoshape: Photorealistic materials for large-scale shape collections. ACM Transactions on Graphics, 2018b.
  51. Swapping autoencoder for deep image manipulation. Neural Information Processing Systems, 33:7198–7211, 2020.
  52. Convolutional generation of textured 3d meshes. Neural Information Processing Systems, 33:870–882, 2020.
  53. Shape as points: A differentiable poisson solver. Neural Information Processing Systems, 34:13032–13044, 2021.
  54. Adversarial latent autoencoders. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  14104–14113, 2020.
  55. Learning to generate textures on 3d meshes. In IEEE Conference on Computer Vision and Pattern RecognitionW, pp.  32–38, 2019.
  56. High-resolution image synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  10684–10695, 2022.
  57. Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural Networks, 61:85–117, 2015.
  58. Graf: Generative radiance fields for 3d-aware image synthesis. Neural Information Processing Systems, 33:20154–20166, 2020.
  59. Texturify: Generating textures on 3d shape surfaces. In European Conference on Computer Vision, pp.  72–88. Springer, 2022.
  60. Finegan: Unsupervised hierarchical disentanglement for fine-grained object generation and discovery. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  6490–6499, 2019.
  61. Epigraf: Rethinking training of 3d gans. Neural Information Processing Systems, 2022.
  62. Gabriel Taubin. A signal processing approach to fair surface design. In ACM SIGGRAPH, pp.  351–358, 1995.
  63. Alex Trevithick and Bo Yang. Grf: Learning a general radiance field for 3d representation and rendering. In IEEE International Conference on Computer Vision, pp.  15182–15192, 2021.
  64. Greg Turk. Texture synthesis on surfaces. In ACM SIGGRAPH, pp.  347–354, 2001.
  65. Geometry-biased transformers for novel view synthesis. arXiv preprint, 2023.
  66. Is attention all nerf needs? arXiv preprint, 2022.
  67. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Neural Information Processing Systems, 2021.
  68. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics, 38(5):1–12, 2019.
  69. Neutex: Neural texture mapping for volumetric neural rendering. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  7119–7128, 2021.
  70. Point-nerf: Point-based neural radiance fields. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  5438–5448, 2022.
  71. Neumesh: Learning disentangled neural mesh-based implicit field for geometry and texture editing. In European Conference on Computer Vision, pp.  597–614. Springer, 2022.
  72. A large-scale car dataset for fine-grained categorization and verification. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  3973–3981, 2015.
  73. Multiview neural surface reconstruction by disentangling geometry and appearance. Neural Information Processing Systems, 33:2492–2502, 2020.
  74. Volume rendering of neural implicit surfaces. Neural Information Processing Systems, 34:4805–4815, 2021.
  75. pixelnerf: Neural radiance fields from one or few images. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  4578–4587, 2021a.
  76. Learning texture generators for 3d shape collections from internet photo sets. In BMVC, 2021b.
  77. Texture generation on 3d meshes with point-uv diffusion. In IEEE International Conference on Computer Vision, 2023.
  78. Lion: Latent point diffusion models for 3d shape generation. In Neural Information Processing Systems, 2022.
  79. Self-attention generative adversarial networks. In ICML, pp.  7354–7363. PMLR, 2019.
  80. Ners: neural reflectance surfaces for sparse-view 3d reconstruction in the wild. Neural Information Processing Systems, 34:29835–29847, 2021.
  81. Adding conditional control to text-to-image diffusion models. In IEEE International Conference on Computer Vision, 2023a.
  82. Papr: Proximity attention point rendering. In Neural Information Processing Systems, 2023b.
  83. 3d menagerie: Modeling the 3d shape and pose of animals. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  6365–6373, 2017.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now