Papers
Topics
Authors
Recent
2000 character limit reached

Auxiliary Features-Guided Super Resolution for Monte Carlo Rendering (2310.13235v1)

Published 20 Oct 2023 in cs.GR and cs.CV

Abstract: This paper investigates super resolution to reduce the number of pixels to render and thus speed up Monte Carlo rendering algorithms. While great progress has been made to super resolution technologies, it is essentially an ill-posed problem and cannot recover high-frequency details in renderings. To address this problem, we exploit high-resolution auxiliary features to guide super resolution of low-resolution renderings. These high-resolution auxiliary features can be quickly rendered by a rendering engine and at the same time provide valuable high-frequency details to assist super resolution. To this end, we develop a cross-modality Transformer network that consists of an auxiliary feature branch and a low-resolution rendering branch. These two branches are designed to fuse high-resolution auxiliary features with the corresponding low-resolution rendering. Furthermore, we design residual densely-connected Swin Transformer groups to learn to extract representative features to enable high-quality super-resolution. Our experiments show that our auxiliary features-guided super-resolution method outperforms both super-resolution methods and Monte Carlo denoising methods in producing high-quality renderings.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (74)
  1. Namhyuk Ahn, Byungkon Kang and Kyung-Ah Sohn “Fast, accurate, and lightweight super-resolution with cascading residual network” In Proceedings of the European Conference on Computer Vision, 2018, pp. 252–268
  2. “Neural frame interpolation for rendered content” In ACM Transactions on Graphics (TOG) 40.6 ACM New York, NY, USA, 2021, pp. 1–13
  3. Mark R Bolin and Gary W Meyer “A perceptually based adaptive sampling algorithm” In Proceedings of the 25th annual conference on Computer graphics and interactive techniques ACM, 1998, pp. 299–309
  4. “Nonlinearly Weighted First-order Regression for Denoising Monte Carlo Renderings” In Computer Graphics Forum 35.4, 2016, pp. 107–117 Wiley Online Library
  5. “Kernel-predicting convolutional networks for denoising Monte Carlo renderings” In ACM Transactions on Graphics 36.4 ACM, 2017, pp. 97
  6. “Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder” In ACM Transactions on Graphics 36.4 ACM, 2017, pp. 98
  7. “End-to-end object detection with transformers” In European conference on computer vision, 2020, pp. 213–229 Springer
  8. Robert L Cook, Thomas Porter and Loren Carpenter “Distributed ray tracing” In ACM SIGGRAPH computer graphics 18.3, 1984, pp. 137–145
  9. “Pre-Trained Image Processing Transformer”, 2021 arXiv:2012.00364 [cs.CV]
  10. “An image is worth 16x16 words: Transformers for image recognition at scale” In arXiv preprint arXiv:2010.11929, 2020
  11. “Learning a deep convolutional network for image super-resolution” In European conference on computer vision, 2014, pp. 184–199 Springer
  12. “Edge-avoiding À-Trous wavelet transform for fast global illumination filtering” In Proceedings of the Conference on High Performance Graphics, 2010, pp. 67–75 Eurographics Association
  13. “Frequency analysis and sheared reconstruction for rendering motion blur” In ACM Transactions on Graphics 28.3, 2009, pp. 93
  14. “ExtraNet: real-time extrapolated rendering for low-latency temporal supersampling” In ACM Transactions on Graphics (TOG) 40.6 ACM New York, NY, USA, 2021, pp. 1–16
  15. “Cmt: Convolutional neural networks meet vision transformers” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 12175–12185
  16. “Sample-based Monte Carlo denoising using a kernel-splatting network” In ACM Transactions on Graphics 38.4 ACM New York, NY, USA, 2019, pp. 1–12
  17. “RSTT: Real-time Spatial Temporal Transformer for Space-Time Video Super-Resolution” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022
  18. “Fast Monte Carlo Rendering via Multi-Resolution Sampling” In Graphics Interface Conference, 2021, pp. 25:1–9
  19. “Densely connected convolutional networks” In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4700–4708
  20. “Neural temporal adaptive sampling and denoising” In Computer Graphics Forum 39.2, 2020, pp. 147–155 Wiley Online Library
  21. Jie Hu, Li Shen and Gang Sun “Squeeze-and-excitation networks” In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132–7141
  22. Muhammad Haris, Gregory Shakhnarovich and Norimichi Ukita “Recurrent back-projection network for video super-resolution” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 3897–3906
  23. Zheng Hui, Xiumei Wang and Xinbo Gao “Fast and accurate single image super-resolution via information distillation network” In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 723–731
  24. “Deep residual learning for image recognition” In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778
  25. Henrik Wann Jensen and Niels Jørgen Christensen “Optimizing path tracing using noise reduction filters” Václav Skala-UNION Agency, 1995
  26. Henrik Wann Jensen “Realistic image synthesis using photon mapping” AK Peters/CRC Press, 2001
  27. James T Kajiya “The rendering equation” In ACM SIGGRAPH computer graphics 20.4 ACM, 1986, pp. 143–150
  28. Diederik P Kingma and Jimmy Ba “Adam: A method for stochastic optimization” In arXiv preprint arXiv:1412.6980, 2014
  29. Nima Khademi Kalantari, Steve Bako and Pradeep Sen “A machine learning approach for filtering Monte Carlo noise.” In ACM Trans. Graph. 34.4, 2015, pp. 122–1
  30. Jiwon Kim, Jung Kwon Lee and Kyoung Mu Lee “Deeply-recursive convolutional network for image super-resolution” In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1637–1645
  31. Alexandr Kuznetsov, Nima Khademi Kalantari and Ravi Ramamoorthi “Deep adaptive sampling for low sample count rendering” In Computer Graphics Forum 37.4, 2018, pp. 35–44
  32. “Swinir: Image restoration using swin transformer” In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 1833–1844
  33. “Swin transformer: Hierarchical vision transformer using shifted windows” In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10012–10022
  34. Mark E Lee and Richard A Redner “A note on the use of nonlinear filtering in computer graphics” In IEEE Computer Graphics and Applications 10.3 IEEE, 1990, pp. 23–29
  35. “Incremental instant radiosity for real-time indirect illumination” In Proceedings of the 18th Eurographics conference on Rendering Techniques, 2007, pp. 277–286 Eurographics Association
  36. “Enhanced deep residual networks for single image super-resolution” In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2017, pp. 136–144
  37. “3D appearance super-resolution with deep learning” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 9671–9680
  38. Tzu-Mao Li, Yu-Ting Wu and Yung-Yu Chuang “SURE-based optimization for adaptive sampling and reconstruction” In ACM Transactions on Graphics 31.6 ACM, 2012, pp. 194
  39. “Hierarchical back projection network for image super-resolution” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2019
  40. “Uniformer: Unifying convolution and self-attention for visual recognition” In arXiv preprint arXiv:2201.09450, 2022
  41. “Statistical acceleration for animated global illumination” In ACM Transactions on Graphics 25.3 ACM, 2006, pp. 1075–1080
  42. Michael D McCool “Anisotropic diffusion for Monte Carlo noise reduction” In ACM Transactions on Graphics 18.2 ACM, 1999, pp. 171–194
  43. “SwiftTron: An Efficient Hardware Accelerator for Quantized Transformers” In arXiv preprint arXiv:2304.03986, 2023
  44. Nigel Meade “Long range forecasting: From crystal ball to computer” In Journal of the Operational Research Society 37.5 Taylor & Francis, 1986, pp. 533–535
  45. “Neural denoising with layer embeddings” In Computer Graphics Forum 39.4, 2020, pp. 1–12 Wiley Online Library
  46. “Adaptive polynomial rendering” In ACM Transactions on Graphics 35.4 ACM, 2016, pp. 40
  47. Ryan S Overbeck, Craig Donner and Ravi Ramamoorthi “Adaptive wavelet rendering.” In ACM Trans. Graph. 28.5, 2009, pp. 140
  48. Fabrice Rousselle, Claude Knaus and Matthias Zwicker “Adaptive sampling and reconstruction using greedy error minimization” In ACM Transactions on Graphics 30.6 ACM, 2011, pp. 159
  49. “Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters” In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020, pp. 3505–3506
  50. Holly E Rushmeier and Gregory J Ward “Energy preserving non-linear filters” In Proceedings of the 21st annual conference on Computer graphics and interactive techniques ACM, 1994, pp. 131–138
  51. “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network” In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1874–1883
  52. “On filtering the noise from the random parameters in Monte Carlo rendering.” In ACM Trans. Graph. 31.3, 2012, pp. 18–1
  53. “Non-interleaved deferred shading of interleaved sample patterns” In Graphics Hardware, 2006, pp. 53–60
  54. “Training data-efficient image transformers distillation through attention” In International Conference on Machine Learning 139, 2021, pp. 10347–10357
  55. “Efficientnet: Rethinking model scaling for convolutional neural networks” In International conference on machine learning, 2019, pp. 6105–6114 PMLR
  56. “Temporally Stable Real-Time Joint Neural Denoising and Supersampling” In Proceedings of the ACM on Computer Graphics and Interactive Techniques 5.3 ACM New York, NY, USA, 2022, pp. 1–22
  57. “Denoising with kernel prediction and asymmetric loss functions” In ACM Transactions on Graphics (TOG) 37.4 ACM New York, NY, USA, 2018, pp. 1–15
  58. “Attention is all you need” In Advances in neural information processing systems 30, 2017
  59. “Multidimensional lightcuts” In ACM Transactions on graphics 25.3 ACM, 2006, pp. 1081–1088
  60. Gregory J Ward, Francis M Rubinstein and Robert D Clear “A ray tracing solution for diffuse interreflection” In ACM SIGGRAPH Computer Graphics 22.4 ACM, 1988, pp. 85–92
  61. Xiangyu Xu, Yongrui Ma and Wenxiu Sun “Towards real scene super-resolution with raw images” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 1723–1731
  62. “Neural supersampling for real-time rendering” In ACM Transactions on Graphics (TOG) 39.4 ACM New York, NY, USA, 2020, pp. 142–1
  63. Ruifeng Xu and Sumanta N Pattanaik “A novel Monte Carlo noise reduction operator” In IEEE Computer Graphics and Applications 25.2 IEEE, 2005, pp. 31–35
  64. “Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation.” In ACM Trans. Graph. 38.6, 2019, pp. 224–1
  65. “Monte Carlo denoising via auxiliary feature guided self-attention.” In ACM Trans. Graph. 40.6, 2021, pp. 273–1
  66. “Wave-vit: Unifying wavelet and transformers for visual representation learning” In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXV, 2022, pp. 328–345 Springer
  67. “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric” In CVPR, 2018
  68. “Recent advances in adaptive sampling and reconstruction for Monte Carlo rendering” In Computer Graphics Forum 34.2, 2015, pp. 667–681
  69. “Image super-resolution using very deep residual channel attention networks” In Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 286–301
  70. “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers” In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 6881–6890
  71. “Residual dense network for image super-resolution” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 2472–2481
  72. “Image super-resolution by neural texture transfer” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 7982–7991
  73. “Ensemble denoising for Monte Carlo renderings” In ACM Transactions on Graphics (TOG) 40.6 ACM New York, NY, USA, 2021, pp. 1–17
  74. Kai Zhang, Wangmeng Zuo and Lei Zhang “Learning a single convolutional super-resolution network for multiple degradations” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 3262–3271
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Video Overview

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Authors (2)

  1. Qiqi Hou
  2. Feng Liu

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up