Papers
Topics
Authors
Recent
Search
2000 character limit reached

Neural Super-Resolution for Real-time Rendering with Radiance Demodulation

Published 13 Aug 2023 in cs.GR | (2308.06699v2)

Abstract: It is time-consuming to render high-resolution images in applications such as video games and virtual reality, and thus super-resolution technologies become increasingly popular for real-time rendering. However, it is challenging to preserve sharp texture details, keep the temporal stability and avoid the ghosting artifacts in real-time super-resolution rendering. To address this issue, we introduce radiance demodulation to separate the rendered image or radiance into a lighting component and a material component, considering the fact that the light component is smoother than the rendered image so that the high-resolution material component with detailed textures can be easily obtained. We perform the super-resolution on the lighting component only and re-modulate it with the high-resolution material component to obtain the final super-resolution image with more texture details. A reliable warping module is proposed by explicitly marking the occluded regions to avoid the ghosting artifacts. To further enhance the temporal stability, we design a frame-recurrent neural network and a temporal loss to aggregate the previous and current frames, which can better capture the spatial-temporal consistency among reconstructed frames. As a result, our method is able to produce temporally stable results in real-time rendering with high-quality details, even in the challenging 4 $\times$ 4 super-resolution scenarios.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Kernel-predicting convolutional networks for denoising monte carlo renderings. ACM Trans. Graph., 36(4):97–1, 2017.
  2. Epipolar-plane image analysis: An approach to determining structure from motion. International journal of computer vision, 1(1):7–55, 1987.
  3. Brent Burley and Walt Disney Animation Studios. Physically-based shading at disney. In ACM SIGGRAPH, volume 2012, pages 1–7. vol. 2012, 2012.
  4. Real-time video super-resolution with spatio-temporal networks and motion compensation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 2848–2857. IEEE Computer Society, 2017.
  5. Basicvsr: The search for essential components in video super-resolution and beyond. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4947–4956, 2021.
  6. Basicvsr++: Improving video super-resolution with enhanced propagation and alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5972–5981, 2022.
  7. Learning temporal coherence via self-supervision for gan-based video generation. ACM Transactions on Graphics (TOG), 39(4):75–1, 2020.
  8. Truly next-gen: Adding deep learning to games and graphics. In In NVIDIA Sponsored Sessions (Game Developers Conference), 2019.
  9. Amd fidelityfx super resolution, 2022.
  10. Efficient video super-resolution through recurrent latent space propagation. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 3476–3485. IEEE, 2019.
  11. Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440–1448, 2015.
  12. Super-resolution by predicting offsets: An ultra-efficient super-resolution network for rasterized images. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision – ECCV 2022, pages 583–598, Cham, 2022. Springer Nature Switzerland.
  13. Classifier guided temporal supersampling for real-time rendering. In Computer Graphics Forum, volume 41, pages 237–246. Wiley Online Library, 2022.
  14. Recurrent back-projection network for video super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3897–3906, 2019.
  15. Bidirectional recurrent convolutional networks for multi-frame super-resolution. Advances in neural information processing systems, 28, 2015.
  16. Video super-resolution with recurrent structure-detail network. In European Conference on Computer Vision, pages 645–660. Springer, 2020.
  17. Video super-resolution with temporal group attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8008–8017, 2020.
  18. Revisiting temporal modeling for video super-resolution. arXiv preprint arXiv:2008.05765, 2020.
  19. Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3224–3232, 2018.
  20. James T. Kajiya. The rendering equation. In David C. Evans and Russell J. Athay, editors, Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1986, Dallas, Texas, USA, August 18-22, 1986, pages 143–150. ACM, 1986.
  21. Video super-resolution with convolutional neural networks. IEEE transactions on computational imaging, 2(2):109–122, 2016.
  22. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  23. Irradiance filtering for monte carlo ray tracing. In Monte Carlo and Quasi-Monte Carlo Methods 2004, pages 259–272. Springer, 2004.
  24. Learning blind video temporal consistency. In Proceedings of the European conference on computer vision (ECCV), pages 170–185, 2018.
  25. Mucan: Multi-correspondence aggregation network for video super-resolution. In European conference on computer vision, pages 335–351. Springer, 2020.
  26. Toward a practical perceptual video quality metric. The Netflix Tech Blog, 6(2), 2016.
  27. Vrt: A video restoration transformer. arXiv preprint arXiv:2201.12288, 2022.
  28. Recurrent video restoration transformer with guided deformable attention. Advances in Neural Information Processing Systems, 35:378–393, 2022.
  29. Video super-resolution via deep draft-ensemble learning. In Proceedings of the IEEE international conference on computer vision, pages 531–539, 2015.
  30. Fdan: Flow-guided deformable alignment network for video super-resolution. arXiv preprint arXiv:2105.05640, 2021.
  31. Learning trajectory-aware transformer for video super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5687–5696, 2022.
  32. Amazon Lumberyard. Amazon lumberyard bistro, July 2017.
  33. Morgan McGuire. Computer graphics archive, July 2017. https://casual-effects.com/data.
  34. Efficient neural supersampling on a novel gaming dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 296–306, 2023.
  35. Nvidia emerald square, July 2017.
  36. pytorch, 2022.
  37. Optical flow estimation using a spatial pyramid network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4161–4170, 2017.
  38. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  39. Frame-recurrent video super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6626–6634, 2018.
  40. Spatiotemporal variance-guided filtering: real-time reconstruction for path-traced global illumination. In Proceedings of High Performance Graphics, pages 1–12. 2017.
  41. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 1874–1883. IEEE Computer Society, 2016.
  42. Convolutional lstm network: A machine learning approach for precipitation nowcasting. Advances in neural information processing systems, 28, 2015.
  43. Taau, 2020.
  44. Advances in real-time rendering in games, part I. In Special Interest Group on Computer Graphics and Interactive Techniques Conference, SIGGRAPH ’14, Vancouver, Canada, August 10-14, 2014, Courses, page 10:1. ACM, 2014.
  45. Nvidia tensorrt, 2022.
  46. Tdan: Temporally-deformable alignment network for video super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3360–3369, 2020.
  47. Unity, 2022.
  48. Edvr: Video restoration with enhanced deformable convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  49. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004.
  50. Mike Winkelmann. Zero-day, open research content archive (orca), November 2019.
  51. Neural supersampling for real-time rendering. ACM Trans. Graph., 39(4), jul 2020.
  52. Frame and feature-context video super-resolution. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 5597–5604, 2019.
  53. Mnss: Neural supersampling framework for real-time rendering on mobile devices. IEEE Transactions on Visualization and Computer Graphics, 2023.
  54. Progressive fusion video super-resolution network via exploiting non-local spatio-temporal correlations. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3106–3115, 2019.
  55. Free-form image inpainting with gated convolution. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 4470–4479. IEEE, 2019.
  56. Temporally reliable motion vectors for real-time ray tracing. Computer Graphics Forum, 40(2):79–90, 2021.
  57. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
  58. Image super-resolution using very deep residual channel attention networks. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VII, volume 11211 of Lecture Notes in Computer Science, pages 294–310. Springer, 2018.
  59. Fusesr: Super resolution for real-time rendering through efficient multi-resolution fusion. In SIGGRAPH Asia 2023 Conference Papers, pages 1–10, 2023.
  60. Real-time denoising using brdf pre-integration factorization. In Computer Graphics Forum, volume 40, pages 173–180. Wiley Online Library, 2021.

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free