Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
149 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Natural & Adversarial Bokeh Rendering via Circle-of-Confusion Predictive Network (2111.12971v3)

Published 25 Nov 2021 in cs.CV

Abstract: Bokeh effect is a natural shallow depth-of-field phenomenon that blurs the out-of-focus part in photography. In recent years, a series of works have proposed automatic and realistic bokeh rendering methods for artistic and aesthetic purposes. They usually employ cutting-edge data-driven deep generative networks with complex training strategies and network architectures. However, these works neglect that the bokeh effect, as a real phenomenon, can inevitably affect the subsequent visual intelligent tasks like recognition, and their data-driven nature prevents them from studying the influence of bokeh-related physical parameters (i.e., depth-of-the-field) on the intelligent tasks. To fill this gap, we study a totally new problem, i.e., natural & adversarial bokeh rendering, which consists of two objectives: rendering realistic and natural bokeh and fooling the visual perception models (i.e., bokeh-based adversarial attack). To this end, beyond the pure data-driven solution, we propose a hybrid alternative by taking the respective advantages of data-driven and physical-aware methods. Specifically, we propose the circle-of-confusion predictive network (CoCNet) by taking the all-in-focus image and depth image as inputs to estimate circle-of-confusion parameters for each pixel, which are employed to render the final image through a well-known physical model of bokeh. With the hybrid solution, our method could achieve more realistic rendering results with the naive training strategy and a much lighter network.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. A. Ignatov, J. Patel, and R. Timofte, “Rendering natural camera bokeh effect with deep learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 418–419.
  2. L. Xiao, A. Kaplanyan, A. Fix, M. Chapman, and D. Lanman, “Deepfocus: Learned image synthesis for computational display,” in ACM SIGGRAPH 2018 Talks, 2018, pp. 1–2.
  3. L. Wang, X. Shen, J. Zhang, O. Wang, Z. Lin, C.-Y. Hsieh, S. Kong, and H. Lu, “Deeplens: Shallow depth of field from a single image,” arXiv preprint arXiv:1810.08100, 2018.
  4. S. Dutta, “Depth-aware blending of smoothed images for bokeh effect generation,” Journal of Visual Communication and Image Representation, vol. 77, p. 103089, 2021.
  5. B. Busam, M. Hog, S. McDonagh, and G. Slabaugh, “Sterefo: Efficient image refocusing with stereo vision,” in Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2019, pp. 0–0.
  6. N. Wadhwa, R. Garg, D. E. Jacobs, B. E. Feldman, N. Kanazawa, R. Carroll, Y. Movshovitz-Attias, J. T. Barron, Y. Pritch, and M. Levoy, “Synthetic depth-of-field with a single-camera mobile phone,” ACM Transactions on Graphics (ToG), vol. 37, no. 4, pp. 1–13, 2018.
  7. S. Dutta, S. D. Das, N. A. Shah, and A. K. Tiwari, “Stacked deep multi-scale hierarchical network for fast bokeh effect rendering from a single image,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 2398–2407.
  8. M. Qian, C. Qiao, J. Lin, Z. Guo, C. Li, C. Leng, and J. Cheng, “Bggan: Bokeh-glass generative adversarial network for rendering realistic bokeh,” in European Conference on Computer Vision.   Springer, 2020, pp. 229–244.
  9. X. Luo, J. Peng, K. Xian, Z. Wu, and Z. Cao, “Bokeh rendering from defocus estimation,” in European Conference on Computer Vision.   Springer, 2020, pp. 245–261.
  10. A. Ignatov, J. Patel, R. Timofte, B. Zheng, X. Ye, L. Huang, X. Tian, S. Dutta, K. Purohit, P. Kandula et al., “Aim 2019 challenge on bokeh effect synthesis: Methods and results,” in 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).   IEEE, 2019, pp. 3591–3598.
  11. J. Lee, H. Son, J. Rim, S. Cho, and S. Lee, “Iterative filter adaptive network for single image defocus deblurring,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
  12. J. Xu, Y. Pu, R. Nie, D. Xu, Z. Zhao, and W. Qian, “Virtual try-on network with attribute transformation and local rendering,” IEEE Transactions on Multimedia, vol. 23, pp. 2222–2234, 2021.
  13. X. Zhang, Y. Song, Z. Li, and J. Jiang, “Pr-rl: Portrait relighting via deep reinforcement learning,” IEEE Transactions on Multimedia, vol. 24, pp. 3240–3255, 2021.
  14. Q. Meng, S. Zhang, Z. Li, C. Wang, W. Zhang, and Q. Huang, “Automatic shadow generation via exposure fusion,” IEEE Transactions on Multimedia, 2023.
  15. R. Wan, B. Shi, H. Li, Y. Hong, L.-Y. Duan, and A. C. Kot, “Benchmarking single-image reflection removal algorithms,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 2, pp. 1424–1441, 2022.
  16. P. Haeberli and K. Akeley, “The accumulation buffer: Hardware support for high-quality rendering,” ACM SIGGRAPH computer graphics, vol. 24, no. 4, pp. 309–318, 1990.
  17. S. Lee, E. Eisemann, and H.-P. Seidel, “Real-time lens blur effects and focus control,” ACM Transactions on Graphics (TOG), vol. 29, no. 4, pp. 1–7, 2010.
  18. C. Soler, K. Subr, F. Durand, N. Holzschuch, and F. Sillion, “Fourier depth of field,” ACM Transactions on Graphics (TOG), vol. 28, no. 2, pp. 1–12, 2009.
  19. J. Wu, C. Zheng, X. Hu, Y. Wang, and L. Zhang, “Realistic rendering of bokeh effect based on optical aberrations,” The Visual Computer, vol. 26, no. 6, pp. 555–563, 2010.
  20. X. Yu, R. Wang, and J. Yu, “Real-time depth of field rendering via dynamic light field generation and filtering,” Computer Graphics Forum, vol. 29, no. 7, pp. 2099–2107, 2010.
  21. X. Shen, X. Tao, H. Gao, C. Zhou, and J. Jia, “Deep automatic portrait matting,” in European conference on computer vision.   Springer, 2016, pp. 92–107.
  22. X. Shen, A. Hertzmann, J. Jia, S. Paris, B. Price, E. Shechtman, and I. Sachs, “Automatic portrait segmentation for image stylization,” Computer Graphics Forum, vol. 35, no. 2, pp. 93–102, 2016.
  23. X. Xu, D. Sun, S. Liu, W. Ren, Y.-J. Zhang, M.-H. Yang, and J. Sun, “Rendering portraitures from monocular camera and beyond,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 35–50.
  24. K. Purohit, M. Suin, P. Kandula, and R. Ambasamudram, “Depth-guided dense dynamic filtering network for bokeh effect rendering,” in 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).   IEEE, 2019, pp. 3417–3426.
  25. Z. Wang, A. Jiang, C. Zhang, H. Li, and B. Liu, “Self-supervised multi-scale pyramid fusion networks for realistic bokeh effect rendering,” Journal of Visual Communication and Image Representation, vol. 87, p. 103580, 2022.
  26. T. Seizinger, M. V. Conde, M. Kolmet, T. E. Bishop, and R. Timofte, “Efficient multi-lens bokeh effect rendering and transformation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1633–1642.
  27. Z. Yang, W. Lian, and S. Lai, “Bokehornot: Transforming bokeh effect with image transformer and lens metadata embedding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1542–1550.
  28. Y. Jeong, S. Y. Baek, Y. Seok, G. B. Lee, and S. Lee, “Real-time dynamic bokeh rendering with efficient look-up table sampling,” IEEE Transactions on Visualization and Computer Graphics, vol. 28, no. 2, pp. 1373–1384, 2020.
  29. H. Nagasubramaniam and R. Younes, “Bokeh effect rendering with vision transformers,” 2022.
  30. S. Zhang, D. Zuo, Y. Yang, and X. Zhang, “A transferable adversarial belief attack with salient region perturbation restriction,” IEEE Transactions on Multimedia, 2022.
  31. C. Wan, F. Huang, and X. Zhao, “Average gradient-based adversarial attack,” IEEE Transactions on Multimedia, 2023.
  32. A. S. Shamsabadi, R. Sanchez-Matilla, and A. Cavallaro, “Colorfool: Semantic adversarial colorization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 1151–1160.
  33. L. Engstrom, B. Tran, D. Tsipras, L. Schmidt, and A. Madry, “Exploring the landscape of spatial robustness,” in International Conference on Machine Learning.   PMLR, 2019, pp. 1802–1811.
  34. Q. Guo, F. Juefei-Xu, X. Xie, L. Ma, J. Wang, B. Yu, W. Feng, and Y. Liu, “Watch out! motion is blurring the vision of your deep neural networks,” arXiv preprint arXiv:2002.03500, 2020.
  35. Y. Cheng, Q. Guo, F. Juefei-Xu, S.-W. Lin, W. Feng, W. Lin, and Y. Liu, “Pasadena: Perceptually aware and stealthy adversarial denoise attack,” IEEE Transactions on Multimedia, vol. 24, pp. 3807–3822, 2021.
  36. Z. Zhao, Z. Liu, and M. Larson, “Adversarial color enhancement: Generating unrestricted adversarial images by optimizing a color filter,” arXiv preprint arXiv:2002.01008, 2020.
  37. A. Abuolaim, M. Delbracio, D. Kelly, M. S. Brown, and P. Milanfar, “Learning to reduce defocus blur by realistically modeling dual-pixel data,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 2289–2298.
  38. K. Endo, M. Tanaka, and M. Okutomi, “Classifying degraded images over various levels of degradation,” in 2020 IEEE International Conference on Image Processing (ICIP).   IEEE, 2020, pp. 1691–1695.
  39. N. H. Thao, O. Soloviev, J. Noom, and M. Verhaegen, “Nonuniform defocus removal for image classification,” arXiv preprint arXiv:2106.13864, 2021.
  40. Y. Pei, Y. Huang, Q. Zou, X. Zhang, and S. Wang, “Effects of image degradation and degradation removal to cnn-based image classification,” IEEE transactions on pattern analysis and machine intelligence, 2019.
  41. A. Kurakin, I. Goodfellow, S. Bengio, Y. Dong, F. Liao, M. Liang, T. Pang, J. Zhu, X. Hu, C. Xie et al., “Adversarial attacks and defences competition,” in The NIPS’17 Competition: Building Intelligent Systems.   Springer, 2018, pp. 195–231.
  42. A. Abuolaim and M. S. Brown, “Defocus deblurring using dual-pixel data,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part X 16.   Springer, 2020, pp. 111–126.
  43. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing, vol. 13, no. 4, pp. 600–612, 2004.
  44. A. Hore and D. Ziou, “Image quality metrics: Psnr vs. ssim,” in 2010 20th international conference on pattern recognition.   IEEE, 2010, pp. 2366–2369.
  45. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 586–595.
  46. X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in Proceedings of the thirteenth international conference on artificial intelligence and statistics.   JMLR Workshop and Conference Proceedings, 2010, pp. 249–256.
  47. A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” arXiv preprint arXiv:1706.06083, 2017.
  48. H. Emami, M. M. Aliabadi, M. Dong, and R. B. Chinnam, “Spa-gan: Spatial attention gan for image-to-image translation,” IEEE Transactions on Multimedia, vol. 23, pp. 391–401, 2020.
  49. H. Tan, B. Yin, K. Wei, X. Liu, and X. Li, “Alr-gan: Adaptive layout refinement for text-to-image synthesis,” IEEE Transactions on Multimedia, 2023.
  50. J. Zhang, L. Jiao, W. Ma, F. Liu, X. Liu, L. Li, P. Chen, and S. Yang, “Transformer based conditional gan for multimodal image fusion,” IEEE Transactions on Multimedia, 2023.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now