Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Shadow Removal by High-Quality Shadow Synthesis (2212.04108v2)

Published 8 Dec 2022 in cs.CV

Abstract: Most shadow removal methods rely on the invasion of training images associated with laborious and lavish shadow region annotations, leading to the increasing popularity of shadow image synthesis. However, the poor performance also stems from these synthesized images since they are often shadow-inauthentic and details-impaired. In this paper, we present a novel generation framework, referred to as HQSS, for high-quality pseudo shadow image synthesis. The given image is first decoupled into a shadow region identity and a non-shadow region identity. HQSS employs a shadow feature encoder and a generator to synthesize pseudo images. Specifically, the encoder extracts the shadow feature of a region identity which is then paired with another region identity to serve as the generator input to synthesize a pseudo image. The pseudo image is expected to have the shadow feature as its input shadow feature and as well as a real-like image detail as its input region identity. To fulfill this goal, we design three learning objectives. When the shadow feature and input region identity are from the same region identity, we propose a self-reconstruction loss that guides the generator to reconstruct an identical pseudo image as its input. When the shadow feature and input region identity are from different identities, we introduce an inter-reconstruction loss and a cycle-reconstruction loss to make sure that shadow characteristics and detail information can be well retained in the synthesized images. Our HQSS is observed to outperform the state-of-the-art methods on ISTD dataset, Video Shadow Removal dataset, and SRD dataset. The code is available at https://github.com/zysxmu/HQSS.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (62)
  1. H. Le and D. Samaras, “Shadow removal via shadow image decomposition,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 8578–8587.
  2. L. Nie, C. Lin, K. Liao, S. Liu, and Y. Zhao, “Unsupervised deep image stitching: Reconstructing stitched features to images,” IEEE Transactions on Image Processing (TIP), vol. 30, pp. 6184–6197, 2021.
  3. J. Zhang, C. Wang, S. Liu, L. Jia, N. Ye, J. Wang, J. Zhou, and J. Sun, “Content-aware unsupervised deep homography estimation,” in Proceedings of the European Conference on Computer Vision (ECCV).   Springer, 2020, pp. 653–669.
  4. X. Shen, F. Darmon, A. A. Efros, and M. Aubry, “Ransac-flow: generic two-stage image alignment,” in Proceedings of the European conference on computer vision (ECCV).   Springer, 2020, pp. 618–637.
  5. Z.-Z. Wang and J.-H. Yong, “Texture analysis and classification with linear regression model based on wavelet transform,” IEEE transactions on image processing (TIP), vol. 17, no. 8, pp. 1421–1430, 2008.
  6. J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang, “Generative image inpainting with contextual attention,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 5505–5514.
  7. G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, and B. Catanzaro, “Image inpainting for irregular holes using partial convolutions,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 85–100.
  8. R. A. Yeh, C. Chen, T. Yian Lim, A. G. Schwing, M. Hasegawa-Johnson, and M. N. Do, “Semantic image inpainting with deep generative models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 5485–5493.
  9. S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Learning enriched features for real image restoration and enhancement,” in Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  10. S. W. Zamir, A. Arora, S. Khan, M. Hayatx, F. S. Khan, M.-H. Yang, and L. Shao, “Multi-stage progressive image restoration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 14 821–14 831.
  11. S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang, “Restormer: Efficient transformer for high-resolution image restoration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 5728–5739.
  12. X. Tao, H. Gao, X. Shen, J. Wang, and J. Jia, “Scale-recurrent network for deep image deblurring,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 8174–8182.
  13. H. Zhang, Y. Dai, H. Li, and P. Koniusz, “Deep stacked hierarchical multi-patch network for image deblurring,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 5978–5986.
  14. X. Huang, G. Hua, J. Tumblin, and L. Williams, “What characterizes a shadow boundary under the sun and sky?” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2011, pp. 898–905.
  15. G. D. Finlayson, S. D. Hordley, C. Lu, and M. S. Drew, “On the removal of shadows from images,” IEEE transactions on pattern analysis and machine intelligence (TPAMI), vol. 28, no. 1, pp. 59–68, 2005.
  16. R. Guo, Q. Dai, and D. Hoiem, “Paired regions for shadow detection and removal,” IEEE transactions on pattern analysis and machine intelligence (TPAMI), vol. 35, no. 12, pp. 2956–2967, 2012.
  17. S. H. Khan, M. Bennamoun, F. Sohel, and R. Togneri, “Automatic shadow detection and removal from a single image,” IEEE transactions on pattern analysis and machine intelligence (TPAMI), vol. 38, no. 3, pp. 431–446, 2015.
  18. L. Zhang, Q. Zhang, and C. Xiao, “Shadow remover: Image shadow removal based on illumination recovering optimization,” IEEE Transactions on Image Processing (TIP), vol. 24, no. 11, pp. 4623–4636, 2015.
  19. Q. Yang, K.-H. Tan, and N. Ahuja, “Shadow removal using bilateral filtering,” IEEE Transactions on Image processing (TIP), vol. 21, no. 10, pp. 4361–4368, 2012.
  20. J. Wang, X. Li, and J. Yang, “Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 1788–1797.
  21. H. Le and D. Samaras, “From shadow segmentation to shadow removal,” in Proceedings of the European Conference on Computer Vision (ECCV).   Springer, 2020, pp. 264–281.
  22. X. Hu, Y. Jiang, C.-W. Fu, and P.-A. Heng, “Mask-shadowgan: Learning to remove shadows from unpaired data,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 2472–2481.
  23. Z. Liu, H. Yin, X. Wu, Z. Wu, Y. Mi, and S. Wang, “From shadow generation to shadow removal,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 4927–4936.
  24. Z. Liu, H. Yin, Y. Mi, M. Pu, and S. Wang, “Shadow removal by a lightness-guided network with training on unpaired data,” IEEE Transactions on Image Processing (TIP), vol. 30, pp. 1853–1865, 2021.
  25. J. Gao, Q. Zheng, and Y. Guo, “Towards real-world shadow removal with a shadow simulation method and a two-stage framework,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2022, pp. 599–608.
  26. Y. Li, S. Tang, R. Zhang, Y. Zhang, J. Li, and S. Yan, “Asymmetric gan for unpaired image-to-image translation,” IEEE Transactions on Image processing (TIP), vol. 28, no. 12, pp. 5881–5896, 2019.
  27. Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo, “Stargan: Unified generative adversarial networks for multi-domain image-to-image translation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 8789–8797.
  28. J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2017, pp. 2223–2232.
  29. M. Gryka, M. Terry, and G. J. Brostow, “Learning to remove soft shadows,” ACM Transactions on Graphics (TOG), vol. 34, no. 5, pp. 1–15, 2015.
  30. Y. Shor and D. Lischinski, “The shadow meets the mask: Pyramid-based shadow removal,” in Computer Graphics Forum, vol. 27, no. 2.   Wiley Online Library, 2008, pp. 577–586.
  31. C. Xiao, R. She, D. Xiao, and K.-L. Ma, “Fast shadow removal using adaptive multi-scale illumination transfer,” in Computer Graphics Forum, vol. 32, no. 8.   Wiley Online Library, 2013, pp. 207–218.
  32. W. Zhang, X. Zhao, J.-M. Morvan, and L. Chen, “Improving shadow suppression for illumination robust face recognition,” IEEE transactions on pattern analysis and machine intelligence (TPAMI), vol. 41, no. 3, pp. 611–624, 2018.
  33. F. Liu and M. Gleicher, “Texture-consistent shadow removal,” in Proceedings of the European Conference on Computer Vision (ECCV).   Springer, 2008.
  34. T. F. Y. Vicente, M. Hoai, and D. Samaras, “Leave-one-out kernel optimization for shadow detection and removal,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 40, no. 3, pp. 682–695, 2017.
  35. C. Fredembach and G. Finlayson, “Hamiltonian path-based shadow removal,” in British Machine Vision Conference (BMVC), vol. 2, no. CONF, 2005, pp. 502–511.
  36. B. Ding, C. Long, L. Zhang, and C. Xiao, “Argan: Attentive recurrent generative adversarial network for shadow detection and removal,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 10 213–10 222.
  37. L. Qu, J. Tian, S. He, Y. Tang, and R. W. Lau, “Deshadownet: A multi-context embedding deep network for shadow removal,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 4067–4075.
  38. X. Hu, C.-W. Fu, L. Zhu, J. Qin, and P.-A. Heng, “Direction-aware spatial context features for shadow detection and removal,” IEEE transactions on pattern analysis and machine intelligence (TPAMI), vol. 42, no. 11, pp. 2795–2808, 2019.
  39. L. Zhang, C. Long, X. Zhang, and C. Xiao, “Ris-gan: Explore residual and illumination with generative adversarial networks for shadow removal,” in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), vol. 34, no. 07, 2020, pp. 12 829–12 836.
  40. X. Cun, C.-M. Pun, and C. Shi, “Towards ghost-free shadow removal via dual hierarchical aggregation network and shadow matting gan,” in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), vol. 34, no. 07, 2020, pp. 10 680–10 687.
  41. L. Fu, C. Zhou, Q. Guo, F. Juefei-Xu, H. Yu, W. Feng, Y. Liu, and S. Wang, “Auto-exposure fusion for single-image shadow removal,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 10 571–10 580.
  42. Z. Chen, C. Long, L. Zhang, and C. Xiao, “Canet: A context-aware network for shadow removal,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 4743–4752.
  43. Y. Jin, A. Sharma, and R. T. Tan, “Dc-shadownet: Single-image hard and soft shadow removal using unsupervised domain-classifier guided network,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 5027–5036.
  44. I. Sato, Y. Sato, and K. Ikeuchi, “Illumination from shadows,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 25, no. 3, pp. 290–300, 2003.
  45. I. Arief, S. McCallum, and J. Y. Hardeberg, “Realtime estimation of illumination direction for augmented reality on mobile devices,” in Color and Imaging Conference, vol. 2012, no. 1.   Society for Imaging Science and Technology, 2012, pp. 111–116.
  46. K. Karsch, V. Hedau, D. Forsyth, and D. Hoiem, “Rendering synthetic objects into legacy photographs,” ACM Transactions on Graphics (TOG), vol. 30, no. 6, pp. 1–12, 2011.
  47. K. Karsch, K. Sunkavalli, S. Hadap, N. Carr, H. Jin, R. Fonte, M. Sittig, and D. Forsyth, “Automatic scene inference for 3d object compositing,” ACM Transactions on Graphics (TOG), vol. 33, no. 3, pp. 1–15, 2014.
  48. M.-A. Gardner, Y. Hold-Geoffroy, K. Sunkavalli, C. Gagné, and J.-F. Lalonde, “Deep parametric indoor lighting estimation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 7175–7183.
  49. J. Zhang, K. Sunkavalli, Y. Hold-Geoffroy, S. Hadap, J. Eisenman, and J.-F. Lalonde, “All-weather deep outdoor lighting estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 10 158–10 166.
  50. Y. Hold-Geoffroy, A. Athawale, and J.-F. Lalonde, “Deep sky modeling for single image outdoor lighting estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 6927–6935.
  51. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2014, pp. 2672–2680.
  52. S. Zhang, R. Liang, and M. Wang, “Shadowgan: Shadow synthesis for virtual objects with conditional adversarial networks,” Computational Visual Media, vol. 5, no. 1, pp. 105–115, 2019.
  53. D. Liu, C. Long, H. Zhang, H. Yu, X. Dong, and C. Xiao, “Arshadowgan: Shadow generative adversarial network for augmented reality in single light scenes,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 8139–8148.
  54. L. Jiang, B. Dai, W. Wu, and C. C. Loy, “Focal frequency loss for image reconstruction and synthesis,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 13 919–13 929.
  55. X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley, “Least squares generative adversarial networks,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2017, pp. 2794–2802.
  56. D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” arXiv preprint arXiv:1607.08022, 2016.
  57. P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1125–1134.
  58. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proceedings of the International Conference on Learning Representations (ICLR), 2014.
  59. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., “Pytorch: An imperative style, high-performance deep learning library,” in Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2019, pp. 8026–8037.
  60. H. Gong and D. Cosker, “Interactive shadow removal and ground truth for variable scene categories,” in British Machine Vision Conference (BMVC).   BMVA Press, 2014.
  61. L. Zhu, Z. Deng, X. Hu, C.-W. Fu, X. Xu, J. Qin, and P.-A. Heng, “Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 121–136.
  62. T. F. Y. Vicente, L. Hou, C.-P. Yu, M. Hoai, and D. Samaras, “Large-scale training of shadow detectors with noisily-annotated shadow examples,” in Proceedings of the European Conference on Computer Vision (ECCV).   Springer, 2016, pp. 816–832.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - zysxmu/HQSS (5 stars)