Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Light Field Reconstruction via Deep Adaptive Fusion of Hybrid Lenses (2102.07085v3)

Published 14 Feb 2021 in eess.IV and cs.CV

Abstract: This paper explores the problem of reconstructing high-resolution light field (LF) images from hybrid lenses, including a high-resolution camera surrounded by multiple low-resolution cameras. The performance of existing methods is still limited, as they produce either blurry results on plain textured areas or distortions around depth discontinuous boundaries. To tackle this challenge, we propose a novel end-to-end learning-based approach, which can comprehensively utilize the specific characteristics of the input from two complementary and parallel perspectives. Specifically, one module regresses a spatially consistent intermediate estimation by learning a deep multidimensional and cross-domain feature representation, while the other module warps another intermediate estimation, which maintains the high-frequency textures, by propagating the information of the high-resolution view. We finally leverage the advantages of the two intermediate estimations adaptively via the learned attention maps, leading to the final high-resolution LF image with satisfactory results on both plain textured areas and depth discontinuous boundaries. Besides, to promote the effectiveness of our method trained with simulated hybrid data on real hybrid data captured by a hybrid LF imaging system, we carefully design the network architecture and the training strategy. Extensive experiments on both real and simulated hybrid data demonstrate the significant superiority of our approach over state-of-the-art ones. To the best of our knowledge, this is the first end-to-end deep learning method for LF reconstruction from a real hybrid input. We believe our framework could potentially decrease the cost of high-resolution LF data acquisition and benefit LF data storage and transmission.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (73)
  1. M. Levoy and P. Hanrahan, “Light field rendering,” in Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, 1996, pp. 31–42.
  2. C. Kim, H. Zimmer, Y. Pritch, A. Sorkine-Hornung, and M. H. Gross, “Scene reconstruction from high spatio-angular resolution light fields.” ACM Transactions on Graphics, vol. 32, no. 4, pp. 73–1, 2013.
  3. J. Fiss, B. Curless, and R. Szeliski, “Refocusing plenoptic images using depth-adaptive splatting,” in IEEE International Conference on Computational Photography (ICCP), 2014, pp. 1–9.
  4. T.-C. Wang, J.-Y. Zhu, E. Hiroaki, M. Chandraker, A. A. Efros, and R. Ramamoorthi, “A 4d light-field dataset and cnn architectures for material recognition,” in European Conference on Computer Vision (ECCV), 2016, pp. 121–138.
  5. N. Li, J. Ye, Y. Ji, H. Ling, and J. Yu, “Saliency detection on light field,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 2806–2813.
  6. M. Guo, J. Jin, H. Liu, and J. Hou, “Learning dynamic interpolation for extremely sparse light fields with wide baselines,” in IEEE International Conference on Computer Vision (ICCV), 2021, pp. 2450–2459.
  7. P. P. Srinivasan, R. Ng, and R. Ramamoorthi, “Light field blind motion deblurring,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2354–2362.
  8. F.-C. Huang, K. Chen, and G. Wetzstein, “The light field stereoscope: immersive computer graphics via factored near-eye light field displays with focus cues,” ACM Transactions on Graphics, vol. 34, no. 4, p. 60, 2015.
  9. J. Yu, “A light-field journey to virtual reality,” IEEE MultiMedia, vol. 24, no. 2, pp. 104–112, 2017.
  10. S. Wanner and B. Goldluecke, “Variational light field analysis for disparity estimation and super-resolution,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 36, no. 3, pp. 606–619, 2014.
  11. M. Rossi and P. Frossard, “Geometry-consistent light field super-resolution via graph-based regularization,” IEEE Transactions on Image Processing, vol. 27, no. 9, pp. 4207–4218, 2018.
  12. Y. Yoon, H.-G. Jeon, D. Yoo, J.-Y. Lee, and I. So Kweon, “Learning a deep convolutional network for light-field image super-resolution,” in IEEE International Conference on Computer Vision Workshops (ICCVW), 2015, pp. 24–32.
  13. J. Jin, J. Hou, H. Yuan, and S. Kwong, “Learning light field angular super-resolution via a geometry-aware network,” in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020, pp. 11 141–11 148.
  14. M. Guo, J. Hou, J. Jin, J. Chen, and L.-P. Chau, “Deep spatial-angular regularization for compressive light field reconstruction over coded apertures,” in European Conference on Computer Vision (ECCV), 2020.
  15. W. F. H. Yeung, J. Hou, J. Chen, Y. Y. Chung, and X. Chen, “Fast light field reconstruction with deep coarse-to-fine modeling of spatial-angular clues,” in European Conference on Computer Vision (ECCV), 2018, pp. 137–152.
  16. J. Jin, J. Hou, J. Chen, H. Zeng, S. Kwong, and J. Yu, “Deep coarse-to-fine dense light field reconstruction with flexible sampling and geometry-aware fusion,” IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1–1, 2020.
  17. J. Jin, J. Hou, J. Chen, W. F. H. Yeung, and S. Kwong, “Light field spatial super-resolution via cnn guided by a single high-resolution rgb image,” in IEEE International Conference on Digital Signal Processing (DSP), 2018, pp. 1–5.
  18. V. Boominathan, K. Mitra, and A. Veeraraghavan, “Improving resolution and depth-of-field of light field cameras using a hybrid imaging system,” in IEEE International Conference on Computational Photography (ICCP), 2014, pp. 1–10.
  19. Y. Wang, Y. Liu, W. Heidrich, and Q. Dai, “The light field attachment: Turning a dslr into a light field camera using a low budget camera ring,” IEEE Transactions on Visualization and Computer Graphics, vol. 23, no. 10, pp. 2357–2364, 2017.
  20. X. Wang, L. Li, and G. Hou, “High-resolution light field reconstruction using a hybrid imaging system,” Applied optics, vol. 55, no. 10, pp. 2580–2593, 2016.
  21. M. Zhao, G. Wu, Y. Li, X. Hao, F. Lu, and Y. Liu, “Cross-scale reference-based light field super-resolution,” IEEE Transactions on Computational Imaging, vol. 4, no. 3, pp. 406–418, 2018.
  22. J. Jin, J. Hou, J. Chen, S. Kwong, and J. Yu, “Light field super-resolution via attention-guided fusion of hybrid lenses,” in ACM International Conference on Multimedia (ACM MM), 2020, p. 193–201.
  23. J.-B. Huang, A. Singh, and N. Ahuja, “Single image super-resolution from transformed self-exemplars,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 5197–5206.
  24. H. Chang, D.-Y. Yeung, and Y. Xiong, “Super-resolution through neighbor embedding,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2004, pp. I–I.
  25. R. Timofte, V. De Smet, and L. Van Gool, “A+: Adjusted anchored neighborhood regression for fast super-resolution,” in IEEE Asian Conference on Computer Vision (ACCV), 2014, pp. 111–126.
  26. J. Sun, Z. Xu, and H.-Y. Shum, “Image super-resolution using gradient profile prior,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008, pp. 1–8.
  27. J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE Transactions on Image Processing, vol. 19, no. 11, pp. 2861–2873, 2010.
  28. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems (NeurIPS), 2012, pp. 1097–1105.
  29. C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a deep convolutional network for image super-resolution,” in European Conference on Computer Vision (ECCV), 2014, pp. 184–199.
  30. ——, “Image super-resolution using deep convolutional networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 2, pp. 295–307, 2016.
  31. J. Kim, J. Kwon Lee, and K. Mu Lee, “Accurate image super-resolution using very deep convolutional networks,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 1646–1654.
  32. W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 1874–1883.
  33. W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Deep laplacian pyramid networks for fast and accurate super-resolution,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 624–632.
  34. Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image super-resolution,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 2472–2481.
  35. B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017, pp. 136–144.
  36. J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in European Conference on Computer Vision (ECCV), 2016, pp. 694–711.
  37. C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. P. Aitken, A. Tejani, J. Totz, Z. Wang et al., “Photo-realistic single image super-resolution using a generative adversarial network,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 105–114.
  38. Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image super-resolution using very deep residual channel attention networks,” in European Conference on Computer Vision (ECCV), 2018, pp. 286–301.
  39. T. Dai, J. Cai, Y. Zhang, S.-T. Xia, and L. Zhang, “Second-order attention network for single image super-resolution,” in IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2019, pp. 11 065–11 074.
  40. J. Tian and K.-K. Ma, “A survey on super-resolution imaging,” Signal, Image and Video Processing, vol. 5, no. 3, pp. 329–342, 2011.
  41. Z. Wang, J. Chen, and S. C. Hoi, “Deep learning for image super-resolution: A survey,” arXiv preprint arXiv:1902.06068, 2019.
  42. H. Zheng, M. Ji, H. Wang, Y. Liu, and L. Fang, “Crossnet: An end-to-end reference-based super resolution network using cross-scale warping,” in European Conference on Computer Vision (ECCV), 2018, pp. 87–104.
  43. Y. Tan, H. Zheng, Y. Zhu, X. Yuan, X. Lin, B. David, and F. Lu, “Crossnet++: Cross-scale large-parallax warping for reference-based super-resolution,” IEEE Transactions on Pattern Analysis & Machine Intelligence, no. 01, pp. 1–1, 2020.
  44. Z. Zhang, Z. Wang, Z. Lin, and H. Qi, “Image super-resolution by neural texture transfer,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 7982–7991.
  45. Y. Xie, J. Xiao, M. Sun, C. Yao, and K. Huang, “Feature representation matters: End-to-end learning for reference-based image super-resolution,” in European Conference on Computer Vision (ECCV), 2020, pp. 230–245.
  46. F. Yang, H. Yang, J. Fu, H. Lu, and B. Guo, “Learning texture transformer network for image super-resolution,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 5791–5800.
  47. G. Shim, J. Park, and I. S. Kweon, “Robust reference-based super-resolution with similarity-aware deformable convolution,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 8425–8434.
  48. R. Shao, G. Wu, Y. Zhou, Y. Fu, L. Fang, and Y. Liu, “Localtrans: A multiscale local transformer network for cross-resolution homography estimation,” in IEEE International Conference on Computer Vision (ICCV), 2021, pp. 14 890–14 899.
  49. Y. Zhou, G. Wu, Y. Fu, K. Li, and Y. Liu, “Cross-mpi: Cross-scale stereo for image super-resolution using multiplane images,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 14 842–14 851.
  50. K. Mitra and A. Veeraraghavan, “Light field denoising, light field superresolution and stereo camera based refocussing using a gmm light field patch prior,” in IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2012, pp. 22–28.
  51. S. Heber and T. Pock, “Shape from light field meets robust pca,” in European Conference on Computer Vision (ECCV), 2014, pp. 751–767.
  52. R. A. Farrugia, C. Galea, and C. Guillemot, “Super resolution of light field images using linear subspace projection of patch-volumes,” IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 7, pp. 1058–1071, 2017.
  53. Y. Yuan, Z. Cao, and L. Su, “Light-field image superresolution using a combined deep cnn based on epi,” IEEE Signal Processing Letters, vol. 25, no. 9, pp. 1359–1363, 2018.
  54. Y. Wang, F. Liu, K. Zhang, G. Hou, Z. Sun, and T. Tan, “Lfnet: A novel bidirectional recurrent convolutional neural network for light-field image super-resolution,” IEEE Transactions on Image Processing, vol. 27, no. 9, pp. 4274–4286, 2018.
  55. W. F. H. Yeung, J. Hou, X. Chen, J. Chen, Z. Chen, and Y. Y. Chung, “Light field spatial super-resolution using deep efficient spatial-angular separable convolution,” IEEE Transactions on Image Processing, vol. 28, no. 5, pp. 2319–2330, 2018.
  56. Y. Wang, L. Wang, J. Yang, W. An, J. Yu, and Y. Guo, “Spatial-angular interaction for light field image super-resolution,” in European Conference on Computer Vision (ECCV), 2020, pp. 290–308.
  57. J. Jin, J. Hou, J. Chen, and S. Kwong, “Light field spatial super-resolution via deep combinatorial geometry embedding and structural consistency regularization,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 2260–2269.
  58. C.-H. Lu, S. Muenzel, and J. W. Fleischer, “High-resolution light-field microscopy,” in Imaging and Applied Optics, 2013.
  59. T.-C. Wang, J.-Y. Zhu, N. K. Kalantari, A. A. Efros, and R. Ramamoorthi, “Light field video capture using a learning-based hybrid imaging system,” ACM Transactions on Graphics, vol. 36, no. 4, p. 133, 2017.
  60. G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 4700–4708.
  61. M. Mathieu, C. Couprie, and Y. LeCun, “Deep multi-scale video prediction beyond mean square error,” International Conference on Learning Representations (ICLR), pp. 1–11, 2016.
  62. O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015, pp. 234–241.
  63. M. Jaderberg, K. Simonyan, A. Zisserman et al., “Spatial transformer networks,” in Advances in Neural Information Processing Systems (NeurIPS), 2015, pp. 2017–2025.
  64. C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in IEEE International Conference on Computer Vision (ICCV), 1998, pp. 839–846.
  65. R. Jonschkowski, A. Stone, J. T. Barron, A. Gordon, K. Konolige, and A. Angelova, “What matters in unsupervised optical flow,” ECCV, pp. 1–1, 2020.
  66. N. K. Kalantari, T.-C. Wang, and R. Ramamoorthi, “Learning-based view synthesis for light field cameras,” ACM Transactions on Graphics, vol. 35, no. 6, p. 193, 2016.
  67. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  68. J. Shi, X. Jiang, and C. Guillemot, “A framework for learning depth from a flexible subset of dense and sparse light field views,” IEEE Transactions on Image Processing, vol. 28, no. 12, pp. 5867–5880, 2019.
  69. K. Honauer, O. Johannsen, D. Kondermann, and B. Goldluecke, “A dataset and evaluation methodology for depth estimation on 4d light fields,” in Asian Conference on Computer Vision (ACCV), 2016, pp. 19–34.
  70. J. Chen, J. Hou, Y. Ni, and L.-P. Chau, “Accurate light field depth estimation with superpixel regularization over partially occluded regions,” IEEE Transactions on Image Processing, vol. 27, no. 10, pp. 4889–4900, 2018.
  71. B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” in European Conference on Computer Vision (ECCV), 2020, pp. 405–421.
  72. C. Wang, X. Wu, Y.-C. Guo, S.-H. Zhang, Y.-W. Tai, and S.-M. Hu, “Nerf-sr: High-quality neural radiance fields using super-sampling,” arXiv preprint arXiv:2112.01759, 2021.
  73. N. Rahaman, A. Baratin, D. Arpit, F. Draxler, M. Lin, F. Hamprecht, Y. Bengio, and A. Courville, “On the spectral bias of neural networks,” in International Conference on Machine Learning (ICML), 2019, pp. 5301–5310.
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now