Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
149 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Iterative approach to reconstructing neural disparity fields from light-field data (2407.15380v2)

Published 22 Jul 2024 in eess.IV and cs.CV

Abstract: This study proposes a neural disparity field (NDF) that establishes an implicit, continuous representation of scene disparity based on a neural field and an iterative approach to address the inverse problem of NDF reconstruction from light-field data. NDF enables seamless and precise characterization of disparity variations in three-dimensional scenes and can discretize disparity at any arbitrary resolution, overcoming the limitations of traditional disparity maps that are prone to sampling errors and interpolation inaccuracies. The proposed NDF network architecture utilizes hash encoding combined with multilayer perceptrons to capture detailed disparities in texture levels, thereby enhancing its ability to represent the geometric information of complex scenes. By leveraging the spatial-angular consistency inherent in light-field data, a differentiable forward model to generate a central view image from the light-field data is developed. Based on the forward model, an optimization scheme for the inverse problem of NDF reconstruction using differentiable propagation operators is established. Furthermore, an iterative solution method is adopted to reconstruct the NDF in the optimization scheme, which does not require training datasets and applies to light-field data captured by various acquisition methods. Experimental results demonstrate that high-quality NDF can be reconstructed from light-field data using the proposed method. High-resolution disparity can be effectively recovered by NDF, demonstrating its capability for the implicit, continuous representation of scene disparities.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. M. Levoy, “Light fields and computational imaging,” Computer, vol. 39, no. 8, pp. 46–55, 2006.
  2. A. Mittal, “Neural radiance fields: Past, present, and future,” arXiv preprint arXiv:2304.10050, 2023.
  3. A. Gershun, “The light field,” Journal of Mathematics and Physics, vol. 18, no. 1-4, pp. 51–151, 1939.
  4. R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, “Light field photography with a hand-held plenoptic camera,” Ph.D. dissertation, Stanford university, 2005.
  5. X. Xiao, B. Javidi, M. Martinez-Corral, and A. Stern, “Advances in three-dimensional integral imaging: sensing, display, and applications,” Applied optics, vol. 52, no. 4, pp. 546–560, 2013.
  6. H.-G. Jeon, J. Park, G. Choe, J. Park, Y. Bok, Y.-W. Tai, and I. So Kweon, “Accurate depth map estimation from a lenslet light field camera,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1547–1555.
  7. S. Wanner and B. Goldluecke, “Variational light field analysis for disparity estimation and super-resolution,” IEEE transactions on pattern analysis and machine intelligence, vol. 36, no. 3, pp. 606–619, 2013.
  8. S. Heber and T. Pock, “Shape from light field meets robust pca,” in European Conference on Computer Vision.   Springer, 2014, pp. 751–767.
  9. M. W. Tao, S. Hadap, J. Malik, and R. Ramamoorthi, “Depth from combining defocus and correspondence using light-field cameras,” in Proceedings of the IEEE International Conference on Computer Vision, 2013, pp. 673–680.
  10. T.-C. Wang, A. A. Efros, and R. Ramamoorthi, “Depth estimation with occlusion modeling using light-field cameras,” IEEE transactions on pattern analysis and machine intelligence, vol. 38, no. 11, pp. 2170–2181, 2016.
  11. H. Sheng, P. Zhao, S. Zhang, J. Zhang, and D. Yang, “Occlusion-aware depth estimation for light field using multi-orientation epis,” Pattern Recognition, vol. 74, pp. 587–599, 2018.
  12. S. Zhang, H. Sheng, C. Li, J. Zhang, and Z. Xiong, “Robust depth estimation for light field via spinning parallelogram operator,” Computer Vision and Image Understanding, vol. 145, pp. 148–159, 2016.
  13. C. Liu, J. Qiu, and S. Zhao, “Iterative reconstruction of scene depth with fidelity based on light field data,” Applied Optics, vol. 56, no. 11, pp. 3185–3192, 2017.
  14. C. Liu, L. Shi, X. Zhao, and J. Qiu, “Adaptive matching norm based disparity estimation from light field data,” Signal Processing, vol. 209, p. 109042, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0165168423001160
  15. S. Heber and T. Pock, “Convolutional networks for shape from light field,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 3746–3754.
  16. C. Shin, H.-G. Jeon, Y. Yoon, I. S. Kweon, and S. J. Kim, “Epinet: A fully-convolutional neural network using epipolar geometry for depth from light field images,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4748–4757.
  17. A. Alperovich, O. Johannsen, M. Strecke, and B. Goldluecke, “Light field intrinsics with a deep encoder-decoder network,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 9145–9154.
  18. J. Peng, Z. Xiong, D. Liu, and X. Chen, “Unsupervised depth estimation from light field using a convolutional neural network,” in 2018 International Conference on 3D Vision (3DV).   IEEE, 2018, pp. 295–303.
  19. W. Zhou, E. Zhou, G. Liu, L. Lin, and A. Lumsdaine, “Unsupervised monocular depth estimation from light field image,” IEEE Transactions on Image Processing, vol. 29, pp. 1606–1617, 2020.
  20. K. Li, J. Zhang, R. Sun, X. Zhang, and J. Gao, “Epi-based oriented relation networks for light field depth estimation,” in British Machine Vision Conference (BMVC), 2020.
  21. Y.-J. Tsai, Y.-L. Liu, M. Ouhyoung, and Y.-Y. Chuang, “Attention-based view selection networks for light-field disparity estimation,” in Proceedings of the 34th Conference on Artificial Intelligence (AAAI), 2020.
  22. Y. Li, Q. Wang, L. Zhang, and G. Lafruit, “A lightweight depth estimation network for wide-baseline light fields,” IEEE Transactions on Image Processing, vol. 30, pp. 2288–2300, 2021.
  23. Z. Huang, X. Hu, Z. Xue, W. Xu, and T. Yue, “Fast light-field disparity estimation with multi-disparity-scale cost aggregation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 6320–6329.
  24. B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Communications of the ACM, vol. 65, no. 1, pp. 99–106, 2021.
  25. J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan, “Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 5855–5864.
  26. J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman, “Mip-nerf 360: Unbounded anti-aliased neural radiance fields,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5470–5479.
  27. D. Verbin, P. Hedman, B. Mildenhall, T. Zickler, J. T. Barron, and P. P. Srinivasan, “Ref-nerf: Structured view-dependent appearance for neural radiance fields,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).   IEEE, 2022, pp. 5481–5490.
  28. T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” ACM Transactions on Graphics (ToG), vol. 41, no. 4, pp. 1–15, 2022.
  29. A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su, “Tensorf: Tensorial radiance fields,” in European Conference on Computer Vision.   Springer, 2022, pp. 333–350.
  30. A. Chen, Z. Xu, X. Wei, S. Tang, H. Su, and A. Geiger, “Factor fields: A unified framework for neural fields and beyond,” arXiv preprint arXiv:2302.01226, 2023.
  31. V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein, “Implicit neural representations with periodic activation functions,” Advances in neural information processing systems, vol. 33, pp. 7462–7473, 2020.
  32. S. Xie, H. Zhu, Z. Liu, Q. Zhang, Y. Zhou, X. Cao, and Z. Ma, “Diner: Disorder-invariant implicit neural representation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 6143–6152.
  33. B. Attal, J.-B. Huang, M. Zollhöfer, J. Kopf, and C. Kim, “Learning neural light fields with ray-space embedding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 19 819–19 829.
  34. Z. Li, L. Song, C. Liu, J. Yuan, and Y. Xu, “Neulf: Efficient novel view synthesis with neural 4d light field,” arXiv preprint arXiv:2105.07112, 2021.
  35. B. Y. Feng and A. Varshney, “Signet: Efficient neural representation for light fields,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 14 224–14 233.
  36. J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove, “Deepsdf: Learning continuous signed distance functions for shape representation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 165–174.
  37. L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger, “Occupancy networks: Learning 3d reconstruction in function space,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 4460–4470.
  38. Y. Xie, T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, F. Tombari, J. Tompkin, V. Sitzmann, and S. Sridhar, “Neural fields in visual computing and beyond,” in Computer Graphics Forum.   Wiley Online Library, 2022, pp. 641–676.
  39. L. Shi, C. Liu, D. He, X. Zhao, and J. Qiu, “Matching entropy based disparity estimation from light field data,” Optics Express, vol. 31, no. 4, pp. 6111–6131, 2023.
  40. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing, vol. 13, no. 4, pp. 600–612, 2004.
  41. L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Physica D: nonlinear phenomena, vol. 60, no. 1-4, pp. 259–268, 1992.
  42. S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa, “Plenoxels: Radiance fields without neural networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5501–5510.
  43. J. N. Martel, D. B. Lindell, C. Z. Lin, E. R. Chan, M. Monteiro, and G. Wetzstein, “Acorn: Adaptive coordinate networks for neural scene representation,” arXiv preprint arXiv:2105.02788, 2021.
  44. S. Wanner, S. Meister, and B. Goldluecke, “Datasets and benchmarks for densely sampled 4d light fields.” in VMV, vol. 13.   Citeseer, 2013, pp. 225–226.
  45. K. Honauer, O. Johannsen, D. Kondermann, and B. Goldluecke, “A dataset and evaluation methodology for depth estimation on 4d light fields,” in Asian conference on computer vision.   Springer, 2016, pp. 19–34.
  46. S. R. Abhilash, L. Michael, S. Raj, and W. Gordon, “Stanford lytro light field database,” http://lightfields.stanford.edu/LF2016.html.
  47. O. Johannsen, K. Honauer, B. Goldluecke, A. Alperovich, F. Battisti, Y. Bok, M. Brizzi, M. Carli, G. Choe, M. Diebold, M. Gutsche, H. Jeon, I. S. Kweon, J. Park, J. Park, H. Schilling, H. Sheng, L. Si, M. Strecke, A. Sulc, Y. Tai, Q. Wang, T. Wang, S. Wanner, Z. Xiong, J. Yu, S. Zhang, and H. Zhu, “A taxonomy and evaluation of dense light field depth estimation algorithms,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017, pp. 1795–1812.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets