Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
153 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Single View Refractive Index Tomography with Neural Fields (2309.04437v2)

Published 8 Sep 2023 in cs.CV and astro-ph.CO

Abstract: Refractive Index Tomography is the inverse problem of reconstructing the continuously-varying 3D refractive index in a scene using 2D projected image measurements. Although a purely refractive field is not directly visible, it bends light rays as they travel through space, thus providing a signal for reconstruction. The effects of such fields appear in many scientific computer vision settings, ranging from refraction due to transparent cells in microscopy to the lensing of distant galaxies caused by dark matter in astrophysics. Reconstructing these fields is particularly difficult due to the complex nonlinear effects of the refractive field on observed images. Furthermore, while standard 3D reconstruction and tomography settings typically have access to observations of the scene from many viewpoints, many refractive index tomography problem settings only have access to images observed from a single viewpoint. We introduce a method that leverages prior knowledge of light sources scattered throughout the refractive medium to help disambiguate the single-view refractive index tomography problem. We differentiably trace curved rays through a neural field representation of the refractive field, and optimize its parameters to best reproduce the observed image. We demonstrate the efficacy of our approach by reconstructing simulated refractive fields, analyze the effects of light source distribution on the recovered field, and test our method on a simulated dark matter mapping problem where we successfully recover the 3D refractive field caused by a realistic dark matter distribution.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (40)
  1. Passive tomography of turbulence strength. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part IV 13, pages 47–60. Springer, 2014a.
  2. Stella maris: Stellar marine refractive imaging sensor. In 2014 IEEE International Conference on Computational Photography (ICCP), pages 1–10. IEEE, 2014b.
  3. Refractive radiative transfer equation. ACM Transactions on Graphics (TOG), 33(2):1–22, 2014.
  4. Time-resolved 3d capture of non-stationary gas flows. ACM transactions on graphics (TOG), 27(5):1–9, 2008.
  5. Weak gravitational lensing. Physics Reports, 340(4-5):291–472, 2001.
  6. Eikonal fields for refractive novel-view synthesis. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1–9, 2022.
  7. JAX: composable transformations of Python+NumPy programs, 2018.
  8. Robert Bridson. Fast poisson disk sampling in arbitrary dimensions. SIGGRAPH sketches, 10(1):1, 2007.
  9. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
  10. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5939–5948, 2019.
  11. Tomographic phase microscopy. Nature methods, 4(9):717–719, 2007.
  12. High-resolution 3d refractive index microscopy of multiple-scattering samples from intensity images. Optica, 6(9):1211–1219, 2019.
  13. A family of embedded runge-kutta formulae. Journal of computational and applied mathematics, 6(1):19–26, 1980.
  14. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015.
  15. Weak gravitational lensing and its cosmological applications. Annual Review of Nuclear and Particle Science, 58:99–123, 2008.
  16. Eikonal rendering: Efficient light transport in refractive objects. ACM Transactions on Graphics (TOG), 26(3):59–es, 2007.
  17. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  18. Euclid definition study report. arXiv preprint arXiv:1110.3193, 2011.
  19. A compressed sensing approach to 3d weak lensing. Astronomy & Astrophysics, 539:A85, 2012.
  20. Glimpse: accurate 3d weak lensing reconstructions using sparsity. Monthly Notices of the Royal Astronomical Society, 440(2):1281–1294, 2014.
  21. Gravitationally lensed black hole emission tomography. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19841–19850, 2022.
  22. End-to-end learned single lens design using fast differentiable ray tracing. Optics Letters, 46(21):5453–5456, 2021.
  23. Eikonal tomography: surface wave tomography by phase front tracking across a regional broad-band seismic array. Geophysical Journal International, 177(3):1091–1110, 2009.
  24. Recovery of continuous 3d refractive index maps from discrete intensity-only measurements using neural fields. Nature Machine Intelligence, 4(9):781–791, 2022.
  25. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021.
  26. Lectures on gravitational lensing. arXiv preprint astro-ph/9606001, 1996.
  27. The illustris simulation: Public data release. Astronomy and Computing, 13:12–37, 2015.
  28. Path tracing estimators for refractive radiative transfer. ACM Transactions on Graphics (TOG), 39(6):1–15, 2020.
  29. Simulating galaxy formation with the illustristng model. Monthly Notices of the Royal Astronomical Society, 473(3):4077–4106, 2018.
  30. Lev Semenovich Pontryagin. Mathematical theory of optimal processes. CRC press, 1987.
  31. Gradient-index fiber-optic microprobes for minimally invasive in vivo low-coherence interferometry. Optics letters, 27(20):1794–1796, 2002.
  32. Gravitational lenses [electronic resource].
  33. Unfolding the matter distribution using three-dimensional weak gravitational lensing. Monthly Notices of the Royal Astronomical Society, 399(1):48–68, 2009.
  34. Optical diffraction tomography for high resolution live cell imaging. Optics express, 17(1):266–277, 2009.
  35. Masahiro Takada. Subaru hyper suprime-cam project. In AIP Conference Proceedings, pages 120–127. American Institute of Physics, 2010.
  36. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537–7547, 2020.
  37. Adjoint nonlinear ray tracing. ACM Transactions on Graphics (TOG), 41(4):1–13, 2022.
  38. Differentiable compound optics and processing pipeline optimization for end-to-end camera design. ACM Transactions on Graphics (TOG), 40(2):1–19, 2021.
  39. Three-dimensional reconstruction of the density field: An svd approach to weak-lensing tomography. The Astrophysical Journal, 727(2):118, 2011.
  40. Refraction wiggles for measuring fluid depth and velocity from video. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part III 13, pages 767–782. Springer, 2014.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now