Papers
Topics
Authors
Recent
Search
2000 character limit reached

Hologram: Realtime Holographic Overlays via LiDAR Augmented Reconstruction

Published 12 May 2024 in cs.CV | (2405.07178v1)

Abstract: Guided by the hologram technology of the infamous Star Wars franchise, I present an application that creates real-time holographic overlays using LiDAR augmented 3D reconstruction. Prior attempts involve SLAM or NeRFs which either require highly calibrated scenes, incur steep computation costs, or fail to render dynamic scenes. I propose 3 high-fidelity reconstruction tools that can run on a portable device, such as a iPhone 14 Pro, which can allow for metric accurate facial reconstructions. My systems enable interactive and immersive holographic experiences that can be used for a wide range of applications, including augmented reality, telepresence, and entertainment.

Authors (1)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (72)
  1. Jens Ahremark and Simon Bazso. 2022. Benchmarking a machine learning model in the transformation from PyTorch to CoreML.
  2. Real-time dynamic holographic display using phase-only spatial light modulator. In Proceedings of the SPIE Photonics West, Vol. 10944. 109440D.
  3. Paul J Besl and Neil D McKay. 1992. A method for registration of 3-D shapes. In IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 14. IEEE, 239–256.
  4. Stan Birchfield and Carlo Tomasi. 1999. Depth discontinuities by pixel-to-pixel stereo. International Journal of Computer Vision 35 (1999), 269–293.
  5. Midas v3. 1–a model zoo for robust monocular relative depth estimation. arXiv preprint arXiv:2307.14460 (2023).
  6. Facewarehouse: A 3d facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics 20, 3 (2013), 413–425.
  7. Structure-Aware Residual Pyramid Depth Estimation. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence.
  8. MonoDepthFormers: Transformers for Computationally Efficient Monocular Depth Estimation. arXiv preprint arXiv:2209.07688 (2022).
  9. Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration. ACM Transactions on Graphics (ToG) 36, 4 (2017), 1.
  10. Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence 38, 2 (2015), 295–307.
  11. Depth map prediction from a single image using a multi-scale deep network. In Advances in Neural Information Processing Systems. 2366–2374.
  12. Efficient neural radiance fields. In arXiv preprint arXiv:2111.07931.
  13. Enhanced computer vision with microsoft kinect sensor: A review. IEEE transactions on cybernetics 43, 5 (2013), 1318–1334.
  14. SneRG: Synthesizing normals to render geometry with learned differentiable intersection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8506–8516.
  15. Revisiting single image depth estimation: Toward higher resolution maps with semi-supervised learning. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 1949–1958.
  16. Recurrent slice networks for 3d segmentation of point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2626–2635.
  17. Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In Proceedings of the 24th annual ACM symposium on User interface software and technology. 559–568.
  18. LiDAR Point Cloud Reconstruction and Completion via RGB-Guided Voxel Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8766–8775.
  19. Leela S Karumbunathan. 2022. NVIDIA Jetson AGX Orin Series.
  20. Poisson surface reconstruction. In Proceedings of the Fourth Eurographics Symposium on Geometry Processing, Vol. 7.
  21. Repurposing diffusion-based image generators for monocular depth estimation. arXiv preprint arXiv:2312.02145 (2023).
  22. GC-Net: Geometric correspondence network for disparity estimation. The British Machine Vision Conference (BMVC) (2017).
  23. Method for measuring stereo camera depth accuracy based on stereoscopic vision. In Three-dimensional imaging, interaction, and measurement, Vol. 7864. SPIE, 168–176.
  24. Deeper depth prediction with fully convolutional residual networks. In 2016 Fourth International Conference on 3D Vision (3DV). IEEE, 239–248.
  25. Neural 3D video synthesis from a single semantic image. In arXiv preprint arXiv:2103.07953.
  26. Learning depth from single monocular images using deep convolutional neural fields. In IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 38. IEEE, 2024–2039.
  27. Neural sparse voxel fields. In Advances in Neural Information Processing Systems, Vol. 33. 15651–15663.
  28. Hugh Christopher Longuet-Higgins. 1981. A computer algorithm for reconstructing a scene from two projections. Nature 293, 5828 (1981), 133–135.
  29. Michael Lucente. 1994. Holographic optical elements for optical data processing. Optical Engineering 33, 9 (1994), 2768–2779.
  30. Michael Lucente. 1995. Optimization of hologram computation for real-time photorealistic augmented reality. In Practical Holography XI, Vol. 2652. International Society for Optics and Photonics, 309–314.
  31. Evaluation of the Apple iPhone 12 Pro LiDAR for an application in geosciences. Scientific reports 11, 1 (2021), 1–9.
  32. LiDAR-Guided Autonomous Vehicle Trajectory Prediction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 1375–1384.
  33. David Marr and Tomaso Poggio. 1976. Cooperative computation of stereo disparity. Science 194, 4262 (1976), 283–287.
  34. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In European Conference on Computer Vision (ECCV). Springer.
  35. Instant neural graphics primitives with a multiresolution hash encoding. In ACM Transactions on Graphics (TOG), Vol. 41. ACM, 1–15.
  36. Real-time 3D reconstruction at scale using voxel hashing. ACM Transactions on Graphics (ToG) 32, 6 (2013), 1–11.
  37. Efficient Parallel Hierarchical Radiosity on a Cluster. In Proceedings of the ACM SIGGRAPH Symposium on High Performance Graphics. 143–152.
  38. A computational model of fresco for 3D reconstruction. In ACM Transactions on Graphics (TOG), Vol. 28. ACM, 1–9.
  39. Probuilding: A robust and memory-efficient voxel reconstruction engine for 3d-scanning systems. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 137–147.
  40. Nick Pears and Christian Duncan. 2016. Automatic 3D modelling of craniofacial form. arXiv:1601.05593 [cs.CV]
  41. Deep learning for computer-generated holography. Advanced Photonics 2, 4 (2020), 046005.
  42. Detailed real-time urban 3D reconstruction from video. International Journal of Computer Vision 78, 2 (2008), 142–167.
  43. Infinitam v3: A framework for large-scale 3d reconstruction with loop closure. arXiv preprint arXiv:1708.00783 (2017).
  44. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10318–10327.
  45. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 12179–12193.
  46. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence 44, 3 (2020), 1623–1637.
  47. DERF: Deformable Neural Radiance Fields. arXiv preprint arXiv:2208.00885 (2022).
  48. Hardware-based rendering of full-parallax synthetic holograms. Applied Optics 38, 8 (1999), 1364–1369.
  49. Monocular depth estimation using diffusion models. arXiv preprint arXiv:2302.14816 (2023).
  50. Daniel Scharstein and Richard Szeliski. 2003. High-accuracy stereo depth maps using structured light. In 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings., Vol. 1. IEEE, I–I.
  51. Staticfusion: Background reconstruction for dense rgb-d slam in dynamic environments. In 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 3849–3856.
  52. Cell-based hardware architecture for full-parallel generation algorithm of digital holograms. Optics Express 19, 9 (2011), 8750–8761.
  53. Towards real-time photorealistic 3D holography with deep generative networks. Nature 591, 7849 (2021), 258–262.
  54. Special-purpose computer for holography HORN-4 with recurrence algorithm. Computer physics communications 148, 2 (2002), 160–170.
  55. Special-purpose computer for holography HORN-3 with PLD technology. Computer physics communications 130, 1-2 (2000), 75–82.
  56. Implicit neural representations with periodic activation functions. In Advances in Neural Information Processing Systems, Vol. 33. 7145–7156.
  57. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7495–7504.
  58. Barf: Bundle-adjusting neural radiance fields. arXiv preprint arXiv:2205.11975 (2022).
  59. Carlo Tomasi and Takeo Kanade. 1992. Shape and motion from image streams under orthography: a factorization method. International Journal of Computer Vision 9, 2 (1992), 137–154.
  60. Alex Trevithick and Bo Yang. 2021. GRF: Learning a general radiance field for 3D scene representation and rendering. In arXiv preprint arXiv:2010.04595.
  61. DeMoN: Depth and motion network for learning monocular stereo. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5038–5047.
  62. DFNet: Exploiting depth-wise separable convolutions for monocular depth estimation. In IEEE Transactions on Image Processing, Vol. 30. IEEE, 4886–4898.
  63. ElasticFusion: Dense SLAM without a pose graph.. In Robotics: science and systems, Vol. 11. Rome, Italy, 3.
  64. Towards complete 3d scene reconstruction from egocentric RGB-D semantic mapping. In 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 3556–3563.
  65. Go-ICP: A globally optimal solution to 3D ICP point-set registration. In IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 38. IEEE, 2241–2254.
  66. Pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4578–4587.
  67. A computational model for the integration of linked data in mobile augmented reality applications. In Proceedings of the 8th International Conference on Semantic Systems. 133–140.
  68. Jure Zbontar and Yann LeCun. 2016. Stereo matching by training a convolutional neural network to compare image patches. The journal of machine learning research 17, 1 (2016), 2287–2318.
  69. Panoramic image and three-axis laser scanner integrated approach for indoor 3D mapping. Remote Sensing 10, 8 (2018), 1269.
  70. Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1851–1858.
  71. Instant Volumetric Head Avatars. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022), 4574–4584. https://api.semanticscholar.org/CorpusID:253761096
  72. Towards Metrical Reconstruction of Human Faces. In European Conference on Computer Vision. https://api.semanticscholar.org/CorpusID:248177832

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 1 like about this paper.

Sign Up for Free