Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
153 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Self-Supervised Monocular Depth Estimation in the Dark: Towards Data Distribution Compensation (2404.13854v1)

Published 22 Apr 2024 in cs.CV

Abstract: Nighttime self-supervised monocular depth estimation has received increasing attention in recent years. However, using night images for self-supervision is unreliable because the photometric consistency assumption is usually violated in the videos taken under complex lighting conditions. Even with domain adaptation or photometric loss repair, performance is still limited by the poor supervision of night images on trainable networks. In this paper, we propose a self-supervised nighttime monocular depth estimation method that does not use any night images during training. Our framework utilizes day images as a stable source for self-supervision and applies physical priors (e.g., wave optics, reflection model and read-shot noise model) to compensate for some key day-night differences. With day-to-night data distribution compensation, our framework can be trained in an efficient one-stage self-supervised manner. Though no nighttime images are considered during training, qualitative and quantitative results demonstrate that our method achieves SoTA depth estimating results on the challenging nuScenes-Night and RobotCar-Night compared with existing methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. Attention attention everywhere: Monocular depth prediction with skip attention. In WACV, pages 5861–5870, 2023.
  2. Recent advances in augmented reality. IEEE Computer Graphics and Applications, 21(6):34–47, 2001.
  3. Deep digging into the generalization of self-supervised monocular depth estimation. In AAAI, 2023.
  4. Adabins: Depth estimation using adaptive bins. In CVPR, pages 4009–4018, 2021.
  5. Unprocessing images for learned raw denoising. In CVPR, pages 11036–11045, 2019.
  6. nuscenes: A multimodal dataset for autonomous driving. In CVPR, pages 11621–11631, 2020.
  7. Learning to see in the dark. In CVPR, pages 3291–3300, 2018.
  8. Towards scene understanding: Unsupervised monocular depth estimation with semantic-aware representation. In CVPR, pages 2624–2632, 2019.
  9. Flare7k: A phenomenological nighttime flare removal dataset. In NeurIPS, 2022.
  10. How do neural networks see depth in single images? In ICCV, 2019.
  11. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv, 2020.
  12. Depth map prediction from a single image using a multi-scale deep network. Advances in Neural Information Processing Systems, 27, 2014.
  13. Learnability enhancement for low-light raw denoising: Where paired real data meets noise modeling. In ACM MM, pages 1436–1444, 2022.
  14. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):1231–1237, 2013.
  15. Digging into self-supervised monocular depth estimation. In ICCV, pages 3828–3838, 2019.
  16. Joseph W Goodman. Introduction to Fourier optics. Roberts and Company publishers, 2005.
  17. 3d packing for self-supervised monocular depth estimation. In CVPR, pages 2485–2494, 2020.
  18. Semantically-guided representation learning for self-supervised monocular depth. arXiv, 2020.
  19. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016.
  20. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NIPS, page 6629–6640, 2017.
  21. Xueqi Hu et al. Qs-attn: Query-selected attention for contrastive learning in i2i translation. In CVPR, 2022.
  22. Neighbor2neighbor: Self-supervised denoising from single noisy images. In CVPR, pages 14781–14790, 2021.
  23. Neighbor2neighbor: A self-supervised framework for deep image denoising. IEEE Transactions on Image Processing, 31:4023–4038, 2022.
  24. Spatial transformer networks. Advances in Neural Information Processing Systems, 28, 2015.
  25. Some properties of the range in samples from tukey’s symmetric lambda distributions. Journal of the American Statistical Association, 66(334):394–399, 1971.
  26. Fine-grained semantics-aware representation enhancement for self-supervised monocular depth estimation. In ICCV, pages 12642–12652, 2021.
  27. Glare generation based on wave optics. In PG, pages 133–140. IEEE, 2004.
  28. Self-supervised monocular depth estimation: Solving the dynamic object problem by semantic guidance. In ECCV, pages 582–600, 2020.
  29. Learning monocular depth in dynamic scenes via instance-aware projection consistency. In AAAI, pages 1863–1872, 2021.
  30. Unsupervised monocular depth learning in dynamic scenes. In CoRL, pages 1908–1917, 2021.
  31. Optical physics. Cambridge University Press, 2010.
  32. Self-supervised monocular depth estimation for all day images using domain separation. In ICCV, pages 12737–12746, 2021.
  33. Decoupled weight decay regularization. arXiv, 2017.
  34. Gaze-dependent simulation of light perception in virtual reality. IEEE Transactions on Visualization and Computer Graphics, 26(12):3557–3567, 2020.
  35. Hr-depth: High resolution self-supervised monocular depth estimation. In AAAI, pages 2294–2301, 2021.
  36. Towards comprehensive representation enhancement in semantics-guided self-supervised monocular depth estimation. In ECCV, pages 304–321, 2022.
  37. 1 year, 1000 km: The oxford robotcar dataset. The International Journal of Robotics Research, 36(1):3–15, 2017.
  38. Object scene flow for autonomous vehicles. In CVPR, pages 3061–3070, 2015.
  39. Multi-modal sensing and robotic manipulation of non-rigid objects: A survey. Robotics, 7(4):74, 2018.
  40. Taesung Park et al. Contrastive learning for unpaired image-to-image translation. In ECCV, 2020.
  41. Exploiting pseudo labels in a self-supervised learning framework for improved monocular depth estimation. In CVPR, pages 1578–1588, 2022.
  42. Bui Tuong Phong. Illumination for computer generated pictures. Communications of the ACM, 18(6):311–317, 1975.
  43. Wim Ruyten. Smear correction for frame transfer charge-coupled-device cameras. Optics Letters, 24(13):878–880, 1999.
  44. Guided curriculum model adaptation and uncertainty-aware evaluation for semantic nighttime image segmentation. In ICCV, 2019.
  45. The monocular depth estimation challenge. In WACV, pages 623–632, 2023.
  46. Night-time scene parsing with a large real dataset. IEEE Transactions on Image Processing, 30:9085–9098, 2021.
  47. Histogram of oriented normal vectors for object recognition with a depth sensor. In ACCV, pages 525–538. Springer, 2013.
  48. Unsupervised monocular depth estimation for night-time images using adversarial domain feature adaptation. In ECCV, pages 443–459, 2020.
  49. When the sun goes down: Repairing photometric losses for all-day depth estimation. In Conference on Robot Learning, pages 1992–2003, 2023.
  50. Self-supervised scale recovery for monocular depth and egomotion estimation. In IROS, pages 2620–2627, 2021.
  51. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  52. Regularizing nighttime weirdness: Efficient self-supervised monocular depth estimation in the dark. In ICCV, pages 16055–16064, 2021.
  53. A physics-based noise formation model for extreme low-light raw denoising. In CVPR, pages 2758–2767, 2020.
  54. Physics-based noise modeling for extreme low-light photography. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
  55. Toward hierarchical self-supervised monocular absolute depth estimation for autonomous driving applications. In IROS, pages 2330–2337, 2020.
  56. Channel-wise attention-based network for self-supervised monocular depth estimation. In 3DV, pages 464–473, 2021.
  57. Neural window fully-connected crfs for monocular depth estimation. In CVPR, pages 3916–3925, 2022.
  58. Rethinking noise synthesis and modeling in raw denoising. In ICCV, pages 4593–4601, 2021.
  59. Monocular depth estimation based on deep learning: An overview. Science China Technological Sciences, 63(9):1612–1627, 2020.
  60. Unsupervised monocular depth estimation in highly complex environments. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(5):1237–1246, 2022.
  61. Monovit: Self-supervised monocular depth estimation with a vision transformer. In 3DV, 2022.
  62. Unsupervised learning of depth and ego-motion from video. In CVPR, pages 1851–1858, 2017.
  63. Unpaired image-to-image translation using cycle-consistent adversarial networks. In CVPR, pages 2223–2232, 2017.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now