Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
133 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Event-Free Moving Object Segmentation from Moving Ego Vehicle (2305.00126v3)

Published 28 Apr 2023 in cs.CV and cs.RO

Abstract: Moving object segmentation (MOS) in dynamic scenes is an important, challenging, but under-explored research topic for autonomous driving, especially for sequences obtained from moving ego vehicles. Most segmentation methods leverage motion cues obtained from optical flow maps. However, since these methods are often based on optical flows that are pre-computed from successive RGB frames, this neglects the temporal consideration of events occurring within the inter-frame, consequently constraining its ability to discern objects exhibiting relative staticity but genuinely in motion. To address these limitations, we propose to exploit event cameras for better video understanding, which provide rich motion cues without relying on optical flow. To foster research in this area, we first introduce a novel large-scale dataset called DSEC-MOS for moving object segmentation from moving ego vehicles, which is the first of its kind. For benchmarking, we select various mainstream methods and rigorously evaluate them on our dataset. Subsequently, we devise EmoFormer, a novel network able to exploit the event data. For this purpose, we fuse the event temporal prior with spatial semantic maps to distinguish genuinely moving objects from the static background, adding another level of dense supervision around our object of interest. Our proposed network relies only on event data for training but does not require event input during inference, making it directly comparable to frame-only methods in terms of efficiency and more widely usable in many application cases. The exhaustive comparison highlights a significant performance improvement of our method over all other methods. The source code and dataset are publicly available at: https://github.com/ZZY-Zhou/DSEC-MOS.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (74)
  1. Vista 2.0: An open, data-driven simulator for multimodal sensing and policy learning for autonomous vehicles. In ICRA. IEEE, 2022.
  2. Time-ordered recent event (tore) volumes for event cameras. IEEE TPAMI, 45(2):2519–2532, 2022.
  3. It’s moving! a probabilistic model for causal motion segmentation in moving camera videos. In ECCV. Springer, 2016.
  4. The 2018 davis challenge on video object segmentation. arXiv preprint arXiv:1803.00557, 2018.
  5. Event-based neuromorphic vision for autonomous driving: A paradigm shift for bio-inspired visual sensing and perception. IEEE SPM, 37(4):34–49, 2020.
  6. Segflow: Joint learning for video object segmentation and optical flow. In ICCV, 2017.
  7. Unsupervised video anomaly detection via normalizing flows with implicit latent features. PR, 129:108703, 2022.
  8. Treating motion as option to reduce motion dependency in unsupervised video object segmentation. In WACV, 2023.
  9. Mose: A new dataset for video object segmentation in complex scenes. In ICCV, 2023.
  10. Spatio-temporal recurrent networks for event-based optical flow estimation. In AAAI, 2022.
  11. Event-based vision: A survey. IEEE TPAMI, 44(1):154–180, 2020.
  12. Dsec: A stereo event camera dataset for driving scenarios. IEEE RAL, 6(3):4947–4954, 2021.
  13. Low cost and latency event camera background activity denoising. IEEE TPAMI, 45(1):785–795, 2022.
  14. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
  15. Unsupervised video object segmentation using motion saliency-guided spatio-temporal propagation. In ECCV, 2018.
  16. Flowformer: A transformer architecture for optical flow. In ECCV. Springer, 2022.
  17. Flownet 2.0: Evolution of optical flow estimation with deep networks. In CVPR, 2017.
  18. Event-based semantic segmentation with posterior attention. IEEE TIP, 32:1829–1842, 2023.
  19. Med-vt: Multiscale encoder-decoder video transformer with application to object segmentation. In CVPR, 2023.
  20. Segment anything. In ICCV, 2023.
  21. Panoptic neural fields: A semantic object-aware neural scene representation. In CVPR, 2022.
  22. Betrayed by motion: Camouflaged object discovery via motion segmentation. In ACCV, 2020.
  23. Mpi-flow: Learning realistic optical flow with multiplane images. In ICCV, 2023.
  24. F2net: Learning to focus on the foreground for unsupervised video object segmentation. In AAAI, 2021.
  25. Video swin transformer. In CVPR, 2022.
  26. See more, know more: Unsupervised video object segmentation with co-attention siamese networks. In CVPR, 2019.
  27. Event-based vision meets deep learning on steering prediction for self-driving cars. In CVPR, 2018.
  28. Bridging the gap between events and frames through unsupervised domain adaptation. IEEE RAL, 7(2):3515–3522, 2022.
  29. Ev-imo: Motion segmentation dataset and learning pipeline for event cameras. In IROS. IEEE, 2019.
  30. Learning-based video motion magnification. In ECCV, 2018.
  31. Hierarchical feature alignment network for unsupervised video object segmentation. In ECCV. Springer, 2022.
  32. A benchmark dataset and evaluation methodology for video object segmentation. In CVPR, 2016.
  33. Analytic collision risk calculation for autonomous vehicle navigation. In ICRA. IEEE, 2019.
  34. E2 (go) motion: Motion augmented event stream for egocentric action recognition. In CVPR, 2022.
  35. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675, 2017.
  36. Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation. In CVPR, 2019.
  37. Reciprocal transformations for unsupervised video object segmentation. In CVPR, 2021.
  38. Secrets of event-based optical flow. In ECCV. Springer, 2022.
  39. Multi domain learning for motion magnification. In CVPR, 2023.
  40. Event-based motion segmentation by motion compensation. In ICCV, 2019.
  41. Unsupervised moving object detection in complex scenes using adversarial regularizations. IEEE TMM, 23:2005–2018, 2020.
  42. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In CVPR, 2018.
  43. Mining relations among cross-frame affinities for video semantic segmentation. In ECCV. Springer, 2022a.
  44. Ess: Learning event-based semantic segmentation from still images. In ECCV. Springer, 2022b.
  45. L2e: Lasers to events for 6-dof extrinsic calibration of lidars and event cameras. In ICRA. IEEE, 2023.
  46. Block-nerf: Scalable large scene neural view synthesis. In CVPR, 2022.
  47. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV. Springer, 2020.
  48. Learning motion patterns in videos. In CVPR, 2017.
  49. Learning to segment moving objects. IJCV, 127:282–301, 2019.
  50. Fusing event-based and rgb camera for robust object detection in adverse conditions. In ICRA. IEEE, 2022.
  51. Video segmentation via object flow. In CVPR, 2016.
  52. Attention is all you need. NeurIPS, 30, 2017.
  53. Learning unsupervised video object segmentation through visual attention. In CVPR, 2019.
  54. Future video synthesis with object motion prediction. In CVPR, 2020.
  55. Cmda: Cross-modality domain adaptation for nighttime semantic segmentation. In ICCV, 2023.
  56. Object discovery in videos as foreground motion clustering. In CVPR, 2019.
  57. Segmenting moving objects via an object-centric layered representation. NeurIPS, 2022.
  58. Deep flow-guided video inpainting. In CVPR, 2019.
  59. Learning motion-appearance co-attention for zero-shot video object segmentation. In ICCV, 2021a.
  60. Unsupervised moving object detection via contextual information separation. In CVPR, 2019.
  61. Dystab: Unsupervised object segmentation via dynamic-static bootstrapping. In CVPR, 2021b.
  62. Associating objects with transformers for video object segmentation. NeurIPS, 34:2491–2502, 2021c.
  63. Isomer: Isomerous transformer for zero-shot video object segmentation. In ICCV, 2023.
  64. Issafe: Improving semantic segmentation in accidents by fusing event-based data. In IROS. IEEE, 2021a.
  65. Deep transport network for unsupervised video object segmentation. In ICCV, 2021b.
  66. A multi-scale recurrent framework for motion segmentation with event camera. IEEE Access, 2023.
  67. Learning discriminative feature with crf for unsupervised video object segmentation. In ECCV. Springer, 2020.
  68. Motion-attentive transition for zero-shot video object segmentation. In AAAI, 2020.
  69. A survey on deep learning technique for video segmentation. IEEE TPAMI, 45(6):7099–7122, 2022.
  70. Event-based motion segmentation with spatio-temporal graph cuts. IEEE TNNLS, 2021a.
  71. Flow-edge guided unsupervised video object segmentation. IEEE TCSVT, 32(12):8116–8127, 2021b.
  72. Rgb-event fusion for moving object detection in autonomous driving. In ICRA. IEEE, 2023.
  73. The multivehicle stereo event camera dataset: An event camera dataset for 3d perception. IEEE RAL, 3(3):2032–2039, 2018.
  74. Devo: Depth-event camera visual odometry in challenging conditions. In ICRA. IEEE, 2022.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - ZZY-Zhou/DSEC-MOS (21 stars)