Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Relating Events and Frames Based on Self-Supervised Learning and Uncorrelated Conditioning for Unsupervised Domain Adaptation (2401.01042v1)

Published 2 Jan 2024 in cs.CV

Abstract: Event-based cameras provide accurate and high temporal resolution measurements for performing computer vision tasks in challenging scenarios, such as high-dynamic range environments and fast-motion maneuvers. Despite their advantages, utilizing deep learning for event-based vision encounters a significant obstacle due to the scarcity of annotated data caused by the relatively recent emergence of event-based cameras. To overcome this limitation, leveraging the knowledge available from annotated data obtained with conventional frame-based cameras presents an effective solution based on unsupervised domain adaptation. We propose a new algorithm tailored for adapting a deep neural network trained on annotated frame-based data to generalize well on event-based unannotated data. Our approach incorporates uncorrelated conditioning and self-supervised learning in an adversarial learning scheme to close the gap between the two source and target domains. By applying self-supervised learning, the algorithm learns to align the representations of event-based data with those from frame-based camera data, thereby facilitating knowledge transfer.Furthermore, the inclusion of uncorrelated conditioning ensures that the adapted model effectively distinguishes between event-based and conventional data, enhancing its ability to classify event-based images accurately.Through empirical experimentation and evaluation, we demonstrate that our algorithm surpasses existing approaches designed for the same purpose using two benchmarks. The superior performance of our solution is attributed to its ability to effectively utilize annotated data from frame-based cameras and transfer the acquired knowledge to the event-based vision domain.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (86)
  1. Ev-segnet: Semantic segmentation for event-based cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  2. Unsupervised domain adaptation by domain invariant projection. In Proceedings of the IEEE international conference on computer vision, pages 769–776, 2013.
  3. Movement detection with event-based cameras: comparison with frame-based cameras in robot object tracking using powerlink communication. Electronics, 7(11):304, 2018.
  4. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 447–463, 2018.
  5. Unsupervised pixel-level domain adaptation with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3722–3731, 2017.
  6. Large scale gan training for high fidelity natural image synthesis, 2019.
  7. Adversarial-learned loss for domain adaptation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 3521–3528, 2020.
  8. A simple framework for contrastive learning of visual representations, 2020.
  9. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006.10029, 2020.
  10. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020.
  11. Learning to super resolve intensity images from events. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2768–2776, 2020.
  12. Optimal transport for domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(9):1853–1865, 2016.
  13. Multi-task self-supervised visual learning. In Proceedings of the IEEE international conference on computer vision, pages 2051–2060, 2017.
  14. Incorporating learnable membrane time constant to enhance learning of spiking neural networks, 2020.
  15. Learning a domain-invariant embedding for unsupervised domain adaptation using class-conditioned distribution alignment. In 2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 352–359, 2019.
  16. Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(1):154–180, 2020.
  17. Event-based camera pose tracking using a generative event model. ArXiv, abs/1510.01972, 2015.
  18. Video to events: Recycling video datasets for event cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3586–3595, 2020.
  19. End-to-end learning of representations for asynchronous event-based data, 2019.
  20. Deep reconstruction-classification networks for unsupervised domain adaptation. In European Conference on Computer Vision, pages 597–613. Springer, 2016.
  21. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
  22. Improving the classification accuracy in label-free flow cytometry using event-based vision and simple logistic regression. IEEE Journal of Selected Topics in Quantum Electronics, 29(2: Optical Computing):1–8, 2023.
  23. Bootstrap your own latent: A new approach to self-supervised learning, 2020.
  24. Deep residual learning for image recognition, 2015.
  25. Learning to exploit multiple vision modalities by using grafted networks. In European Conference on Computer Vision, pages 85–101. Springer, 2020.
  26. v2e: From video frames to realistic dvs events. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1312–1321, 2021.
  27. Unsupervised domain adaptation for training event-based networks using contrastive learning and uncorrelated conditioning. In International Conference on Computer Vision, 2023.
  28. Alexia Jolicoeur-Martineau. The relativistic discriminator: a key element missing from standard gan, 2018.
  29. Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
  30. Deep multi-wasserstein unsupervised domain adaptation. Pattern Recognition Letters, 2019.
  31. Sliced wasserstein discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10285–10295, 2019.
  32. Cifar10-dvs Li. an event-stream dataset for object classification, front.
  33. Neuromorphic data augmentation for training spiking neural networks, 2022.
  34. Exploring uncertainty in pseudo-label guided unsupervised domain adaptation. Pattern Recognition, page 106996, 2019.
  35. A 128\t⁢i⁢m⁢e⁢s⁢128120⁢d⁢b⁢15\m⁢u\absent\𝑡𝑖𝑚𝑒𝑠128120𝑑𝑏15𝑚𝑢\backslash times128120db15\backslash mu\ italic_t italic_i italic_m italic_e italic_s 128120 italic_d italic_b 15 \ italic_m italic_u s latency asynchronous temporal contrast vision sensor. IEEE journal of solid-state circuits, 43(2):566–576, 2008.
  36. On the variance of the adaptive learning rate and beyond, 2019.
  37. Learning transferable features with deep adaptation networks. In Proceedings of International Conference on Machine Learning, pages 97–105, 2015.
  38. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pages 1640–1650, 2018.
  39. Deep transfer learning with joint adaptation networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 2208–2217. JMLR. org, 2017.
  40. Event-based vision meets deep learning on steering prediction for self-driving cars. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5419–5427, 2018.
  41. UMAP: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):861, 2018.
  42. Bridging the gap between events and frames through unsupervised domain adaptation. 2022.
  43. Minimal-entropy correlation alignment for unsupervised deep domain adaptation. In ICLR, 2018.
  44. Properties of artificial networks evolved to contend with natural spectra. Proceedings of the National Academy of Sciences, 111(supplement_3):10868–10872, 2014.
  45. Event guided depth sensing. 2021.
  46. Image to image translation for domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4500–4509, 2018.
  47. Converting static image datasets to spiking neuromorphic datasets using saccades, 2015.
  48. Transferrable prototypical networks for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2239–2247, 2019.
  49. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406–1415, 2019.
  50. Da4event: towards bridging the sim-to-real gap for event cameras using domain adaptation. IEEE Robotics and Automation Letters, 6(4):6616–6623, 2021.
  51. Retinomorphic event-based vision sensors: bioinspired cameras with spiking output. Proceedings of the IEEE, 102(10):1470–1484, 2014.
  52. Dart: Distribution aware retinal transform for event-based cameras, 2017.
  53. Esim: an open event camera simulator. In Conference on robot learning, pages 969–982. PMLR, 2018.
  54. High speed and high dynamic range video with an event camera. IEEE transactions on pattern analysis and machine intelligence, 43(6):1964–1980, 2021.
  55. High speed and high dynamic range video with an event camera, 2019.
  56. A. Redko, I.and Habrard and M. Sebban. Theoretical analysis of domain adaptation with optimal transport. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 737–753. Springer, 2017.
  57. Mohammad Rostami. Lifelong domain adaptation via consolidated internal distribution. Advances in neural information processing systems, 34:11172–11183, 2021.
  58. Mohammad Rostami. Transfer Learning Through Embedding Spaces. CRC Press, 2021.
  59. Mohammad Rostami. Increasing model generalizability for unsupervised visual domain adaptation. In Conference on Lifelong Learning Agents, pages 281–293. PMLR, 2022.
  60. Domain adaptation for sentiment analysis using robust internal representations. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11484–11498, 2023.
  61. Cognitively inspired learning of incremental drifting concepts. In International Joint Conference on Artificial Intelligence, 2023.
  62. Overcoming concept shift in domain-aware settings through consolidated internal distributions. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 9623–9631, 2023.
  63. A crowdsourcing triage algorithm for geopolitical event forecasting. In Proceedings of the 12th ACM Conference on Recommender Systems, pages 377–381. ACM, 2018.
  64. Deep transfer learning for few-shot sar image classification. Remote Sensing, 11(11):1374, 2019.
  65. Sar image classification using few-shot cross-domain transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  66. Generative continual concept learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5545–5552, 2020.
  67. Hats: Histograms of averaged time surfaces for robust event-based object classification, 2018.
  68. Unsupervised model adaptation for continual semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2593–2601, 2021.
  69. Domain adaptation for the segmentation of confidential medical images. In Proceedings of the British Machine Vision Conference, 2022.
  70. Secure domain adaptation with multiple sources. Transactions on Machine Learning Research, 2022.
  71. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pages 443–450. Springer, 2016.
  72. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7167–7176, 2017.
  73. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5018–5027, 2017.
  74. Event-based high dynamic range image and very high frame rate video generation using conditional generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10081–10090, 2019.
  75. Eventsr: From asynchronous events to image reconstruction, restoration, and super-resolution via end-to-end adversarial learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8315–8325, 2020.
  76. Deep visual domain adaptation: A survey. Neurocomputing, 312:135–153, 2018.
  77. A survey of unsupervised deep domain adaptation. ACM Transactions on Intelligent Systems and Technology (TIST), 11(5):1–46, 2020.
  78. Unsupervised domain adaptation for graph-structured data using class-conditional distribution alignment. arXiv preprint arXiv:2301.12361, 2023.
  79. Simple and efficient learning using privileged information, 2016.
  80. Learning domain-shared group-sparse representation for unsupervised domain adaptation. Pattern Recognition, 81:615–632, 2018.
  81. S4l: Self-supervised semi-supervised learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1476–1485, 2019.
  82. Learning to see in the dark with events. In European Conference on Computer Vision, pages 666–682. Springer, 2020.
  83. Collaborative and adversarial network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3801–3809, 2018.
  84. Transformer-based domain adaptation for event data classification. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4673–4677. IEEE, 2022.
  85. Eventgan: Leveraging large scale image datasets for event cameras. In 2021 IEEE International Conference on Computational Photography (ICCP), pages 1–11. IEEE, 2021.
  86. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, 2017.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Mohammad Rostami (64 papers)
  2. Dayuan Jian (2 papers)

Summary

We haven't generated a summary for this paper yet.