2000 character limit reached
ANAVI: Audio Noise Awareness using Visuals of Indoor environments for NAVIgation (2410.18932v1)
Published 24 Oct 2024 in cs.RO, cs.AI, and cs.CV
Abstract: We propose Audio Noise Awareness using Visuals of Indoors for NAVIgation for quieter robot path planning. While humans are naturally aware of the noise they make and its impact on those around them, robots currently lack this awareness. A key challenge in achieving audio awareness for robots is estimating how loud will the robot's actions be at a listener's location? Since sound depends upon the geometry and material composition of rooms, we train the robot to passively perceive loudness using visual observations of indoor environments. To this end, we generate data on how loud an 'impulse' sounds at different listener locations in simulated homes, and train our Acoustic Noise Predictor (ANP). Next, we collect acoustic profiles corresponding to different actions for navigation. Unifying ANP with action acoustics, we demonstrate experiments with wheeled (Hello Robot Stretch) and legged (Unitree Go2) robots so that these robots adhere to the noise constraints of the environment. See code and data at https://anavi-corl24.github.io/
- iRobot Home Support. Article 32709 from irobot home support. https://homesupport.irobot.com/s/article/32709, 2024.
- Sonicverse: A multisensory simulation platform for training household agents that see and hear. In ICRA, 2023.
- SAVi++: Towards end-to-end object-centric learning from real-world videos. In Advances in Neural Information Processing Systems, 2022.
- Soundspaces 2.0: A simulation platform for visual-acoustic learning. NeuriPS 2022 Datasets and Benchmarks Track, 2022.
- Calculating the acoustical room response by the use of a ray tracing technique. Journal of Sound and Vibration, 8:118–125, 1968. URL https://api.semanticscholar.org/CorpusID:89608438.
- P. J. Huber. Robust Estimation of a Location Parameter. The Annals of Mathematical Statistics, 35(1):73 – 101, 1964. doi:10.1214/aoms/1177703732. URL https://doi.org/10.1214/aoms/1177703732.
- Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
- Matterport3d: Learning from rgb-d data in indoor environments. International Conference on 3D Vision (3DV), 2017.
- W. Lin and G. Ghinea. Progress and opportunities in modelling just-noticeable difference (jnd) for multimedia. IEEE Transactions on Multimedia, 24:3706–3721, 2021.
- Evaluation of smartphone sound measurement applicationsa). The Journal of the Acoustical Society of America, 135(4):EL186–EL192, 03 2014. ISSN 0001-4966. doi:10.1121/1.4865269. URL https://doi.org/10.1121/1.4865269.
- A. Owens and A. A. Efros. Audio-visual scene analysis with self-supervised multisensory features. In Proceedings of the European conference on computer vision (ECCV), pages 631–648, 2018.
- Novel-view acoustic synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6409–6419, 2023.
- Visually indicated sounds. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2405–2413, 2016.
- Soundingactions: Learning how actions sound from narrated egocentric videos. arXiv preprint arXiv:2404.05206, 2024.
- Learning state-aware visual representations from audible interactions. Advances in Neural Information Processing Systems, 35:23765–23779, 2022.
- Visual acoustic matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18858–18868, 2022.
- Learning representations from audio-visual spatial alignment. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4733–4744. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/328e5d4c166bb340b314d457a208dc83-Paper.pdf.
- Learning audio-visual dereverberation. In ICASSP, 2023.
- Sim2real transfer for audio-visual navigation with frequency-adaptive acoustic field prediction. arXiv preprint arXiv:2405.02821, 2024.
- Move2hear: Active audio-visual source separation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 275–285, October 2021.
- Real acoustic fields: An audio-visual room acoustics dataset and benchmark. 2024.
- Learning audio feedback for estimating amount and flow of granular material. In A. Billard, A. Dragan, J. Peters, and J. Morimoto, editors, Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pages 529–550. PMLR, 29–31 Oct 2018. URL https://proceedings.mlr.press/v87/clarke18a.html.
- A. Thankaraj and L. Pinto. That sounds right: Auditory self-supervision for dynamic robot manipulation. arXiv preprint arXiv:2210.01116, 2022.
- Swoosh! rattle! thump!–actions that sound. arXiv preprint arXiv:2007.01851, 2020.
- Play it by ear: Learning skills amidst occlusion through audio-visual imitation learning. arXiv preprint arXiv:2205.14850, 2022.
- How do i sound like? forward models for robot ego-noise prediction. pages 246–251, 09 2016. doi:10.1109/DEVLRN.2016.7846826.
- Audio visual language maps for robot navigation. In Proceedings of the International Symposium on Experimental Robotics (ISER), Chiang Mai, Thailand, 2023.
- Blind as a bat: Audible echolocation on small robots. IEEE Robotics and Automation Letters, 8(3):1271–1278, 2022.
- Yell at your robot: Improving on-the-fly from language corrections. arXiv preprint arXiv:2403.12910, 2024.
- Learning visual-audio representations for voice-controlled robots. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9508–9514. IEEE, 2023.
- Q. Zhang. The application of audio control in social robotics. In Proceedings of the 2022 4th International Conference on Robotics, Intelligent Control and Artificial Intelligence, RICAI ’22, page 963–966, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450398343. doi:10.1145/3584376.3584548. URL https://doi.org/10.1145/3584376.3584548.
- L. Cao and J. Gross. Cultural differences in perceiving sounds generated by others: Self matters. Frontiers in psychology, 6:163665, 2015.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.