Papers
Topics
Authors
Recent
Search
2000 character limit reached

SoundShift: Exploring Sound Manipulations for Accessible Mixed-Reality Awareness

Published 20 Jan 2024 in cs.HC, cs.SD, and eess.AS | (2401.11095v2)

Abstract: Mixed-reality (MR) soundscapes blend real-world sound with virtual audio from hearing devices, presenting intricate auditory information that is hard to discern and differentiate. This is particularly challenging for blind or visually impaired individuals, who rely on sounds and descriptions in their everyday lives. To understand how complex audio information is consumed, we analyzed online forum posts within the blind community, identifying prevailing challenges, needs, and desired solutions. We synthesized the results and propose SoundShift for increasing MR sound awareness, which includes six sound manipulations: Transparency Shift, Envelope Shift, Position Shift, Style Shift, Time Shift, and Sound Append. To evaluate the effectiveness of SoundShift, we conducted a user study with 18 blind participants across three simulated MR scenarios, where participants identified specific sounds within intricate soundscapes. We found that SoundShift increased MR sound awareness and minimized cognitive load. Finally, we developed three real-world example applications to demonstrate the practicality of SoundShift.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (80)
  1. 2023a. AirPods redefine the personal audio experience. https://www.apple.com/newsroom/2023/06/airpods-redefine-the-personal-audio-experience/
  2. 2023b. AppleVis. https://www.applevis.com/
  3. 2023a. Blind and Visually Impaired Community. https://www.reddit.com/r/Blind/
  4. 2023b. BlindSquare. https://www.blindsquare.com/
  5. 2023. Microsoft Soundscape. https://www.microsoft.com/en-us/research/product/soundscape/
  6. Sound for 3D cinema and the sense of presence. Georgia Institute of Technology.
  7. Greg Ballou. 1987. Handbook for Sound Engineers: The New Audio Cyclopedia, Howard W. Sams and Co., Indianapolis (1987).
  8. Earcons and icons: Their structure and common design principles. Human–Computer Interaction 4, 1 (1989), 11–44.
  9. The effect of 3D audio and other audio techniques on virtual reality experience. Annual Review of Cybertherapy and Telemedicine 2015 (2015), 44–48.
  10. OmniScribe: Authoring Immersive Audio Descriptions for 360° Videos. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (Bend, OR, USA) (UIST ’22). Association for Computing Machinery, New York, NY, USA, Article 15, 14 pages. https://doi.org/10.1145/3526113.3545613
  11. Predicting Opportune Moments to Deliver Notifications in Virtual Reality. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 186, 18 pages. https://doi.org/10.1145/3491102.3517529
  12. Thematic analysis. Qualitative psychology: A practical guide to research methods 3 (2015), 222–248.
  13. Materialising contexts: virtual soundscapes for real-world exploration. Personal and Ubiquitous Computing 25 (2021), 623–636.
  14. Everyday Proxy Objects for Virtual Reality. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI EA ’21). Association for Computing Machinery, New York, NY, USA, Article 101, 6 pages. https://doi.org/10.1145/3411763.3441343
  15. A system to aid blind people in the mobility: A usability test and its results. In Second International Conference on Systems (ICONS’07). IEEE, Institute of Electrical and Electronics Engineers, New York, NY, USA, 35–35.
  16. ModularHMD: A Reconfigurable Mobile Head-Mounted Display Enabling Ad-Hoc Peripheral Interactions with the Real World. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21). Association for Computing Machinery, New York, NY, USA, 100–117. https://doi.org/10.1145/3472749.3474738
  17. Centers for Disease Control and Prevention. 2023. What Noises Cause Hearing Loss? https://www.cdc.gov/nceh/hearing_loss/what_noises_cause_hearing_loss.html
  18. William W Gaver and Donald A Norman. 1988. Everyday listening and auditory icons. Ph. D. Dissertation. University of California, San Diego, Department of Cognitive Science and Psychology.
  19. NotifiVR: Exploring Interruptions and Notifications in Virtual Reality. IEEE Transactions on Visualization and Computer Graphics 24, 4 (2018), 1447–1456. https://doi.org/10.1109/TVCG.2018.2793698
  20. Robert H Gilkey and Janet M Weisenberger. 1995. The sense of presence for the suddenly deafened adult: Implications for virtual environments. Presence: Teleoperators & Virtual Environments 4, 4 (1995), 357–363.
  21. VRception: Rapid Prototyping of Cross-Reality Systems in Virtual Reality. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 611, 15 pages. https://doi.org/10.1145/3491102.3501821
  22. Can’t You Hear Me? Investigating Personal Soundscape Curation. In Proceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia (Cairo, Egypt) (MUM ’18). Association for Computing Machinery, New York, NY, USA, 59–69. https://doi.org/10.1145/3282894.3282897
  23. Mack Hagood. 2011. Quiet comfort: Noise, otherness, and the mobile production of personal space. American Quarterly 63, 3 (2011), 573–589.
  24. Sandra G Hart. 2006. NASA-task load index (NASA-TLX); 20 years later. In Proceedings of the human factors and ergonomics society annual meeting, Vol. 50. Sage publications Sage CA: Los Angeles, CA, 904–908.
  25. RealityCheck: Blending Virtual Environments with Situated Physical Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300577
  26. Making Mobile Augmented Reality Applications Accessible. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, Greece) (ASSETS ’20). Association for Computing Machinery, New York, NY, USA, Article 3, 14 pages. https://doi.org/10.1145/3373625.3417006
  27. Anuruddha Hettiarachchi and Daniel Wigdor. 2016. Annexing Reality: Enabling Opportunistic Use of Everyday Objects as Tangible Proxies in Augmented Reality. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 1957–1967. https://doi.org/10.1145/2858036.2858134
  28. Bridging the Virtual and Real Worlds: A Preliminary Study of Messaging Notifications in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376228
  29. ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard-of-Hearing Users. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 305, 16 pages. https://doi.org/10.1145/3491102.3502020
  30. Towards Sound Accessibility in Virtual Reality. In Proceedings of the 2021 International Conference on Multimodal Interaction (Montréal, QC, Canada) (ICMI ’21). Association for Computing Machinery, New York, NY, USA, 80–91. https://doi.org/10.1145/3462244.3479946
  31. ”I Want to Figure Things Out”: Supporting Exploration in Navigation for People with Visual Impairments. Proc. ACM Hum.-Comput. Interact. 7, CSCW1, Article 63 (apr 2023), 28 pages. https://doi.org/10.1145/3579496
  32. VRBubble: Enhancing Peripheral Awareness of Avatars for People with Visual Impairments in Social Virtual Reality. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (Athens, Greece) (ASSETS ’22). Association for Computing Machinery, New York, NY, USA, Article 3, 17 pages. https://doi.org/10.1145/3517428.3544821
  33. Characterising Soundscape Research in Human-Computer Interaction. In Designing Interactive Systems Conference (Virtual Event, Australia) (DIS ’22). Association for Computing Machinery, New York, NY, USA, 1394–1417. https://doi.org/10.1145/3532106.3533458
  34. Suede: A Wizard of Oz Prototyping Tool for Speech User Interfaces. In Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology (San Diego, California, USA) (UIST ’00). Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/354401.354406
  35. Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics. arXiv preprint arXiv:2205.10646 (2022).
  36. Nearmi: A Framework for Designing Point of Interest Techniques for VR Users with Limited Mobility. In Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, USA) (ASSETS ’21). Association for Computing Machinery, New York, NY, USA, Article 5, 14 pages. https://doi.org/10.1145/3441852.3471230
  37. Ubicoustics: Plug-and-Play Acoustic Activity Recognition. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (Berlin, Germany) (UIST ’18). Association for Computing Machinery, New York, NY, USA, 213–224. https://doi.org/10.1145/3242587.3242609
  38. Auditory-induced presence in mixed reality environments and related technology. In The engineering of mixed reality systems. Springer, 143–163.
  39. HOMERE: a multimodal system for visually impaired people to explore virtual environments. In IEEE Virtual Reality, 2003. Proceedings. Institute of Electrical and Electronics Engineers, New York, NY, USA, 251–258. https://doi.org/10.1109/VR.2003.1191147
  40. CollabAlly: Accessible Collaboration Awareness in Document Editing. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 596, 17 pages. https://doi.org/10.1145/3491102.3517635
  41. ImageExplorer: Multi-Layered Touch Exploration to Encourage Skepticism Towards Imperfect AI-Generated Image Captions. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 462, 15 pages. https://doi.org/10.1145/3491102.3501966
  42. The snackbot: documenting the design of a robot for long-term human-robot interaction. In Proceedings of the 4th ACM/IEEE international conference on Human robot interaction. 7–14.
  43. SoundVizVR: Sound Indicators for Accessible Sounds in Virtual Reality for Deaf or Hard-of-Hearing Users. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (Athens, Greece) (ASSETS ’22). Association for Computing Machinery, New York, NY, USA, Article 5, 13 pages. https://doi.org/10.1145/3517428.3544817
  44. David Lindlbauer and Andy D. Wilson. 2018. Remixed Reality: Manipulating Space and Time in Augmented Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3173703
  45. Mary J Lindstrom and Douglas M Bates. 1988. Newton—Raphson and EM algorithms for linear mixed-effects models for repeated-measures data. J. Amer. Statist. Assoc. 83, 404 (1988), 1014–1022.
  46. Matthew Lombard and Theresa Ditton. 1997. At the heart of it all: The concept of presence. Journal of computer-mediated communication 3, 2 (1997), JCMC321.
  47. Audio Description in the UK: What works, what doesn’t, and understanding the need for personalising access. British journal of visual impairment 36, 3 (2018), 274–291.
  48. Seeing films through sound: Sound design, spatial audio, and accessibility for visually impaired audiences. British Journal of Visual Impairment 40, 2 (2022), 117–144.
  49. Attentive Headphones: Augmenting Conversational Attention with a Real World TiVo. In Extended Abstracts of CHI, Vol. 5.
  50. A Dose of Reality: Overcoming Usability Challenges in VR Head-Mounted Displays. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 2143–2152. https://doi.org/10.1145/2702123.2702382
  51. Acoustic Transparency and the Changing Soundscape of Auditory Mixed Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–16. https://doi.org/10.1145/3313831.3376702
  52. Paul Milgram and Fumio Kishino. 1994. A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems 77, 12 (1994), 1321–1329.
  53. Accessible by Design: An Opportunity for Virtual Reality. In 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). 451–454. https://doi.org/10.1109/ISMAR-Adjunct.2019.00122
  54. “I Just Went into It Assuming That I Wouldn’t Be Able to Have the Full Experience”: Understanding the Accessibility of Virtual Reality for People with Limited Mobility. In Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, Greece) (ASSETS ’20). Association for Computing Machinery, New York, NY, USA, Article 43, 13 pages. https://doi.org/10.1145/3373625.3416998
  55. Presence accompanying induced hearing loss: Implications for immersive virtual environments. Presence 9, 2 (2000), 137–148.
  56. NavStick: Making Video Games Blind-Accessible via the Ability to Look Around. In The 34th Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, New York, NY, USA, 538–551. https://doi.org/10.1145/3472749.3474768
  57. Uncovering Visually Impaired Gamers’ Preferences for Spatial Awareness Tools Within Video Games. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility. 1–16.
  58. Towards a Generalized Acoustic Minimap for Visually Impaired Gamers. In The Adjunct Publication of the 34th Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, New York, NY, USA, 89–91. https://doi.org/10.1145/3474349.3480177
  59. Towards a Generalized Acoustic Minimap for Visually Impaired Gamers. In Adjunct Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21 Adjunct). Association for Computing Machinery, New York, NY, USA, 89–91. https://doi.org/10.1145/3474349.3480177
  60. A Fieldwork of the Future with User Enactments. In Proceedings of the Designing Interactive Systems Conference (Newcastle Upon Tyne, United Kingdom) (DIS ’12). Association for Computing Machinery, New York, NY, USA, 338–347. https://doi.org/10.1145/2317956.2318008
  61. Unpacking the Thinking and Making behind a User Enactments Project. In Proceedings of the 2014 Conference on Designing Interactive Systems (Vancouver, BC, Canada) (DIS ’14). Association for Computing Machinery, New York, NY, USA, 513–522. https://doi.org/10.1145/2598510.2602960
  62. Rescribe: Authoring and Automatically Editing Audio Descriptions. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, New York, NY, USA, 747–759. https://doi.org/10.1145/3379337.3415864
  63. Donald A Ramsdell. 1978. The psychology of the hard-of-hearing and the deafened adult. Hearing and deafness 4 (1978), 499–510.
  64. Joan Sol Roo and Martin Hachet. 2017. One Reality: Augmenting How the Physical World is Experienced by Combining Multiple Mixed Reality Modalities. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST ’17). Association for Computing Machinery, New York, NY, USA, 787–795. https://doi.org/10.1145/3126594.3126638
  65. Monika Rychtarikova. 2015. How do blind people perceive sound and soundscape. Akustika 23, 1 (2015), 6–9.
  66. Substitutional Reality: Using the Physical Environment to Design Virtual Reality Experiences. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 3307–3316. https://doi.org/10.1145/2702123.2702389
  67. Virtual Reality Without Vision: A Haptic and Auditory White Cane to Navigate Complex Virtual Worlds. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376353
  68. Advanced Hearing Solutions. 2023. How Does Active Noise Cancelling Work? https://hearlife.org/how-does-active-noise-cancelling-work/
  69. Going Beyond One-Size-Fits-All Image Descriptions to Satisfy the Information Wants of People Who Are Blind or Have Low Vision. In Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, USA) (ASSETS ’21). Association for Computing Machinery, New York, NY, USA, Article 16, 15 pages. https://doi.org/10.1145/3441852.3471233
  70. Cyclic co-learning of sounding object visual grounding and sound separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2745–2754.
  71. The New York Times. 2020. What Your Noise-Cancelling Headphones Can and Can’t Do. https://www.nytimes.com/wirecutter/blog/what-noise-cancelling-headphones-do/
  72. Rob van Rijswijk and Jeroen Strijbos. 2013. Sounds in Your Pocket: Composing Live Soundscapes with an App. Leonardo Music Journal 23 (12 2013), 27–29. https://doi.org/10.1162/LMJ_a_00149 arXiv:https://direct.mit.edu/lmj/article-pdf/doi/10.1162/LMJ_a_00149/1674871/lmj_a_00149.pdf
  73. Real-Time Target Sound Extraction. arXiv preprint arXiv:2211.02250 (2022).
  74. World Wide Web Consortium (W3C). 2021. XR Accessibility User Requirements. https://www.w3.org/TR/xaur/
  75. RealityLens: A User Interface for Blending Customized Physical World View into Virtual Reality. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (Bend, OR, USA) (UIST ’22). Association for Computing Machinery, New York, NY, USA, Article 49, 11 pages. https://doi.org/10.1145/3526113.3545686
  76. Slice of Light: Transparent and Integrative Transition Among Realities in a Multi-HMD-User Environment. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’20). Association for Computing Machinery, New York, NY, USA, 805–817. https://doi.org/10.1145/3379337.3415868
  77. Immersive Notification Framework: Adaptive & Plausible Notifications in Virtual Reality. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI EA ’18). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3170427.3188505
  78. Enabling People with Visual Impairments to Navigate Virtual Reality with a Haptic and Auditory Cane Simulation. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3173574.3173690
  79. SeeingVR: A Set of Tools to Make Virtual Reality More Accessible to People with Low Vision. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3290605.3300341
  80. Andreas Zimmermann and Andreas Lorenz. 2008. LISTEN: a user-adaptive audio-augmented museum guide. User Modeling and User-Adapted Interaction 18, 5 (2008), 389–416.
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Tweets

Sign up for free to view the 4 tweets with 53 likes about this paper.

Sign Up for Free