Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
133 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Erie: A Declarative Grammar for Data Sonification (2402.00156v2)

Published 31 Jan 2024 in cs.HC and cs.SD

Abstract: Data sonification-mapping data variables to auditory variables, such as pitch or volume-is used for data accessibility, scientific exploration, and data-driven art (e.g., museum exhibitions) among others. While a substantial amount of research has been made on effective and intuitive sonification design, software support is not commensurate, limiting researchers from fully exploring its capabilities. We contribute Erie, a declarative grammar for data sonification, that enables abstractly expressing auditory mappings. Erie supports specifying extensible tone designs (e.g., periodic wave, sampling, frequency/amplitude modulation synthesizers), various encoding channels, auditory legends, and composition options like sequencing and overlaying. Using standard Web Audio and Web Speech APIs, we provide an Erie compiler for web environments. We demonstrate the expressiveness and feasibility of Erie by replicating research prototypes presented by prior work and provide a sonification design gallery. We discuss future steps to extend Erie toward other audio computing environments and support interactive data sonification.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. [n. d.]. vega-datasets. Last accessed Sep. 1, 2023. https://github.com/vega/vega-datasets.
  2. n.d.. CSound. Last accessed Sep. 5, 2023. https://csound.com/.
  3. n.d.. Tone.JS. Last accessed Sep 12, 2023. https://tonejs.github.io/.
  4. Sonify. https://hcii.cmu.edu/mhci/capstone/2016/bloomberg/index.html.
  5. Apple. n.d.. Attack, decay, sustain, and release. Last accessed Aug 5, 2023. https://support.apple.com/guide/logicpro/attack-decay-sustain-and-release-lgsife419620/mac.
  6. Apple Inc. n.d.. Audio Graph — Apple Developer Documentation. https://developer.apple.com/documentation/accessibility/audio_graphs.
  7. Stephen Barrass. 1995. Personify: a Toolkit for Perceptually Meaningful Sonification. In Proceedings of the Australian Computer Music Association Conference.
  8. Exploring D3 Implementation Challenges on Stack Overflow. In 2022 IEEE Visualization and Visual Analytics (VIS). 1–5. https://doi.org/10.1109/VIS54862.2022.00009
  9. SonART: The sonification application research toolbox. In Proceedings of the 2002 International Conference on Auditory Display (Kyoto, Japan) (ICAD ’02). ICAD, 151–153.
  10. D³ Data-Driven Documents. IEEE Trans. Visual Comput. Graphics 17, 12 (2011), 2301–2309. https://doi.org/10.1109/TVCG.2011.185
  11. Ethan Brown. 2011. Play It By R. Last accessed July 20, 2023. http://playitbyr.org/gettingstarted.html.
  12. Nine Rounds a Second: How the Las Vegas Gunman Outfitted a Rifle to Fire Faster. Last accessed Sep. 3, 2023. https://www.nytimes.com/interactive/2017/10/02/us/vegas-guns.html.
  13. xSonify Sonification Tool for Space Physics. In Proceedings of the 2006 International Conference on Auditory Display (London, UK) (ICAD ’06). ICAD.
  14. Highcharts Sonification Studio: An Online, Open-source, extensible, and Accessible Data Sonification Tool. In Proceedings of the 2021 International Conference on Auditory Display (ICAD ’21). ICAD, 211–216. https://doi.org/10.21785/icad2021.005
  15. TactualPlot: Spatializing Data as Sound using Sensory Substitution for Touchscreen Accessibility. IEEE Transactions on Visualization and Computer Graphics (2023), 1–11. https://doi.org/10.1109/TVCG.2023.3326937
  16. André Cibils. 2020. SODA: SOnification of DAta for Learning Analytics. https://github.com/AndreCI/Soda4LA.
  17. Translating Color: Sonification as a Method of Sensory Substitution within the Museum. In Proceedings of the 20th International Web for All Conference (W4A ’23). Association for Computing Machinery, 162–163. https://doi.org/10.1145/3587281.3587706
  18. SonifYer: A Concept, a Software, a Platform. In Proceedings of the 2008 International Conference on Auditory Display (Paris, France) (ICAD ’08). ICAD.
  19. Gaël Dubus and Roberto Bresin. 2013. A Systematic Review of Mapping Strategies for the Sonification of Physical Quantities. PLOS ONE 8, 12 (12 2013), 1–28. https://doi.org/10.1371/journal.pone.0082491
  20. John Dunn and Mary Anne Clark. 1999. Life Music: The Sonification of Proteins. Leonardo 32, 1 (02 1999), 25–32. https://doi.org/10.1162/002409499552966
  21. Evaluating a Tool for Improving Accessibility to Charts and Graphs. ACM Trans. Comput.-Hum. Interact. 20, 5 (2013). https://doi.org/10.1145/2533682.2533683
  22. Pallab Ghosh. 2010. God particle signal is simulated as sound. Last accessed Aug 2, 2023. https://www.bbc.co.uk/news/10385675.
  23. Jeffrey Hannam. 2014. StarSound. https://www.jeffreyhannam.com/starsound.
  24. Thomas Hermann. 2008. Taxonomy and definitions for sonification and auditory display. (2008), 1–8.
  25. Accessible Data Representation with Natural Sound. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 826. https://doi.org/10.1145/3544548.3581087
  26. Visualization Accessibility in the Wild: Challenges Faced by Visualization Designers. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 83. https://doi.org/10.1145/3491102.3517630
  27. Web sonification sandbox-an easy-to-use web application for sonifying data and equations. (2017).
  28. Sonification report: Status of the field and research agenda. (1997). Report prepared for the National Science Foundation.
  29. Muse: A musical data sonification toolkit. In Proceedings of the 1997 International Conference on Auditory Display (ICAD ’97). ICAD.
  30. MDN. n.d.a. AudioNode - Web APIs. Last accessed Sep 12, 202. https://developer.mozilla.org/en-US/docs/Web/API/AudioNode.
  31. MDN. n.d.b. Web Audio API. Last accessed Aug 5, 2023. https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API.
  32. MDN. n.d.c. Web Speech API. Last accessed Aug 5, 2023. https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API.
  33. Sandra Pauletto and Andy Hunt. 2004. A Toolkit for Interactive Sonification. In Proceedings of the 2004 International Conference on Auditory Display (ICAD ’04). ICAD.
  34. Sean Phillips and Andres Cabrera. 2019. Sonification workstation. In Proceedings of the 1997 International Conference on Auditory Display (Newcastle upon Tyne, UK) (ICAD ’19). ICAD.
  35. Andrea Polli. 2005. Atmospherics/Weather Works: A Spatialized Meteorological Data Sonification Project. Leonardo 38, 1 (02 2005), 31–36. https://doi.org/10.1162/leon.2005.38.1.31
  36. Proof of Concept of a Generic Toolkit for Sonification: The Sonification Cell in Ossia Score. In Proceedings of the 2003 International Conference on Auditory Display (Norrköping, Sweden) (ICAD ’03). ICAD.
  37. The Audible Spectrum. In Neuroscience (2nd ed.). Sinauer Associates. https://www.ncbi.nlm.nih.gov/books/NBK10924/.
  38. Donya Quick and Paul Hudak. 2013. Grammar-Based Automated Music Composition in Haskell. In Proceedings of the First ACM SIGPLAN Workshop on Functional Art, Music, Modeling & Design (Boston, Massachusetts, USA) (FARM ’13). ACM, 59–70. https://doi.org/10.1145/2505341.2505345
  39. Vega-Lite: A Grammar of Interactive Graphics. IEEE Trans. Visualization & Comp. Graphics (Proc. InfoVis) (2017). https://doi.org/10.1109/TVCG.2016.2599030
  40. Reactive Vega: A Streaming Dataflow Architecture for Declarative Interactive Visualization. In IEEE Trans. Visualization & Comp. Graphics (Proc. InfoVis ’15). https://doi.org/10.1109/TVCG.2015.2467091
  41. Carla Scaletti. 1994. Sound synthesis algorithms for auditory data representations. In Auditory Display, Sonification: Audification, and Auditory INterfaces, Gregory Kramer (Ed.). 223–251.
  42. Arnold Seong and Joonyoung Seo. 2020. DataGoBoop. Last accessed July 20, 2023. https://github.com/akseong/datagoboop.
  43. Ather Sharif. 2022. Sonifer.JS. Last accessed July 20, 2023. https://github.com/athersharif/sonifier.
  44. “What Makes Sonification User-Friendly?” Exploring Usability and User-Friendliness of Sonified Responses. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (Athens, Greece) (ASSETS ’22). Association for Computing Machinery, New York, NY, USA, Article 45. https://doi.org/10.1145/3517428.3550360
  45. Supporting Accessible Data Visualization Through Audio Data Narratives. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). ACM. https://doi.org/10.1145/3491102.3517678
  46. Alexandra Supper. 2014. Sublime frequencies: The construction of sublime listening experiences in the sonification of scientific data. Social Studies of Science 44, 1 (2014), 34–58. https://doi.org/10.1177/0306312713496875
  47. Chart Reader: Accessible Visualization Experiences Designed with Screen Reader Users. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 802. https://doi.org/10.1145/3544548.3581186
  48. Solar system sonification: exploring earth and its neighbors through sound. In The 23rd international conference on auditory display (ICAD 2017). 128–134. https://doi.org/doi.org/10.21785/icad2017.027
  49. James Trayford. 2021. STRAUSS. https://github.com/james-trayford/strauss.
  50. Samuel Van Ransbeeck. n.d.. Sonification Art. Last accessed Aug 2, 2023 https://sonificationart.wordpress.com/.
  51. Bruce N Walker. 2002. Magnitude estimation of conceptual data dimensions for use in sonification. Journal of experimental psychology: Applied 8, 4 (2002), 211.
  52. Bruce N Walker. 2007. Consistency of magnitude estimations with conceptual data dimensions used for sonification. Applied Cognitive Psychology: The Official Journal of the Society for Applied Research in Memory and Cognition 21, 5 (2007), 579–599.
  53. Bruce N. Walker and Joshua T. Cothran. 2003. Sonification Sandbox: A Graphical Toolkit for Auditory Graphs. In Proceedings of the 2003 International Conference on Auditory Display (Boston, MA, USA) (ICAD ’03). ICAD, 161–163.
  54. Aquarium sonification: Soundscapes for accessible dynamic informal learning environments. In Proceedings of the 12th International Conference on Auditory Display. 238–241. http://www.icad.org/Proceedings/2006/WalkerGodfrey2006.pdf.
  55. B. N. Walker and L. M. Mauney. 2010. Universal Design of Auditory Graphs: A Comparison of Sonification Mappings for Visually Impaired and Sighted Listeners. ACM Trans. Access. Comput. 2, 3, Article 12 (mar 2010). https://doi.org/10.1145/1714458.1714459
  56. Seeing Through Sounds: Mapping Auditory Dimensions to Data and Charts for People with Visual Impairments. Computer Graphics Forum 41, 3 (2022), 71–83. https://doi.org/10.1111/cgf.14523
  57. Hadley Wickham. 2010. A Layered Grammar of Graphics. Journal of Computational and Graphical Statistics 19, 1 (2010), 3–28. https://doi.org/10.1198/jcgs.2009.07098
  58. Wikipedia. n.d.. Pulse-code modulation. Last accessed Sep. 6, 2023. https://en.wikipedia.org/wiki/Pulse-code_modulation.
  59. Leland Wilkinson. 2012. The grammar of graphics. Springer.
  60. Catherine M Wilson and Suresh K Lodha. 1996. Listen: A data sonification toolkit. (1996).
  61. Data Sonification for Users with Visual Impairment: A Case Study with Georeferenced Data. ACM Trans. Comput.-Hum. Interact. 15, 1 (2008). https://doi.org/10.1145/1352782.1352786
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets