Interactive Sonification for Health and Energy using ChucK and Unity (2404.08813v1)
Abstract: Sonification can provide valuable insights about data but most existing approaches are not designed to be controlled by the user in an interactive fashion. Interactions enable the designer of the sonification to more rapidly experiment with sound design and allow the sonification to be modified in real-time by interacting with various control parameters. In this paper, we describe two case studies of interactive sonification that utilize publicly available datasets that have been described recently in the International Conference on Auditory Display (ICAD). They are from the health and energy domains: electroencephalogram (EEG) alpha wave data and air pollutant data consisting of nitrogen dioxide, sulfur dioxide, carbon monoxide, and ozone. We show how these sonfications can be recreated to support interaction utilizing a general interactive sonification framework built using ChucK, Unity, and Chunity. In addition to supporting typical sonification methods that are common in existing sonification toolkits, our framework introduces novel methods such as supporting discrete events, interleaved playback of multiple data streams for comparison, and using frequency modulation (FM) synthesis in terms of one data attribute modulating another. We also describe how these new functionalities can be used to improve the sonification experience of the two datasets we have investigated.
- B. N. Walker and J. T. Cothran, “Sonification sandbox: A graphical toolkit for auditory graphs,” in Proceedings of the Ninth International Conference on Auditory Display (ICAD 2003), 2003, p. 3.
- B. K. Davison and B. N. Walker, “Sonification sandbox reconstruction: Software standard for auditory graphs,” in Proceedings of the Thirteenth International Conference on Auditory Display (ICAD 2007), 2007, p. 4.
- S. Pauletto and A. Hunt, “A toolkit for interactive sonification,” in Proceedings of the Tenth International Conference on Auditory Display (ICAD 2004), 2004, p. 7.
- S. research group at Berne University of the Arts (BUA), “Sonifyer.” [Online]. Available: http://www.sonifyer.org/softhardware/?lang=en
- S. Phillips and A. Cabrera, “Sonification workstation,” in Proceedings of the 25th International Conference on Auditory Display (ICAD 2019). Newcastle upon Tyne: Department of Computer and Information Sciences, Northumbria University, Jun 2019, p. 184–190. [Online]. Available: http://hdl.handle.net/1853/61529
- H. Lindetorp and K. Falkenberg, “Sonification for everyone everywhere - evaluating the webaudioxml sonification toolkit for browsers,” in Proceedings of the 26th International Conference on Auditory Display (ICAD 2021). Virtual Conference: International Community for Auditory Display, Jun 2021, p. 15–21. [Online]. Available: http://hdl.handle.net/1853/66351
- Sonify, “Twotone: An open-source data sonification and data-driven music web app,” Jun 2022. [Online]. Available: https://github.com/sonifydata/twotone
- A. L. A. Blanco, S. Grautoff, and T. Hermann, “Cardiosounds: A portable system to sonify ecg rhythm disturbances in real-time,” in Proceedings of the 24th International Conference on Auditory Display - ICAD 2018. Houghton, Michigan: The International Community for Auditory Display, Jun 2018, p. 20–27. [Online]. Available: http://hdl.handle.net/1853/60089
- R. M. Winters, A. Kalra, and B. N. Walker, “Hearing artificial intelligence: Sonification guidelines & results from a case-study in melanoma diagnosis,” in Proceedings of the 25th International Conference on Auditory Display (ICAD 2019). Newcastle upon Tyne: Department of Computer and Information Sciences, Northumbria University, Jun 2019, p. 262–267. [Online]. Available: http://hdl.handle.net/1853/61501
- T. J. Mitchell, J. Thom, M. Pountney, and J. Hyde, “The alchemy of chaos: A sound art sonification of a year of tourette’s episodes,” in Proceedings of the 25th International Conference on Auditory Display (ICAD 2019). Newcastle upon Tyne: Department of Computer and Information Sciences, Northumbria University, Jun 2019, p. 163–168. [Online]. Available: http://hdl.handle.net/1853/61514
- P. R. Kantan, E. G. Spaich, and S. Dahl, “A metaphor-based technical framework for musical sonification in movement rehabilitation,” in Proceedings of the 26th International Conference on Auditory Display (ICAD 2021). Virtual Conference: International Community for Auditory Display, Jun 2021, p. 176–183. [Online]. Available: http://hdl.handle.net/1853/66350
- P. Cowden and L. Dosiek, “Auditory displays of electric power grids,” in Proceedings of the 24th International Conference on Auditory Display - ICAD 2018. Houghton, Michigan: The International Community for Auditory Display, Jun 2018, p. 113–120. [Online]. Available: http://hdl.handle.net/1853/60076
- K. Groß-Vogt, M. Weger, R. Höldrich, T. Hermann, T. Bovermann, and S. Reichmann, “Augmentation of an institute’s kitchen: An ambient auditory display of electric power consumption,” in Proceedings of the 24th International Conference on Auditory Display - ICAD 2018. Houghton, Michigan: The International Community for Auditory Display, Jun 2018, p. 105–112. [Online]. Available: http://hdl.handle.net/1853/60088
- M. St Pierre and M. Droumeva, “Sonifying for public engagement: A context-based model for sonifying air pollution data,” in Proceedings of the 22nd International Conference on Auditory Display (ICAD 2016). Canberra, Australia: The International Community for Auditory Display, Jul 2016, p. 60–64. [Online]. Available: http://hdl.handle.net/1853/56580
- G. Wang and P. R. Cook, “Chuck: A concurrent, on-the-fly, audio programming language,” in Proceedings of the 2003 International Computer Music Conference, 2003, p. 8.
- Unity Technologies, “Unity real-time development platform.” [Online]. Available: https://unity.com/
- J. Atherton and G. Wang, “Chunity: Integrated audiovisual programming in unity,” in Proceedings of the 2018 conference on new interfaces for musical expression, p. 6.
- A. de Campo, “Toward a data sonification design space map,” in Proceedings of the 13th International Conference on Auditory Displa (ICAD 2007), 2007, p. 6.
- M. Puckette, “Pure data,” in Proceedings of the International Computer Music Conference, Jan 1996, p. 41.
- M. St Pierre and M. Droumeva, “Sonification of air quality data in four canadian cities.” [Online]. Available: https://soundcloud.com/marcstpierre/sets/sonification-of-air-quality
- M. Wright and A. Freed, “Open soundcontrol: A new protocol for communicating with sound synthesizers,” in Proceedings of the International Computer Music Conference, p. 4.