Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

EmoWear: Exploring Emotional Teasers for Voice Message Interaction on Smartwatches (2402.07174v1)

Published 11 Feb 2024 in cs.HC and cs.AI

Abstract: Voice messages, by nature, prevent users from gauging the emotional tone without fully diving into the audio content. This hinders the shared emotional experience at the pre-retrieval stage. Research scarcely explored "Emotional Teasers"-pre-retrieval cues offering a glimpse into an awaiting message's emotional tone without disclosing its content. We introduce EmoWear, a smartwatch voice messaging system enabling users to apply 30 animation teasers on message bubbles to reflect emotions. EmoWear eases senders' choice by prioritizing emotions based on semantic and acoustic processing. EmoWear was evaluated in comparison with a mirroring system using color-coded message bubbles as emotional cues (N=24). Results showed EmoWear significantly enhanced emotional communication experience in both receiving and sending messages. The animated teasers were considered intuitive and valued for diverse expressions. Desirable interaction qualities and practical implications are distilled for future design. We thereby contribute both a novel system and empirical knowledge concerning emotional teasers for voice messaging.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (83)
  1. MeowPlayLive: Enhancing Animal Live Streaming Experience Through Voice Message-Based Real-Time Viewer-Animal Interaction. In Designing Interactive Systems Conference (Virtual Event, Australia) (DIS ’22). Association for Computing Machinery, New York, NY, USA, 849–864. https://doi.org/10.1145/3532106.3533553
  2. Beyond Subtitles: Captioning and Visualizing Non-Speech Sounds to Improve Accessibility of User-Generated Videos. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (Athens, Greece) (ASSETS ’22). Association for Computing Machinery, New York, NY, USA, Article 26, 12 pages. https://doi.org/10.1145/3517428.3544808
  3. Affective Affordance of Message Balloon Animations: An Early Exploration of AniBalloons. (2023), 138–143. https://doi.org/10.1145/3584931.3607017
  4. VibEmoji: Exploring User-authoring Multi-modal Emoticons in Social Communication. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (, New Orleans, LA, USA,) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 493, 17 pages. https://doi.org/10.1145/3491102.3501940
  5. EmoBalloon - Conveying Emotional Arousal in Text Chats with Speech Balloons. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 527, 16 pages. https://doi.org/10.1145/3491102.3501920
  6. Apple. 2023. Use Memoji on your iPhone or iPad Pro. https://support.apple.com/en-us/HT208986
  7. Inclusion of other in the self scale and the structure of interpersonal closeness. Journal of personality and social psychology 63, 4 (1992), 596.
  8. Brian. 2012. PLATO Emoticons, revisited.
  9. Keith Broni. 2022. New Emojis In 2022-2023. https://blog.emojipedia.org/new-emojis-in-2022-2023/
  10. Personal Mobile Messaging in Context: Chat Augmentations for Expressiveness and Awareness. ACM Trans. Comput.-Hum. Interact. 25, 4, Article 23 (aug 2018), 33 pages. https://doi.org/10.1145/3201404
  11. Bubble Coloring to Visualize the Speech Emotion. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI EA ’21). Association for Computing Machinery, New York, NY, USA, Article 361, 6 pages. https://doi.org/10.1145/3411763.3451698
  12. Designing voice-aware text in voice media with background color and typography. Journal of the International Colour Association (2022). https://aic-color.org/resources/Documents/jaic_v28_10.pdf
  13. Emoji-Powered Sentiment and Emotion Detection from Software Developers’ Communication Data. ACM Trans. Softw. Eng. Methodol. 30, 2, Article 18 (Jan. 2021), 48 pages. https://doi.org/10.1145/3424308
  14. Animations 25 years later: New roles and opportunities. In Proceedings of the international working conference on advanced visual interfaces. 280–287.
  15. I Share, You Care: Private Status Sharing and Sender-Controlled Notifications in Mobile Instant Messaging. Proc. ACM Hum.-Comput. Interact. 4, CSCW1, Article 34 (may 2020), 25 pages. https://doi.org/10.1145/3392839
  16. Saemi Choi and Kiyoharu Aizawa. 2019. Emotype: Expressing emotions by changing typeface in mobile messenger texting. Multimedia Tools and Applications 78, 11 (2019), 14155–14172.
  17. S. Davis and P. Mermelstein. 1980. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE Transactions on Acoustics, Speech, and Signal Processing 28, 4 (1980), 357–366. https://doi.org/10.1109/TASSP.1980.1163420
  18. Irene de la Torre-Arenas and Pedro Cruz. 2017. A taxonomy of motion applications in data visualization. In Proceedings of the symposium on Computational Aesthetics. 1–2.
  19. Visualization of Speech Prosody and Emotion in Captions: Accessibility for Deaf and Hard-of-Hearing Users. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 831, 15 pages. https://doi.org/10.1145/3544548.3581511
  20. Emotions Understanding Model from Spoken Language using Deep Neural Networks and Mel-Frequency Cepstral Coefficients. In 2020 IEEE Conference on Evolving and Adaptive Intelligent Systems (EAIS). 1–5. https://doi.org/10.1109/EAIS48028.2020.9122698
  21. GoEmotions: A Dataset of Fine-Grained Emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 4040–4054. https://doi.org/10.18653/v1/2020.acl-main.372
  22. Pieter Desmet. 2018. Measuring emotion: Development and application of an instrument to measure emotional responses to products. Funology 2: From Usability to Enjoyment (2018), 391–404.
  23. Paul Ekman. 1992a. Are there basic emotions? Psychological Review 3, 99 (1992), 550–553.
  24. Paul Ekman. 1992b. An argument for basic emotions. Cognition & emotion 6, 3-4 (1992), 169–200.
  25. ThermalWear: Exploring Wearable On-Chest Thermal Displays to Augment Voice Messages with Affect. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376682
  26. Challenges of Emotion Detection Using Facial Expressions and Emotion Visualisation in Remote Communication. In Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers (Virtual, USA) (UbiComp ’21). Association for Computing Machinery, New York, NY, USA, 230–236. https://doi.org/10.1145/3460418.3479341
  27. Facebook. 2023. Messenger. https://www.messenger.com/
  28. Scott Fahlman. Retrieved 2021. Smiley Lore :-).
  29. “Hey, Can We Talk?”: Exploring How Revealing Implicit Emotional Responses Tangibly Could Foster Empathy During Mobile Texting. In Proceedings of the Seventeenth International Conference on Tangible, Embedded, and Embodied Interaction (Warsaw, Poland) (TEI ’23). Association for Computing Machinery, New York, NY, USA, Article 53, 7 pages. https://doi.org/10.1145/3569009.3573124
  30. Using a Touch-Sensitive Wristband for Text Entry on Smart Watches. In CHI ’14 Extended Abstracts on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI EA ’14). Association for Computing Machinery, New York, NY, USA, 2305–2310. https://doi.org/10.1145/2559206.2581143
  31. Google. 2023. Gboard. https://apps.apple.com/us/app/gboard-the-google-keyboard/id1091700242
  32. WatchWriter: Tap and Gesture Typing on a Smartwatch Miniature Keyboard with Statistical Decoding. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 3817–3821. https://doi.org/10.1145/2858036.2858242
  33. Exploring Haptic Feedback for Common Message Notification Between Intimate Couples with Smartwatches. In Proceedings of the 32nd Australian Conference on Human-Computer Interaction (Sydney, NSW, Australia) (OzCHI ’20). Association for Computing Machinery, New York, NY, USA, 245–252. https://doi.org/10.1145/3441000.3441012
  34. Saul Greenberg and Bill Buxton. 2008. Usability Evaluation Considered Harmful (Some of the Time). In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy) (CHI ’08). Association for Computing Machinery, New York, NY, USA, 111–120. https://doi.org/10.1145/1357054.1357074
  35. Customizations and Expression Breakdowns in Ecosystems of Communication Apps. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 26 (nov 2019), 26 pages. https://doi.org/10.1145/3359128
  36. Mediating Intimacy with DearBoard: A Co-Customizable Keyboard for Everyday Messaging. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Article 342, 16 pages.
  37. “They Like to Hear My Voice”: Exploring Usage Behavior in Speech-Based Mobile Instant Messaging. In 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services (Oldenburg, Germany) (MobileHCI ’20). Association for Computing Machinery, New York, NY, USA, Article 35, 10 pages. https://doi.org/10.1145/3379503.3403561
  38. VoiceMessage++: Augmented Voice Recordings for Mobile Instant Messaging. In 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services (Oldenburg, Germany) (MobileHCI ’20). Association for Computing Machinery, New York, NY, USA, Article 30, 10 pages. https://doi.org/10.1145/3379503.3403560
  39. Emoji Use in Twitter White Nationalism Communication. In Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing (Austin, TX, USA) (CSCW ’19). 201–205. https://doi.org/10.1145/3311957.3359495
  40. Mitsuhiko Hanada. 2018. Correspondence analysis of color–emotion associations. Color Research & Application 43, 2 (2018), 224–237. https://doi.org/10.1002/col.22171 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/col.22171
  41. Kineticons: Using Iconographic Motion in Graphical User Interface Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). 1999–2008. https://doi.org/10.1145/1978942.1979232
  42. Emojilization: An Automated Method For Speech to Emoji-Labeled Text. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA ’19). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3290607.3313071
  43. Students’ Colour Perception and Preference: An Empirical Analysis of its Relationship. Procedia - Social and Behavioral Sciences 90 (2013), 575–582. https://doi.org/10.1016/j.sbspro.2013.07.128 6th International Conference on University Learning and Teaching (InCULT 2012).
  44. Understanding Diverse Interpretations of Animated GIFs. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI EA ’17). 1726–1732. https://doi.org/10.1145/3027063.3053139
  45. Gestural Text Input Using a Smartwatch. In Proceedings of the International Working Conference on Advanced Visual Interfaces (Bari, Italy) (AVI ’16). Association for Computing Machinery, New York, NY, USA, 220–223. https://doi.org/10.1145/2909132.2909273
  46. Ryan Kelly and Leon Watts. 2015. Characterising the inventive appropriation of emoji as relationally meaningful in mediated close personal relationships. Experiences of technology appropriation: Unanticipated users, usage, circumstances, and design 2 (2015), 7 pages.
  47. Visible Nuances: A Caption System to Visualize Paralinguistic Speech Cues for Deaf and Hard-of-Hearing Individuals. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 54, 15 pages. https://doi.org/10.1145/3544548.3581130
  48. Messaging Beyond Texts with Real-Time Image Suggestions. In 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services (Oldenburg, Germany) (MobileHCI ’20). Article 28, 12 pages. https://doi.org/10.1145/3379503.3403553
  49. Kineticharts: Augmenting Affective Expressiveness of Charts in Data Stories with Animation Design. IEEE Transactions on Visualization and Computer Graphics 28, 1 (2022), 933–943. https://doi.org/10.1109/TVCG.2021.3114775
  50. John Lasseter. 1987. Principles of traditional animation applied to 3D computer animation. In Proceedings of the 14th annual conference on Computer graphics and interactive techniques. 35–44.
  51. Context-Dependent Domain Adversarial Neural Network for Multimodal Emotion Recognition. In Proc. Interspeech 2020. 394–398. https://doi.org/10.21437/Interspeech.2020-1705
  52. Animo: Sharing Biosignals on a Smartwatch for Lightweight Social Connection. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 1, Article 18 (mar 2019), 19 pages. https://doi.org/10.1145/3314405
  53. Significant Otter: Understanding the Role of Biosignals in Communication. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 334, 15 pages. https://doi.org/10.1145/3411764.3445200
  54. Steven R. Livingstone and Frank A. Russo. 2018. The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PLOS ONE 13 (05 2018), 1–35. https://doi.org/10.1371/journal.pone.0196391
  55. Guidelines for Depicting Emotions in Storyboard Scenarios.
  56. Tap ’n’ Shake: Gesture-Based Smartwatch-Smartphone Communications System. In Proceedings of the 28th Australian Conference on Computer-Human Interaction (Launceston, Tasmania, Australia) (OzCHI ’16). Association for Computing Machinery, New York, NY, USA, 442–446. https://doi.org/10.1145/3010915.3010983
  57. Remote Handshaking: Touch Enhances Video-Mediated Social Telepresence. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 2143–2152. https://doi.org/10.1145/2556288.2557169
  58. Donald A Norman. 2003. Designing emotions pieter desmet. The Design Journal 6, 2 (2003), 60–62.
  59. Emotional Subtitles: A System and Potential Applications for Deaf and Hearing Impaired People.. In CVHI.
  60. A Preliminary Study on Understanding Voice-Only Online Meetings Using Emoji-Based Captioning for Deaf or Hard of Hearing Users. In Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, Greece) (ASSETS ’20). Association for Computing Machinery, New York, NY, USA, Article 54, 4 pages. https://doi.org/10.1145/3373625.3418032
  61. MagTouch: Robust Finger Identification for a Smartwatch Using a Magnet Ring and a Built-in Magnetometer. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376234
  62. Young-Woo Park and Tek-Jin Nam. 2013. POKE: A New Way of Sharing Emotional Touches during Phone Conversations. In CHI ’13 Extended Abstracts on Human Factors in Computing Systems (Paris, France) (CHI EA ’13). Association for Computing Machinery, New York, NY, USA, 2859–2860. https://doi.org/10.1145/2468356.2479548
  63. Gulnar Rakhmetulla and Ahmed Sabbir Arif. 2023. Crownboard: A One-Finger Crown-Based Smartwatch Keyboard for Users with Limited Dexterity. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 46, 22 pages. https://doi.org/10.1145/3544548.3580770
  64. Designing Emotional Expressions of Conversational States for Voice Assistants: Modality and Engagement. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI EA ’18). 1–6. https://doi.org/10.1145/3170427.3188560
  65. AniSAM & AniAvatar: Animated Visualizations of Affective States. 4828–4837.
  66. Telegram. 2023. Messenger. https://telegram.org/
  67. The illusion of life: Disney animation. Hyperion New York.
  68. Bringing Lou to Life: A Study in Creating Lou. In ACM SIGGRAPH 2017 Talks (Los Angeles, California) (SIGGRAPH ’17). Article 1, 2 pages. https://doi.org/10.1145/3084363.3085089
  69. Social signal processing: Survey of an emerging domain. Image and vision computing 27, 12 (2009), 1743–1759.
  70. Communicating Emotions in Online Chat Using Physiological Sensors and Animated Text. In CHI ’04 Extended Abstracts on Human Factors in Computing Systems (Vienna, Austria) (CHI EA ’04). 1171–1174. https://doi.org/10.1145/985921.986016
  71. Keep in Touch: Channel, Expectation and Experience. Association for Computing Machinery, New York, NY, USA, 139–148. https://doi.org/10.1145/2207676.2207697
  72. Enhancing Social Messaging with Mediated Social Touch. International Journal of Human–Computer Interaction 0, 0 (2022), 1–20. https://doi.org/10.1080/10447318.2022.2148883 arXiv:https://doi.org/10.1080/10447318.2022.2148883
  73. Sarah Wiseman and Sandy J. J. Gould. 2018. Repurposing Emoji for Personalised Communication: Why Pizza-emoji Means “I Love You”. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). 1–10. https://doi.org/10.1145/3173574.3173726
  74. Creating Emordle: Animating Word Cloud for Emotion Expression. IEEE Transactions on Visualization and Computer Graphics (2023), 1–14. https://doi.org/10.1109/TVCG.2023.3286392
  75. A social emotion classification approach using multi-model fusion. Future Generation Computer Systems 102 (2020), 347–356. https://doi.org/10.1016/j.future.2019.07.007
  76. Affective Profile Pictures: Exploring the Effects of Changing Facial Expressions in Profile Pictures on Text-Based Communication. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 270, 17 pages. https://doi.org/10.1145/3544548.3581061
  77. ProxiTalk: Activate Speech Input by Bringing Smartphone to the Mouth. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 3, Article 118 (sep 2019), 25 pages. https://doi.org/10.1145/3351276
  78. Temporal Modeling Matters: A Novel Temporal Emotional Modeling Approach for Speech Emotion Recognition. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 1–5. https://doi.org/10.1109/ICASSP49357.2023.10096370
  79. Speech-Text Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Toronto, Canada, 7900–7913. https://doi.org/10.18653/v1/2023.acl-long.438
  80. Using Student Annotated Hashtags and Emojis to Collect Nuanced Affective States. In Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale (Cambridge, Massachusetts, USA) (L@S ’17). 319–322. https://doi.org/10.1145/3051457.3054014
  81. Touch without Touching: Overcoming Social Distancing in Semi-Intimate Relationships with SansTouch. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 651, 13 pages. https://doi.org/10.1145/3411764.3445612
  82. VisualTouch: Enhancing Affective Touch Communication with Multi-Modality Stimulation. In 2019 International Conference on Multimodal Interaction (Suzhou, China) (ICMI ’19). Association for Computing Machinery, New York, NY, USA, 114–123. https://doi.org/10.1145/3340555.3353733
  83. Goodbye Text, Hello Emoji: Mobile Communication on WeChat in China. 748–759.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets