Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Memoro: Using Large Language Models to Realize a Concise Interface for Real-Time Memory Augmentation (2403.02135v1)

Published 4 Mar 2024 in cs.HC

Abstract: People have to remember an ever-expanding volume of information. Wearables that use information capture and retrieval for memory augmentation can help but can be disruptive and cumbersome in real-world tasks, such as in social settings. To address this, we developed Memoro, a wearable audio-based memory assistant with a concise user interface. Memoro uses a LLM to infer the user's memory needs in a conversational context, semantically search memories, and present minimal suggestions. The assistant has two interaction modes: Query Mode for voicing queries and Queryless Mode for on-demand predictive assistance, without explicit query. Our study of (N=20) participants engaged in a real-time conversation demonstrated that using Memoro reduced device interaction time and increased recall confidence while preserving conversational quality. We report quantitative results and discuss the preferences and experiences of users. This work contributes towards utilizing LLMs to design wearable memory augmentation systems that are minimally disruptive.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (72)
  1. Remembering can cause forgetting: retrieval dynamics in long-term memory. Journal of Experimental Psychology: Learning, Memory, and Cognition 20, 5 (1994), 1063. https://doi.org/10.1037/0278-7393.20.5.1063
  2. Investigating Proactive Search Support in Conversations. In Proceedings of the 2018 Designing Interactive Systems Conference (Hong Kong, China) (DIS ’18). Association for Computing Machinery, New York, NY, USA, 1295–1307. https://doi.org/10.1145/3196709.3196734
  3. SearchBot: Supporting Voice Conversations with Proactive Search. In Companion of the 2018 ACM Conference on Computer Supported Cooperative Work and Social Computing (Jersey City, NJ, USA) (CSCW ’18). Association for Computing Machinery, New York, NY, USA, 9–12. https://doi.org/10.1145/3272973.3272990
  4. Memory. Psychology Press.
  5. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of usability studies 4, 3 (2009), 114–123.
  6. Marine Beaudoin and Olivier Desrichard. 2011. Are memory self-efficacy and memory performance related? A meta-analysis. Psychological bulletin 137, 2 (2011), 211. https://doi.org/doi/10.1037/a0022106
  7. VIMES: A Wearable Memory Assistance System for Automatic Information Retrieval. (2020), 3191–3200. https://doi.org/10.1145/3394171.3413663
  8. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77–101. https://doi.org/10.1191/1478088706qp063oa
  9. John Brooke et al. 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4–7.
  10. Roger Brown and David McNeill. 1966. The “tip of the tongue” phenomenon. Journal of Verbal Learning and Verbal Behavior 5, 4 (1966), 325–337. https://doi.org/10.1016/S0022-5371(66)80040-3
  11. Language Models are Few-Shot Learners. arXiv:2005.14165 [cs.CL]
  12. Vannevar Bush et al. 1945. As we may think. The atlantic monthly 176, 1 (1945), 101–108.
  13. ParaGlassMenu: Towards Social-Friendly Subtle Interactions in Conversations. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–21. https://doi.org/10.1145/3544548.3581065
  14. Samantha Chan. 2020. Biosignal-Sensitive Memory Improvement and Support Systems. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1–7. https://doi.org/10.1145/3334480.3375031
  15. Samantha Chan. 2022. Augmenting Human Prospective Memory through Cognition-Aware Technologies. Ph. D. Dissertation. ResearchSpace@ Auckland. https://hdl.handle.net/2292/58810
  16. KinVoices: Using voices of friends and family in voice interfaces. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–25. https://doi.org/10.1145/3479590
  17. Prompto: Investigating Receptivity to Prompts Based on Cognitive Load from Memory Training Conversational Agent. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 4, Article 121 (dec 2020), 23 pages. https://doi.org/10.1145/3432190
  18. Prospero: A personal wearable memory coach. In Proceedings of the 10th Augmented Human International Conference 2019. 1–5. https://doi.org/10.1145/3311823.3311870
  19. Wearable Reasoner: towards enhanced human rationality through a wearable device with an explainable AI assistant. In Proceedings of the Augmented Humans International Conference. 1–12. https://doi.org/10.1145/3384657.3384799
  20. Ishita Dasgupta and Samuel J. Gershman. 2021. Memory as a Computational Resource. Trends in Cognitive Sciences 25, 3 (2021), 240–251. https://doi.org/10.1016/j.tics.2020.12.008
  21. Security and privacy implications of pervasive memory augmentation. IEEE Pervasive Computing 14, 1 (2015), 44–53. https://doi.org/10.1109/MPRV.2015.13
  22. Richard W. Devaul and Alex P. Pentland. 2004. The Memory Glasses: Wearable Computing for Just-in-Time Memory Support. Ph. D. Dissertation. USA. http://dspace.mit.edu/handle/1721.1/7582
  23. Acceptability of a lifelogging wearable camera in older adults with mild cognitive impairment: a mixed-method study. BMC geriatrics 19, 1 (2019), 1–10. https://doi.org/10.1186/s12877-019-1132-0
  24. News summarization and evaluation in the era of gpt-3. arXiv preprint arXiv:2209.12356 (2022). https://doi.org/10.48550/arXiv.2209.12356
  25. Ido Guy. 2016. Searching by Talking: Analysis of Voice Queries on Mobile Web Search. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (Pisa, Italy) (SIGIR ’16). Association for Computing Machinery, New York, NY, USA, 35–44. https://doi.org/10.1145/2911451.2911525
  26. You Know What I’m Saying: Designing Conversational Strategies of AI Agent for Tip of the Tongue Phenomenon. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems. 1–6. https://doi.org/10.1145/3544549.3585670
  27. Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology. Vol. 52. Elsevier, 139–183. https://doi.org/10.1016/S0166-4115(08)62386-9
  28. Remembering through lifelogging: A survey of human memory augmentation. Pervasive and Mobile Computing 27 (2016), 14–26. https://doi.org/10.1016/j.pmcj.2015.12.002
  29. The Personal Audio Loop: Designing a Ubiquitous Audio-Based Memory Aid. In Mobile Human-Computer Interaction - MobileHCI 2004, Stephen Brewster and Mark Dunlop (Eds.). Springer Berlin Heidelberg, 168–179. https://doi.org/10.1007/978-3-540-28637-0_15
  30. Designing a willing-to-use-in-public hand gestural interaction technique for smart glasses. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 4203–4215. https://doi.org/10.1145/2858036.2858436
  31. Nick Hunn. 2014. Hearables—the new wearables. Wearable Technologies (2014).
  32. Muhammad Zahid Iqbal and Abraham G. Campbell. 2023. Adopting smart glasses responsibly: potential benefits, ethical, and privacy concerns with Ray-Ban stories. AI and Ethics 3, 1 (2023), 325–327. https://doi.org/10.1007/s43681-022-00155-7
  33. Atlas: Few-shot Learning with Retrieval Augmented Language Models. arXiv:2208.03299 [cs.CL]
  34. Memento: An emotion-driven lifelogging system with wearables. ACM Transactions on Sensor Networks (TOSN) 15, 1 (2019), 1–23. https://doi.org/10.1145/3281630
  35. Norene Kelly and Stephen B. Gilbert. 2018. The wearer, the device, and its use: advances in understanding the social acceptability of wearables. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 62. SAGE Publications Sage CA: Los Angeles, CA, 1027–1031. https://doi.org/10.1177/1541931218621237
  36. PAL: A Wearable Platform for Real-time, Personalized and Context-Aware Health and Cognition Support. CoRR abs/1905.01352 (2019). arXiv:1905.01352 http://arxiv.org/abs/1905.01352
  37. Visual Captions: Augmenting Verbal Communication with On-the-fly Visuals. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–21. https://doi.org/10.1145/3544548.3581566
  38. Social acceptability in HCI: A survey of methods, measures, and design strategies. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–19. https://doi.org/10.1145/3313831.3376162
  39. Overview of lifelogging: current challenges and advances. IEEE Access 9 (2021), 62630–62641. https://doi.org/10.1109/ACCESS.2021.3073469
  40. Efficient search for approximate nearest neighbor in high dimensional spaces. In Proceedings of the thirtieth annual ACM symposium on Theory of computing. 614–623. https://doi.org/10.1145/276698.276877
  41. The Design of a Human Memory Prosthesis. Comput. J. 37, 3 (01 1994), 153–163. https://doi.org/10.1093/comjnl/37.3.153 arXiv:https://academic.oup.com/comjnl/article-pdf/37/3/153/1127676/370153.pdf
  42. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401 [cs.CL]
  43. Memory and decision making. Handbook of consumer behavior (1991), 1–9.
  44. Steve Mann. 1996. Wearable Tetherless Computer-Mediated Reality: WearCam as a Wearable Face-Recognizer, and Other Applications for the Disabled Papers. (1996).
  45. Natalia Marmasse. 1999. comMotion: a context-aware communication system. In CHI’99 Extended Abstracts on Human Factors in Computing Systems. 320–321. https://doi.org/10.1145/632716.632910
  46. Exploring User Expectations of Proactive AI Systems. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 4, Article 146 (dec 2020), 22 pages. https://doi.org/10.1145/3432193
  47. Neville Moray. 1959. Attention in dichotic listening: Affective cues and the influence of instructions. Quarterly journal of experimental psychology 11, 1 (1959), 56–60. https://doi.org/10.1080/17470215908416289
  48. Wearable Systems without Experiential Disruptions: Exploring the Impact of Device Feedback Changes on Explicit Awareness, Physiological Synchrony, Sense of Agency, and Device-Body Ownership. Frontiers in Computer Science 5 (2023), 1289869. https://doi.org/10.3389/fcomp.2023.1289869
  49. Cloudbits: Supporting Conversations through Augmented Zero-Query Search Visualization. In Proceedings of the 5th Symposium on Spatial User Interaction (Brighton, United Kingdom) (SUI ’17). Association for Computing Machinery, New York, NY, USA, 30–38. https://doi.org/10.1145/3131277.3132173
  50. Khalid Nassiri and Moulay Akhloufi. 2023. Transformer models used for text-based question answering systems. Applied Intelligence 53, 9 (2023), 10602–10635. https://doi.org/10.1007/s10489-022-04052-8
  51. Everyday memory errors in older adults. Aging, Neuropsychology, and Cognition 20, 2 (2013), 220–242. https://doi.org/10.1080/13825585.2012.690365 arXiv:https://doi.org/10.1080/13825585.2012.690365 PMID: 22694275.
  52. The global prevalence of dementia: A systematic review and metaanalysis. Alzheimer’s Dementia 9, 1 (2013), 63–75.e2. https://doi.org/10.1016/j.jalz.2012.11.007
  53. Aung Pyae and Tapani N. Joelsson. 2018. Investigating the usability and user experiences of voice user interface: a case of Google home smart speaker. In Proceedings of the 20th international conference on human-computer interaction with mobile devices and services adjunct. 127–131. https://doi.org/10.1145/3236112.3236130
  54. Scaling Language Models: Methods, Analysis & Insights from Training Gopher. arXiv:2112.11446 [cs.CL]
  55. Björn Rasch and Jan Born. 2013. About sleep’s role in memory. Physiological reviews (2013). https://doi.org/10.1152/physrev.00032.2012
  56. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019). https://doi.org/10.48550/arXiv.1908.10084
  57. Juniper Research. 2019. Digital Voice Assistants in Use to Triple to 8 Billion by 2023, Driven by Smart Home Devices. (2019).
  58. Speaker verification using adapted Gaussian mixture models. Digital signal processing 10, 1-3 (2000), 19–41. https://doi.org/10.1006/dspr.1999.0361
  59. Bradley J. Rhodes. 1997. The wearable remembrance agent: A system for augmented memory. Personal Technologies 1, 4 (01 Dec 1997), 218–224. https://doi.org/10.1007/BF01682024
  60. B. J. Rhodes and P. Maes. 2000. Just-in-time information retrieval agents. IBM Systems Journal 39, 3.4 (2000), 685–704. https://doi.org/10.1147/sj.393.0685
  61. Multimodal inductive transfer learning for detection of Alzheimer’s dementia and its severity. arXiv preprint arXiv:2009.00700 (2020).
  62. Daniel L. Schacter. 1999. The seven sins of memory: insights from psychology and cognitive neuroscience. American psychologist 54, 3 (1999), 182. https://doi.org/doi/10.1037/0003-066X.54.3.182
  63. Valentin Schwind and Niels Henze. 2020. Anticipated User Stereotypes Systematically Affect the Social Acceptability of Mobile Devices. In Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society. 1–12. https://doi.org/10.1145/3419249.3420113
  64. Lifelogging: Archival and retrieval of continuously recorded audio using wearable devices. In 2012 IEEE International Conference on Emerging Signal Processing Applications. 99–102. https://doi.org/10.1109/ESPA.2012.6152455
  65. Improving the domain adaptation of retrieval augmented generation (RAG) models for open domain question answering. Transactions of the Association for Computational Linguistics 11 (2023), 1–17. https://doi.org/10.1162/tacl_a_00530
  66. Conciseness: An Overlooked Language Task. arXiv:2211.04126 [cs.CL]
  67. Investigating Users’ Preferences and Expectations for Always-Listening Voice Assistants. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 4, Article 153 (sep 2020), 23 pages. https://doi.org/10.1145/3369807
  68. Christophe Van Gysel. 2023. Modeling Spoken Information Queries for Virtual Assistants: Open Problems, Challenges and Opportunities. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. https://doi.org/10.48550/arXiv.2304.13149
  69. An Audio-Based Personal Memory Aid. (2004), 400–417.
  70. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 (2021).
  71. Multimodal Mobile Interactions: Usability Studies in Real World Settings. In Proceedings of the 13th International Conference on Multimodal Interfaces (Alicante, Spain) (ICMI ’11). Association for Computing Machinery, New York, NY, USA, 361–368. https://doi.org/10.1145/2070481.2070551
  72. Kiichiro Yamano and Katunobu Itou. 2009. Browsing Audio Life-log Data Using Acoustic and Location Information. In 2009 Third International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies. 96–101. https://doi.org/10.1109/UBICOMM.2009.57
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets