Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Neighbor-Environment Observer: An Intelligent Agent for Immersive Working Companionship (2403.18331v1)

Published 27 Mar 2024 in cs.HC

Abstract: Human-computer symbiosis is a crucial direction for the development of artificial intelligence. As intelligent systems become increasingly prevalent in our work and personal lives, it is important to develop strategies to support users across physical and virtual environments. While technological advances in personal digital devices, such as personal computers and virtual reality devices, can provide immersive experiences, they can also disrupt users' awareness of their surroundings and enhance the frustration caused by disturbances. In this paper, we propose a joint observation strategy for artificial agents to support users across virtual and physical environments. We introduce a prototype system, neighbor-environment observer (NEO), that utilizes non-invasive sensors to assist users in dealing with disruptions to their immersive experience. System experiments evaluate NEO from different perspectives and demonstrate the effectiveness of the joint observation strategy. A user study is conducted to evaluate its usability. The results show that NEO could lessen users' workload with the learned user preference. We suggest that the proposed strategy can be applied to various smart home scenarios.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (70)
  1. https://github.com/cvzone/cvzone.
  2. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
  3. On-the-fly detection of user engagement decrease in spontaneous human–robot interaction using recurrent and deep neural networks. International Journal of Social Robotics, 11:815–828, 2019.
  4. A neural probabilistic language model. Advances in neural information processing systems, 13, 2000.
  5. Smart-home environment to support homework activities for children. IEEE Access, 8:160251–160267, 2020.
  6. Habitat: An iot solution for independent elderly. Sensors, 19(5):1258, 2019.
  7. G. Bradski. The opencv library. Dr. Dobb’s Journal: Software Tools for the Professional Programmer, 25(11):120–123, 2000.
  8. Multimodal human-human-robot interactions (mhhri) dataset for studying personality and engagement. IEEE Transactions on Affective Computing, 10(4):484–497, 2017.
  9. A virtual reality based fire training simulator integrated with fire dynamics data. Fire Safety Journal, 50:12–24, 2012.
  10. Virtual reality sickness: a review of causes and measurements. International Journal of Human–Computer Interaction, 36(17):1658–1682, 2020.
  11. Natural language processing (almost) from scratch. Journal of machine learning research, 12(ARTICLE):2493–2537, 2011.
  12. M. Danninger and R. Stiefelhagen. A context-aware virtual secretary in a smart office environment. In Proceedings of the 16th ACM international conference on Multimedia, pages 529–538, 2008.
  13. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  14. Carla: An open urban driving simulator. arXiv preprint arXiv:1711.03938, 2017.
  15. A. Dowdall and M. Perry. The millennium home: Domestic technology to support independent-living older people. In Proceedings of the 1st Equator IRC Workshop, pages 1–15. Citeseer, 2001.
  16. Examining the robustness of sensor-based statistical models of human interruptibility. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 207–214, 2004.
  17. Automatically classifying user engagement for dynamic multi-party human–robot interaction. International Journal of Social Robotics, 9(5):659–674, 2017.
  18. Splitnet: Sim2sim and task2task transfer for embodied visual navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1022–1031, 2019.
  19. Embodied intelligence via learning and evolution. Nature communications, 12(1):5721, 2021.
  20. Learning latent dynamics for planning from pixels. In Proceedings of the International Conference on Machine Learning, pages 2555–2565. PMLR, 2019.
  21. Illuminating the dark spaces of healthcare with ambient intelligence. Nature, 585(7824):193–202, 2020.
  22. Development of nasa-tlx (task load index): Results of empirical and theoretical research. In Advances in psychology, volume 52, pages 139–183. Elsevier, 1988.
  23. J. Ho and S. S. Intille. Using context-aware computing to reduce the perceived burden of interruptions from mobile devices. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 909–918, 2005.
  24. A human touch: Social touch increases the perceived human-likeness of agents in virtual reality. In Proceedings of the 2020 CHI conference on human factors in computing systems, pages 1–11. ACM, 2020.
  25. E. Horvitz and J. Apacible. Learning and reasoning about interruption. In Proceedings of the 5th international conference on Multimodal interfaces, pages 20–27, 2003.
  26. Predicting human interruptibility with sensors: a wizard of oz feasibility study. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 257–264, 2003.
  27. Spatial transformer networks. Advances in neural information processing systems, 28, 2015.
  28. A cordial sync: Going beyond marginal policies for multi-agent embodied tasks. In Proceedings of European Conference on Computer Vision (ECCV), pages 471–490. Springer, 2020.
  29. Vroom: virtual robot overlay for online meetings. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–10, 2020.
  30. The aware home: A living laboratory for ubiquitous computing research. In Cooperative Buildings: Integrating Information, Organizations, and Architecture, pages 191–198. Springer, 1999.
  31. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474, 2017.
  32. Utilising context ontology in mobile device application personalisation. In Proceedings of the 3rd international conference on Mobile and ubiquitous multimedia, pages 133–140, 2004.
  33. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84–90, 2017.
  34. S. M. LaValle. Planning algorithms. Cambridge university press, 2006.
  35. Deep learning. Nature, 521(7553):436–444, 2015.
  36. Visual fatigue induced by watching virtual reality device and the effect of anisometropia. Ergonomics, 64(12):1522–1531, 2021.
  37. High performance visual tracking with siamese region proposal network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  38. Earthquake safety training through virtual drills. Proceedings of IEEE Transactions on Visualization & Computer Graph (TVCG), 23(4):1275–1284, 2017.
  39. Igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. arXiv preprint arXiv:2108.03272, 2021.
  40. J. C. Licklider. Man-computer symbiosis. IRE transactions on human factors in electronics, (1):4–11, 1960.
  41. Mediapipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172, 2019.
  42. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
  43. Engagement in human-agent interaction: An overview. Frontiers in Robotics and AI, 7:92, 2020.
  44. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8494–8502, 2018.
  45. A review of attention detection in online learning. Artificial Intelligence in Education and Teaching Assessment, pages 87–100, 2021.
  46. Exploring smart agents for the interaction with multimodal mediated environments. Multimodal Technologies and Interaction, 4(2):27, 2020.
  47. Personalized estimation of engagement from videos using active learning with deep reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 217–226. IEEE, 2019.
  48. Multi-modal active learning from human data: A deep reinforcement learning approach. In Proceedings of the International Conference on Multimodal Interaction, pages 6–15, 2019.
  49. Design and evaluation of a short version of the user experience questionnaire (ueq-s). International Journal of Interactive Multimedia and Artificial Intelligence, 4(6):103–108, 2017.
  50. Validation of a novel virtual reality simulator for robotic surgery. The Scientific World Journal, 2014, 2014.
  51. C. Shen. A transdisciplinary review of deep learning research and its relevance for water resources scientists. Water Resources Research, 54(11):8558–8593, 2018.
  52. A. Shrestha and A. Mahmood. Review of deep learning algorithms and architectures. IEEE access, 7:53040–53065, 2019.
  53. Theory of minds: Understanding behavior in groups through inverse planning. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 6163–6170, 2019.
  54. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. In Proceedings of the Conference on Robot Learning, pages 477–490. PMLR, 2022.
  55. Reinforcement learning: An introduction. MIT press, 2018.
  56. D. Szafir and B. Mutlu. Pay attention! designing adaptive agents that monitor and improve user engagement. In Proceedings of the SIGCHI conference on human factors in computing systems, pages 11–20, 2012.
  57. Y. Tao and P. Lopes. Integrating real-world distractions into virtual reality. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pages 1–16, 2022.
  58. N. Thakur and C. Y. Han. An ambient intelligence-based human behavior monitoring framework for ubiquitous environments. Information, 12(2):81, 2021.
  59. S. Thrun. Probabilistic robotics. Communications of the ACM, 45(3):52–57, 2002.
  60. Z. Tu and S.-C. Zhu. Image segmentation by data-driven markov chain monte carlo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(5):657–673, 2002.
  61. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  62. Drivers’ attention detection: a systematic literature review. arXiv preprint arXiv:2204.03741, 2022.
  63. Interactive gibson benchmark: A benchmark for interactive navigation in cluttered environments. IEEE Robotics and Automation Letters, 5(2):713–720, 2020.
  64. Sedentary behavior and cardiovascular morbidity and mortality: a science advisory from the american heart association. Circulation, 134(13):e262–e279, 2016.
  65. Z. Zhang. Symmetrical cognition between physical humans and virtual agents. In Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pages 587–588. IEEE, 2021.
  66. Z. Zhang. Building symmetrical reality systems for cooperative manipulation. In Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pages 751–752. IEEE, 2023.
  67. Symmetrical reality: Toward a unified framework for physical and virtual reality. In Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 1275–1276. IEEE, 2019.
  68. Inverse augmented reality: A virtual agent’s perspective. In Proceedings of the IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pages 154–157. IEEE, 2018.
  69. Mediated atmospheres: A multimodal mediated work environment. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(2):1–23, 2017.
  70. A stochastic grammar of images. Foundations and Trends® in Computer Graphics and Vision, 2(4):259–362, 2007.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets