Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

On the Emergence of Symmetrical Reality (2401.15132v1)

Published 26 Jan 2024 in cs.HC and cs.AI

Abstract: AI has revolutionized human cognitive abilities and facilitated the development of new AI entities capable of interacting with humans in both physical and virtual environments. Despite the existence of virtual reality, mixed reality, and augmented reality for several years, integrating these technical fields remains a formidable challenge due to their disparate application directions. The advent of AI agents, capable of autonomous perception and action, further compounds this issue by exposing the limitations of traditional human-centered research approaches. It is imperative to establish a comprehensive framework that accommodates the dual perceptual centers of humans and AI agents in both physical and virtual worlds. In this paper, we introduce the symmetrical reality framework, which offers a unified representation encompassing various forms of physical-virtual amalgamations. This framework enables researchers to better comprehend how AI agents can collaborate with humans and how distinct technical pathways of physical-virtual integration can be consolidated from a broader perspective. We then delve into the coexistence of humans and AI, demonstrating a prototype system that exemplifies the operation of symmetrical reality systems for specific tasks, such as pouring water. Subsequently, we propose an instance of an AI-driven active assistance service that illustrates the potential applications of symmetrical reality. This paper aims to offer beneficial perspectives and guidance for researchers and practitioners in different fields, thus contributing to the ongoing research about human-AI coexistence in both physical and virtual environments.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (150)
  1. Neural additive models: Interpretable machine learning with neural nets. Advances in Neural Information Processing Systems, 34:4699–4711, 2021.
  2. Guidelines for human-ai interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–13, 2019.
  3. Extended reality in medical practice. Current treatment options in cardiovascular medicine, 21(4):1–12, 2019.
  4. A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5):469–483, 2009.
  5. R. T. Azuma. A survey of augmented reality. Presence: Teleoperators & Virtual Environments, 6(4):355–385, 1997.
  6. J. Bach. A motivational system for cognitive ai. In Proceedings of the International Conference on Artificial General Intelligence, pages 232–242. Springer, 2011.
  7. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS One, 10(7):e0130140, 2015.
  8. Merging virtual objects with the real world: Seeing ultrasound imagery within the patient. ACM SIGGRAPH Computer Graphics, 26(2):203–210, 1992.
  9. On-line simultaneous learning and recognition of everyday activities from virtual reality performances. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3510–3515. IEEE, 2017.
  10. Design, modeling and control of a 5-dof light-weight robot arm for aerial manipulation. In Proceedings of the 23rd Mediterranean Conference on Control and Automation (MED), pages 853–858. IEEE, 2015.
  11. Shared spaces: Transportation, artificiality, and spatiality. In Proceedings of the 1996 ACM Conference on Computer Supported Cooperative Work, pages 77–86, 1996.
  12. Crowded collaborative virtual environments. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, pages 59–66, 1997.
  13. Understanding and constructing shared spaces with mixed-reality boundaries. ACM Transactions on Computer-Human Interaction (TOCHI), 5(3):185–223, 1998.
  14. Magnetosensitive e-skins with directional perception for augmented reality. Science Advances, 4(1):eaao2623, 2018.
  15. Media spaces: Bringing people together in a video, audio, and computing environment. Communications of the ACM, 36(1):28–46, 1993.
  16. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
  17. S. Boschert and R. Rosen. Digital twin—the simulation aspect. In Mechatronic Futures, pages 59–74. Springer, 2016.
  18. F. P. Brooks. What’s real about virtual reality? IEEE Computer Graphics and Applications, 19(6):16–27, 1999.
  19. Project gropehaptic displays for scientific visualization. ACM SIGGRAPH Computer Graphics, 24(4):177–185, 1990.
  20. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901, 2020.
  21. Ai challenges in human-robot cognitive teaming. arXiv preprint arXiv:1707.04775, 2017.
  22. B. Coleman. Using sensor inputs to affect virtual and real environments. IEEE Pervasive Computing, 8(3):16–23, 2009.
  23. Mixed reality: A survey. In Human Machine Interaction, pages 47–68. Springer, 2009.
  24. M. Crang and S. M. H. Mohamad. Researching virtual communities. Key Methods in Geography, page 270, 2016.
  25. Interactive storytelling and gaming environments for museums: The interactive storytelling exhibition project. In Proceedings of the International Conference on Technologies for E-Learning and Digital Entertainment, pages 104–115. Springer, 2007.
  26. Embodied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 2054–2063, 2018.
  27. A. Das and P. Rad. Opportunities and challenges in explainable artificial intelligence (xai): A survey. arXiv preprint arXiv:2006.11371, 2020.
  28. A review of extended reality (xr) technologies for manufacturing training. Technologies, 8(4):77, 2020.
  29. A survey of embodied ai: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 2022.
  30. A tale of two explanations: Enhancing human trust by explaining robot behavior. Science Robotics, 4(37), 2019.
  31. Video see-through design for merging of real and virtual environments. In Proceedings of the IEEE Virtual Reality Annual International Symposium, pages 223–233. IEEE, 1993.
  32. Understanding representations learned in deep architectures. Technical report, Canada, Tech. Rep, 1355, 2010.
  33. A space based model for user interaction in shared synthetic environments. In Proceedings of the INTERACT’93 and CHI’93 Conference on Human Factors in Computing Systems, pages 43–48, 1993.
  34. D. Fass. Augmented human engineering: A theoretical and experimental approach to human systems integration. In Systems Engineering-Practice and Theory, pages 257–276. InTech Open Access Publisher, 2012.
  35. Testing and validating extended reality (xr) technologies in manufacturing. Procedia Manufacturing, 25:31–38, 2018.
  36. Windows on the world: 2d windows for 3d augmented reality. In Proceedings of the 6th Annual ACM Symposium on User Interface Software and Technology, pages 145–155, 1993.
  37. Knowledge-based augmented reality. Communications of the ACM, 36(7):53–62, 1993.
  38. Physical spaces, virtual places and social worlds: A study of work in the virtual. In Proceedings of the 1996 ACM Conference on Computer Supported Cooperative Work, pages 334–343, 1996.
  39. D. B. Fogel. Evolutionary computation: Toward a new philosophy of machine intelligence, volume 1. John Wiley & Sons, 2006.
  40. Companion cognitive systems: A step toward human-level ai. AI Magazine, 27(2):83–83, 2006.
  41. Threedworld: A platform for interactive multi-modal physical simulation. arXiv preprint arXiv:2007.04954, 2020.
  42. Joint mind modeling for explanation generation in complex human-robot collaborative tasks. In Proceedings of the 29th IEEE international conference on robot and human interactive communication (RO-MAN), pages 1119–1126. IEEE, 2020.
  43. Show me what you can do: Capability calibration on reachable workspace for human-robot collaboration. IEEE Robotics and Automation Letters, 7(2):2644–2651, 2022.
  44. Intuitive robot tasks with augmented reality and virtual obstacles. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 6026–6031. IEEE, 2014.
  45. Realizing a video environment: Europarc’s rave system. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 27–35, 1992.
  46. D. Gelernter. Mirror worlds: Or the day software puts the universe in a shoebox… How it will happen and what it will mean. Oxford University Press, 1993.
  47. B. Goertzel. Artificial general intelligence: Concept, state of the art, and future prospects. Journal of Artificial General Intelligence, 5(1):1, 2014.
  48. Augmented artificial intelligence: A conceptual framework. arXiv preprint arXiv:1802.02172, 2018.
  49. C. Greenhalgh and S. Benford. Massive: A collaborative virtual environment for teleconferencing. ACM Transactions on Computer-Human Interaction (TOCHI), 2(3):239–261, 1995.
  50. Robot companion for domestic health assistance: Implementation, test and case study under everyday conditions in private apartments. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5992–5999. IEEE, 2015.
  51. Exploring the differences of visual discomfort caused by long-term immersion between virtual environments and physical environments. In Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 443–452. IEEE, 2020.
  52. Evaluation of maslows hierarchy of needs on long-term use of hmds-a case study of office environment. In Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 948–949. IEEE, 2019.
  53. S. Haag and R. Anderl. Digital twin–proof of concept. Manufacturing Letters, 15:64–66, 2018.
  54. Learning from humans: Agent modeling with individual human behaviors. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 41(1):1–9, 2010.
  55. irobot-factory: An intelligent robot factory based on cognitive manufacturing and edge computing. Future Generation Computer Systems, 90:569–577, 2019.
  56. S. E. Hudson and I. Smith. Techniques for addressing fundamental privacy and disruption tradeoffs in awareness support systems. In Proceedings of the 1996 ACM Conference on Computer Supported Cooperative Work, pages 248–257, 1996.
  57. Global explanations of neural networks: Mapping the landscape of predictions. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 279–287, 2019.
  58. Robovie: An interactive humanoid robot. Industrial Robot: An International Journal, 2001.
  59. Integration of interpersonal space and shared workspace: Clearboard design and experiments. ACM Transactions on Information Systems (TOIS), 11(4):349–375, 1993.
  60. H. Ishii and B. Ullmer. Tangible bits: Towards seamless interfaces between people, bits and atoms. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, pages 234–241, 1997.
  61. M. H. Jarrahi. Artificial intelligence and the future of work: Human-ai symbiosis in organizational decision making. Business Horizons, 61(4):577–586, 2018.
  62. Deployment of a service robot to help older people. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5990–5995. IEEE, 2010.
  63. L. Jensen and F. Konradsen. A review of the use of virtual reality head-mounted displays in education and training. Education and Information Technologies, 23(4):1515–1529, 2018.
  64. Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks? In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 746–753. IEEE, 2017.
  65. Gpt-4 passes the bar exam. Available at SSRN 4389233, 2023.
  66. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In Proceedings of the International Conference on Machine Learning, pages 2668–2677. PMLR, 2018.
  67. D. Koller and N. Friedman. Probabilistic graphical models: Principles and techniques. MIT press, 2009.
  68. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474, 2017.
  69. M. Kosinski. Theory of mind may have spontaneously emerged in large language models. arXiv preprint arXiv:2302.02083, 2023.
  70. J. Krumm. Ubiquitous computing fundamentals. CRC Press, 2018.
  71. R. Kurzweil. How to create a mind: The secret of human thought revealed. Penguin, 2013.
  72. Evaluation of a remote training approach for teaching seniors to use a telehealth system. International Journal of Medical Informatics, 78(11):732–744, 2009.
  73. Unmasking clever hans predictors and assessing what machines really learn. Nature Communications, 10(1):1–8, 2019.
  74. Deep learning. Nature, 521(7553):436–444, 2015.
  75. Visual-inertial hand motion tracking with robustness against occlusion, interference, and contact. Science Robotics, 6(58):eabe1315, 2021.
  76. Towards fully autonomous driving: Systems and algorithms. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV), pages 163–168. IEEE, 2011.
  77. The internet of things: A survey. Information Systems Frontiers, 17:243–259, 2015.
  78. Artificial intelligence for hci: A modern approach. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–8, 2020.
  79. B. Libet. Do we have free will? Journal of Consciousness Studies, 6(8-9):47–57, 1999.
  80. J. Lifton and J. A. Paradiso. Dual reality: Merging the real and virtual. In Proceedings of the International Conference on Facets of Virtual Environments, pages 12–28. Springer, 2009.
  81. A virtual reality platform for dynamic human-scene interaction. In SIGGRAPH ASIA 2016 virtual reality meets physical reality: Modelling and simulating virtual humans and environments, pages 1–4. 2016.
  82. Baxter’s homunculus: Virtual reality spaces for teleoperation in manufacturing. IEEE Robotics and Automation Letters, 3(1):179–186, 2018.
  83. A glove-based system for studying hand-object manipulation via joint pose and force sensing. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6617–6624. IEEE, 2017.
  84. Interactive robot knowledge patching using augmented reality. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 1947–1954. IEEE, 2018.
  85. Brain intelligence: Go beyond artificial intelligence. Mobile Networks and Applications, 23(2):368–375, 2018.
  86. An observational and interview study on personal digital assistant (pda) uses by clinicians in different contexts. Human-Centred Computing: Cognitive, Social and Ergonomic Aspects, pages 93–97, 2019.
  87. A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30, 2017.
  88. All reality: Virtual, augmented, mixed (x), mediated (x, y), and multimediated reality. arXiv preprint arXiv:1804.08386, 2018.
  89. G. Mantovani and G. Riva. “real” presence: How different ontologies generate different criteria for presence, telepresence, and virtual presence. Presence, 8(5):540–550, 1999.
  90. A. H. Maslow. A theory of human motivation. Psychological Review, 50(4):370, 1943.
  91. S. Mateus and J. Branch. Intelligent virtual environment using a methodology oriented to agents. In Proceedings of the International Conference on Virtual, Augmented and Mixed Reality, pages 714–723. Springer, 2016.
  92. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6):1–35, 2021.
  93. A taxonomy of real and virtual world display integration. Mixed Reality: Merging Real and Virtual Worlds, 1(1999):1–26, 1999.
  94. P. Milgram and F. Kishino. A taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems, 77(12):1321–1329, 1994.
  95. Augmented reality: A class of displays on the reality-virtuality continuum. In Telemanipulator and Telepresence Technologies, volume 2351, pages 282–292. International Society for Optics and Photonics, 1995.
  96. Applications of augmented reality for human-robot communication. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’93), volume 3, pages 1467–1472. IEEE, 1993.
  97. M. Minsky. Society of Mind. Simon and Schuster, 1988.
  98. R. R. Nadikattu. The emerging role of artificial intelligence in modern society. International Journal of Creative Research Thoughts, 2016.
  99. OpenAI. Gpt-4 technical report. arXiv, pages 2303–08774, 2023.
  100. Z. Pan and D. Manocha. Feedback motion planning for liquid pouring using supervised learning. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1252–1259. IEEE, 2017.
  101. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023.
  102. The tong test: Evaluating artificial general intelligence through dynamic embodied physical and social interactions. Engineering, 2023.
  103. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8494–8502, 2018.
  104. Human-robot interaction in a shared augmented reality workspace. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11413–11418. IEEE, 2020.
  105. Machine behaviour. Nature, 568(7753):477–486, 2019.
  106. Extended reality (xr) remote research: A survey of drawbacks and opportunities. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–13, 2021.
  107. When a robot teaches humans: Automated feedback selection accelerates motor learning. Science Robotics, 4(27):eaav1560, 2019.
  108. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135–1144, 2016.
  109. The mirror world: Preparing for mixed-reality living. IEEE Pervasive Computing, 14(2):60–63, 2015.
  110. J. S. Roo and M. Hachet. One reality: Augmenting how the physical world is experienced by combining multiple mixed reality modalities. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pages 787–795, 2017.
  111. S. Russell. Human compatible: Artificial intelligence and the problem of control. Penguin, 2019.
  112. M. Ryan. In ai we trust: Ethics, artificial intelligence, and reliability. Science and Engineering Ethics, 26(5):2749–2767, 2020.
  113. G. Salomon. Ai in reverse: Computer tools that turn cognitive. Journal of Educational Computing Research, 4(2):123–139, 1988.
  114. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9339–9347, 2019.
  115. igibson 1.0: A simulation environment for interactive tasks in large realistic scenes. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7520–7527. IEEE, 2020.
  116. A. Sheth. Internet of things to smart iot through semantic, cognitive, and perceptual computing. IEEE Intelligent Systems, 31(2):108–112, 2016.
  117. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
  118. Revisiting milgram and kishino’s reality-virtuality continuum. Frontiers in Virtual Reality, 2:27, 2021.
  119. D. Sperber and D. Wilson. Relevance: Communication and cognition, volume 142. Citeseer, 1986.
  120. extended reality (xr) environments. Handbook of Human Factors and Ergonomics, pages 782–815, 2021.
  121. Cognitive science: An introduction. MIT press, 1995.
  122. Neighbor-environment observer: An intelligent agent for immersive working companionship. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–14, 2023.
  123. Reinforcement learning: An introduction. MIT press, 2018.
  124. “studierstube”: An environment for collaboration in augmented reality. Virtual Reality, 3(1):37–48, 1998.
  125. H. Takemura and F. Kishino. Cooperative work environment using virtual workspace. In Proceedings of the 1992 ACM Conference on Computer-Supported Cooperative Work, pages 226–232, 1992.
  126. Digital twin in industry: State-of-the-art. IEEE Transactions on Industrial Informatics, 15(4):2405–2415, 2018.
  127. P. Thomas and W. David. Augmented reality: An application of heads-up display technology to manual manufacturing processes. In Proceedings of the Hawaii International Conference on System Sciences, pages 659–669, 1992.
  128. Smart healthcare: Making medical care more intelligent. Global Health Journal, 3(3):62–65, 2019.
  129. D. Van Krevelen and R. Poelman. A survey of augmented reality technologies, applications and limitations. International Journal of Virtual Reality, 9(2):1–20, 2010.
  130. J. Wallén. The history of the industrial robot. Linköping University Electronic Press, 2008.
  131. Spatially perturbed collision sounds attenuate perceived causality in 3d launching events. In Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 259–266. IEEE, 2018.
  132. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023.
  133. C. Wienrich and M. E. Latoschik. extended artificial intelligence: New prospects of human-ai interaction research. Frontiers in Virtual Reality, 2:686783, 2021.
  134. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9068–9079, 2018.
  135. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11097–11107, 2020.
  136. Vrgym: A virtual testbed for physical and interactive ai. In Proceedings of the ACM Turing Celebration Conference-China, pages 1–6, 2019.
  137. In situ bidirectional human-robot value alignment. Science Robotics, 7(68):eabm4183, 2022.
  138. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 1–8. IEEE, 2018.
  139. W. Zhang and B. Y. Lim. Towards relatable explainable ai with the perceptual process. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1–24, 2022.
  140. Inverse virtual reality: Intelligence-driven mutually mirrored world. In Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 735–736. IEEE, 2018.
  141. Symmetrical reality: Toward a unified framework for physical and virtual reality. In Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 1275–1276. IEEE, 2019.
  142. Z. Zhang and X. Wang. Machine intelligence matters: Rethink human-robot collaboration based on symmetrical reality. In Proceedings of the IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pages 225–228. IEEE, 2020.
  143. Inverse augmented reality: A virtual agent’s perspective. In Proceedings of the IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pages 154–157. IEEE, 2018.
  144. Graph-based hierarchical knowledge representation for robot task transfer from virtual to physical world. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020.
  145. Q. Zhao. A survey on virtual reality. Science in China Series F: Information Sciences, 52(3):348–400, 2009.
  146. Virtual reality: A state-of-the-art survey. International Journal of Automation and Computing, 6(4):319–325, 2009.
  147. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 3357–3364. IEEE, 2017.
  148. Understanding tools: Task-oriented object modeling, learning and recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2855–2864, 2015.
  149. A research framework of smart education. Smart Learning Environments, 3:1–17, 2016.
  150. Exploring ai ethics of chatgpt: A diagnostic analysis. arXiv preprint arXiv:2301.12867, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Zhenliang Zhang (35 papers)
  2. Zeyu Zhang (143 papers)
  3. Ziyuan Jiao (19 papers)
  4. Yao Su (25 papers)
  5. Hangxin Liu (32 papers)
  6. Wei Wang (1793 papers)
  7. Song-Chun Zhu (216 papers)
Citations (1)

Summary

We haven't generated a summary for this paper yet.

X Twitter Logo Streamline Icon: https://streamlinehq.com