Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Real-and-Present: Investigating the Use of Life-Size 2D Video Avatars in HMD-Based AR Teleconferencing (2401.02171v1)

Published 4 Jan 2024 in cs.HC

Abstract: Augmented Reality (AR) teleconferencing allows separately located users to interact with each other in 3D through agents in their own physical environments. Existing methods leveraging volumetric capturing and reconstruction can provide a high-fidelity experience but are often too complex and expensive for everyday usage. Other solutions target mobile and effortless-to-setup teleconferencing on AR Head Mounted Displays (HMD). They directly transplant the conventional video conferencing onto an AR-HMD platform or use avatars to represent remote participants. However, they can only support either a high fidelity or a high level of co-presence. Moreover, the limited Field of View (FoV) of HMDs could further influence users' immersive experience. To achieve a balance between fidelity and co-presence, we explore using life-size 2D video-based avatars (video avatars for short) in AR teleconferencing. Specifically, with the potential effect of FoV on users' perception of proximity, we first conduct a pilot study to explore the local-user-centered optimal placement of video avatars in small-group AR conversations. With the placement results, we then implement a proof-of-concept prototype of video-avatar-based teleconferencing. We conduct user evaluations with the prototype to verify its effectiveness in balancing fidelity and co-presence. Following the indication in the pilot study, we further quantitatively explore the effect of FoV size on the video avatar's optimal placement through a user study involving more FoV conditions in a VR-simulated environment. We regress placement models to serve as references for computationally determining video avatar placements in such teleconferencing applications on various existing AR HMDs and future ones with bigger FoVs.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (72)
  1. J. Lawrence, D. B. Goldman, S. Achar, G. M. Blascovich, J. G. Desloge, T. Fortes, E. M. Gomez, S. Häberling, H. Hoppe, A. Huibers, C. Knaus, B. Kuschak, R. Martin-Brualla, H. Nover, A. I. Russell, S. M. Seitz, and K. Tong, “Project starline: A high-fidelity telepresence system,” ACM Transactions on Graphics (Proc. SIGGRAPH Asia), vol. 40(6), 2021.
  2. S. Orts-Escolano, C. Rhemann, S. Fanello, W. Chang, A. Kowdle, Y. Degtyarev, D. Kim, P. L. Davidson, S. Khamis, M. Dou, V. Tankovich, C. Loop, Q. Cai, P. A. Chou, S. Mennicken, J. Valentin, V. Pradeep, S. Wang, S. B. Kang, P. Kohli, Y. Lutchyn, C. Keskin, and S. Izadi, “Holoportation: Virtual 3d teleportation in real-time,” in Proceedings of the 29th Annual Symposium on User Interface Software and Technology, ser. UIST ’16.   New York, NY, USA: Association for Computing Machinery, 2016, p. 741–754. [Online]. Available: https://doi.org/10.1145/2984511.2984517
  3. Y. Zhang, J. Yang, Z. Liu, R. Wang, G. Chen, X. Tong, and B. Guo, “Virtualcube: An immersive 3d video communication system,” IEEE Transactions on Visualization and Computer Graphics, vol. 28, no. 5, pp. 2146–2156, 2022.
  4. Microsoft, “Remote assist,” https://dynamics.microsoft.com/en-us/mixed-reality/remote-assist.
  5. I. Magic Leap, “Assist app,” https://www.magicleap.com/on-device/solutions/assist.
  6. Microsoft, “Microsoft mesh,” https://www.microsoft.com/mesh.
  7. Meta, “Horizon workrooms,” https://www.oculus.com/workrooms.
  8. T. Piumsomboon, G. A. Lee, J. D. Hart, B. Ens, R. W. Lindeman, B. H. Thomas, and M. Billinghurst, “Mini-me: An adaptive avatar for mixed reality remote collaboration,” in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, ser. CHI ’18.   New York, NY, USA: Association for Computing Machinery, 2018, p. 1–13. [Online]. Available: https://doi.org/10.1145/3173574.3173620
  9. L. Yoon, D. Yang, J. Kim, C. Chung, and S.-H. Lee, “Placement retargeting of virtual avatars to dissimilar indoor environments,” IEEE Transactions on Visualization and Computer Graphics, vol. 28, no. 3, pp. 1619–1633, 2022.
  10. X. Wang, H. Ye, C. Sandor, W. Zhang, and H. Fu, “Predict-and-drive: Avatar motion adaption in room-scale augmented reality telepresence with heterogeneous spaces,” IEEE Transactions on Visualization and Computer Graphics, vol. 28, no. 11, pp. 3705–3714, 2022.
  11. S. Junuzovic, K. Inkpen, J. Tang, M. Sedlins, and K. Fisher, “To see or not to see: A study comparing four-way avatar, video, and audio conferencing for work,” in Proceedings of the 2012 ACM International Conference on Supporting Group Work, ser. GROUP ’12.   New York, NY, USA: Association for Computing Machinery, 2012, p. 31–34. [Online]. Available: https://doi.org/10.1145/2389176.2389181
  12. Y. Pan and A. Steed, “A comparison of avatar-, video-, and robot-mediated interaction on users’ trust in expertise,” Frontiers in Robotics and AI, vol. 3, p. 12, 2016.
  13. F. De Simone, J. Li, H. G. Debarba, A. E. Ali, S. N. Gunkel, and P. Cesar, “Watching videos together in social virtual reality: An experimental study on user’s qoe,” in 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019, pp. 890–891.
  14. K. Yu, G. Gorbachev, U. Eck, F. Pankratz, N. Navab, and D. Roth, “Avatars for teleconsultation: Effects of avatar embodiment techniques on user perception in 3d asymmetric telepresence,” IEEE Transactions on Visualization and Computer Graphics, vol. 27, no. 11, pp. 4129–4139, 2021.
  15. R. Vertegaal, “The gaze groupware system: Mediating joint attention in multiparty communication and collaboration,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ser. CHI ’99.   New York, NY, USA: Association for Computing Machinery, 1999, p. 294–301. [Online]. Available: https://doi.org/10.1145/302979.303065
  16. M. Billinghurst and H. Kato, “Out and about—real world teleconferencing,” BT technology journal, vol. 18, no. 1, pp. 80–82, 2000.
  17. K. Otsuka, “Mmspace: Kinetically-augmented telepresence for small group-to-group conversations,” in 2016 IEEE Virtual Reality (VR), 2016, pp. 19–28.
  18. Z. He, K. Wang, B. Y. Feng, R. Du, and K. Perlin, “Gazechat: Enhancing virtual conferences with gaze-aware 3d photos,” in The 34th Annual ACM Symposium on User Interface Software and Technology, ser. UIST ’21.   New York, NY, USA: Association for Computing Machinery, 2021, p. 769–782. [Online]. Available: https://doi.org/10.1145/3472749.3474785
  19. S.-Y. Lee, I.-J. Kim, S. C. Ahn, M.-T. Lim, and H.-G. Kim, “Toward immersive telecommunication: 3d video avatar with physical interaction,” in Proceedings of the 2005 International Conference on Augmented Tele-Existence, ser. ICAT ’05.   New York, NY, USA: Association for Computing Machinery, 2005, p. 56–61. [Online]. Available: https://doi.org/10.1145/1152399.1152411
  20. T. Pejsa, J. Kantor, H. Benko, E. Ofek, and A. Wilson, “Room2room: Enabling life-size telepresence in a projected augmented reality environment,” in Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, ser. CSCW ’16.   New York, NY, USA: Association for Computing Machinery, 2016, p. 1716–1725. [Online]. Available: https://doi.org/10.1145/2818048.2819965
  21. A. Maimone, X. Yang, N. Dierk, A. State, M. Dou, and H. Fuchs, “General-purpose telepresence with head-worn optical see-through displays and projector-based lighting,” in 2013 IEEE Virtual Reality (VR), 2013, pp. 23–26.
  22. J. A. Insley, D. J. Sandin, and T. A. DeFanti, “Using video to create avators in virtual reality,” in ACM SIGGRAPH 97 Visual Proceedings: The art and interdisciplinary programs of SIGGRAPH’97, 1997, p. 128.
  23. C. W. Borst, N. G. Lipari, and J. W. Woodworth, “Teacher-guided educational vr: Assessment of live and prerecorded teachers guiding virtual field trips,” in 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2018, pp. 467–474.
  24. X. Cui, D. Khan, Z. He, and Z. Cheng, “Fusing surveillance videos and three-dimensional scene: A mixed reality system,” Computer Animation and Virtual Worlds, p. e2129, 2022.
  25. S. K. Pathi, A. Kristoffersson, A. Kiselev, and A. Loutfi, “F-formations for social interaction in simulation using virtual agents and mobile robotic telepresence systems,” Multimodal Technologies and Interaction, vol. 3, no. 4, p. 69, 2019.
  26. P. Marshall, Y. Rogers, and N. Pantidi, “Using f-formations to analyse spatial patterns of interaction in physical environments,” in Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work, ser. CSCW ’11.   New York, NY, USA: Association for Computing Machinery, 2011, p. 445–454. [Online]. Available: https://doi.org/10.1145/1958824.1958893
  27. E. T. Hall, R. L. Birdwhistell, B. Bock, P. Bohannan, A. R. Diebold Jr, M. Durbin, M. S. Edmonson, J. Fischer, D. Hymes, S. T. Kimball et al., “Proxemics [and comments and replies],” Current anthropology, vol. 9, no. 2/3, pp. 83–108, 1968.
  28. H. Hecht, R. Welsch, J. Viehoff, and M. R. Longo, “The shape of personal space,” Acta psychologica, vol. 193, pp. 113–122, 2019.
  29. A. Huang, P. Knierim, F. Chiossi, L. L. Chuang, and R. Welsch, “Proxemics for human-agent interaction in augmented reality,” in Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, ser. CHI ’22.   New York, NY, USA: Association for Computing Machinery, 2022. [Online]. Available: https://doi.org/10.1145/3491102.3517593
  30. Z.-M. Ye, J.-L. Chen, M. Wang, and Y.-L. Yang, “Paval: Position-aware virtual agent locomotion for assisted virtual reality navigation,” in 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2021, pp. 239–247.
  31. M. Czerwinski, D. S. Tan, and G. G. Robertson, “Women take a wider view,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ser. CHI ’02.   New York, NY, USA: Association for Computing Machinery, 2002, p. 195–202. [Online]. Available: https://doi.org/10.1145/503376.503412
  32. N. Kishishita, K. Kiyokawa, J. Orlosky, T. Mashita, H. Takemura, and E. Kruijff, “Analysing the effects of a wide field of view augmented reality display on search performance in divided attention tasks,” in 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2014, pp. 177–186.
  33. E. D. Ragan, D. A. Bowman, R. Kopper, C. Stinson, S. Scerbo, and R. P. McMahan, “Effects of field of view and visual complexity on virtual reality training effectiveness for a visual scanning task,” IEEE Transactions on Visualization and Computer Graphics, vol. 21, no. 7, pp. 794–807, 2015.
  34. J. Kim, A. Madhusudan, B. Watson, B. Boudaoud, R. Tarrazo, and J. Spjut, “Display size and targeting performance: Small hurts, large may help,” in SIGGRAPH Asia 2022 Conference Papers, ser. SA ’22.   New York, NY, USA: Association for Computing Machinery, 2022. [Online]. Available: https://doi.org/10.1145/3550469.3555396
  35. M. Lee, G. Bruder, T. Höllerer, and G. Welch, “Effects of unaugmented periphery and vibrotactile feedback on proxemics with virtual humans in ar,” IEEE Transactions on Visualization and Computer Graphics, vol. 24, no. 4, pp. 1525–1534, 2018.
  36. I. Wang, J. Smith, and J. Ruiz, “Exploring virtual agents for augmented reality,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, ser. CHI ’19.   New York, NY, USA: Association for Computing Machinery, 2019, p. 1–12. [Online]. Available: https://doi.org/10.1145/3290605.3300511
  37. D. Ren, T. Goldschwendt, Y. Chang, and T. Höllerer, “Evaluating wide-field-of-view augmented reality with mixed reality simulation,” in 2016 IEEE Virtual Reality (VR), 2016, pp. 93–102.
  38. D. Medeiros, M. McGill, A. Ng, R. McDermid, N. Pantidi, J. Williamson, and S. Brewster, “From shielding to avoidance: Passenger augmented reality and the layout of virtual displays for productivity in shared transit,” IEEE Transactions on Visualization and Computer Graphics, vol. 28, no. 11, pp. 3640–3650, 2022.
  39. J. a. M. Evangelista Belo, M. N. Lystbæk, A. M. Feit, K. Pfeuffer, P. Kán, A. Oulasvirta, and K. Grønbæk, “Auit – the adaptive user interfaces toolkit for designing xr applications,” in Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, ser. UIST ’22.   New York, NY, USA: Association for Computing Machinery, 2022. [Online]. Available: https://doi.org/10.1145/3526113.3545651
  40. S. Prince, A. Cheok, F. Farbiz, T. Williamson, N. Johnson, M. Billinghurst, and H. Kato, “3d live: real time captured content for mixed reality,” in Proceedings. International Symposium on Mixed and Augmented Reality, 2002, pp. 7–317.
  41. A. Jones, M. Lang, G. Fyffe, X. Yu, J. Busch, I. McDowall, M. Bolas, and P. Debevec, “Achieving eye contact in a one-to-many 3d video teleconferencing system,” ACM Trans. Graph., vol. 28, no. 3, jul 2009. [Online]. Available: https://doi.org/10.1145/1531326.1531370
  42. S. Beck, A. Kunert, A. Kulik, and B. Froehlich, “Immersive group-to-group telepresence,” IEEE Transactions on Visualization and Computer Graphics, vol. 19, no. 4, pp. 616–625, 2013.
  43. G. Gamelin, A. Chellali, S. Cheikh, A. Ricca, C. Dumas, and S. Otmane, “Point-cloud avatars to improve spatial communication in immersive collaborative virtual environments,” Personal and Ubiquitous Computing, vol. 25, pp. 467–484, 2021.
  44. S. Yura, T. Usaka, and K. Sakamura, “Video avatar: embedded video for collaborative virtual environment,” in Proceedings IEEE International Conference on Multimedia Computing and Systems, vol. 2, 1999, pp. 433–438 vol.2.
  45. Y. Wang, D. M. Krum, E. M. Coelho, and D. A. Bowman, “Contextualized videos: Combining videos with environment models to support situational understanding,” IEEE Transactions on Visualization and Computer Graphics, vol. 13, no. 6, pp. 1568–1575, 2007.
  46. M. J. Prins, S. N. B. Gunkel, H. M. Stokking, and O. A. Niamut, “Togethervr: A framework for photorealistic shared media experiences in 360-degree vr,” SMPTE Motion Imaging Journal, vol. 127, no. 7, pp. 39–44, 2018.
  47. M. Billinghurst and H. Kato, “Real world teleconferencing,” in CHI ’99 Extended Abstracts on Human Factors in Computing Systems, ser. CHI EA ’99.   New York, NY, USA: Association for Computing Machinery, 1999, p. 194–195. [Online]. Available: https://doi.org/10.1145/632716.632838
  48. C. Neustaedter, S. Singhal, R. Pan, Y. Heshmat, A. Forghani, and J. Tang, “From being there to watching: Shared and dedicated telepresence robot usage at academic conferences,” ACM Trans. Comput.-Hum. Interact., vol. 25, no. 6, dec 2018. [Online]. Available: https://doi.org/10.1145/3243213
  49. Y. Furuya and K. Takashio, “Telepresence robot blended with a real landscape and its impact on user experiences,” in 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), 2020, pp. 406–411.
  50. D. Jo, K.-H. Kim, and G. J. Kim, “Spacetime: adaptive control of the teleported avatar for improved ar tele-conference experience,” Computer Animation and Virtual Worlds, vol. 26, no. 3-4, pp. 259–269, 2015.
  51. Y. Lang, W. Liang, and L.-F. Yu, “Virtual agent positioning driven by scene semantics in mixed reality,” in 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019, pp. 767–775.
  52. D. Lindlbauer, A. M. Feit, and O. Hilliges, “Context-aware online adaptation of mixed reality interfaces,” in Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, ser. UIST ’19.   New York, NY, USA: Association for Computing Machinery, 2019, p. 147–160. [Online]. Available: https://doi.org/10.1145/3332165.3347945
  53. J. N. Bailenson, J. Blascovich, A. C. Beall, and J. M. Loomis, “Interpersonal distance in immersive virtual environments,” Personality and social psychology bulletin, vol. 29, no. 7, pp. 819–833, 2003.
  54. J. E. Swan, A. Jones, E. Kolstad, M. A. Livingston, and H. S. Smallman, “Egocentric depth judgments in optical, see-through augmented reality,” IEEE Transactions on Visualization and Computer Graphics, vol. 13, no. 3, pp. 429–442, 2007.
  55. J. W. Woodworth, N. G. Lipari, and C. W. Borst, “Evaluating teacher avatar appearances in educational vr,” in 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019, pp. 1235–1236.
  56. D. Gonzalez–Morin, E. Gonzalez–Sosa, P. Perez–Garcia, and A. Villegas, “Bringing real body as self-avatar into mixed reality: A gamified volcano experience,” in 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 2022, pp. 794–795.
  57. D. G. Morin, E. Gonzalez-Sosa, P. Perez, and A. Villegas, “Full body video-based self-avatars for mixed reality: from e2e system to user study,” arXiv preprint arXiv:2208.12639, 2022.
  58. G. A. Lee, J. Chen, M. Billinghurst, and R. Lindeman, “Enhancing immersive cinematic experience with augmented virtuality,” in 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), 2016, pp. 115–116.
  59. S. Jung, G. Bruder, P. J. Wisniewski, C. Sandor, and C. E. Hughes, “Over my hand: Using a personalized hand in vr to improve object size estimation, body ownership, and presence,” in Proceedings of the 2018 ACM Symposium on Spatial User Interaction, ser. SUI ’18.   New York, NY, USA: Association for Computing Machinery, 2018, p. 60–68. [Online]. Available: https://doi.org/10.1145/3267782.3267920
  60. K. Caine, “Local standards for sample size at CHI,” in Proceedings of the 2016 CHI conference on human factors in computing systems, 2016, pp. 981–992.
  61. D. Kim, J.-e. Shin, J. Lee, and W. Woo, “Adjusting relative translation gains according to space size in redirected walking for mixed reality mutual space generation,” in 2021 IEEE Virtual Reality and 3D User Interfaces (VR), 2021, pp. 653–660.
  62. A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, Q. V. Le, and H. Adam, “Searching for mobilenetv3,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  63. K. L. Nowak and F. Biocca, “The effect of the agency and anthropomorphism on users’ sense of telepresence, copresence, and social presence in virtual environments,” Presence: Teleoperators and Virtual Environments, vol. 12, no. 5, pp. 481–494, 10 2003. [Online]. Available: https://doi.org/10.1162/105474603322761289
  64. T. Rhee, L. Petikam, B. Allen, and A. Chalmers, “Mr360: Mixed reality rendering for 360° panoramic videos,” IEEE Transactions on Visualization and Computer Graphics, vol. 23, no. 4, pp. 1379–1388, 2017.
  65. Z. He, R. Du, and K. Perlin, “Lookatchat: Visualizing gaze awareness for remote small-group conversations,” arXiv preprint arXiv:2107.06265, 2021.
  66. B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Commun. ACM, vol. 65, no. 1, p. 99–106, dec 2021. [Online]. Available: https://doi.org/10.1145/3503250
  67. A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer, “D-nerf: Neural radiance fields for dynamic scenes,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp. 10 318–10 327.
  68. G. Gafni, J. Thies, M. Zollhofer, and M. Niessner, “Dynamic neural radiance fields for monocular 4d facial avatar reconstruction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp. 8649–8658.
  69. Y. Kim, H. Park, S. Bang, and S.-H. Lee, “Retargeting human-object interaction to virtual avatars,” IEEE Transactions on Visualization and Computer Graphics, vol. 22, no. 11, pp. 2405–2412, 2016.
  70. T. Tahara, T. Seno, G. Narita, and T. Ishikawa, “Retargetable ar: Context-aware augmented reality in indoor scenes based on 3d scene graph,” in 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2020, pp. 249–255.
  71. G. Song, T.-J. Cham, J. Cai, and J. Zheng, “Real-time shadow-aware portrait relighting in virtual backgrounds for realistic telepresence,” in 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2022, pp. 729–738.
  72. M. Elgharib, M. Mendiratta, J. Thies, M. Nießner, H.-P. Seidel, A. Tewari, V. Golyanik, and C. Theobalt, “Egocentric videoconferencing,” ACM Transactions on Graphics, vol. 39, no. 6, Dec 2020.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Xuanyu Wang (5 papers)
  2. Weizhan Zhang (17 papers)
  3. Christian Sandor (3 papers)
  4. Hongbo Fu (67 papers)
Citations (1)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets