Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Learning High-Quality Navigation and Zooming on Omnidirectional Images in Virtual Reality (2405.00351v1)

Published 1 May 2024 in cs.HC, cs.AI, cs.CV, and cs.MM

Abstract: Viewing omnidirectional images (ODIs) in virtual reality (VR) represents a novel form of media that provides immersive experiences for users to navigate and interact with digital content. Nonetheless, this sense of immersion can be greatly compromised by a blur effect that masks details and hampers the user's ability to engage with objects of interest. In this paper, we present a novel system, called OmniVR, designed to enhance visual clarity during VR navigation. Our system enables users to effortlessly locate and zoom in on the objects of interest in VR. It captures user commands for navigation and zoom, converting these inputs into parameters for the Mobius transformation matrix. Leveraging these parameters, the ODI is refined using a learning-based algorithm. The resultant ODI is presented within the VR media, effectively reducing blur and increasing user engagement. To verify the effectiveness of our system, we first evaluate our algorithm with state-of-the-art methods on public datasets, which achieves the best performance. Furthermore, we undertake a comprehensive user study to evaluate viewer experiences across diverse scenarios and to gather their qualitative feedback from multiple perspectives. The outcomes reveal that our system enhances user engagement by improving the viewers' recognition, reducing discomfort, and improving the overall immersive experience. Our system makes the navigation and zoom more user-friendly.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. A. Vermast and W. Hürst, “Introducing 3d thumbnails to access 360-degree videos in virtual reality,” IEEE Transactions on Visualization and Computer Graphics, vol. 29, no. 5, pp. 2547–2556, 2023.
  2. Z. Luo, B. Chai, Z. Wang, M. Hu, and D. Wu, “Masked360: Enabling robust 360-degree video streaming with ultra low bandwidth consumption,” IEEE Transactions on Visualization and Computer Graphics, vol. 29, no. 5, pp. 2690–2699, 2023.
  3. M. Dasari, E. Lu, M. W. Farb, N. Pereira, I. Liang, and A. Rowe, “Scaling vr video conferencing,” in 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR).   IEEE, 2023, pp. 648–657.
  4. Q. Zhang, J. Wei, S. Wang, S. Ma, and W. Gao, “Realvr: Efficient, economical, and quality-of-experience-driven vr video system based on mpeg omaf,” IEEE Transactions on Multimedia, 2022.
  5. J. Lou, Y. Wang, C. Nduka, M. Hamedi, I. Mavridou, F.-Y. Wang, and H. Yu, “Realistic facial expression reconstruction for vr hmd users,” IEEE Transactions on Multimedia, vol. 22, no. 3, pp. 730–743, 2019.
  6. P. Szabo, A. Simiscuka, S. Masneri, M. Zorrilla, and G.-M. Muntean, “A cnn-based framework for enhancing 360 vr experiences with multisensorial effects,” IEEE Transactions on Multimedia, 2022.
  7. S. Verma, L. Warrier, B. Bolia, and S. Mehta, “Past, present, and future of virtual tourism-a literature review,” International Journal of Information Management Data Insights, vol. 2, no. 2, p. 100085, 2022.
  8. A. Mohammad and H. Ismail, “Development and evaluation of an interactive 360 virtual tour for tourist destinations,” J. Inform. Technol. Impact, vol. 9, pp. 137–182, 2009.
  9. A. Azmi, R. Ibrahim, M. Abdul Ghafar, and A. Rashidi, “Smarter real estate marketing using virtual reality to influence potential homebuyers’ emotions and purchase intention,” Smart and Sustainable Built Environment, vol. 11, no. 4, pp. 870–890, 2022.
  10. J. Singh, M. Malhotra, and N. Sharma, “Metaverse in education: An overview,” Applying metalytics to measure customer experience in the metaverse, pp. 135–142, 2022.
  11. H. Chang and M. F. Cohen, “Panning and zooming high-resolution panoramas in virtual reality devices,” in Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, 2017, pp. 279–288.
  12. Y. Yoon, I. Chung, L. Wang, and K.-J. Yoon, “Spheresr: 360deg image super-resolution with arbitrary projection via continuous spherical image representation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5677–5686.
  13. M. Kwon, R. Liu, and L. Chien, “Compensation for blur requires increase in field of view and viewing time,” PLoS One, vol. 11, no. 9, p. e0162711, 2016.
  14. L. O’Hare and P. B. Hibbard, “Visual discomfort and blur,” Journal of vision, vol. 13, no. 5, pp. 7–7, 2013.
  15. D. M. Hoffman, A. R. Girshick, K. Akeley, and M. S. Banks, “Vergence–accommodation conflicts hinder visual performance and cause visual fatigue,” Journal of vision, vol. 8, no. 3, pp. 33–33, 2008.
  16. S. Ang and J. Quarles, “Gingervr: An open source repository of cybersickness reduction techniques for unity,” in 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW).   IEEE, 2020, pp. 460–463.
  17. R. Hussain, M. Chessa, and F. Solari, “Mitigating cybersickness in virtual reality systems through foveated depth-of-field blur,” Sensors, vol. 21, no. 12, p. 4006, 2021.
  18. X. Meng, R. Du, M. Zwicker, and A. Varshney, “Kernel foveated rendering,” Proceedings of the ACM on Computer Graphics and Interactive Techniques, vol. 1, no. 1, pp. 1–20, 2018.
  19. Z. Cao, H. Ai, Y.-P. Cao, Y. Shan, X. Qie, and L. Wang, “Omnizoomer: Learning to move and zoom in on sphere at high-resolution,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 12 897–12 907.
  20. X. Deng, H. Wang, M. Xu, Y. Guo, Y. Song, and L. Yang, “Lau-net: Latitude adaptive upscaling network for omnidirectional image super-resolution,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 9189–9198.
  21. V. Fakour-Sevom, E. Guldogan, and J.-K. Kämäräinen, “360 panorama super-resolution using deep convolutional networks,” in Int. Conf. on Computer Vision Theory and Applications (VISAPP), vol. 1, 2018.
  22. A. Nishiyama, S. Ikehata, and K. Aizawa, “360 single image super resolution via distortion-aware network and distorted perspective images,” in 2021 IEEE International Conference on Image Processing (ICIP).   IEEE, 2021, pp. 1829–1833.
  23. F. Yu, X. Wang, M. Cao, G. Li, Y. Shan, and C. Dong, “Osrt: Omnidirectional image super-resolution with distortion-aware transformer,” arXiv preprint arXiv:2302.03453, 2023.
  24. X. Sun, W. Li, Z. Zhang, Q. Ma, X. Sheng, M. Cheng, H. Ma, S. Zhao, J. Zhang, J. Li et al., “Opdn: Omnidirectional position-aware deformable network for omnidirectional image super-resolution,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1293–1301.
  25. L. Penaranda, L. Velho, and L. Sacht, “Real-time correction of panoramic images using hyperbolic möbius transformations,” Journal of Real-Time Image Processing, vol. 15, pp. 725–738, 2018.
  26. L. S. Ferreira, L. Sacht, and L. Velho, “Local moebius transformations applied to omnidirectional images,” Computers & Graphics, vol. 68, pp. 77–83, 2017.
  27. L. S. Ferreira and L. Sacht, “Bounded biharmonic blending of möbius transformations for flexible omnidirectional image rectification,” Computers & Graphics, vol. 93, pp. 51–60, 2020.
  28. S. Schleimer and H. Segerman, “Squares that look round: transforming spherical images,” arXiv preprint arXiv:1605.01396, 2016.
  29. J. Wu, C. Xia, T. Yu, and J. Li, “View-aware salient object detection for 360 {{\{{\\\backslash\deg}}\}} omnidirectional image,” arXiv preprint arXiv:2209.13222, 2022.
  30. S. Zhou, J. Zhang, H. Jiang, T. Lundh, and A. Y. Ng, “Data augmentation with mobius transformations,” Machine Learning: Science and Technology, vol. 2, no. 2, p. 025016, 2021.
  31. N. Özdemir, B. B. İskender, and N. Y. Özgür, “Complex valued neural network with möbius activation function,” Communications in Nonlinear Science and Numerical Simulation, vol. 16, no. 12, pp. 4698–4703, 2011.
  32. N. Azizi, H. Possegger, E. Rodolà, and H. Bischof, “3d human pose estimation using möbius graph convolutional networks,” in Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part I.   Springer, 2022, pp. 160–178.
  33. T. W. Mitchel, N. Aigerman, V. G. Kim, and M. Kazhdan, “Möbius convolutions for spherical cnns,” in ACM SIGGRAPH 2022 Conference Proceedings, 2022, pp. 1–9.
  34. S. Kato and P. McCullagh, “Möbius transformation and a cauchy family on the sphere,” arXiv: Statistics Theory, 2015.
  35. B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2017, pp. 136–144.
  36. W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1874–1883, 2016.
  37. K. Eybpoosh, M. Rezghi, and A. Heydari, “Applying inverse stereographic projection to manifold learning and clustering,” Applied Intelligence, vol. 52, pp. 4443–4457, 2021.
  38. J. P. Fatelo and N. Martins-Ferreira, “Mobility spaces and geodesics for the n-sphere,” 2021.
  39. H. Ai, Z. Cao, J. Zhu, H. Bai, Y. Chen, and L. Wang, “Deep learning for omnidirectional vision: A survey and new perspectives,” arXiv preprint arXiv:2205.10468, 2022.
  40. X. Deng, H. Wang, M. Xu, L. Li, and Z. Wang, “Omnidirectional image super-resolution via latitude adaptive network,” IEEE Transactions on Multimedia, 2022.
  41. J. Xiao, K. A. Ehinger, A. Oliva, and A. Torralba, “Recognizing scene viewpoint using panoramic place representation,” computer vision and pattern recognition, 2012.
  42. Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image super-resolution using very deep residual channel attention networks,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 286–301.
  43. C. Ozcinar, A. Rana, and A. Smolic, “Super-resolution of omnidirectional images using adversarial learning,” in 2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP).   IEEE, 2019, pp. 1–6.
  44. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  45. Y. Sun, A. Lu, and L. Yu, “Weighted-to-spherically-uniform quality evaluation for omnidirectional video,” IEEE signal processing letters, vol. 24, no. 9, pp. 1408–1412, 2017.
  46. Y. Zhou, M. Yu, H. Ma, H. Shao, and G. Jiang, “Weighted-to-spherically-uniform ssim objective quality evaluation for panoramic video,” in 2018 14th IEEE International Conference on Signal Processing (ICSP).   IEEE, 2018, pp. 54–57.
  47. M. Cao, C. Mou, F. Yu, X. Wang, Y. Zheng, J. Zhang, C. Dong, G. Li, Y. Shan, R. Timofte et al., “Ntire 2023 challenge on 360deg omnidirectional image and video super-resolution: Datasets, methods and results,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1731–1745.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets