Papers
Topics
Authors
Recent
2000 character limit reached

GaitGuard: Towards Private Gait in Mixed Reality

Published 7 Dec 2023 in cs.HC and cs.CR | (2312.04470v5)

Abstract: Augmented/Mixed Reality (AR/MR) technologies usher in a new era of immersive, collective experiences, differentiating them from traditional mobile systems. As these technologies evolve, prioritizing privacy and security is critical. This paper focuses on gait privacy, where gait, the way a person walks, can reveal sensitive information such as age, ethnicity, or disorders. We present GaitGuard, a real-time system that protects gait privacy against video-based gait extraction attacks in MR environments. GaitGuard leverages a multi-threaded framework to efficiently process video frames, incorporating dedicated modules for stream capture, body detection and tracking, and privacy leak mitigation. We compare and combine multiple mitigation techniques, offering guidance to navigate the privacy-utility tradeoff. Through extensive experiments covering 248 settings across mitigation regions, types, and tunable parameters, we assess the impact of these techniques on privacy, video quality, and system performance. GaitGuard reduces the confidence of video-based gait extraction attacks by introducing a substantial distribution shift (Jensen-Shannon Divergence of 0.63, indicating highly altered gait features) and a decrease in identification risks by up to 68%, while maintaining 29 FPS and preserving video clarity. GaitGuard provides a practical real-time solution for privacy-preserving MR applications without affecting the MR user experience based on 20 subjective user surveys.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. 2013. The CPRA. https://thecpra.org/.
  2. 2018. Likert Scale: Definition, Examples & How to Use It. https://www.questionpro.com/blog/what-is-likert-scale/.
  3. 2022. App Quality Criteria - Mixed Reality. https://learn.microsoft.com/en-us/windows/mixed-reality/develop/advanced-concepts/app-quality-criteria-overview.
  4. 2022. Introduction to the Multi-user Capabilities Tutorials - Mixed Reality. https://learn.microsoft.com/en-us/windows/mixed-reality/develop/unity/tutorials/mr-learning-sharing-01.
  5. 2022. Mixed Reality Capture Overview. https://learn.microsoft.com/en-us/windows/mixed-reality/develop/advanced-concepts/mixed-reality-capture-overview.
  6. 2022. Performance - MRTK 2. https://learn.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk2/performance/perf-getting-started?view=mrtkunity-2022-05.
  7. 2023. Apple Vision Pro. {https://www.apple.com/apple-vision-pro/}.
  8. 2023. Bouvet. https://www.microsoft.com/en-us/p/lens-by-bouvet/9p29ft8z34zs.
  9. 2023. Catapult Mixed Reality Client and Development Application. https://www.makesea.com/download/.
  10. 2023. General Data Protection Regulation (GDPR). https://gdpr-info.eu/.
  11. 2023. HoloLens 2 Hardware. https://learn.microsoft.com/en-us/hololens/hololens2-hardware.
  12. 2023a. HoloLens Microsoft 2. {https://www.microsoft.com/en-us/hololens/}.
  13. 2023b. HoloLens2ForCV. https://github.com/microsoft/HoloLens2ForCV/tree/main/Samples/CameraWithCVAndCalibration.
  14. 2023. Meta Quest 3: Mixed Reality VR Headset. {https://www.meta.com/quest/quest-3/}.
  15. 2023. Microsoft Apps. https://apps.microsoft.com/home?hl=en-US&gl=US.
  16. 2023. OpenCV. {https://docs.opencv.org/3.4/d5/d33/structcv_1_1HOGDescriptor.html}.
  17. 2023. Overview of Dynamics 365 Remote Assist on HoloLens and HoloLens 2. https://learn.microsoft.com/en-us/dynamics365/mixed-reality/remote-assist/overview-hololens.
  18. 2023. Photon Unity Networking for Unity Multiplayer Games PUN2. {https://www.photonengine.com/PUN}.
  19. 2023. Pose Estimation. https://paperswithcode.com/task/pose-estimation.
  20. 2023. Sklearn.Ensemble.GradientBoostingClassifier. https://scikit-learn/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html.
  21. 2023. Sphere XR. https://apps.microsoft.com/detail/9P11TRNWG2HR?hl=en-US.
  22. 2023. What Is Mixed Reality? https://learn.microsoft.com/en-us/windows/mixed-reality/discover/mixed-reality.
  23. Walk to show your identity: gait-based seamless user authentication framework using deep neural network. In The 5th ACM Workshop on Wearable Systems and Applications. 53–58.
  24. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7291–7299.
  25. Aurelio Cappozzo. 1984. Gait analysis methodology. Human movement science 3, 1-2 (1984), 27–50.
  26. Exploring user reactions and mental models towards perceptual manipulation attacks in mixed reality. In USENIX Security, Vol. 18.
  27. Gait-based authentication using a wrist-worn device. In Proceedings of the 13th International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services. 208–217.
  28. BystandAR: Protecting Bystander Visual Data in Augmented Reality Systems. In Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services. 370–382.
  29. IMU-based gait recognition using convolutional neural networks and multi-sensor fusion. Sensors 17, 12 (2017), 2735.
  30. GaitPrivacyON: Privacy-preserving mobile gait biometrics using unsupervised learning. Pattern Recognition Letters 161 (2022), 30–37.
  31. Improved cycle detection for accelerometer based gait authentication. In 2010 Sixth International Conference on Intelligent Information Hiding and Multimedia Signal Processing. IEEE, 312–317.
  32. Juan C Dibene and Enrique Dunn. 2022. HoloLens 2 Sensor Streaming. arXiv preprint arXiv:2211.02648 (2022).
  33. {{\{{LocIn}}\}}: Inferring Semantic Location from Spatial Maps in Mixed Reality. In 32nd USENIX Security Symposium (USENIX Security 23). 877–894.
  34. Full body gait analysis with Kinect. In 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 1964–1967.
  35. Gait authentication and identification using wearable accelerometer sensor. In 2007 IEEE workshop on automatic identification advanced technologies. IEEE, 220–225.
  36. Hidden reality: caution, your hand gesture inputs in the immersive virtual world are visible to all!. In 32nd USENIX Security Symposium (USENIX Security 23). 859–876.
  37. Understanding Person Identification through Gait. arXiv preprint arXiv:2203.04179 (2022).
  38. Sudhakar Kumawat and Hajime Nagahara. 2022. Privacy-Preserving Action Recognition via Motion Difference Quantization. In European Conference on Computer Vision. Springer, 518–534.
  39. Towards security and privacy for multi-user augmented reality: Foundations with end users. In 2018 IEEE Symposium on Security and Privacy (SP). IEEE, 392–408.
  40. Rationalstat LLC. 2023. Mixed Reality Market Is Set to Double by 2030 — Mixed Reality Market Share, Size, Growth, Forecast, 2023-2030 — RationalStat Study. https://www.globenewswire.com/news-release/2023/09/22/2747789/0/en/Mixed-Reality-Market-is-Set-to-Double-by-2030-Mixed-Reality-Market-Share-Size-Growth-Forecast-2023-2030-RationalStat-Study.html.
  41. Video-based pose estimation for gait analysis in stroke survivors during clinical assessments: a proof-of-concept study. Digital Biomarkers 6, 1 (2022), 9–18.
  42. Muhammad Muaaz and Rene Mayrhofer. 2014. Orientation independent cell phone based gait authentication. In Proceedings of the 12th International Conference on Advances in Mobile Computing and Multimedia. 161–164.
  43. Modern methods for affordable clinical gait analysis: theories and applications in healthcare systems. Academic Press.
  44. A reliable gait phase detection system. IEEE Transactions on neural systems and rehabilitation engineering 9, 2 (2001), 113–125.
  45. Comparative abilities of Microsoft Kinect and Vicon 3D motion capture for gait analysis. Journal of medical engineering & technology 38, 5 (2014), 274–280.
  46. Wearable sensor-based real-time gait detection: A systematic review. Sensors 21, 8 (2021), 2727.
  47. SecSpace: prototyping usable privacy and security for mixed reality collaborative environments. In Proceedings of the 2014 ACM SIGCHI symposium on Engineering interactive computing systems. 273–282.
  48. Secure {{\{{Multi-User}}\}} content sharing for augmented reality applications. In 28th USENIX Security Symposium (USENIX Security 19). 141–158.
  49. Warit Sirichotedumrong and Hitoshi Kiya. 2021. A gan-based image transformation scheme for privacy-preserving deep neural networks. In 2020 28th European Signal Processing Conference (EUSIPCO). IEEE, 745–749.
  50. Quantitative gait analysis in Parkinson’s disease: comparison with a healthy control group. Archives of physical medicine and rehabilitation 86, 5 (2005), 1007–1013.
  51. Two-dimensional video-based analysis of human gait using pose estimation. PLoS computational biology 17, 4 (2021), e1008935.
  52. Christopher L Vaughan. 2003. Theories of bipedal walking: an odyssey. Journal of biomechanics 36, 4 (2003), 513–523.
  53. A study on gait-based gender classification. IEEE Transactions on image processing 18, 8 (2009), 1905–1910.
  54. Adversarial framing for image and video classification. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 10077–10078.
  55. Ethnicity classification based on gait using multi-view fusion. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops. IEEE, 108–115.
  56. It’s all in your head (set): Side-channel attacks on ar/vr systems. In USENIX Security.
  57. The detection of age groups by dynamic gait outcomes using machine learning approaches. Scientific reports 10, 1 (2020), 4426.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free