Co-designing a Sub-millisecond Latency Event-based Eye Tracking System with Submanifold Sparse CNN (2404.14279v1)
Abstract: Eye-tracking technology is integral to numerous consumer electronics applications, particularly in the realm of virtual and augmented reality (VR/AR). These applications demand solutions that excel in three crucial aspects: low-latency, low-power consumption, and precision. Yet, achieving optimal performance across all these fronts presents a formidable challenge, necessitating a balance between sophisticated algorithms and efficient backend hardware implementations. In this study, we tackle this challenge through a synergistic software/hardware co-design of the system with an event camera. Leveraging the inherent sparsity of event-based input data, we integrate a novel sparse FPGA dataflow accelerator customized for submanifold sparse convolution neural networks (SCNN). The SCNN implemented on the accelerator can efficiently extract the embedding feature vector from each representation of event slices by only processing the non-zero activations. Subsequently, these vectors undergo further processing by a gated recurrent unit (GRU) and a fully connected layer on the host CPU to generate the eye centers. Deployment and evaluation of our system reveal outstanding performance metrics. On the Event-based Eye-Tracking-AIS2024 dataset, our system achieves 81% p5 accuracy, 99.5% p10 accuracy, and 3.71 Mean Euclidean Distance with 0.7 ms latency while only consuming 2.29 mJ per inference. Notably, our solution opens up opportunities for future eye-tracking systems. Code is available at https://github.com/CASR-HKU/ESDA/tree/eye_tracking.
- Event-based eye tracking - ais2024 cvpr workshop. https://www.kaggle.com/competitions/event-based-eye-tracking-ais2024/.
- PYNQ — pynq.io. https://www.pynq.io/.
- Nullhop: A flexible convolutional neural network accelerator based on sparse representations of feature maps. IEEE transactions on neural networks and learning systems, 30(3):644–656, 2018.
- Event based, near eye gaze tracking beyond 10,000 hz. arXiv preprint arXiv:2004.03577, 2020.
- Simultaneous optical flow and intensity estimation from an event camera. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 884–892, 2016.
- 3et: Efficient event-based eye tracking using a change-based convlstm network. In 2023 IEEE Biomedical Circuits and Systems Conference (BioCAS), pages 1–5. IEEE, 2023.
- Cnn-based pupil center detection for wearable gaze estimation system. Applied Computational Intelligence and Soft Computing, 2017, 2017.
- Eye pupil localization algorithm using convolutional neural networks. Multimedia Tools and Applications, 79(43):32563–32574, 2020.
- 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3075–3084, 2019.
- Model-platform optimized deep neural network accelerator generation through mixed-integer geometric programming. In 2023 IEEE 31st Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), pages 83–93, 2023.
- Leveling the playing field: A comparative reevaluation of unmodified eye tracking as an input and interaction modality for vr. IEEE Transactions on Visualization and Computer Graphics, 29(5):2269–2279, 2023.
- Pupilnet: Convolutional neural networks for robust pupil detection. arXiv preprint arXiv:1601.04902, 2016a.
- Else: Ellipse selection for robust pupil detection in real-world environments. In Proceedings of the ninth biennial ACM symposium on eye tracking research & applications, pages 123–130, 2016b.
- Teyed: Over 20 million real-world eye images with pupil, eyelid, and iris 2d and 3d segmentations, 2d and 3d landmarks, 3d eyeball, gaze vector, and eye movement types. In 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 367–375. IEEE, 2021.
- A composable dynamic sparse dataflow architecture for efficient event-based vision processing on fpga. In Proceedings of the 2024 ACM/SIGDA International Symposium on Field Programmable Gate Arrays, page 246–257, New York, NY, USA, 2024. Association for Computing Machinery.
- Recurrent vision transformers for object detection with event cameras. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13884–13893, 2023.
- Cascade learning from adversarial synthetic images for accurate pupil detection. Pattern Recognition, 88:584–594, 2019.
- Eigen v3. http://eigen.tuxfamily.org, 2010.
- Hierarchical neural memory network for low latency event processing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22867–22876, 2023.
- In the eye of the beholder: A survey of models for eyes and gaze. IEEE transactions on pattern analysis and machine intelligence, 32(3):478–500, 2009.
- Energy efficient pupil tracking based on rule distillation of cascade regression forest. Sensors, 20(18):5141, 2020.
- Eye tracking algorithms, techniques, tools, and applications with an emphasis on machine learning and internet of things technologies. Expert Systems with Applications, 166:114037, 2021.
- Accurate pupil center detection in off-the-shelf eye tracking systems using convolutional neural networks. Sensors, 21(20):6847, 2021.
- Deep learning-based pupil center detection for fast and accurate eye tracking system. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIX 16, pages 36–52. Springer, 2020.
- Tma: Temporal motion aggregation for event-based optical flow. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9685–9694, 2023.
- Adaptive time-slice block-matching optical flow algorithm for dynamic vision sensors. BMVC, 2018.
- Ssd: Single shot multibox detector. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, pages 21–37. Springer, 2016.
- Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
- Get: group event transformer for event-based vision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6038–6048, 2023.
- Event-based temporally dense optical flow estimation with sequential learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9827–9836, 2023.
- A real-time high precision eye center localizer. Journal of Real-Time Image Processing, 19(2):475–486, 2022.
- Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
- Event-based kilohertz eye tracking using coded differential lighting. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2515–2523, 2022.
- Menet: A memory-based network with dual-branch for efficient event stream processing. In European Conference on Computer Vision, pages 214–234. Springer, 2022.
- Robust real-time pupil tracking in highly off-axis images. In Proceedings of the symposium on eye tracking research and applications, pages 173–176, 2012.
- Event-Based Eye Tracking. AIS 2024 Challenge Survey. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2024.
- Pupil center detection inspired by multi-task auxiliary learning characteristic. Multimedia Tools and Applications, 81(28):40067–40088, 2022.
- Hawq-v3: Dyadic neural network quantization. In International Conference on Machine Learning, pages 11875–11886. PMLR, 2021.
- Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.
Sponsor
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.