Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Eat-Radar: Continuous Fine-Grained Intake Gesture Detection Using FMCW Radar and 3D Temporal Convolutional Network with Attention (2211.04253v2)

Published 8 Nov 2022 in cs.CV, eess.IV, and eess.SP

Abstract: Unhealthy dietary habits are considered as the primary cause of various chronic diseases, including obesity and diabetes. The automatic food intake monitoring system has the potential to improve the quality of life (QoL) of people with diet-related diseases through dietary assessment. In this work, we propose a novel contactless radar-based approach for food intake monitoring. Specifically, a Frequency Modulated Continuous Wave (FMCW) radar sensor is employed to recognize fine-grained eating and drinking gestures. The fine-grained eating/drinking gesture contains a series of movements from raising the hand to the mouth until putting away the hand from the mouth. A 3D temporal convolutional network with self-attention (3D-TCN-Att) is developed to detect and segment eating and drinking gestures in meal sessions by processing the Range-Doppler Cube (RD Cube). Unlike previous radar-based research, this work collects data in continuous meal sessions (more realistic scenarios). We create a public dataset comprising 70 meal sessions (4,132 eating gestures and 893 drinking gestures) from 70 participants with a total duration of 1,155 minutes. Four eating styles (fork & knife, chopsticks, spoon, hand) are included in this dataset. To validate the performance of the proposed approach, seven-fold cross-validation method is applied. The 3D-TCN-Att model achieves a segmental F1-score of 0.896 and 0.868 for eating and drinking gestures, respectively. The results of the proposed approach indicate the feasibility of using radar for fine-grained eating and drinking gesture detection and segmentation in meal sessions.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. “Obesity and overweight.” https://www.who.int/news-room/fact-sheets/detail/obesity-and-overweight (accessed Jun. 05, 2022).
  2. P. V. Rouast and M. T. P. Adam, “Single-stage intake gesture detection using CTC loss and extended prefix beam search,” IEEE J. Biomed. Heal. Informatics, vol. 25, no. 7, pp. 2733–2743, 2021.
  3. P. V. Rouast and M. T. P. Adam, “Learning deep representations for video-based intake gesture detection,” IEEE J. Biomed. Heal. Informatics, vol. 24, no. 6, pp. 1727–1737, 2020.
  4. J. Qiu, F. P. W. Lo, S. Jiang, Y. Y. Tsai, Y. Sun, and B. Lo, “Counting bites and recognizing consumed food from videos for passive dietary monitoring,” IEEE J. Biomed. Heal. Informatics, vol. 25, no. 5, pp. 1471–1482, 2021.
  5. K. Kyritsis, C. Diou, and A. Delopoulos, “A data driven end-to-end approach for in-the-wild monitoring of eating behavior using smartwatches,” IEEE J. Biomed. Heal. Informatics, vol. 25, no. 1, pp. 22–34, 2020.
  6. S. Zhang, R. Alharbi, W. Stogin, M. Pourhomayun, B. Spring, and N. Alshurafa, “Food watch: Detecting and characterizing eating episodes through feeding gestures,” BodyNets Int. Conf. Body Area Networks, 2017.
  7. A. Doulah, T. Ghosh, D. Hossain, M. H. Imtiaz, and E. Sazonov, “‘Automatic Ingestion Monitor Version 2’ - A novel wearable device for automatic food intake detection and passive capture of food images,” IEEE J. Biomed. Heal. Informatics, vol. 25, no. 2, pp. 568–576, 2021.
  8. K. S. Lee, “Joint audio-ultrasound food recognition for noisy environments,” IEEE J. Biomed. Heal. Informatics, vol. 24, no. 5, pp. 1477–1489, 2020.
  9. G. Mertes, L. Ding, W. Chen, H. Hallez, J. Jia, and B. Vanrumste, “Measuring and localizing individual bites using a sensor augmented plate during unrestricted eating for the aging population,” IEEE J. Biomed. Heal. Informatics, vol. 24, no. 5, pp. 1509–1518, 2020.
  10. M. P. Lasschuijt, E. Brouwer-Brolsma, M. Mars, E. Siebelink, E. Feskens, K. de Graaf and G. Camps, “Concept development and use of an automated food intake and eating behavior assessment method,” J. Vis. Exp. (168), e62144.
  11. H. Zhao, “A noncontact breathing disorder recognition system using 2.4-GHz digital-IF doppler radar,” IEEE J. Biomed. Heal. Informatics, vol. 23, no. 1, pp. 208–217, 2019.
  12. J. Maitre, K. Bouchard, and S. Gaboury, “Fall detection with UWB radars and CNN-LSTM architecture,” IEEE J. Biomed. Heal. Informatics, vol. 25, no. 4, pp. 1273–1283, 2021.
  13. Z. Sun, Q. Ke, H. Rahmani, M. Bennamoun, G. Wang, and J. Liu, “Human action recognition from various data modalities: A review,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 3, pp. 3200–3225, 2023.
  14. F. Luo, S. Poslad, and E. Bodanese, “Kitchen activity detection for healthcare using a low-power radar-enabled sensor network,” IEEE Int. Conf. Commun., 2019, pp. 1-7.
  15. A. Gorji, T. Gielen, M. Bauduin, H. Sahli, and A. Bourdoux, “A multi-radar architecture for human activity recognition in indoor kitchen environments,” 2021 IEEE Radar Conf., 2021, pp. 1-6.
  16. J. Maitre, K. Bouchard, C. Bertuglia, and S. Gaboury, “Recognizing activities of daily living from UWB radars and deep learning,” Expert Syst. Appl., vol. 164, no. 113994, 2021.
  17. Y. Xie, R. Jiang, X. Guo, Y. Wang, J. Cheng, and Y. Chen, “mmEat: Millimeter wave-enabled environment-invariant eating behavior monitoring,” Smart Heal., vol. 23, no. 100236, 2022.
  18. Y. Wu, Y. Chen, S. Shirmohammadi, and C. Hsu, “AI-assisted food intake activity recognition using 3D mmWave radars,” In Proceedings of the 7th International Workshop on Multimedia Assisted Dietary Management (MADiMa ’22). Oct, 2022, pp. 81–89.
  19. S. Zhu, R. G. Guendel, G. S. Member, A. Yarovoy, F. Fioranelli, and S. Member, “Continuous human activity recognition with distributed radar sensor networks and CNN–RNN architectures,” IEEE Trans. Geosci. Remote Sens., vol. 60, pp.1-15, 2022.
  20. Z. Ni and B. Huang, “Gait-based person identification and intruder detection using mm-Wave sensing in multi-person scenario,” IEEE Sens. J., vol. 22, no. 10, pp. 9713-9723, 2022.
  21. H. Sloetjes and P. Wittenburg, “Annotation by category - ELAN and ISO DCR,” In Proc. 6th Int. Conf. Lang. Resour. Eval. Lr., 2008, pp. 816–820.
  22. C. Lea, M. D. Flynn, R. Vidal, A. Reiter, and G. D. Hager, “Temporal convolutional networks for action segmentation and detection,” In Proc. 30th IEEE Conf. Comput. Vis. Pattern Recognition (CVPR), 2017, pp. 1003–1012.
  23. B. Filtjens, P. Ginis, A. Nieuwboer, P. Slaets, and B. Vanrumste, “Automated freezing of gait assessment with marker-based motion capture and multi-stage spatial-temporal graph convolutional neural networks,” J. Neuroeng. Rehabil., vol. 19, no. 1, pp. 1–14, 2022.
  24. D. Jarrett, J. Yoon, and M. Van Der Schaar, “Dynamic prediction in clinical survival analysis using temporal convolutional networks,” IEEE J. Biomed. Heal. Informatics, vol. 24, no. 2, pp. 424–436, Feb. 2020.
  25. S. Bai, J. Z. Kolter, and V. Koltun, “An empirical evaluation of generic convolutional and recurrent networks for sequence modeling,” 2018, [Online]. Available: http://arxiv.org/abs/1803.01271.
  26. A. Vaswani, N. Shazeer, N. Parmar, and J. Uszkoreit, “Attention is all you need,” 2017, arXiv:1706.03762.
  27. J. Selva, A. S. Johansen, S. Escalera, K. Nasrollahi, T. B. Moeslund, and A. Clapes, “Video transformers: A survey,” IEEE Trans. Pattern Anal. Mach. Intell., pp. 1–20, 2023.
  28. X. Zhang, L. Han, W. Zhu, L. Sun, and D. Zhang, “An explainable 3D residual self-attention deep neural network for joint atrophy localization and alzheimer’s disease diagnosis using structural MRI,” IEEE J. Biomed. Heal. Informatics, vol. 26, no. 11, pp. 5289–5297, 2021.
  29. S. P. Singh, M. K. Sharma, A. Lay-Ekuakille, D. Gangwar, and S. Gupta, “Deep convLSTM with self-attention for human activity decoding using wearable sensors,” IEEE Sens. J., vol. 21, no. 6, pp. 8575–8582, 2021.
  30. R. Xue and X. Bai, “2D-Temporal convolution for target recognition of SAR sequence image,” 2019 6th Asia-Pacific Conf. Synth. Aperture Radar, APSAR 2019, pp. 2–5.
  31. Y. A. Farha and J. Gall, “MS-TCN: Multi-stage temporal convolutional network for action segmentation,” In Proc. 32th IEEE Conf. Comput. Vis. Pattern Recognition (CVPR), 2019, pp. 3570–3579.
  32. A. Ullah, J. Ahmad, K. Muhammad, M. Sajjad, and S. W. Baik, “Action recognition in video sequences using deep bi-directional LSTM with CNN features,” IEEE Access, vol. 6, pp. 1155–1166, 2017.
  33. K. Hara, H. Kataoka, and Y. Satoh, “Can spatiotemporal 3D CNNs retrace the history of 2D CNNs and ImageNet?,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2018, pp. 6546–6555.
  34. C. Feichtenhofer, H. Fan, J. Malik, and K. He, “Slowfast networks for video recognition,” Proc. IEEE Int. Conf. Comput. Vis., vol. 2019-Octob, 2019, pp. 6201–6210.
  35. C. Feichtenhofer, “X3D: Expanding architectures for efficient video recognition,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2020, pp. 200–210.
  36. J. Carreira and A. Zisserman, “Quo vadis, action recognition? A new model and the Kinetics dataset,” in CVPR, 2017, pp. 6299–6308.
  37. A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić, and C. Schmid, “ViViT: A video vision transformer,” Proc. IEEE Int. Conf. Comput. Vis., 2021, pp. 6816–6826.
  38. X. Li, Y. He, and X. Jing, “A survey of deep learning-based human activity recognition in radar,” Remote Sens., vol. 11, no. 9, 2019.
  39. Y. Shen, J. Salley, E. Muth, and A. Hoover, “Assessing the accuracy of a wrist motion tracking method for counting bites across demographic and food variables,” IEEE J. Biomed. Heal. Informatics, vol. 21, no. 3, pp. 599–606, 2017.
  40. D. Hossain, T. Ghosh, and E. Sazonov, “Automatic count of bites and chews from videos of eating episodes,” IEEE Access, vol. 8, pp. 101934–101945, 2020.
  41. E. S. Sazonov, O. Makeyev, S. Schuckers, P. Lopez-Meyer, E. L. Melanson, and M. R. Neuman, “Automatic detection of swallowing events by acoustical means for applications of monitoring of ingestive behavior,” IEEE Trans. Biomed. Eng., vol. 57, no. 3, pp. 626–633, 2010.
  42. K. Lee, “Food intake detection using ultrasonic doppler sonar,” IEEE Sens. J., vol. 17, no. 18, pp. 6056–6068, 2017.
  43. Z. Qiu, T. Yao, and T. Mei, “Learning spatio-temporal representation with pseudo-3D residual networks,” Proc. IEEE Int. Conf. Comput. Vis., vol. 2017-Octob, 2017, pp. 5534–5542.
  44. N. Gillani and T. Arslan, “Unobtrusive detection and monitoring of tremors using non-contact radar sensor,” Proc. 4th Int. Conf. Bio-Engineering Smart Technol.(BioSMART), 2021, pp. 1–4.
  45. R. Feng, E. De Greef, M. Rykunov, H. Sahli, S. Pollin, and A. Bourdoux, “Multipath ghost recognition for indoor MIMO radar,” IEEE Trans. Geosci. Remote Sens., vol. 60, pp. 1–10, 2022.
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now