Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
140 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

On the Efficiency and Robustness of Vibration-based Foundation Models for IoT Sensing: A Case Study (2404.02461v1)

Published 3 Apr 2024 in cs.LG and eess.SP

Abstract: This paper demonstrates the potential of vibration-based Foundation Models (FMs), pre-trained with unlabeled sensing data, to improve the robustness of run-time inference in (a class of) IoT applications. A case study is presented featuring a vehicle classification application using acoustic and seismic sensing. The work is motivated by the success of foundation models in the areas of natural language processing and computer vision, leading to generalizations of the FM concept to other domains as well, where significant amounts of unlabeled data exist that can be used for self-supervised pre-training. One such domain is IoT applications. Foundation models for selected sensing modalities in the IoT domain can be pre-trained in an environment-agnostic fashion using available unlabeled sensor data and then fine-tuned to the deployment at hand using a small amount of labeled data. The paper shows that the pre-training/fine-tuning approach improves the robustness of downstream inference and facilitates adaptation to different environmental conditions. More specifically, we present a case study in a real-world setting to evaluate a simple (vibration-based) FM-like model, called FOCAL, demonstrating its superior robustness and adaptation, compared to conventional supervised deep neural networks (DNNs). We also demonstrate its superior convergence over supervised solutions. Our findings highlight the advantages of vibration-based FMs (and FM-inspired selfsupervised models in general) in terms of inference robustness, runtime efficiency, and model adaptation (via fine-tuning) in resource-limited IoT settings.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. S. Liu, T. Kimura, D. Liu, R. Wang, J. Li, S. Diggavi, M. Srivastava, and T. Abdelzaher, “Focal: Contrastive learning for multimodal time-series sensing signals in factorized orthogonal latent space,” in Advances in Neural Information Processing Systems, 2023.
  2. T. Wang, D. Kara, J. Li, S. Liu, T. Abdelzaher, and B. Jalaian, “The methodological pitfall of dataset-driven research on deep learning: An iot example,” in MILCOM 2022-2022 IEEE Military Communications Conference (MILCOM).   IEEE, 2022, pp. 1082–1087.
  3. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
  4. B. Chatterjee, N. Cao, A. Raychowdhury, and S. Sen, “Context-aware intelligence in resource-constrained iot nodes: Opportunities and challenges,” IEEE Design & Test, vol. 36, no. 2, pp. 7–40, 2019.
  5. S. Yao, Y. Zhao, H. Shao, S. Liu, D. Liu, L. Su, and T. Abdelzaher, “Fastdeepiot: Towards understanding and optimizing neural network execution time on mobile and embedded devices,” in Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems, 2018, pp. 278–291.
  6. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
  7. K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 16 000–16 009.
  8. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” arXiv preprint arXiv:1910.13461, 2019.
  9. T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in International conference on machine learning.   PMLR, 2020, pp. 1597–1607.
  10. Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen, “Graph contrastive learning with augmentations,” Advances in neural information processing systems, vol. 33, pp. 5812–5823, 2020.
  11. C.-Y. Chuang, J. Robinson, Y.-C. Lin, A. Torralba, and S. Jegelka, “Debiased contrastive learning,” Advances in neural information processing systems, vol. 33, pp. 8765–8775, 2020.
  12. D. Liu, T. Wang, S. Liu, R. Wang, S. Yao, and T. Abdelzaher, “Contrastive self-supervised representation learning for sensing signals from the time-frequency perspective,” in 2021 International Conference on Computer Communications and Networks (ICCCN).   IEEE, 2021, pp. 1–10.
  13. S. Yao, S. Hu, Y. Zhao, A. Zhang, and T. Abdelzaher, “Deepsense: A unified deep learning framework for time-series mobile sensing data processing,” in International Conference on World Wide Web (WWW), 2017.
  14. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in IEEE/CVF International Conference on Computer Vision (CVPR), 2021.
  15. I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in International Conference on Learning Representations, 2018.
  16. ——, “Sgdr: Stochastic gradient descent with warm restarts,” in International Conference on Learning Representations, 2016.
  17. D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015.
  18. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in International Conference on Learning Representations, 2020.
  19. H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” in International Conference on Learning Representations, 2018.
  20. B. Salehi, G. Reus-Muns, D. Roy, Z. Wang, T. Jian, J. Dy, S. Ioannidis, and K. Chowdhury, “Deep learning on multimodal sensor data at the wireless edge for vehicular network,” IEEE Transactions on Vehicular Technology, vol. 71, no. 7, pp. 7639–7655, 2022.
  21. V. Radu, C. Tong, S. Bhattacharya, N. D. Lane, C. Mascolo, M. K. Marina, and F. Kawsar, “Multimodal deep learning for activity and context recognition,” Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies, vol. 1, no. 4, pp. 1–27, 2018.
  22. R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill et al., “On the opportunities and risks of foundation models,” arXiv preprint arXiv:2108.07258, 2021.
  23. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019.
  24. M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, R. Howes, P.-Y. Huang, H. Xu, V. Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski, “Dinov2: Learning robust visual features without supervision,” 2023.
  25. J. G. Almaraz-Rivera, J. A. Cantoral-Ceballos, and J. F. Botero, “Enhancing iot network security: Unveiling the power of self-supervised learning against ddos attacks,” Sensors, vol. 23, no. 21, p. 8701, 2023.
  26. Z. Zhang, S. Bu, Y. Zhang, and Z. Han, “Market-level integrated detection against cyber attacks in real-time market operations by self-supervised learning,” IEEE Transactions on Smart Grid, 2024.
  27. S. Zhang, O. T. Ajayi, and Y. Cheng, “A self-supervised learning approach for accelerating wireless network optimization,” IEEE Transactions on Vehicular Technology, 2023.
  28. M. S. Towhid and N. Shahriar, “Encrypted network traffic classification using self-supervised learning,” in 2022 IEEE 8th International Conference on Network Softwarization (NetSoft).   IEEE, 2022, pp. 366–374.
  29. G. Mai, W. Huang, J. Sun, S. Song, D. Mishra, N. Liu, S. Gao, T. Liu, G. Cong, Y. Hu et al., “On the opportunities and challenges of foundation models for geospatial artificial intelligence,” arXiv preprint arXiv:2304.06798, 2023.
  30. X. Chen, S. Xie, and K. He, “An empirical study of training self-supervised vision transformers,” in IEEE/CVF International Conference on Computer Vision (CVPR), 2021.
  31. X. Ouyang, X. Shuai, J. Zhou, I. W. Shi, Z. Xie, G. Xing, and J. Huang, “Cosmo: Contrastive fusion learning with small data for multimodal human activity recognition,” in International Conference on Mobile Computing And Networking (MobiCom), 2022.
  32. P. Poklukar, M. Vasco, H. Yin, F. S. Melo, A. Paiva, and D. Kragic, “Geometric multimodal contrastive representation learning,” in International Conference on Machine Learning (ICML), 2022.
  33. S. Tonekaboni, D. Eytan, and A. Goldenberg, “Unsupervised representation learning for time series with temporal neighborhood coding,” in International Conference on Learning Representations (ICLR), 2021.
  34. Z. Yue, Y. Wang, J. Duan, T. Yang, C. Huang, Y. Tong, and B. Xu, “Ts2vec: Towards universal representation of time series,” in AAAI Conference on Artificial Intelligence (AAAI), 2022.
  35. T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in International Conference on Machine Learning (ICML), 2020.
  36. Y. Tian, D. Krishnan, and P. Isola, “Contrastive multiview coding,” in European Conference on Computer Vision (ECCV), 2020.
  37. J. Li, M. Jing, H. Su, K. Lu, L. Zhu, and H. T. Shen, “Faster domain adaptation networks,” IEEE Transactions on Knowledge and Data Engineering, vol. 34, no. 12, pp. 5770–5783, 2021.
  38. Y. Zhao, D. Saxena, and J. Cao, “Memory-efficient domain incremental learning for internet of things,” in Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems, 2022, pp. 1175–1181.
  39. L. Zhang, X. Lei, Y. Shi, H. Huang, and C. Chen, “Federated learning for iot devices with domain generalization,” IEEE Internet of Things Journal, 2023.
  40. Y. Huang, M. Du, H. Zheng, and X. Feng, “Incremental unsupervised adversarial domain adaptation for federated learning in iot networks,” in 2022 18th International Conference on Mobility, Sensing and Networking (MSN).   IEEE, 2022, pp. 186–190.
  41. I. Achituve, H. Maron, and G. Chechik, “Self-supervised learning for domain adaptation on point clouds,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021, pp. 123–133.
  42. J. Xu, L. Xiao, and A. M. López, “Self-supervised domain adaptation for computer vision tasks,” IEEE Access, vol. 7, pp. 156 694–156 706, 2019.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets