Scaling Wearable Foundation Models

Abstract: Wearable sensors have become ubiquitous thanks to a variety of health tracking features. The resulting continuous and longitudinal measurements from everyday life generate large volumes of data; however, making sense of these observations for scientific and actionable insights is non-trivial. Inspired by the empirical success of generative modeling, where large neural networks learn powerful representations from vast amounts of text, image, video, or audio data, we investigate the scaling properties of sensor foundation models across compute, data, and model size. Using a dataset of up to 40 million hours of in-situ heart rate, heart rate variability, electrodermal activity, accelerometer, skin temperature, and altimeter per-minute data from over 165,000 people, we create LSM, a multimodal foundation model built on the largest wearable-signals dataset with the most extensive range of sensor modalities to date. Our results establish the scaling laws of LSM for tasks such as imputation, interpolation and extrapolation, both across time and sensor modalities. Moreover, we highlight how LSM enables sample-efficient downstream learning for tasks like exercise and activity recognition.

• The paper introduces the Large Sensor Model (LSM), a foundation model for wearable data trained on 40 million hours from 165,000 users, and empirically evaluates scaling laws showing performance benefits but also saturation.
• The LSM demonstrates significant gains over conventional methods on generative tasks like imputation and downstream tasks such as exercise detection, indicating strong potential for transfer learning.
• Key implications include the applicability of scaling laws to multimodal wearable data, potential for enhanced personalized health tracking, and challenges related to handling missing data and generalization.

Scaling Wearable Foundation Models: Insights and Implications

The paper "Scaling Wearable Foundation Models" presents a comprehensive study on the development and scaling of foundation models tailored for wearable sensor data. This research is motivated by the proliferation of wearable devices and their potential to generate large volumes of multimodal data that can provide actionable insights for health and wellness.

Key Contributions

The paper introduces the Large Sensor Model (LSM), a foundation model trained on an extensive dataset of wearable sensor readings. This dataset includes up to 40 million hours of data collected from over 165,000 users, spanning multiple modalities such as heart rate, accelerometer, electrodermal activity, and skin temperature. The research focuses on several core areas:

1. Empirical Evaluation of Scaling Laws: The study investigates the effectiveness of scaling data, computing resources, and model size in improving the performance of foundation models. These evaluations reveal that, similar to other domains like language and vision, wearable sensor models benefit from scaling. However, they show signs of performance saturation at the upper bounds of model size and data volume.
2. Generative and Discriminative Tasks: The LSM is evaluated on tasks such as imputation, interpolation, and extrapolation, demonstrating significant performance gains over conventional methods. The model's performance in downstream tasks like exercise detection and activity recognition indicates its potential for efficient transfer learning and generalization across diverse activities.
3. Self-Supervised Learning with Masked Autoencoders: The authors employ a masked autoencoder approach, which allows the model to learn rich representations of the sensor data without extensive labeling. This method shows promise in addressing the challenges of large-scale data utilization and label efficiency.

Implications and Future Directions

The implications of this study are multifaceted, impacting both theoretical understanding and practical applications:

• Theoretical Implications: The findings suggest that the principles underlying scaling laws in neural networks apply to multimodal wearable data. This insight underscores the potential to extend foundational model architectures across varying domains of data, contributing to a more unified theory of model scaling.
• Practical Implications for Health and Wellness: Wearable sensors already play a pivotal role in health monitoring and behavior tracking. The advancements reported in this paper could enhance the predictive accuracy and robustness of health trackers, paving the way for personalized interventions and proactive health management.
• Challenges and Opportunities: Despite the promising results, challenges remain, particularly in handling the inherent missingness of wearable data and ensuring the generalization of models to unseen populations. Future work can explore more sophisticated methods for managing missing data, possibly incorporating domain knowledge about the physiological processes being monitored.

Conclusion

This research is a significant step towards harnessing the full potential of wearable sensor data through scalable and efficient foundation models. While there are challenges to be addressed in model scaling and data handling, the demonstrated benefits hold great promise for advancing wearable technology applications, particularly in personalized health and fitness. As foundation models continue to mature, the insights from this paper will likely inform ongoing efforts to integrate multimodal data into comprehensive health monitoring systems.