Wearable AI in the Era of Large Sensor Models

Abstract: As an effective approach to understanding the human-centric physical world, Wearable AI, which leverages multimodal wearable sensors to understand human physiology and behavior, has attracted increasing attention in recent years. However, existing sensor models remain largely siloed by modality and task, lacking a unified paradigm for integrating diverse wearable modalities, training strategies, and achieving robust generalization in real-world applications. Motivated by the success of multimodal foundation models, which learn transferable representations from massive multimodal data, we argue that Large Sensor Models (LSMs), defined as foundation models trained on large-scale and multimodal wearable data, offer a promising pathway toward a more general and scalable framework for wearable AI. In this position paper, we formalize the data substrate underlying LSMs, analyze the unique challenges of large-scale wearable sensing, and articulate two directions: (i) LSMs without language capability and (ii) LSMs with language capability. We further discuss representative application areas that can be unlocked by such models. Through this paper, we encourage the community to explore LSMs as a foundational approach for the next generation of human-centric AI systems.

• The paper introduces Large Sensor Models (LSMs) as a foundation model for wearable AI, unifying fragmented sensor pipelines.
• It details generative, contrastive, and hybrid pretraining methods that enhance the representation of multimodal, continuous wearable data.
• It proposes integrating language models for cross-modal semantic alignment, paving the way for real-world applications in healthcare and assistive tech.

Large Sensor Models: A Foundation for Wearable AI

Introduction to the LSM Paradigm

The paper "Wearable AI in the Era of Large Sensor Models" (2604.10172) proposes Large Sensor Models (LSMs) as a principled foundation model (FM) paradigm for wearable sensing. Drawing inspiration from language and vision FMs, the authors argue that LSMs—trained on massive, multimodal, and continuous wearable sensor data—can unify currently fragmented, task-specific pipelines across wearable AI. Key contributions include a critical appraisal of the data substrate for wearable sensing, a bifurcation of LSM development between models with and without language capabilities, and a progressive research roadmap for application and deployment.

Figure 1: Overview of LSMs: (a) LSMs without language capability; (b) LSMs integrating LLMs for cross-modal semantic alignment.

Evolution of Sensor Models and the LSM Roadmap

The historical development of sensor models is categorized into four generations: statistical, neural, pre-trained, and now LSMs. The progression reflects increasing feature learning capacity and generalizability across tasks and modalities.

Figure 2: The four-generational roadmap of Sensor Models from manually engineered features to large-scale, multimodal foundation models.

Statistical models relied on handcrafted features and strong priors; neural sensor models leveraged deep architectures for raw data representation. Pre-trained sensor models adopted self-supervised and transfer learning paradigms to mitigate the lack of labels. The emergence of LSMs marks a transition toward large-scale models capable of unifying and transferring knowledge across diverse sensor modalities and tasks, paralleling advances in language and vision FMs.

Wearable Data Substrate: Modalities, Challenges, and Design Considerations

LSMs depend on complex, heterogeneous, and temporally structured data substrates. The sensor modalities relevant to LSMs are:

• Physiological: Low-dimension, high-structure signals like ECG, EDA, and PPG, crucial for cardiac and stress analysis, but susceptible to noise, motion artifacts, and device differences.
• Motion: Inertial sensors (accelerometer, gyroscope, magnetometer) capture movement across multiple scales but introduce challenges linked to sensor placement and calibration.
• Contextual/Position: Bluetooth, acoustic, UWB, and GNSS provide relational and spatial context, yet are sparse, asynchronous, and noise-prone.

Wearable data pose unique challenges: cross-modality heterogeneity, embodied and context-dependent distribution shifts, low semantic density at raw signal level, missing modalities, and irregular sampling. The lack of standardized "tokens" as in NLP complicates both modeling and labeling, with supervised learning limited by annotation cost and ambiguity in aligning signals to semantic concepts.

LSMs Without Language Capability

Large-scale self-supervised and contrastive pretraining form the dominant paradigm for current LSMs without language capabilities. Representation learning methods for wearable data segregate into:

• Generative: Signal reconstruction or prediction via autoencoding, including masked reconstruction strategies to capture contextual dependencies and robustness to missingness.
• Contrastive: Alignment across samples, subjects, modalities, or temporal windows, aiming for invariance while suppressing spurious correlations.
• Hybrid: Jointly optimizes generative and contrastive objectives to capture local signal fidelity and global invariance.

  Figure 4: Conceptual workflow of LSMs trained via large-scale generative and contrastive pretraining, covering local signal structure and cross-modal invariance.

A strong claim in the paper is that power-law scaling behavior reported in language/vision FMs may extend to wearable sensor domains, but the existence and stability of such scaling laws in highly heterogeneous and embodied environments remain open empirical questions.

The authors stress the need for inductive biases in model design that reflect physical invariance, embodiment, and nonstationarity inherent to sensor data, rather than naively copying paradigms from other FM domains.

LSMs With Language Capability

The integration of LLMs enables new interaction modalities and semantic capabilities for LSMs, with four principal directions:

1. Prompting: Serializing sensor data streams into pseudo-tokens for direct LLM input, enabling zero-shot or instruction-guided adaptation.
2. Tokenization: Designing tokenization mechanisms that discretize continuous sensor data for LLM consumption, preserving temporal and semantic structure.
3. Alignment: Mapping sensor and textual embeddings into shared semantic spaces, either via similarity-based (e.g., contrastive learning) or fusion-based (projecting sensor features into the LLM input embedding space) techniques.
4. Agentic Architectures: Employing LLMs as agents to coordinate task planning, retrieval, and multimodal reasoning in wearable pipelines.

   Figure 3: Visualization of prompting and tokenization strategies: raw signals become pseudo-tokens or are mapped to interpretable token sequences for LLM input.

   

   Figure 7: Schematic of alignment strategies: (a) similarity-based joint embedding of sensor and language; (b) fusion-based projection of sensor features into LLM-compatible spaces.

Despite rapid advances, current language-enabled LSMs are typically limited to single-modality or well-curated paired sensor-text corpora, with high computational and memory demand that impedes on-device deployment.

Application Areas and Implications

The unification of wearable sensing through LSMs has significant implications:

• Personal Assistants: Real-time integration of multimodal data for adaptive, context-aware recommendations and health state inference.
• Healthcare Monitoring: Continuous disease/accident risk stratification and early warning via comprehensive multimodal signals.
• Fitness and Lifestyle: Holistic support for activity, sleep, and dietary monitoring leveraging generalizable LSM representations.
• Embodied and Assistive Intelligence: Fine-grained user intent and context-awareness for physical interaction with surroundings, robotics, and virtual reality.
• Affective Computing and Emotion Support: Robust emotion inference and regulation in naturalistic settings through multimodal, longitudinal sensing.
• Transportation and Mobility: Large-scale behavioral modeling for transportation mode identification and human-centered mobility intelligence.

Future Directions: Development, Deployment, and Evaluation

Unresolved issues highlighted include:

• Data scale and diversity: The scarcity of large, diverse multimodal wearable datasets is a critical bottleneck.
• Model evaluation: Standardized, fair, and task-agnostic evaluation metrics for LSMs remain lacking; interpretability and mechanistic understanding lag behind capability.
• Deployment constraints: Real-world applications are constrained by resource limitations, computational cost, and privacy concerns, necessitating compressible and federated LSM variants.

The authors assert that addressing privacy and robust embedding of embodied physical context will be crucial for deployment at societal scale, especially in health-related applications.

Conclusion

This paper consolidates the paradigm shift in wearable sensing from siloed, task-specific models to LSMs as foundational, scalable, and unified representations across human-centric AI. The theoretical motivation, modeling strategies, and targeted application domains delineated herein establish a research agenda that will shape the pervasive deployment of sensor intelligence. The ultimate realization of LSMs—whether language-enabled or not—demands advances in scalable multimodal learning, privacy-preserving computation, embodied representation, and evaluation frameworks attuned to the unique constraints of wearable data and real-world context.