WatchHAR: Real-Time On-Device Activity Recognition

• WatchHAR is a smartwatch-based human activity recognition system that integrates audio and inertial sensing for on-device processing.
• It employs a two-stage pipeline with a depthwise CNN event detector and a multimodal classifier using gated fusion, ensuring efficient energy and privacy management.
• Evaluations on Apple Watch Series 7 demonstrate improved F1 scores, faster processing times, and reduced FLOPs compared to traditional offloading methods.

WatchHAR is a real-time on-device human activity recognition system for smartwatches that combines audio and inertial sensing, runs fully on the watch, and is designed to address privacy and latency constraints associated with external data processing in unconstrained environments (Yeon et al., 5 Sep 2025). In the adjacent literature, the term also appears in a broader watch-based HAR context; however, HARMES explicitly states that “WatchHAR” in its paper refers to Yeon et al. as a system, not to a separate dataset, which is important for distinguishing the system itself from later wrist-worn benchmarks and datasets (Burchard et al., 4 May 2026).

1. Definition and scope

WatchHAR targets smartwatch-based recognition of Activities of Daily Living using a multimodal pipeline built around a wrist-worn microphone and a 6-axis IMU. Its central design premise is that smartwatches are always-on, worn close to the hand, and equipped with complementary sensors, yet fully on-device multimodal HAR remains difficult because offloading introduces privacy risk and latency, audio processing is power- and compute-intensive, watch memory and compute are limited, and unconstrained environments induce variable noise and motion artifacts (Yeon et al., 5 Sep 2025).

Within this framing, WatchHAR should be understood as a deployable system architecture rather than a benchmark dataset. That distinction is sometimes obscured because later papers place WatchHAR alongside datasets such as SAMoSa, HARMES, and OpenWatch when discussing the broader watch-based HAR ecosystem. HARMES makes the distinction explicit: its comparison section uses “WatchHAR” to denote the Yeon et al. real-time on-device system, while positioning HARMES itself as a larger multimodal wrist dataset for benchmarking cross-subject generalization in realistic home environments (Burchard et al., 4 May 2026).

The system also occupies a specific methodological niche. It preserves the two-stage structure of prior inertial–acoustic pipelines, notably SAMoSA’s event detector and multimodal classifier, but re-engineers both stages for watch-native inference. This suggests a transition in the literature from offline or phone-assisted wearable HAR toward end-to-end systems whose sensing, preprocessing, fusion, and inference all execute on the smartwatch itself (Yeon et al., 5 Sep 2025).

2. End-to-end architecture

WatchHAR consists of two tightly coupled components: an IMU-only event detector that runs continuously, and a multimodal activity classifier that is activated only when the detector signals an activity event. The event detector operates on 6-axis IMU data at 50 Hz using a 3 s window and a 20 ms hop. Its architecture is a depthwise 1D CNN with four convolutional blocks, max pooling, fully connected layers of sizes $512 \rightarrow 256 \rightarrow 128$, and a final sigmoid for binary event versus no-event prediction. A 2 s moving average is applied to the detector output to suppress spurious detections (Yeon et al., 5 Sep 2025).

The detector’s output gates the microphone. When no event is detected, the microphone remains off to preserve energy; when an event is detected, the system activates the microphone and runs the multimodal classifier on synchronized 1 s windows of audio and IMU with a 20 ms hop. This gating policy is both an efficiency mechanism and a privacy mechanism, because raw audio and IMU remain on the watch and the microphone is not continuously active (Yeon et al., 5 Sep 2025).

A defining feature of WatchHAR is the unification of audio preprocessing and inference into a single differentiable module. Instead of treating acoustic feature extraction as an external, fixed preprocessing step, the system computes the Short-Time Fourier Transform by 1D convolutions for real and imaginary parts, implements mel filterbanks as a trainable linear layer initialized from triangular mel filters, and applies amplitude-to-dB conversion through a logarithmic activation. These layers feed an audio encoder based on MobileNetV3 pretrained on AudioSet. In parallel, the IMU branch uses a ConvBoost-style CNN2D for multivariate time series, with $5 \times 1$ kernels, max pooling after the first two convolutional layers, ReLU activations, and a fully connected layer with dropout $p=0.5$ (Yeon et al., 5 Sep 2025).

Fusion is performed by a Gated Fusion mechanism. The IMU and audio embeddings are first projected into a shared 256-dimensional space, after which modality-specific gating networks produce element-wise weights in $[0,1]$ through linear layers followed by sigmoids. The gated representations are summed, mapped to a 256-dimensional fused representation, and passed to a classifier head. All components are differentiable and jointly optimized, so gradients propagate through both the acoustic preprocessing layers and the sensor encoders (Yeon et al., 5 Sep 2025).

The deployment path is likewise integral to the architecture. PyTorch models are traced with torch.jit.trace at batch size 1, converted to CoreML, quantized to float16, and executed on the Apple Watch Series 7 GPU. The paper attributes part of the runtime reduction to the elimination of separate CPU-bound audio feature extraction, since the STFT, mel, and log operations are embedded within the network and run on the mobile neural processing stack (Yeon et al., 5 Sep 2025).

3. Data sources and evaluation protocols

WatchHAR is evaluated on three datasets with distinct operating conditions. SAMoSA contains 20 participants, synchronized smartwatch IMU at 50 Hz, and audio originally at 16 kHz but resampled to 1 kHz for privacy preservation; it spans 26 activities across kitchen, bathroom, workshop, and miscellaneous contexts and totals 14.2 h, of which 5.9 h are labeled activities and 8.3 h are transition “Other.” The Semi-Naturalistic dataset of Bhattacharya et al. contains 15 participants, accelerometer and gyroscope at 50 Hz, audio at 22.05 kHz, and 23 activities recorded in continuous sessions with manual annotation. The In-the-Wild dataset of Bhattacharya et al. contains 5 participants, smartwatch sensing, and chest-mounted smartphone egocentric video for sparse labeling support (Yeon et al., 5 Sep 2025).

Two evaluation regimes are used: Leave-One-Participant-Out and Personalized-LOPO, where a subset of the held-out user’s data is available during training. For the In-the-Wild setting, weighted F1 is reported under P-LOPO, and the system is pretrained on Semi-Naturalistic data before fine-tuning with within-session data. The WatchHAR paper does not provide explicit optimizer, learning rate, batch size, or epoch count, but it does state that fusion and classification are trained end-to-end and that the gated fusion is trained with binary cross-entropy loss (Yeon et al., 5 Sep 2025).

These protocol choices matter because WatchHAR is not optimized merely for in-dataset accuracy on short, isolated clips. Its central claim is feasibility under streaming, watch-resident operation with hop-synchronous windows and online microphone gating. That makes latency, FLOPs, and per-window processing time first-order evaluation criteria alongside recognition accuracy (Yeon et al., 5 Sep 2025).

4. Empirical performance and efficiency

On Apple Watch Series 7 hardware, the event detector reaches an F1 score of 93.5%, compared with 88.0% for the SAMoSA baseline. Its processing time per IMU window is 9.3 ms, compared with 55.4 ms, while onset latency is reduced from 0.62 s to 0.27 s and offset latency from 0.16 s to 0.07 s. For the authors’ deployment argument, these numbers establish that continuous inertial gating is practical at smartwatch timescales (Yeon et al., 5 Sep 2025).

For activity classification, the strongest reported results are also tied to concrete efficiency gains. On SAMoSA, WatchHAR attains 92.34% context-wise accuracy with 11.8 ms processing time and 0.036 GFLOPs, versus 92.2%, 56.4 ms, and 1.71 GFLOPs for the baseline. On the Semi-Naturalistic dataset, it reports LOPO 90.4% and P-LOPO 93.8%, with 71.1 ms processing time and 0.917 GFLOPs, compared with 89.7%, 94.3%, 438.3 ms, and 4.24 GFLOPs for the Bhattacharya et al. baseline. In the In-the-Wild setting, it reports 56.7% weighted F1 under P-LOPO versus 55.8%, and 28.5% F1 without per-user fine-tuning versus 26.8% (Yeon et al., 5 Sep 2025).

The paper attributes these gains to several compounding substitutions. Replacing a Random Forest plus eight hand-crafted IMU features with the depthwise 1D CNN detector reduces feature computation from 28.76 ms to 3 ms and lowers total event-detection time to 9.3 ms while improving F1 by 5.5 percentage points. Embedding audio preprocessing inside the network removes heavy CPU preprocessing and contributes to the reported 5× speedup on SAMoSA. Among evaluated audio backbones, MobileNetV3 attains the best P-LOPO value, 86.7%, while being approximately 30× smaller and approximately 13× lighter in FLOPs than ResNet-22. Among IMU backbones, CNN2D achieves the highest LOPO value, 85.5%, at the lowest FLOPs, 0.13 G. Gated Fusion nearly matches the best LOPO result, 90.4%, and improves P-LOPO to 93.8% with negligible FLOPs increase and fewer than 0.5M parameters (Yeon et al., 5 Sep 2025).

Taken together, these measurements define WatchHAR less as a single classifier than as a systems-level optimization stack. Its reported contribution is not simply accuracy above 90% across more than 25 activity classes, but the combination of that accuracy with 5–47× lower FLOPs than baselines and fully on-watch execution (Yeon et al., 5 Sep 2025).

5. Relation to datasets, benchmarks, and adjacent watch-based HAR research

Later watch-based datasets place WatchHAR in a broader multimodal landscape. HARMES introduces a dataset collected from 20 participants performing household activities in their own homes, with dual-wrist IMUs, wrist audio at 44.1 kHz, and atmospheric sensing through humidity, temperature, and pressure. It totals 80.53 h of recorded data, with 60.9–61 h of fully labeled ADLs across 15 classes, and is described as nearly six times larger than the previously largest wrist-inertial-acoustic HAR dataset. In its comparison section, HARMES positions itself as directly relevant to watch-based HAR and to prior wrist inertial–acoustic efforts sometimes referred to as “WatchHAR,” while clarifying that WatchHAR itself is a system rather than a dataset (Burchard et al., 4 May 2026).

HARMES is especially relevant because it exposes the strengths and limits of the modality pairing on which WatchHAR is built. In its LOPO benchmark, audio alone reaches $\mathrm{F1}_{\text{macro}} \approx 0.700$, IMU$_R$+Audio reaches $\mathrm{F1}_{\text{macro}} \approx 0.736$, and IMU$_L$+IMU$_R$+Audio reaches $\mathrm{F1}_{\text{macro}} \approx 0.763$ with accuracy $5 \times 1$0. Its ablation study shows that audio resolves ambiguities such as floor cleaning versus vacuum cleaning, washing hands versus washing dishes, disinfecting hands versus applying hand cream, and making tea versus watering plants. Humidity provides clear contextual cues for water-related activities but does not improve aggregate F1 under naive short-window fusion because the sensor is sampled at 1 Hz and exhibits response delay (Burchard et al., 4 May 2026).

OpenWatch broadens the watch-HAR space in a different direction. It is an open-access smartwatch benchmark for hand-gesture recognition using synchronized IMU and PPG on a commercial watch, with 50 participants, 78 sessions, more than 10 hours of data, and a taxonomy of 59 labelled gesture sequences. Its core benchmark focuses on five gestures—double_clench, double_pinch, pinch_up, pinch_down, and slide—under subject-independent evaluation. In this setting, MixToken reaches clip-level macro-F1 $5 \times 1$1 and accuracy $5 \times 1$2 with approximately 223k parameters, while NormWear-LoRA reaches clip-level macro-F1 $5 \times 1$3 and accuracy $5 \times 1$4, and PPG contributes a clip-level macro-F1 gain of approximately $5 \times 1$5 percentage points for foundation-model inputs. OpenWatch therefore complements WatchHAR by emphasizing gesture recognition, IMU–PPG fusion, and the trade-off between specialized compact architectures and adapted foundation models (Bonazzi et al., 6 May 2026).

An earlier but methodologically relevant line is self-supervised wrist HAR using wrist-worn accelerometers. The 2021 work on self-supervised neural networks trains a 5-block CNN encoder producing 256-dimensional embeddings from tri-axial wrist accelerometer windows, using approximately 42,000 hours of unlabeled Project Baseline Health Study data at 30 Hz. It pairs this representation learning with a salience-based segmentation algorithm for continuous streams and reports cross-dataset accuracies such as 83.3% on PAMAP2 and 93.4% on MHealth, exceeding a simple statistical baseline. Although it is not an on-watch multimodal system, it is directly relevant as a device-agnostic representation-learning strategy for wrist HAR under heterogeneous conditions (Sridhar et al., 2021).

6. Constraints, misconceptions, and future directions

A persistent misconception is that WatchHAR names a benchmark corpus. The literature summarized here does not support that reading. WatchHAR is the Yeon et al. on-device audio-and-inertial system; HARMES and OpenWatch are datasets or benchmarks that sit alongside it in the watch-based HAR literature rather than constituting alternate releases of the same resource (Burchard et al., 4 May 2026).

The principal constraints of WatchHAR are also explicit. Audio is power-hungry, so microphone gating is central to the design. watchOS does not expose fine-grained battery metrics, and energy measurements were not reported. Hardware variability matters: older watches may exhibit slower inference or higher drain, whereas newer devices with NPUs may further improve efficiency. Domain shift remains a challenge, since performance degrades from semi-controlled to in-the-wild conditions. Longitudinal performance over multi-week or multi-month deployment was not studied (Yeon et al., 5 Sep 2025).

Adjacent benchmarks sharpen these limitations. HARMES shows that home environments introduce substantial environmental diversity, including different taps, kettles, vacuum cleaners, and room layouts, which is beneficial for generalization but challenging for models. It also highlights privacy-sensitive handling of audio by prohibiting speech during recording and muting 47.5 s where speech occurred. OpenWatch, by contrast, shows that multimodal smartwatch recognition need not be restricted to audio and IMU: synchronized PPG can materially improve subject-independent recognition, especially at clip level, and lightweight specialized architectures can outperform large adapted foundation models in both F1 and memory efficiency (Burchard et al., 4 May 2026, Bonazzi et al., 6 May 2026).

A plausible implication is that future WatchHAR-style systems will combine several directions that are presently distributed across separate papers: on-device gating and differentiable preprocessing from WatchHAR, broader in-home multimodal benchmarking from HARMES, physiological fusion and compact subject-independent models from OpenWatch, and self-supervised wrist representation learning plus salience-based streaming segmentation from earlier accelerometer-only work (Yeon et al., 5 Sep 2025, Sridhar et al., 2021). The current literature already indicates the technical rationale for such convergence: modality complementarity is activity-dependent, sequence-level aggregation can stabilize predictions, and cross-subject or cross-device robustness is strengthened by explicit attention to deployment constraints rather than by classifier accuracy alone.