- The paper introduces HARMES, an openly licensed dataset with over 80 hours of synchronized recordings from 20 participants across 15 household activities, combining dual-wrist IMUs, 44.1 kHz audio, and atmospheric sensors.
- The benchmark shows that both wrist IMUs plus audio achieve the strongest performance, reaching 0.794 accuracy and 0.763 macro F1 in leave-one-participant-out evaluation, while audio alone outperforms inertial-only configurations.
- The paper finds that humidity sensing does not improve classification with short windows despite measurable responses during water-related activities, highlighting the need for longer temporal context and improved multimodal fusion methods.
Overview and motivation
HARMES is a multi-modal wearable dataset for Human Activity Recognition (HAR) that combines three wrist-recorded modalities: inertial motion from two IMUs, atmospheric environmental sensing (humidity, temperature, pressure), and audio. The dataset was collected from 20 participants performing household Activities of Daily Living (ADLs) in their own homes across eight locations in Germany. It totals over 80 hours of recordings, of which 60.90 hours are fully labeled across 15 ADL classes, with an additional ~20 hours of partially labeled free-form background activities (2605.02596).
The authors' central research question is the extent to which environmental and acoustic sensing complement inertial measurements for recognizing ADLs that are ambiguous from motion data alone — for example, washing hands versus washing dishes, which share similar repetitive motion patterns but differ acoustically and in humidity response. To their knowledge, no prior public dataset jointly captures wrist-worn IMU data, audio, and atmospheric sensing; HARMES is also nearly six times larger than SAMoSa (14.2 h), previously the largest wrist-inertial-acoustic HAR dataset.
Position relative to existing datasets
The paper situates HARMES against a landscape where large multi-modal datasets typically pair egocentric video with head-mounted IMU (Epic-Kitchens, Ego4D, Ego-Exo4D) or combine multiple body-worn IMUs without audio or environmental sensing (WEAR, CSL-SHARE). The authors argue that head-mounted IMUs mostly decode head movement, whereas wrist-mounted IMUs provide substantially more discriminative signals for object manipulation. Among wrist-inertial-acoustic datasets, Berkeley MHAD provides only 82 minutes, ActionSense 9 h, and SAMoSa 14.2 h with a single right-wrist IMU and no left-handed participants. HARMES improves on these along several axes simultaneously:
| Property |
HARMES |
SAMoSa |
| Participants |
20 (incl. 3 left-handed) |
20 (right-handed only) |
| Labeled duration |
61 h |
14.2 h |
| IMU placement |
Both wrists |
Right wrist |
| Audio |
Yes (44.1 kHz raw) |
Subsampled |
| Atmospheric sensors |
Yes |
No |
| Recording setting |
Participants' homes |
Indoor environments |
The inclusion of three left-handed participants (15%) matches the estimated population share of handedness (9–18%), addressing a known bias in single-wrist setups. The out-of-lab recording in eight different homes introduces substantial variability in appliances, taps, and room layouts, which the authors argue prevents models from overfitting to specific devices — though this same variability is not quantified as a controlled factor.
Data collection protocol
Each participant wore a WearOS smartwatch on the right wrist (IMU + microphone at 44.1 kHz) and a Puck.js device with an I²C-attached BME280 sensor on the left wrist (52.5 Hz IMU plus humidity, temperature, pressure at 1 Hz). A researcher supervised each session, performing instantaneous on-the-fly labeling via a Python application, following the argument that post-hoc labeling is time-consuming and imperfect. Synchronization relied on a "clap three times" gesture visible in both IMU streams and audible in the microphone; the annotator pressed start exactly on the third clap. All activity repetitions (n = 2818) were visually inspected via stacked time-series plots to verify alignment, and two sensor failures spanning 40 instances were identified and excluded from downstream analysis.
Privacy was handled by prohibiting speech during sessions (activities were communicated silently via screen), muting seven segments totaling 47.5 s where third parties could be heard, and releasing the full raw audio signal rather than degraded versions. Each participant performed 47 repetitions of 15 activities per labeled recording in randomized order, plus one hour of free-form background activities annotated as supplementary labels. The background data is positioned as usable for null-class training, false-positive evaluation, unsupervised pre-training, or domain adaptation, though it was not used in the benchmark experiments.
Validation and benchmark results
The validation pipeline uses non-overlapping majority-vote-labeled windows of 5 s and 10 s, weighted cross-entropy loss, ADAM optimization, and leave-one-participant-out (LOPO) cross-validation. The baseline model is a late-fusion architecture with four branches: a CNN+LSTM audio branch on log-mel spectrograms, two DeepConvLSTM branches (one per wrist), and a humidity branch using 14 hand-crafted features, concatenated into a two-layer classification head.
Key LOPO results (5 s windows, macro F1):
| Sensor configuration |
Accuracy |
F1 (macro) |
| Humidity only |
0.143 |
0.098 |
| Left IMU only |
0.475 |
0.456 |
| Right IMU only |
0.599 |
0.570 |
| Both IMUs |
0.639 |
0.619 |
| Audio only |
0.738 |
0.700 |
| Right IMU + audio |
0.771 |
0.736 |
| Both IMUs + audio |
0.794 |
0.763 |
| All sensors (+ humidity) |
0.789 |
0.754 |
Three findings stand out. First, audio alone outperforms any IMU configuration (0.70 vs. 0.62 macro F1), and combining both IMUs with audio yields the best result at 0.763, confirming the complementarity of inertial and acoustic sensing. Second, adding the humidity branch did not improve performance — the all-sensors configuration slightly underperforms the IMU+audio configuration (0.754 vs. 0.763). This is a notable negative result given that visual inspection shows a clear average humidity increase of roughly 10% relative humidity during water-related activities such as washing hands and dishes. The authors attribute this gap to sensor latency (humidity peaks only after ~20 s during hand washing) and window lengths too short to capture slow temporal trends, and they explicitly frame better integration of atmospheric sensing as an open problem rather than claiming the modality is useless. Third, temperature and pressure showed no activity-dependent patterns comparable to humidity and were excluded from the ML experiments, although barometric pressure has been shown useful for elevation-related activities elsewhere.
Per-participant analysis shows the multi-modal model exceeds a macro F1 of 0.61 for every participant (maximum 0.84, mean 0.76, median 0.77) with reduced variance compared to IMU-only models (sd = 0.05 vs. 0.08). Notably, the IMU-only model struggles with the three left-handed participants, while adding audio equalizes performance across handedness — an implication that acoustic sensing can compensate for placement asymmetries that disadvantage minority-handedness users in single- or dual-IMU systems.
Confusion analysis identifies the hardest class pairs: putting away dishes vs. cleaning out the dishwasher, and applying hand cream vs. disinfecting hands. These pairs share nearly identical motion and sound patterns, and the latter two are also the smallest classes. The authors suggest these could be merged into single labels if researchers prefer, or treated as targets for context-based HAR beyond sensor fusion.
Limitations and open questions
The paper concedes several limitations directly. The dataset contains no speech at all, so classifiers trained on it will likely not learn robustness against speech masking activity sounds, although artificial noise augmentation could partially address this. The use of two different wearable devices may not reflect typical consumer setups and introduces cross-device variability. With 20 participants, larger cohorts would strengthen generalization claims, and the indoor home focus limits transferability to outdoor or industrial settings. Most importantly, the semi-naturalistic protocol does not permit measurement of true in-the-wild performance; a fully unscripted collection would be required.
Several questions remain open. Whether shorter windows than 5 s are viable for audio-inertial HAR on this data is untested, constrained by the 1 Hz humidity sampling. Whether a second wrist IMU justifies its cost is unresolved: single right-IMU + audio reached 0.736 versus 0.763 for dual-IMU + audio, hinting that a single smartwatch with microphone may suffice, but this requires further evaluation. How to incorporate slow humidity trends (e.g., augmenting a 10 s IMU window with the preceding 100 s of humidity context) into fusion architectures is explicitly proposed but not demonstrated.
Conclusion
HARMES contributes a large, synchronized, openly licensed multi-modal wearable dataset that fills a concrete gap: no prior public dataset combined dual-wrist IMU, high-resolution audio, and atmospheric sensing, and none approached its scale among inertial-acoustic collections. The benchmark validates that the data supports training deep models that generalize to unseen participants, and the ablation study quantifies modality contributions — establishing audio as strongly complementary to inertial sensing while showing that current state-of-the-art integration of humidity sensing fails to convert a clearly measurable signal response into classification gains. The dataset's principal value lies in enabling systematic study of sensor-fusion strategies, null-class handling, self-supervised pre-training, and improved atmospheric-sensing integration, all of which remain open problems that the authors document rather than resolve.