WHAR Arena: Benchmarking the State of the Art in Efficient Wearable Human Activity Recognition

Abstract: Deep learning has become the dominant paradigm in Wearable Human Activity Recognition (WHAR), yet progress is obscured by a comparability crisis. Results are often reported using inconsistent datasets, custom data processing, and varying evaluation protocols, making state-of-the-art claims fragile. We address this with a large-scale, open-source benchmark that integrates 30 diverse datasets under standardized processing, unified model interfaces, and a shared cross-subject evaluation protocol. Evaluating 17 representative architectures across 4760 training runs, we jointly measure predictive performance alongside on-device latency, peak memory, and model size on an Android reference device. Our results reveal that the WHAR state of the art is distributed rather than dominated by a single architecture. While CNN-HAR achieves the highest mean macro-F1, top-performing models cluster tightly, indicating contemporary architectures have converged near a predictive performance ceiling. When accounting for deployment efficiency, compact neural models, such as TinierHAR, and classical Random Forests define the practically relevant Pareto frontier, whereas larger recurrent and hybrid models incur high hardware costs without corresponding performance gains. Consequently, while predictive performance has plateaued, substantial potential for future progress remains in optimizing deployment efficiency and improving adaptation to domain shifts. We release our full framework to support transparent reuse and extension.

• The paper introduces WHAR Arena, a unified benchmark that standardizes data processing, evaluation, and deployment metrics across 30 diverse datasets.
• The paper employs a rigorous experimental protocol with cross-subject splits and measures key metrics including macro-F1, latency, memory usage, and model size.
• The study reveals that no single model dominates, with compact architectures and classical baselines offering competitive trade-offs for real-world edge deployments.

WHAR Arena: A Standardized Benchmark for Efficient Wearable Human Activity Recognition

Introduction and Motivation

Wearable Human Activity Recognition (WHAR) is critical for embedded health, fitness, and assistive technologies, yet the field has been hindered by a proliferation of ad hoc evaluation practices and non-standardized benchmarks. Existing studies have predominantly focused on isolated datasets, customized data-processing pipelines, and inconsistent evaluation protocols, leading to a comparability crisis that impedes rigorous assessment and longitudinal progress. Moreover, most prior work only partially considers constraints fundamental for deployment on edge and wearable devices—namely, efficiency in latency, memory, and model footprint.

To address these methodological deficits, the paper introduces WHAR Arena, a large-scale, open benchmark ecosystem that harmonizes data processing, model evaluation, and deployment metrics across an extensive and heterogeneous suite of real-world WHAR datasets and modeling approaches (2606.13194).

Dataset Suite and Standardization

The benchmark collates 30 carefully curated datasets encompassing a broad array of sensor modalities, device types, application contexts, and activity taxonomies. This deliberate selection ensures that both historic and emerging use cases—spanning daily living, locomotion, fitness, fall detection, and health monitoring—are represented. Heterogeneity is manifest in both configuration (e.g., single device, multimodal body sensor networks) and signal diversity (e.g., accelerometers, gyroscopes, physiological sensors).

Figure 2: Categorical attributes of selected WHAR datasets, highlighting the breadth of application settings, device types, and sensor modalities.

Figure 1: Bubble chart mapping the multimodal complexity and class granularity of each dataset, with points scaled by subject count and shaded by publication year.

Standardization is operationalized through a dataset library with configuration-driven preprocessing. Each dataset is harmonized to a session-centric tabular format (activity, session, window metadata), enabling separation and reproducibility of preprocessing, split generation, postprocessing, and loading. This is complemented by explicit session, subject, and activity identifiers, enabling rigorous cross-subject evaluation fundamental to real-world generalizability. An entity-relationship diagram formalizes the metadata schema.

Figure 3: Metadata schema for standardized WHAR dataset representation based on session/activity/window separation.

Model Suite and Benchmark Protocol

The study assembles 17 representative models: 14 diverse deep learning architectures—spanning convolutional and recurrent networks, attention mechanisms, and compact MLPs—alongside 3 strong classical baselines (Random Forest, k-NN, SVM). Model selection prioritizes code availability, generality, and feasibility for edge deployment.

A unified experimental protocol, enforced across the entire benchmark, features:

• Standardized preprocessing and 3s windows (50% overlap) at native sampling rates
• k-Subject-Groups splitting ($k=10$) to enforce strict subject-wise test sets
• Joint evaluation of predictive performance (macro-F1), inference latency, peak memory, and model size, with hardware measurements conducted on a Google Pixel 8 using ExecuTorch for deployment fidelity

Results: Predictive Performance and Generalization Landscape

Across 30 datasets and 4760 model splits, predictive performance is measured via macro-F1 on held-out subject groups. Results reveal that no single model dominates; top architectures cluster tightly with marginal differences. CNN-HAR achieves the highest mean macro-F1 (67.7%), but is closely trailed by TinyHAR and TripleCrossDomainAttention. Per-dataset wins are distributed, with Random Forest leading 9 datasets and a mix of deep and classical models on others.

Figure 4: Model-dataset predictive macro-F1 matrix; best result in each dataset highlighted, demonstrating tight clustering and distributed wins.

Figure 5: Summary statistics (mean/min/max) across models and datasets, illustrating performance bands and dataset-dependent variation.

This clustering of state-of-the-art macro-F1 scores—with multiple models achieving statistically indistinguishable results—suggests that the field may be near a predictive performance ceiling under current methodologies, especially for saturated datasets.

Deployment Efficiency: Latency, Memory, and Model Size

Edge deployment constraints are quantified along latency (ms), peak memory (MB), and model size (MB). Measurements demonstrate that compact neural architectures (TinierHAR, TinyHAR, CNN-HAR) and Random Forest collectively define the Pareto frontier—offering graduated trade-offs between performance and efficiency—while larger recurrent and hybrid models consistently incur substantial costs without performance benefit.

Figure 6: Pareto front trade-offs between latency and predictive performance (macro-F1), visualizing which models optimally balance accuracy with real-time inference.

Efficiency evaluations are contextualized across deployment axes (latency, memory, footprint) via a log-scale, min-max normalized Efficiency-Index, and a joint aggregate index integrating all costs with accuracy. TinierHAR dominates in joint efficiency, while Random Forest is optimal at the extreme of lowest latency.

Figure 7: Efficiency-index-based leaderboards for latency, memory, size, and aggregate deployment cost relative to accuracy.

Per-Dataset Analysis and Model Robustness

Detailed per-dataset matrices (not shown here due to limits) corroborate that deployment cost structure varies significantly depending on input sequence dimensionality and signal complexity. Nonetheless, robust model selection for novel settings is best informed by benchmarking relative rankings rather than single-dataset scores, given moderate rank correlations between datasets (mean Spearman $0.485$).

Methodological and Practical Implications

• The convergence of architectures on a macro-F1 ceiling under standardized evaluation indicates diminishing returns from incremental architectural changes without addressing dataset, annotation, or protocol-inherent noise and domain shift.
• Efficiency, rather than accuracy, emerges as a principal axis for future gains in WHAR, as deployment constraints remain unsolved for increasingly embedded scenarios (ultra-low-power MCUs, edge AI).
• Distributed leaderboard wins and dataset-sensitivity expose the necessity of broad, heterogeneous benchmarking to avoid misleading claims of universality or superiority.

Outlook and Future Directions

The benchmark provides essential infrastructure for reproducible, comparable, and deployment-aware progress in WHAR research. Remaining challenges include:

• Expansion to cover additional device classes (e.g., wearables, microcontrollers) and energy/throughput metrics
• Continuous integration of new datasets and models through open-source community workflows
• Exploration of domain-adaptive methods and semi-supervised learning for robust generalization under domain shift

Conclusion

WHAR Arena addresses long-standing methodological deficits in human activity recognition by instituting a rigorously standardized, large-scale, and deployment-centric benchmark. The empirical study demonstrates that, at the current state of the art, no single architecture consistently dominates in predictive accuracy or efficiency; instead, compact models and classical baselines yield competitive—and often superior—trade-offs for real-world constraints. This benchmark sets a transparent foundation for future WHAR advances, both methodological and practical, and underlines the imperative for future work to look beyond marginal architectural advances towards adaptive, efficient, and robust deployment under real-world variability.

(2606.13194)