Triple Spectral Fusion for Sensor-based Human Activity Recognition

Abstract: The field of sensor-based human activity recognition (HAR) mainly uses posture, motion and context data of Inertial Measurement Units (IMUs) to identify daily activities. Despite the advancements in learning-based methods, it is challenging to perform information fusion from the temporal perspective due to the complexities in fusing heterogeneous sensor data and establishing long-term context correlations. This paper proposes a novel triple spectral fusion framework tailored for HAR. First, we develop an adaptive complementary filtering technique for noise suppression and organize each IMU's sensors into posture and motion modality nodes. Given that IMU nodes form a dynamic heterogeneous graph, we then apply adaptive filtering within the graph Fourier domain to merge both homogeneous and heterogeneous node information. Furthermore, an adaptive wavelet frequency selection approach is implemented to suppress context redundancy and shorten the length of features. This approach enhances both timestamp-based graph aggregation and the correlation of long-term contexts. Our framework uses adaptive filtering in the Fourier, graph Fourier, and wavelet domains, enabling effective multi-sensor fusion and context correlation. Extensive experiments on ten benchmark datasets demonstrate the superior performance of our framework. Project page: https://github.com/crocodilegogogo/TSF-TPAMI2026.

• The paper introduces an innovative triple spectral fusion framework that adaptively processes signals in the Fourier, graph Fourier, and wavelet domains.
• It employs adaptive filtering techniques, including causal convolutions and attention-based noise suppression, to optimize intra-IMU and inter-modal fusion.
• Experimental results reveal state-of-the-art performance with up to 4.6% improvement in F1 scores and efficient compute-accuracy tradeoffs across multiple HAR benchmarks.

Triple Spectral Fusion for Sensor-based Human Activity Recognition: An Expert Overview

Introduction

The proliferation of sensor-based Human Activity Recognition (HAR) systems has necessitated robust methodologies for fusing multimodal IMU signals in unconstrained, real-world environments. Existing learning-based systems have considerably advanced both sensor fusion mechanisms and temporal modeling; however, critical deficiencies persist in intra-IMU noise handling, exploitation of modality heterogeneity, and suppression of temporal redundancy. "Triple Spectral Fusion for Sensor-based Human Activity Recognition" (TSF) (2605.02743) introduces a formal framework leveraging spectral filtering in the Fourier, graph Fourier, and wavelet domains to address these open problems, establishing measurable gains over prevailing architectures on a broad benchmark suite.

Figure 1: Overall TSF framework depicting the IMU fusion block, modality node fusion block, and temporal information fusion block leading to final activity classification.

Triple Spectral Fusion Architecture

IMU Fusion Block: Adaptive Complementary Filtering

The IMU fusion block segregates IMU signals into posture (gravimeter, gyroscope) and motion (linear accelerometer) modalities, acknowledging their unique physical and noise characteristics. Unlike traditional approaches that uniformly process all IMU sensors, the posture branch applies adaptive, trainable complementary filtering using wide-kernel causal convolutions (CConv) combined with an attention weight. This generalizes classical UAV domain complementary filtering to deep learning, making filter hyperparameters learned end-to-end under varying noise distributions.

Figure 2: IMU fusion block: Separate processing of posture and motion using causal convolutions and adaptive sensor attention mechanism.

Here, the filter's cut-off frequency is adaptively learned via an attention mechanism, optimizing noise suppression dynamically with respect to both high-frequency gyroscope drift and low-frequency gravimeter interference. The architecture facilitates expeditious temporal information aggregation while automatically tuning the balance between gravimeter and gyroscope cues for robust intra-IMU fusion.

Modality Node Fusion Block: Adaptive Graph Fourier Filtering

To advance sensor fusion beyond homogeneous GNNs, TSF constructs a fully-connected graph where each node represents the posture or motion component of a body-worn IMU. In the graph Fourier domain, both low-pass and high-pass filters are applied and adaptively weighted through a learned attention scheme. The filtering spectrum can thus shift smoothly from extracting homogeneous (shared) information to preserving heterogeneity (node-specificity), regulated by dynamically estimated signed adjacency matrices learned via MLPs.

Figure 3: Modality node fusion: Dynamic, adaptive filtering of posture and motion branches in the graph Fourier domain.

This method circumvents the limitations of static, hand-designed edge connectivity and allows TSF to reconfigure spatial correlation structures per dataset, activity, subject, and even timestamp, effectively supporting transfer to arbitrary multi-IMU layouts and modalities.

Temporal Information Fusion Block: Adaptive Wavelet Frequency Selection

Central to TSF’s handling of context redundancy and diverse temporal scales is the temporal information fusion block, which recursively applies discrete wavelet transforms (DWT) to downsample and decompose features into high/low-frequency subbands with subsequent adaptive primary-band selection. The main selection is governed by Gumbel-softmax, providing differentiable routing for arbitrary time-frequency signatures. Local and global temporal aggregation (via convolution and attention) operate solely on the adaptively chosen primary wavelet branch, minimizing computational waste on irrelevant context.

Figure 4: Temporal information fusion with adaptive frequency selection, operating prior to temporal fusion and graph fusion layers, using the Gumbel softmax trick for binary gating.

This mechanism is essential in mitigating overfitting to spurious context and curbing the computational budget of Transformers and GNN layers, especially as sensor window size increases.

Experimental Results and Quantitative Evaluation

TSF is exhaustively evaluated on ten public HAR datasets, encompassing both single-IMU (smartphone) and multi-IMU (wearable) configurations under LOSO and K-fold protocols. Across virtually all benchmarks, TSF demonstrates highest F1 and WF1 scores compared to recent neural, GNN, and attention-based models. Notably, it achieves performance improvements of up to 4.6% in F1 and 2.9% in weighted F1 over the baseline ConvLSTM reference, with consistent outperformance versus IF-ConvTransformer and ConvBoost.

The architectural gains are accompanied by a favorable compute-accuracy tradeoff. FLOPs and parameter counts (see Figure 5) reveal that TSF not only compresses feature dimension via multistage adaptive wavelet reduction but also avoids the parameter scaling associated with fully parameterized sensor-fusion MLPs and convolutional layers, maintaining model size under practical regimes even as the number of sensor nodes increases.

Figure 5: (a) FLOPs and (b) parameter count analysis across models/datasets demonstrate that TSF achieves improved compute-accuracy efficiency compared to prior deep and transformer-based approaches.

Ablation studies further substantiate the necessity of each design module, with the most severe degradations yielded by removal of adaptive filtering (−2.34% F1) and replacement of learned DWT routing with fixed pooling strategies (−1.6% F1). Both low-pass and high-pass constituents in the graph domain, as well as adaptive, rather than static, frequency selection, contribute significant and orthogonal performance increments.

Analytical Insights

Noise Robustness and Dynamic Feature Selection

Extensive analysis on sensor-noise-corrputed data shows that the posture-sensor attention mechanism dynamically shifts focus in accordance with the spectral characteristics of injected noise, converging on optimal cut-off frequency regimes for each batch (Figure 6). This substantiates the block's dynamic noise compensation capacity, critical for robust deployment in real-world, variable, and adversarial noise environments.

Figure 6: IMU fusion block confers robust activity classification and dynamically adjusts sensor attention under increasing sensor noise intensities.

Graph Fourier Weights Interpretation

Analysis of intra-/inter-edge weights in the learned signed graph show that TSF automatically accentuates homogeneous features for structurally constrained activities (e.g., treadmill), while reserving high-pass weighting for more varied, unconstrained actions (e.g., basketball, subway), confirming that the global node fusion mechanism selectively prioritizes node similarity or uniqueness as required (Figure 7).

Figure 7: Histograms detailing learned intra- and inter-edge graph weights for diverse activities, illustrating adaptive fusion of homogeneous and heterogeneous sensor information.

Dynamic and Sample-wise Time-Frequency Routing

The visualizations in Figure 8 demonstrate that TSF's DWT routing varies not only per activity type but also on a per-subject and per-trial basis, with faster actions routed to higher-frequency branches and vice versa. This adaptive routing is essential for properly capturing the temporal variability across real-world HAR recordings.

Figure 8: Sample-dependent DWT routing mapped onto the time-frequency spectrum, showing TSF’s adaptability to subject, activity, and trial variations.

Theoretical and Practical Implications

TSF’s principled triple-domain spectral approach exhibits strong generalization to arbitrary IMU layouts, activity structures, and noise regimes. The demonstrated adaptability and efficient compute scaling position TSF as a compelling baseline for future HAR systems, particularly in applications involving privacy-sensitive, large-scale wearable deployments. The modularity of spectral filtering stages opens the path for extending the framework to domains such as multimodal fusion with vision, spatio-temporal pattern discovery, and context-aware privacy diagnostics.

From a theoretical standpoint, the findings suggest new directions in end-to-end spectral feature selection, graph attention mechanisms, and differentiable context compression within deep time-series models, with potential cross-pollination to more general sensor fusion and event detection at scale.

Conclusion

TSF defines a comprehensive architecture for sensor-based HAR, combining adaptive IMU-level noise suppression, flexible multi-modality node fusion, and sample-driven adaptive time-frequency context selection. Experimental results confirm strong, state-of-the-art performance and compute efficiency across HAR benchmarks. The robust, extensible design of TSF sets a technical foundation for future advances in multimodal activity recognition and efficient context modeling in resource-constrained sensor environments.