IMU Temporal Action Localization (IMU-TAL)

• IMU-TAL is a method that adapts video-temporal action localization to continuous, multivariate IMU data, enabling dynamic human activity segmentation.
• It employs multi-scale feature pyramids and per-timestamp prediction with regression boundaries to overcome fixed-window limits in traditional HAR.
• The framework integrates supervised and weakly supervised techniques, achieving enhanced temporal precision and improved detection metrics (e.g., up to +26% F1).

Inertial Measurement Unit Temporal Action Localization (IMU-TAL) is the adaptation of Temporal Action Localization (TAL) paradigms—originally developed for video analysis—to continuous, multivariate IMU data streams for human activity recognition (HAR). IMU-TAL aims to simultaneously predict labeled action segments—both class and temporal boundaries—within arbitrary-length IMU recordings, advancing beyond the traditional fixed-window classification paradigm that dominates inertial HAR. This transition enables more coherent detection of activities with variable and unknown durations, facilitates higher temporal precision, and aligns IMU-HAR with the increasingly segment-centric methodologies of the video TAL community. The formalization, algorithmic frameworks, evaluation protocols, empirical results, and emerging weakly supervised learning approaches for IMU-TAL are detailed below (Bock et al., 2023, &&&1&&&).

1. Formalization and Problem Shift

IMU-TAL transforms HAR from window-level classification to a segment localization task. In the standard fixed-window paradigm, the IMU time series $X$ is windowed into short segments (e.g., $1\ \mathrm{s}$, 50% overlap), each $x_\mathrm{SW} \in \mathbb{R}^{W\times S}$, where $W$ is window length (timesteps), $S$ is the number of sensor channels, and classified into one of $C$ activity classes (plus a NULL/background class).

IMU-TAL abandons this rigid association by applying a per-timestamp prediction across the entire windowed sequence. For every timestamp $t$, the network outputs:

• A categorical probability vector $p(a_t)$ over $C+1$ classes
• Two real-valued offsets $d_t^s > 0$ and $d_t^e > 0$ that regress the distance from $t$ to the start and end of the current (hypothesized) segment

Decoded, this yields the candidate segment $$(s_i, e_i, a_i) = (t - d_t^s, t + d_t^e, \arg\max p(a_t))$$. The entire output $\hat Y = \{ (p(a_t), d_t^s, d_t^e) \}_t$ defines a variable-length, temporally-localized prediction for complex activity sequences, breaking the “one window → one label” constraint in favor of “one timestamp → (label + temporal boundaries)” (Bock et al., 2023).

2. Model Architectures: Backbone and Detection Head Variants

IMU-TAL is instantiated by adapting single-stage TAL architectures from the video domain—ActionFormer, TemporalMaxer, and TriDet—retaining their core feature pyramids and detection heads but modifying input encodings for IMU data (Bock et al., 2023):

• Input Vectorization: Each window $x_\mathrm{SW}$ is flattened as $$\mathrm{vec}(x_\mathrm{SW}) \in \mathbb{R}^{W \cdot S}$$.
• Feature Pyramid Backbone: Models construct $L$-level pyramids $\{z_\ell\}$ to enable multi-scale temporal perception.
  • ActionFormer: Projects inputs with 1D convolutions, stacks $L$ transformer layers with windowed attention and local multi-head self-attention, and applies downsampling by depthwise 1D convolutions.
  • TemporalMaxer: Uses the same input projection; replaces transformers with 1D max-pooling and $1\times 1$ convolutions, omitting attention.
  • TriDet: Employs Scalable-Granularity Perception blocks that aggregate features with pointwise and depthwise convolutions at varying receptive fields; downsampled by max-pooling.
• Detection Heads: On every pyramid level and time index, a classification head (predicting $p(a_t)$ via softmax) and regression head (predicting $[d_t^s, d_t^e]$) operate.
  • TriDet incorporates a “trident” regression head with three branches (start, center-offset, end), aggregating with SoftArgmax over local windows.

Post-processing aggregates all segment proposals across pyramid levels and applies Soft-NMS (soft non-maximum suppression) with $\sigma=0.75$ to eliminate highly overlapping, low-confidence detections. Final segments are filtered by activity-class probability threshold $\tau$ (Bock et al., 2023).

3. Training Formulation and Losses

IMU-TAL is optimized with a multi-task loss at each timestamp and pyramid level. The training regime differs from classic window-classification through its direct boundary-aware regression and class balancing.

• Classification Loss (Focal Loss):

$$FL(p(a_t), y_t) = - \alpha_{y_t} \cdot (1 - p_{y_t})^\gamma \cdot \log p_{y_t}$$

where $y_t$ is the one-hot ground truth (0=NULL), $\alpha_{NULL} < \alpha_{act}$ to counter background class imbalance, and $\gamma=2$.

• Regression Loss (Generalized IoU):

For positive timestamps, the model computes the 1D Generalized IoU loss:

$L_{loc}(t) = 1 - \text{GIoU}(I_t^p, I_t^g)$

where $I_t^p = [t - d_t^s, t + d_t^e]$, $I_t^g = [s^g, e^g]$ is the ground-truth interval, and GIoU is calculated per Rezatofighi et al.

• Overall Loss:

$$L = \frac{1}{T} \sum_{t,\ell} FL(p_{t, \ell}, y_t) + \lambda \frac{1}{N_{pos}} \sum_{t \in Pos, \ell} L_{loc}(t, \ell)$$

where $N_{pos}$ is the number of positive (within-segment) timestamps, $\lambda$ is typically set to 1.

• Auxiliary Trick (Center Sampling): Regression loss is restricted to timestamps within a fraction of the center of ground-truth segments, minimizing label ambiguity noise (Bock et al., 2023).

4. Inference Pipelines: Offline and Near-Online

Two inference modes enable both retrospective (offline) and streaming (near-online) prediction.

• Offline: The full signal is windowed, vectorized, and processed in chunks (if exceeding $T_{max}$), with segment proposals merged post hoc. Segment filtering uses Soft-NMS and dataset-tuned thresholds.
• Near-Online: Maintains a buffer of the most recent $B$ windows, updating as new data arrives. Segments with centers near the buffer head are decoded and emitted in timestamp order, with local Soft-NMS for overlap mitigation.

This bifurcated approach supports both retrospective analysis and low-latency applications (Bock et al., 2023).

5. Evaluation Protocols and Empirical Performance

Evaluation uses leave-one-subject-out (LOSO) cross-validation over six canonical HAR datasets, with 1 s windows (50% overlap). Metrics include (Bock et al., 2023):

• Frame-level: Precision, Recall, F1 on 1 s steps after segment rasterization.
• Misalignment Ratios (UODIFM): Deletion, Underfill, Fragmentation, Insertion, Overfill, Merge, normalized per-class.
• Segment-level Mean Average Precision (mAP): AP per class at multiple temporal IoU thresholds (0.3, 0.4, 0.5, 0.6, 0.7), then mean-averaged.

Key quantitative findings:

• TAL models reach up to +26% F1 over DeepConvLSTM and other baselines; e.g., +25% on SBHAR, +10% on Hang-Time, +5% on Opportunity/Wetlab.
• mAP exceeds 45% in most cases (51% on Opportunity, 95% on SBHAR, 35% on Wetlab), compared to inertial baselines rarely exceeding 15% with majority-vote smoothing.
• TAL models produce higher NULL-class accuracy and reduced background confusion as per DETAD analysis.

A table summarizing reported segment-level mAP (τ-average):

Dataset	ActionFormer/TriDet (Supervised)	TemporalMaxer (Supervised)	TinyHAR/DeepConvLSTM (Baseline)
SBHAR	94.74	—	≈ 70
Opportunity	50.82	—	≈ 36
WetLab	37.14	—	< 20
Hang-Time	29.35	—	< 18

(Specific numbers from related works and LOSO splits; see (Bock et al., 2023), Table 2.)

6. Weakly Supervised IMU-TAL: Transfer and Benchmarking

IMU-TAL’s label-intensive, fully supervised regime motivates research into weakly supervised IMU-based temporal action localization (WS-IMU-TAL), which is benchmarked in WS-IMUBench (Li et al., 2 Feb 2026). In WS-IMU-TAL, only sequence-level multi-hot class labels are provided (no segment boundaries). Candidate segment proposals are instantiated using multiple-instance learning (MIL): short slices or proposals within the sequence are scored, and sequence-level aggregation functions optimize multi-label objectives. Localization arises from instances with high activation.

Key findings from WS-IMUBench include:

• Temporal-domain transfer: Audio (WSSED) and video (WSVAL) paradigms outperform image-based proposal methods due to the 1D sequential nature of IMU data.
• Dataset dependency: Weak supervision can match supervised models on datasets with long actions and high sensor dimensionality (e.g., RSKP achieves 53.93% mAP on RWHAR, 63.29% on WEAR), but fails (mAP ≤ 1%) on short/low-dimensional datasets (SBHAR, Opportunity).
• Failure Modes: Short actions, temporal ambiguity, and low-quality proposals (especially from image-based WSOD) dominate errors.
• Key metric profiles: On SBHAR, RSKP records underfill (UR) of 0.33%, overfill (OR) of 12.74%, deletion (DR) of 1.59%, insertion (IR) of 9.31%, highlighting the fine-grained error insights not captured by mAP alone (Li et al., 2 Feb 2026).

7. Analysis, Limitations, and Future Directions

IMU-TAL delivers segment-level coherence, reducing label fragmentation and improving background discrimination by leveraging boundary regression and multi-scale context modeling. Explicit segment-based IoU losses and feature pyramids enable models to learn activities of arbitrary duration. Null-class accuracy is enhanced by decoupling regression (applied only to positive segments) from background classification (Bock et al., 2023).

Limitations include poor segment coherence for immediate (short-window) classification, as fixed-window classifiers lack future context. In weak supervision, dominant bottlenecks include (i) inability to localize short actions due to MIL pooling’s limited temporal focus, (ii) sensitivity to proposal quality, and (iii) loss of performance on temporally ambiguous or low-dimensional sensor streams (Li et al., 2 Feb 2026).

Future research directions for IMU-TAL and WS-IMU-TAL include the development of content-aware temporal proposal generation (e.g., change-point or edge-detection methods), boundary-aware objective functions, multi-resolution architectures for co-occurring actions, memory-augmented and stronger temporal reasoning models, and self-supervised foundation model pretraining on multi-dataset IMU corpora. These steps are projected as critical to scalable and annotation-efficient IMU-based temporal action localization (Li et al., 2 Feb 2026).