FedTime-MAGNET: Federated Multimodal HAR

• The paper introduces FedTime-MAGNET, a framework that leverages multimodal fusion via a graph neural expert transformer and a lightweight T5 encoder for time-series data.
• It integrates depth cameras, pressure mats, and accelerometers to capture both spatial relationships and long-range temporal dependencies in HAR tasks.
• The system achieves high accuracy with a centralized F1 of 0.934 and a federated F1 of 0.881, balancing privacy with robust performance in real-world applications.

Searching arXiv for the specified paper and closely related methodological references. FedTime-MAGNET is a multimodal federated learning framework for human activity recognition (HAR) introduced in the paper "MHARFedLLM: Multimodal Human Activity Recognition Using Federated LLM" (Bandyopadhyay et al., 3 Aug 2025). It combines heterogeneous sensing modalities—depth cameras, pressure mats, and accelerometers—within a unified architecture centered on the Multimodal Adaptive Graph Neural Expert Transformer (MAGNET), and augments this fusion mechanism with a customized lightweight T5 encoder-only model for time-series representation learning. In the reported experiments, the framework achieves a centralized F1 Score of 0.934 and a federated F1 Score of 0.881 on the MEx dataset, positioning it as a privacy-aware HAR system that integrates multimodal fusion, time-series LLM techniques, and federated optimization (Bandyopadhyay et al., 3 Aug 2025).

1. Conceptual scope and problem setting

FedTime-MAGNET is designed for HAR scenarios in which single-modality pipelines are insufficiently robust in real-world environments. The framework addresses this limitation by combining heterogeneous data sources: depth cameras, pressure mats, and accelerometers. The stated application context includes fitness tracking, smart homes, and healthcare monitoring, where multimodal sensing can improve robustness and accuracy relative to systems that rely on a single sensor family (Bandyopadhyay et al., 3 Aug 2025).

At the architectural level, the framework is organized around three coupled components. The first is MAGNET, which performs multimodal fusion through graph attention and a Mixture of Experts (MoE). The second is a customized lightweight T5 encoder-only model adapted for time-series inputs. The third is a federated learning setup based on FedAvg-style rounds, intended to preserve user privacy while maintaining competitive recognition performance (Bandyopadhyay et al., 3 Aug 2025).

This composition suggests that FedTime-MAGNET is not merely a late-fusion ensemble, but a structured multimodal sequence-learning system in which sensor interactions, temporal dependencies, and distributed optimization are co-designed. A plausible implication is that the framework targets deployments in wearable and ambient computing settings where centralized aggregation of raw data may be undesirable or infeasible.

2. MAGNET: multimodal fusion architecture

The core fusion module is the Multimodal Adaptive Graph Neural Expert Transformer. Its input is a set of modality-specific embeddings,

$\{\,\emb_m\}_{m=1}^M,$

with the modalities instantiated as $\emb_{\text{act}}$ and $\emb_{\text{acw}}$ from two accelerometers, $\emb_{\text{dc}}$ from the depth camera, and $\emb_{\text{pm}}$ from the pressure mat (Bandyopadhyay et al., 3 Aug 2025).

The framework constructs an adjacency structure in three stages. First, it computes a dynamic cosine-based adjacency:

$A_{\text{dyn},ij} = \frac{\cos\bigl(\hat\emb_i,\hat\emb_j\bigr)+1}{2} \quad\text{with }\hat\emb_i=\frac{\emb_i}{\|\emb_i\|}.$

Second, it introduces learnable adjacency logits,

$$A_{\text{learn}} = \sigma\bigl(W_{\text{adj}}\bigr), \quad W_{\text{adj}}\in\R^{M\times M}.$$

Third, it combines these terms with self-loops to obtain the final adjacency:

$$A_{\text{final}} = A_{\text{dyn}}\;\odot\;A_{\text{learn}} \;+\;0.5\,I_{M\times M}\,.$$

This design makes the inter-modality graph simultaneously data-adaptive and learnable (Bandyopadhyay et al., 3 Aug 2025).

On top of this graph, MAGNET applies a multi-head Graph Attention Layer. For each node $i$ and its neighbors $j$, it computes standard GAT coefficients $\emb_{\text{act}}$0, then masks them through the final adjacency:

$\emb_{\text{act}}$1

The reported interpretation is that graph-attention plus MoE enables MAGNET to discover and exploit rich inter-modal relationships while weighting each sensor channel adaptively (Bandyopadhyay et al., 3 Aug 2025).

The MoE sublayer introduces sparse expert routing. A gating network $\emb_{\text{act}}$2 produces

$\emb_{\text{act}}$3

after which the model selects top-$\emb_{\text{act}}$4 experts. Each expert $\emb_{\text{act}}$5 is a small feed-forward block with skip-connections and RMSNorm, and only the top-$\emb_{\text{act}}$6 experts receive nonzero weight. Load balancing is enforced through the entropy penalty

$\emb_{\text{act}}$7

This introduces conditional computation into multimodal fusion rather than treating all sensor interactions as uniformly informative (Bandyopadhyay et al., 3 Aug 2025).

The fusion stage culminates in what is described as Fusion Layer-III. Modality importance weights are computed by averaging hidden states:

$\emb_{\text{act}}$8

A final projection through an FFN plus RMSNorm yields a unified vector for classification. The paper’s reported t-SNE visualization shows well-separated clusters in the learned feature space, which is presented as evidence that the learned multimodal representation is discriminative (Bandyopadhyay et al., 3 Aug 2025).

3. Lightweight T5 encoder-only adaptation for time series

FedTime-MAGNET incorporates a customized lightweight T5 encoder-only architecture to model temporal structure in multivariate sensor streams. For a multivariate series $\emb_{\text{act}}$9, the model extracts a context window $\emb_{\text{acw}}$0 and treats each row $\emb_{\text{acw}}$1 as a token. This tokenization converts time-series patches into a transformer-compatible sequence representation (Bandyopadhyay et al., 3 Aug 2025).

Patch embedding and positional encoding are defined as

$\emb_{\text{acw}}$2

where $\emb_{\text{acw}}$3 is a sinusoidal or learned encoding. Self-attention uses LoRA adapters, with the projection matrices decomposed as

$\emb_{\text{acw}}$4

and

$\emb_{\text{acw}}$5

so that only the low-rank matrices $\emb_{\text{acw}}$6 are trainable (Bandyopadhyay et al., 3 Aug 2025).

The transformer block configuration is specified as model dimension $\emb_{\text{acw}}$7, 8 layers, 8 heads with $\emb_{\text{acw}}$8, FFN dimension 2048, dropout 0.1, and layer-norm $\emb_{\text{acw}}$9, with relative positional biases in 32 buckets. Output pooling concatenates average and max pooled last hidden states and passes the result through an FFN to produce $\emb_{\text{dc}}$0 (Bandyopadhyay et al., 3 Aug 2025).

The paper states that this T5 encoder-only variant, patched at each time step and augmented with LoRA, efficiently captures long-range temporal dependencies in high-frequency sensor streams. This suggests that the model treats temporal abstraction as a first-class component rather than as a purely post-fusion refinement stage.

4. Federated learning formulation and systems considerations

The federated component of FedTime-MAGNET optimizes the global objective

$\emb_{\text{dc}}$1

Here, $\emb_{\text{dc}}$2 clients each hold local data $\emb_{\text{dc}}$3, and client contributions are weighted by dataset size (Bandyopadhyay et al., 3 Aug 2025).

Training proceeds through FedAvg-style rounds. At round $\emb_{\text{dc}}$4, a fraction $\emb_{\text{dc}}$5 of clients is sampled, with $\emb_{\text{dc}}$6, and the server broadcasts the current global parameters $\emb_{\text{dc}}$7. Each selected client performs $\emb_{\text{dc}}$8 local epochs of

$\emb_{\text{dc}}$9

with gradient clipping constrained by $\emb_{\text{pm}}$0. Clients then return $\emb_{\text{pm}}$1, and the server aggregates according to

$\emb_{\text{pm}}$2

The use of $\emb_{\text{pm}}$3 in the local optimization objective indicates that sparsity and load-balancing in expert routing are preserved during federated updates (Bandyopadhyay et al., 3 Aug 2025).

Several communication and efficiency measures are explicitly reported. LoRA adapters greatly reduce parameter-exchange size. Gradient accumulation over 6 steps is used to emulate a larger batch. CPU–GPU memory swaps and cache cleanup are employed to handle limited VRAM. Early stopping is applied on validation loss (Bandyopadhyay et al., 3 Aug 2025).

The paper further states that federated training preserves user privacy, accommodates non-IID data via client-specific learning rates and early stopping, and still achieves a strong F1 of 0.881 versus the centralized 0.934. The mention of client-specific learning rates appears in the key insights, although the detailed algorithmic description emphasizes FedAvg-style aggregation, gradient clipping, and early stopping. This suggests that the implementation is intended to remain practical under heterogeneous local conditions rather than assuming homogeneous client distributions.

5. Dataset, preprocessing, and experimental protocol

The experimental evaluation uses the MEx dataset, described as containing 30 subjects performing 7 physiotherapy exercises of up to 60 s, recorded with two accelerometers located at wrist and thigh at 100 Hz, depth camera frames downsampled to $\emb_{\text{pm}}$4 at 15 Hz, and a pressure mat of size $\emb_{\text{pm}}$5 at 15 Hz (Bandyopadhyay et al., 3 Aug 2025).

The subject split is 21 subjects (70%) for training, 3 subjects (10%) for validation, and 6 subjects (20%) for testing. Preprocessing includes z-score normalization per modality, resampling or interpolation to uniform rates, and a sliding window of 5 s with 1 s stride, yielding 500 accelerometer frames and 75 depth-camera or pressure-mat frames per window. Data augmentation adds Gaussian noise $\emb_{\text{pm}}$6 (Bandyopadhyay et al., 3 Aug 2025).

The reported hyperparameters specify AdamW with learning rate $\emb_{\text{pm}}$7, weight decay $\emb_{\text{pm}}$8, gradient clipping 1.0, and ReduceLROnPlateau with factor $\emb_{\text{pm}}$9, patience 3, and minimum learning rate $A_{\text{dyn},ij} = \frac{\cos\bigl(\hat\emb_i,\hat\emb_j\bigr)+1}{2} \quad\text{with }\hat\emb_i=\frac{\emb_i}{\|\emb_i\|}.$0. The batch configuration is $A_{\text{dyn},ij} = \frac{\cos\bigl(\hat\emb_i,\hat\emb_j\bigr)+1}{2} \quad\text{with }\hat\emb_i=\frac{\emb_i}{\|\emb_i\|}.$1-fold accumulation, corresponding to an effective batch size of 48. The T5 backbone uses 8 layers of width 512 with LoRA rank $A_{\text{dyn},ij} = \frac{\cos\bigl(\hat\emb_i,\hat\emb_j\bigr)+1}{2} \quad\text{with }\hat\emb_i=\frac{\emb_i}{\|\emb_i\|}.$2 and $A_{\text{dyn},ij} = \frac{\cos\bigl(\hat\emb_i,\hat\emb_j\bigr)+1}{2} \quad\text{with }\hat\emb_i=\frac{\emb_i}{\|\emb_i\|}.$3. Comparative and supporting sequence modules are described with DART-CNN convolution channels $A_{\text{dyn},ij} = \frac{\cos\bigl(\hat\emb_i,\hat\emb_j\bigr)+1}{2} \quad\text{with }\hat\emb_i=\frac{\emb_i}{\|\emb_i\|}.$4, CBAM $A_{\text{dyn},ij} = \frac{\cos\bigl(\hat\emb_i,\hat\emb_j\bigr)+1}{2} \quad\text{with }\hat\emb_i=\frac{\emb_i}{\|\emb_i\|}.$5, bidirectional LSTM/RNN/GRU hidden size 256, and dropout 0.1. MAGNET itself is configured with 4 modalities, 3 fusion layers, 4 experts with top-2 routing, 8-head GAT, and load-balancing $A_{\text{dyn},ij} = \frac{\cos\bigl(\hat\emb_i,\hat\emb_j\bigr)+1}{2} \quad\text{with }\hat\emb_i=\frac{\emb_i}{\|\emb_i\|}.$6 (Bandyopadhyay et al., 3 Aug 2025).

The evaluation metrics are accuracy, macro-F1, and Precision-Recall AUC curves. Figures reported in the paper show faster convergence in centralized training, more fluctuation under federated learning, Precision-Recall AUC greater than 0.95 for many classes, and well-separated t-SNE clusters in the learned representation space (Bandyopadhyay et al., 3 Aug 2025).

6. Quantitative performance, ablations, and interpretation

The central quantitative comparison is between MAGNET and several baselines under centralized and federated training.

Model	Centralized (Acc./F1)	Federated (Acc./F1)
MAGNET	0.934 / 0.934	0.880 / 0.881
Concat	0.876 / 0.876	0.808 / 0.801
Attention	0.898 / 0.897	0.837 / 0.836
LSTM	0.828 / 0.822	0.778 / 0.776
DART-CNN	0.876 / 0.874	0.657 / 0.651

These values indicate that MAGNET attains the best reported performance in both training regimes, with centralized MAGNET F1 = 0.934 and federated MAGNET F1 = 0.881 (Bandyopadhyay et al., 3 Aug 2025).

The ablation results further show that the best three-modality subset, $A_{\text{dyn},ij} = \frac{\cos\bigl(\hat\emb_i,\hat\emb_j\bigr)+1}{2} \quad\text{with }\hat\emb_i=\frac{\emb_i}{\|\emb_i\|}.$7, yields centralized F1 = 0.927 and federated F1 = 0.872. This suggests that most of the full model’s performance is retained even when one modality is removed, although the complete four-modality configuration remains superior in the reported results (Bandyopadhyay et al., 3 Aug 2025).

The figures described in the source emphasize several qualitative properties. Faster convergence in centralized training and greater fluctuation under federated learning are consistent with the expected optimization gap between fully pooled and distributed non-IID training. Precision-Recall AUC greater than 0.95 for many classes suggests that the classification boundaries are strong for a substantial portion of the label space. The t-SNE visualization of well-separated clusters is presented as evidence that the fused representation is structured in a way that supports discriminative classification (Bandyopadhyay et al., 3 Aug 2025).

A common misconception in multimodal HAR is that simple concatenation is sufficient once enough sensor channels are available. The reported comparison with a Concat baseline does not support that assumption in this setting, since MAGNET exceeds Concat under both centralized and federated training. Another possible misconception is that federated learning necessarily renders multimodal sequence models impractical. The reported federated F1 of 0.881 suggests otherwise, although the observed training fluctuations indicate a measurable cost relative to centralized optimization (Bandyopadhyay et al., 3 Aug 2025).

7. Position within multimodal HAR and reported implications

FedTime-MAGNET occupies the intersection of multimodal HAR, graph-based sensor fusion, transformer-based time-series modeling, and privacy-aware federated learning. The paper frames its contribution around three pillars: structured multimodal fusion through graph-attention and MoE, LLM-style time-series modeling through a lightweight T5 encoder-only design, and communication-efficient federated learning enabled in part by LoRA-based parameter reduction (Bandyopadhyay et al., 3 Aug 2025).

The reported conclusion is that these components jointly yield a robust, accurate, and practical HAR system suitable for real-world wearable and ambient computing deployments. Because the system combines depth cameras, pressure mats, and accelerometers, it explicitly spans both wearable and ambient sensing modalities rather than remaining confined to a single instrumentation paradigm (Bandyopadhyay et al., 3 Aug 2025).

From an interpretive standpoint, the framework suggests a broader trend in HAR toward architectures that model cross-modal relational structure explicitly rather than relying on fixed fusion heuristics. It also suggests that parameter-efficient adaptation mechanisms such as LoRA can serve a dual role in time-series modeling and federated communication efficiency. A plausible implication is that future multimodal HAR systems may increasingly combine sparse expert routing, graph-structured fusion, and lightweight transformer adaptation to balance accuracy, privacy, and deployment constraints.

In the source paper’s own synthesis, graph-attention plus MoE discovers and exploits rich inter-modal relationships, the T5 encoder-only component efficiently captures long-range temporal dependencies in high-frequency sensor streams, and federated training preserves user privacy while maintaining strong performance. Within the confines of the reported experiments, FedTime-MAGNET is therefore defined less by any single module than by the coordinated interaction of these three design commitments (Bandyopadhyay et al., 3 Aug 2025).