Expansion-Squeeze-Excitation Fusion Network

• The paper introduces a novel multi-modal fusion approach for elderly activity recognition using an expansion-squeeze-excitation mechanism to capture both local and global dependencies.
• It employs dedicated modal-wise and channel-wise attention modules to dynamically reweight feature channels and modalities for enhanced feature integration.
• A novel multi-modal loss function ensures that the fused branch outperforms the best single-modal predictor, achieving state-of-the-art accuracy.

The Expansion-Squeeze-Excitation Fusion Network (ESE-FN) is a two-stage multi-modal fusion architecture designed for elderly activity recognition that combines RGB video and skeleton sequence features. ESE-FN extends Squeeze-and-Excitation Networks (SENet) by introducing an “Expansion” operation prior to conventional squeeze-excitation, allowing both local and global dependency modeling across modalities and feature channels. The architecture employs dedicated attention mechanisms at the modal and channel levels and incorporates a novel loss function to ensure the fused multi-modal representation maintains or improves upon the performance of single-modal branches (Shu et al., 2021).

1. Network Architecture

ESE-FN processes video data using a two-stream feature extraction paradigm, with one branch for RGB frames and another for skeleton-based features. The processing pipeline is as follows:

• From each input video, $T$ clips (e.g., $T=64$) are sampled, each with one RGB frame and a corresponding skeleton frame.
• The RGB branch utilizes ResNeXt–101 (or ResNeXt–18), pre-trained on ImageNet and fine-tuned on ETRI-Activity3D, producing feature vectors $f_r \in \mathbb{R}^{d_1 \times 1}$.
• The skeleton branch employs Shift-GCN, pre-trained on NTU and fine-tuned, generating $f_s \in \mathbb{R}^{d_2 \times 1}$.
• Two MLPs, $H_r$ and $H_s$, project $f_r$ and $f_s$ to a shared dimension $d$, forming $$f = \operatorname{Concat}(H_r(f_r), H_s(f_s)) \in \mathbb{R}^{d \times 2}$$.
• The Modal-fusion Module (M-Net) applies modal-wise Expansion-Squeeze-Excitation (M-ESE) attention to $T=64$0 (transposed to $T=64$1), producing $T=64$2.
• The Channel-fusion Module (C-Net) further transposes and processes $T=64$3 using channel-wise Expansion-Squeeze-Excitation (C-ESE), outputting $T=64$4.
• Final global fusion is achieved by summing across the modal dimension: $T=64$5.
• Three classifiers ($T=64$6, $T=64$7, $T=64$8) perform prediction on the RGB, skeleton, and fused features, respectively.
• The architectural flow ensures modality-dependent and channel-wise attentive reweighting of features prior to classification.

2. Expansion-Squeeze-Excitation Attention Mechanisms

ESE-FN generalizes the attention paradigm of SENet by interspersing an expansion step with the conventional squeeze-excitation sequence. This is operationalized in two hierarchically organized modules:

2.1 Modal-wise ESE (M-Net)

• Input: $T=64$9, where $f_r \in \mathbb{R}^{d_1 \times 1}$0 modalities, $f_r \in \mathbb{R}^{d_1 \times 1}$1 channels.
• Expansion: Sequential 1D convolutions with varying kernel sizes (e.g., 3, 5, 7) yield $f_r \in \mathbb{R}^{d_1 \times 1}$2 with $f_r \in \mathbb{R}^{d_1 \times 1}$3.
• Squeeze: Global pooling across the expanded channel dimension:

$f_r \in \mathbb{R}^{d_1 \times 1}$4

• Excitation: Two fully connected layers (with ReLU, sigmoid) transform $f_r \in \mathbb{R}^{d_1 \times 1}$5 into attention weights:

$f_r \in \mathbb{R}^{d_1 \times 1}$6

• Re-weight: Input $f_r \in \mathbb{R}^{d_1 \times 1}$7 is reweighted by $f_r \in \mathbb{R}^{d_1 \times 1}$8, broadcasting along the channel dimension.

2.2 Channel-wise ESE (C-Net)

• Input: $f_r \in \mathbb{R}^{d_1 \times 1}$9
• Expansion: A $f_s \in \mathbb{R}^{d_2 \times 1}$0 convolution produces $f_s \in \mathbb{R}^{d_2 \times 1}$1, $f_s \in \mathbb{R}^{d_2 \times 1}$2.
• Squeeze: Global pooling over the modal dimension:

$f_s \in \mathbb{R}^{d_2 \times 1}$3

• Excitation: Two-layer MLP computes channel-wise attention weights:

$f_s \in \mathbb{R}^{d_2 \times 1}$4

• Re-weight: $f_s \in \mathbb{R}^{d_2 \times 1}$5 is reweighted per channel by $f_s \in \mathbb{R}^{d_2 \times 1}$6.

2.3 Fusion Effect

M-ESE models interactions between modalities, enabling suppression or enhancement of RGB or skeleton cues at a global level. C-ESE further emphasizes or suppresses individual feature channels within the fused representation. The sequence yields coarse-to-fine attentive fusion, first aligning modalities and then refining at the channel level.

3. Multi-modal Loss Function

The Multi-modal Loss (ML) is designed to enforce the consistency and superiority of the fused branch over single-modal predictors. It is defined as:

• $f_s \in \mathbb{R}^{d_2 \times 1}$7
• $f_s \in \mathbb{R}^{d_2 \times 1}$8
• $f_s \in \mathbb{R}^{d_2 \times 1}$9
• The combined loss is:

$H_r$0

with $H_r$1 (empirically, $H_r$2, $H_r$3 on ETRI-Activity3D).

• The penalty term encourages the fused output not to underperform the best unimodal branch. This approach ensures fusion effectiveness and guides the network toward optimal multimodal integration (Shu et al., 2021).

4. Training Protocol and Implementation

4.1 Input and Preprocessing

• Videos are segmented into $H_r$4 clips. Each clip yields one RGB frame and one 2D skeleton frame, resized to approximately $H_r$5.

4.2 Feature Extraction

Branch	Backbone	Initialization	Training Details
RGB	ResNeXt-101	ImageNet pre-trained	SGD, momentum=0.9, weight decay=$H_r$6, lr=0.1, batch size=32, epochs=120 (ResNeXt-18: bs=128)
Skeleton	Shift-GCN	NTU pre-trained	SGD, momentum=0.9, weight decay=$H_r$7, lr=0.1, batch size=32, epochs=140

• Extracted features $H_r$8, $H_r$9 are projected into a shared embedding of $H_s$0.

4.3 Fusion Training

• SGD optimizer: momentum=0.9, weight decay=$H_s$1.
• Initial learning rate: 0.1; batch size: 32; total epochs: 30, with step decay by 0.1 at epochs 15 and 25.
• Activation: ReLU in hidden layers, sigmoid in excitation operations.
• Regularization: weight decay, multi-modal consistency loss (ML).

5. Experimental Performance and Ablation

Experimental evaluation on ETRI-Activity3D (110,000+ videos, 50+ activity categories) demonstrates the efficacy of ESE-FN.

5.1 Baseline Accuracies

Model	Modality	Accuracy (%)
ResNeXt-101	RGB	93.5
Shift-GCN	Skeleton	88.6
Concat + Softmax	RGB+Skeleton	94.0

5.2 Component Ablation

Configuration	Accuracy (%)
w/o C-Net (modal-wise ESE + ML)	95.3
w/o M-Net (channel-wise ESE + ML)	94.5
w/o ML (full ESE, single loss $H_s$2)	95.7
Full ESE-FN (M-Net + C-Net + ML)	95.9

5.3 Comparison with State-of-the-Art

Method	Modalities	Accuracy (%)
Deep Bilinear Learning	RGB+Depth+Skeleton	88.4
Evolution Pose Map	RGB+Skeleton	93.6
c-ConvNet	RGB+Depth	91.3
FSA-CNN	RGB+Skeleton	93.7
ESE-FN	RGB+Skeleton	95.9

Precision and recall were not separately reported; overall accuracy served as the primary metric (Shu et al., 2021).

6. Architectural Significance and Extensions

ESE-FN introduces an expansion stage prior to squeeze-excitation, enhancing the capacity to integrate dependencies locally (via convolutional expansion) and globally (via pooling). This two-stage attentive fusion—modal then channel-wise—yields more effective aggregation of heterogeneous cues (e.g., skeleton and RGB) than prior linear or single-stage fusion paradigms. The multi-modal loss provides an explicit mechanism for ensuring that fusion never underperforms the best available unimodal pathway.

A plausible implication is that ESE-FN’s core mechanisms can be generalized to other multi-modal recognition tasks beyond elderly activity recognition, such as audio-visual or RGB-depth-inertial fusion. Future extensions may involve replacing simple convolutions with spatial transformers to further enrich the expansion operation or adapting the architecture to handle more than two modalities by increasing the modal dimension in ESE modules (Shu et al., 2021).