AMNN: Adaptive Multimodal Fusion

• Attention-based Multimodal Neural Network (AMNN) is a deep architecture that integrates heterogeneous modalities like text, audio, visual, and sentiment features using adaptive and cross-modal attention.
• The Adaptive Multimodal Recurrent Fusion (AMRF) mechanism employs cyclic-shift operations and learnable weights to fuse modality pair features beyond simple concatenation.
• This approach, exemplified by CANAMRF, achieves superior performance in tasks such as depression detection by enabling fine-grained, dynamic integration across multiple input streams.

An Attention-based Multimodal Neural Network (AMNN) is a deep network architecture that integrates heterogeneous modalities—text, audio, visual, and hand-engineered features—using adaptive fusion and cross-modal attention mechanisms. The CANAMRF framework (Wei et al., 2024) exemplifies the operational principles of AMNN in the context of multimodal depression detection, combining novel architectural modules that move beyond naive fusion toward modality-sensitive, fine-grained representation learning.

1. Architectural Components and Modality-specific Feature Extraction

CANAMRF consists of three principal modules in a feed-forward pipeline: multimodal feature extractors, the Adaptive Multimodal Recurrent Fusion (AMRF) block, and a Hybrid Attention module. Four input modalities are considered:

• Textual features are encoded via a pretrained BERT model and further processed by temporal 1D convolution, yielding a fixed-dimensional vector per time step.
• Acoustic features are derived as low-level audio descriptors (e.g., prosody, MFCCs) using OpenSMILE, followed by 1D convolution.
• Visual features are extracted as facial action units and landmark vectors from OpenFace, convolved temporally.
• Sentiment-structural features are aggregated from eight hand-engineered statistics (five word-level, three sentence-level), convolved to ensure temporal coherence.

All feature streams are independently reshaped into a common embedding space of dimension $d$, facilitating subsequent pairwise fusion.

2. Adaptive Multimodal Recurrent Fusion (AMRF) Mechanism

AMRF adaptively fuses each non-text modality $M$ (visual, acoustic, sentiment) with the text stream. For each pair $(x, y)$, $x \in \mathbb{R}^m, y \in \mathbb{R}^n$:

• Project $x$ and $y$ to $\mathbb{R}^d$ using learned weights $W_1, W_2$:

$X = x W_1^T,\, Y = y W_2^T$

• Cyclic-shift matrices: Construct $A, B \in \mathbb{R}^{d \times d}$, where each row is a circular shift of $X$ or $Y$.
• Elementwise fusion:

$$X' = \frac{1}{d} \sum_{i=1}^{d} a_i \odot A, \quad Y' = \frac{1}{d} \sum_{i=1}^{d} b_i \odot B$$

($a_i, b_i$: $i$-th row; $\odot$: Hadamard product.)

• Weighted merge:

$$Z = (\alpha X' + \beta Y') W_3^T, \quad \alpha, \beta \in [0, 1]$$

where $\alpha, \beta$ are learned fusion weights. Output is the fused embedding $Z \in \mathbb{R}^k$.

AMRF's recurrent cyclic-shift operation and learnable weights permit nuanced, sample-dependent fusion dynamics, eschewing static concatenation or summation.

3. Cross-modal Hybrid Attention Formulation

The Hybrid Attention module orchestrates joint reasoning over fused pairwise embeddings. For pairwise AMRF outputs $X_{VT}, X_{AT}, X_{ST}$, and defining

• $Q = X_{ST}$
• $K_{VT} = V_{VT} = X_{VT}$
• $K_{AT} = V_{AT} = X_{AT}$

Two parallel scaled dot-product attentions are computed: $$Z_{VTS} = \text{softmax}\left( \frac{Q K_{VT}^T}{\sqrt{d_k}} \right) V_{VT}, \quad Z_{ATS} = \text{softmax}\left( \frac{Q K_{AT}^T}{\sqrt{d_k}} \right) V_{AT}$$ with $d_k = d$, producing two cross-modal fusion streams.

4. Final Fusion and Self-Attention Alignment

CANAMRF further fuses $Z_{VTS}, Z_{ATS}$ using a second AMRF, aligning the multimodal streams. The output is passed through standard self-attention: $$Z_f = \mathrm{SelfAttention}\left(\mathrm{AMRF}(Z_{ATS}, Z_{VTS})\right)$$ where $Z_f \in \mathbb{R}^{d \times T'}$ for sequence length $T'$. This process ensures global consistency and multi-stream alignment in the final representation.

The flattened $Z_f$ is fed into one or more fully-connected layers culminating in a sigmoid activation for binary or multi-class prediction.

5. Training Objective and Optimization Protocol

The network's parameters—including all extractor heads, AMRF weights, fusion matrices, attention projections, and classifier heads—are jointly optimized using a focal loss: $$\mathcal{L}_{fl} = - (1 - \tilde y)^\gamma \log(\tilde y)$$ with $\tilde y = \hat y$, $\gamma \geq 0$, emphasizing hard (uncertain) samples. Backpropagation minimizes $\mathcal{L}_{fl}$ across all layers.

6. Implementation Results and Comparative Performance

CANAMRF demonstrates state-of-the-art accuracy on benchmark depression-detection datasets. The modular AMRF and hybrid attention pipeline outperform prior approaches that rely on fixed-weight or naive concatenation fusion, yielding superior discriminative capacity of the multimodal representation. The recurrent fusion mechanism adapts to varying signal strengths (e.g., when acoustic or facial cues dominate), a property not present in earlier fusion schemes.

7. Methodological Significance and AMNN Positioning

CANAMRF illustrates the paradigm shift from static multimodal fusion to dynamic, attention-weighted joint encoding:

• Cross-modal relevance: Attention blocks precisely measure and apply relevance between modalities rather than enforce uniform fusion.
• Adaptive integration: AMRF's recurrent and learnable weighted mechanism grants fine-grained control over modality contribution per sample and per pair.
• Hierarchical composition: By stacking AMRF and hybrid attention, the architecture achieves increasingly discriminative representations, improving downstream task performance.

AMNN as instantiated in CANAMRF is applicable to various domains requiring robust multimodal understanding, such as affect recognition, medical diagnostics, and complex event detection—whenever task performance is constrained by naive multimodal integration.

---

Summary Table: Core Computational Steps in CANAMRF

Stage	Key Operation	Output Shape
Feature Extraction	BERT / OpenSMILE / OpenFace / Sentiment stat + 1D conv	$\mathbb{R}^d$ per stream
Adaptive Fusion (AMRF)	Recurrent cyclic-shift + elementwise Hadamard + learned merge	$\mathbb{R}^k$
Cross-modal Attention	Scaled dot-product, $QK^T/\sqrt{d_k}$, softmax	$\mathbb{R}^d \times 1$
Hybrid Fusion + Self-Attention	AMRF on cross-modal outputs, then standard self-attention	$\mathbb{R}^{d \times T'}$
Classification	FC layers + sigmoid	$\mathbb{R}^{C}$

---

Attention-based Multimodal Neural Networks, and specifically CANAMRF (Wei et al., 2024), represent the current frontier in adaptive, modality-sensitive deep learning for psychological state and related multimodal tasks. The defining characteristics are fine-grained, per-pair dynamic fusion and explicit cross-modal alignment, superseding legacy architectures that inadequately model cross-modal nuance.