AMF-Net: Asymmetric Audio-Visual Micro-Expression Fusion

• The paper's main contribution is the design of a multimodal fusion network that combines a holistic visual summary with a temporally evolving audio sequence using an asymmetric cross-attention framework.
• It utilizes modality-specific branches where the visual stream produces a single global descriptor while the audio stream maintains temporal dynamics, ensuring modality-respectful fusion.
• Empirical results on the MMED dataset demonstrate improved accuracy over visual-only models, though addressing class imbalance remains a challenge.

Searching arXiv for AMF-Net and closely related asymmetric multimodal fusion papers. The Asymmetric Multimodal Fusion Network (AMF-Net) is a multimodal micro-expression recognition architecture introduced together with the MMED dataset to fuse a global visual summary with a dynamic audio sequence through an asymmetric cross-attention framework. In this formulation, asymmetry is not merely a naming convention but an architectural commitment: the visual modality is represented as a single holistic descriptor extracted from an entire micro-expression video, whereas the audio modality is preserved as a temporally evolving sequence, and the fusion module is explicitly designed around this structural mismatch rather than forcing both modalities into a symmetric representation pattern (Wang et al., 18 Sep 2025).

1. Conceptual basis and problem setting

AMF-Net was proposed for audio-visual micro-expression recognition in a setting where visual-only analysis is regarded as insufficient for fully resolving subtle and ambiguous expressions. The associated MMED work states that micro-expressions are often ambiguous or subtle in the visual channel alone, whereas co-occurring audio such as prosody, tone, and breath provides valuable, emotionally discriminative cues (Wang et al., 18 Sep 2025). The network therefore operationalizes a multimodal fusion principle described as “modality-respectful” fusion: it combines a global summary of the visual modality with a dynamic, temporally evolving sequence from the audio modality, rather than imposing identical structural assumptions on both streams (Wang et al., 18 Sep 2025).

This design places AMF-Net within a broader line of asymmetric multimodal fusion research that treats modalities as inherently different in their statistics, abstraction rates, or reliability. In infrared-visible fusion, MMA-UNet argues that symmetric fusion can cause loss of shallow information or bias toward a single modality and therefore adopts separate specialized encoders and cross-scale asymmetric fusion (Huang et al., 2024). In multimodal feature learning for pixel-aligned vision tasks, asymmetric multi-layer fusion was introduced to avoid redundant representations under bidirectional fusion (Wang et al., 2021). In medical multimodal fusion, HyPCA-Net identifies information loss in cascaded attention and explicitly contrasts its hybrid design with prior asymmetric or cascaded models including AMF-Net (Dhar et al., 18 Feb 2026). These comparisons do not redefine AMF-Net’s original scope, but they clarify that its asymmetry belongs to a wider methodological family concerned with non-identical modality structure, non-uniform information value, and non-symmetric fusion operators.

2. Architectural composition

AMF-Net comprises two modality-specific branches and a fusion module (Wang et al., 18 Sep 2025). The visual branch uses MMNet as its backbone and processes an entire micro-expression video sequence into a single global feature vector representing holistic facial movement. Its output is denoted $\mathbf{v}_{seq} \in \mathbb{R}^{1 \times D_v}$ (Wang et al., 18 Sep 2025).

The audio branch uses a 1D convolutional neural network and preserves the temporal structure of the audio, producing a sequence of feature vectors rather than a single pooled descriptor. Its output is denoted $\mathbf{A} \in \mathbb{R}^{T_a \times D_a}$, where $T_a$ is the number of audio frames (Wang et al., 18 Sep 2025).

The architectural asymmetry is thus explicit at the representation level. The visual stream compresses an entire video into one global token-like vector, while the audio stream remains sequential. A plausible implication is that the model treats facial dynamics as a holistic summary signal and voice as a temporally fine-grained signal whose local variations may disambiguate the expression over time. The paper’s formulation supports this interpretation by stating that the network is specifically designed to fuse a global visual summary with a dynamic audio sequence through an asymmetric cross-attention framework (Wang et al., 18 Sep 2025).

The fusion stage does not rely on simple concatenation or late fusion. Instead, it introduces a bi-directional cross-attention mechanism tailored to the mismatch between a static summary and a dynamic sequence (Wang et al., 18 Sep 2025). This point is central to the identity of AMF-Net: asymmetry is implemented in both the branch outputs and the interaction operator.

3. Asymmetric cross-attention mechanism

Before cross-attention is applied, both modalities are projected into a shared latent space of dimension $D_c$ through separate linear layers: $\mathbf{v}'_{seq} = W_v \cdot \mathbf{v}_{seq}$ and $\mathbf{A}' = W_a \cdot \mathbf{A}$, with $W_v \in \mathbb{R}^{D_v \times D_c}$ and $W_a \in \mathbb{R}^{D_a \times D_c}$ (Wang et al., 18 Sep 2025). This projection creates a common interaction space while preserving the distinction between a single-vector visual summary and a multi-step audio sequence.

The fusion module contains two parallel cross-attention paths (Wang et al., 18 Sep 2025). In the first, the visual vector acts as the Query and the audio sequence acts as Key and Value. This operation refines the visual summary using audio context: $$\tilde{\mathbf{v}}_{seq} = \operatorname{MultiHead}\bigl(\mathbf{v}'_{seq},\, \mathbf{A}',\, \mathbf{A}'\bigr) \in \mathbb{R}^{1 \times D_c}$$ (Wang et al., 18 Sep 2025).

In the second, each temporal element of the audio sequence acts as Query, while the visual vector provides Key and Value. This contextualizes each audio step with respect to the global facial representation: $$\tilde{\mathbf{A}} = \operatorname{MultiHead}\bigl(\mathbf{A}',\, \mathbf{v}'_{seq},\, \mathbf{v}'_{seq}\bigr) \in \mathbb{R}^{T_a \times D_c}$$ (Wang et al., 18 Sep 2025).

Each cross-attention path uses standard multi-head scaled dot-product attention: $\mathbf{A} \in \mathbb{R}^{T_a \times D_a}$0 and

$\mathbf{A} \in \mathbb{R}^{T_a \times D_a}$1

(Wang et al., 18 Sep 2025).

This design makes the asymmetry operational in two directions at once. The visual summary interrogates the audio sequence for contextually relevant temporal evidence, and the audio sequence is simultaneously grounded by the global facial representation (Wang et al., 18 Sep 2025). This suggests that the model is not enforcing symmetry through reciprocal identical streams; rather, it allows the same pair of modalities to interact under two different relational roles.

4. Output formation and training protocol

After cross-attention, the fused representations are flattened through pooling and passed to a final multilayer classifier to predict the micro-expression class (Wang et al., 18 Sep 2025). The paper summary does not specify further architectural details of the classifier, and no additional loss formulation beyond the evaluation protocol is provided in the supplied material.

AMF-Net is evaluated using Leave-One-Subject-Out Cross-Validation (LOSO-CV), selected for person-independent assessment and to avoid overfitting to individual traits (Wang et al., 18 Sep 2025). In each round, one subject is held out for testing while the model is trained on the remaining subjects, and this is repeated for all subjects (Wang et al., 18 Sep 2025). The reported metrics are Accuracy and Unweighted F1-score (UF1), with UF1 emphasized as important for class-imbalanced settings (Wang et al., 18 Sep 2025).

The dataset context is integral to the network’s interpretation. MMED is described as the first dataset capturing spontaneous vocal cues that co-occur with micro-expressions in ecologically valid, high-stakes interactions, and the experimental setup is presented as evidence that audio offers critical, disambiguating information for micro-expression analysis (Wang et al., 18 Sep 2025). AMF-Net should therefore be understood not only as a fusion architecture but also as a baseline tied to a new problem formulation in which vocal and facial micro-signals are treated as synchronized evidence sources.

5. Empirical findings on MMED

The reported LOSO-CV results compare visual-only, audio-only, and fused audio-visual settings (Wang et al., 18 Sep 2025).

Modality Type	Accuracy (%)	UF1
Visual Only	78.54	0.7057
Audio Only	75.16	0.6914
Visual + Audio (AMF-Net)	81.90	0.7060

These results establish several points directly stated in the paper summary (Wang et al., 18 Sep 2025). First, audio alone achieves high performance, which is taken as evidence that voice contains discriminative prosodic or emotional cues concurrent with micro-expressions. Second, audio-visual fusion via AMF-Net improves over the best unimodal setting by up to $\mathbf{A} \in \mathbb{R}^{T_a \times D_a}$2 Accuracy and $\mathbf{A} \in \mathbb{R}^{T_a \times D_a}$3 UF1 over visual-only performance. Third, the UF1 gain is modest relative to the accuracy gain, and the summary explicitly notes that this implies most gains are in majority classes, indicating a remaining class-imbalance issue (Wang et al., 18 Sep 2025).

The paper describes these experiments as providing conclusive evidence that audio offers critical, disambiguating information for micro-expression analysis (Wang et al., 18 Sep 2025). A cautious interpretation is that the architectural asymmetry is validated not only by absolute performance but also by the fact that the network is designed around a modality pairing in which one stream is holistic and the other temporal, and that this pairing yields measurable benefit over either modality in isolation.

6. Relation to other asymmetric fusion paradigms

AMF-Net is one instance of asymmetric multimodal fusion, but the broader literature represented in the supplied sources shows that asymmetry can arise from different causes and be implemented through different mechanisms.

In MMA-UNet, asymmetry is motivated by the finding that visible and infrared features reach high-level semantic abstraction at different rates and network depths. The model therefore separately trains specialized encoders and fuses visible features from the first four layers with infrared features from the last four layers, using a cross-scale asymmetric fusion strategy guided by CKA analysis (Huang et al., 2024). Here, asymmetry concerns depth alignment across modalities.

In “Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion,” asymmetry addresses the inadequacy of symmetric fusion operations for bidirectional multi-layer fusion. The framework introduces channel shuffle and pixel shift as parameter-free asymmetric fusion operations and combines them with shared convolutional weights plus modality-specific batch normalization (Wang et al., 2021). Here, asymmetry concerns the directional non-equivalence of the fusion operator itself.

In CAL, framed in the supplied material as an Asymmetric Multimodal Fusion Network paradigm, asymmetry is contribution-guided: strong modalities are enhanced while weak modalities are compressed or accelerated according to a modality contribution metric $\mathbf{A} \in \mathbb{R}^{T_a \times D_a}$4, with explicit mechanisms for asymmetric gradient acceleration and contribution-aware Asymmetric Information Bottleneck compression (Xu et al., 30 Oct 2025). Here, asymmetry concerns modality imbalance and noise robustness.

In SAC$\mathbf{A} \in \mathbb{R}^{T_a \times D_a}$5-Net for multimodal micro-expression recognition, asymmetry is reliability-aware and spatially local. Motion magnification and optical flow are first aligned through Semantic Anchoring Soft Alignment, then fused through Complementary-Consensus Fusion, which repairs unreliable local evidence and enforces a shared spatial focus (Zheng et al., 24 Jun 2026). Here, asymmetry concerns spatially varying reliability and asymmetric failure patterns.

Compared with these variants, AMF-Net’s distinctive form of asymmetry is the fusion of a single global visual summary with a temporally structured audio sequence through bi-directional cross-attention (Wang et al., 18 Sep 2025). This is neither cross-scale asymmetry nor contribution-weighted compression nor local reliability repair. Its defining problem is structural heterogeneity between one summary vector and one evolving sequence.

7. Significance, interpretation, and limitations

AMF-Net is described as a novel method that effectively fuses a global visual summary with a dynamic audio sequence via an asymmetric cross-attention framework, and together with MMED it provides a validated analytical approach for micro-expression recognition (Wang et al., 18 Sep 2025). The paper further presents it as the first strong audio-visual MER baseline with clear empirical validation and as a benchmark resource for the community (Wang et al., 18 Sep 2025).

Its significance lies in the combination of two claims supported by the supplied material. First, micro-expression recognition had been constrained by a reliance on silent, visual-only data, whereas MMED introduces spontaneous vocal cues in ecologically valid, high-stakes interactions (Wang et al., 18 Sep 2025). Second, AMF-Net shows that audio is not merely auxiliary background information but a source of critical, non-redundant evidence for recognition in this setting (Wang et al., 18 Sep 2025). This suggests that the historical framing of micro-expression recognition as predominantly visual may be incomplete when the target domain includes naturalistic social interactions.

At the same time, the reported UF1 gain is minimal, and the summary explicitly notes that future work is needed on class-imbalance mitigation (Wang et al., 18 Sep 2025). A plausible implication is that while AMF-Net validates the usefulness of audio, it does not fully resolve the skewed class-distribution problem in MER. The architecture also relies on a specific pairing of modalities and representational forms—global visual summary and dynamic audio sequence—so its conclusions are most direct for settings with synchronized face-voice micro-expression evidence rather than arbitrary multimodal combinations.

Within the asymmetric fusion literature, AMF-Net occupies an early audio-visual MER position later complemented by methods that address finer-grained issues such as semantic alignment and local reliability repair in multimodal micro-expression analysis (Zheng et al., 24 Jun 2026). Even so, its central contribution remains clear: it established an explicit asymmetric cross-attention formulation for fusing holistic facial evidence with temporal audio evidence, and empirically demonstrated that this formulation improves person-independent micro-expression recognition on MMED (Wang et al., 18 Sep 2025).