BiLSTM-based Multimodal Fusion

• BiLSTM-based multimodal fusion is a framework that fuses heterogeneous data (images, text, audio) using bidirectional LSTM to capture spatial and sequential dependencies.
• It employs modality-specific encoders, such as CNNs for images and BERT for text, to extract and align features for tasks like malware detection and sentiment analysis.
• Empirical results demonstrate that integrating attention and gating mechanisms with BiLSTM significantly enhances accuracy and mitigates challenges of modality misalignment.

A BiLSTM-based multimodal fusion system is a deep learning architecture in which Bidirectional Long Short-Term Memory (BiLSTM) networks are employed within a broader pipeline to integrate and jointly reason over heterogeneous modalities (e.g., images, audio, text, time series, or code) for classification or regression. Fusion can be performed at feature, decision, or hybrid levels, with the BiLSTM capturing temporal, sequential, or context dependencies bidirectionally across modalities. Such architectures are prevalent in high-performance applications involving malware detection, affective computing, medical prognostics, and sentiment analysis.

1. Core Methodology of BiLSTM-Based Multimodal Fusion

The central principle in BiLSTM-based multimodal fusion is the parallel extraction of complementary features via modality-specific encoders—such as CNNs for images, Transformers or BERT for text, or domain-tailored pre-processing for biosignals—yielding structured embeddings per modality. The BiLSTM framework then enables sequence modeling over either the fused joint feature vector (early fusion) or over higher-level discriminant outputs (decision-level fusion), with parameter sharing or dedicated branches contingent on the desired abstraction and alignment.

A canonical BiLSTM-based feature-fusion model for malware detection exemplifies the architecture (Zhang et al., 2024):

• Feature extraction: Binary executables are converted to grayscale images using a MinHash algorithm, with GIST and LBP applied for global and local texture features, while IDA Pro is used for opcode extraction followed by N-gram and TF-IDF vectorization.
• CNN branch: Processes image/textural features (GIST+LBP or the grayscale image) via stacked convolutions, activations, and pooling, yielding a flattened feature vector $v_I$.
• BiLSTM branch: Models the opcode sequence’s temporal structure, inputting the N-gram+TF-IDF vector sequence into a BiLSTM, producing bidirectional hidden states aggregated into a vector $v_O$.
• Fusion: $v_I$ and $v_O$ are concatenated and optionally refined by a fully connected layer. Final classification is performed by a softmax output layer.

This approach captures both spatial and sequential/semantic relationships for robust detection, demonstrating the structural utility of BiLSTM as a context integrator in complex multimodal systems.

2. Canonical Architectures and Variants

Multiple architectural patterns leverage BiLSTM for multimodal fusion, with variations in modality encoders, fusion strategies, and depth.

• Parallel encoding and concatenation: Textual (BERT/BiLSTM), visual (ResNet/Transformer), and acoustic (VGGish/TCN) features are extracted separately, then sequences are aligned and concatenated for downstream BiLSTM processing (Wang et al., 2023, Lee et al., 2 Jul 2025, Khatooni et al., 27 Nov 2025).
• Decision-level hybrid fusion: Modality-specific discriminant modules (e.g., transformer + BiLSTM per modality) are trained, producing unimodal scores that are combined with (possibly transformed) raw features. A shared BiLSTM then fuses these for final prediction, allowing both intra- and cross-modal dependencies to be modeled (Xu et al., 2022).
• Recursive or temporal gating mechanisms: Fused cross-modal features from recursive attention steps are processed by a BiLSTM, followed by softmax gating to yield temporally adaptive fusion weights, as in emotion recognition frameworks (Lee et al., 2 Jul 2025).
• Longitudinal medical prediction: Multimodal sequences consist of spatial features (e.g., CNN+ViT representations from MRI) and scalar biomarkers per timepoint. A BiLSTM models the sequence of multimodal visit vectors, yielding robust prediction of disease progression (Khatooni et al., 27 Nov 2025).

These structures are not mutually exclusive. They share central elements: feature-level BiLSTM modeling for temporal or contextual alignment, decision-level BiLSTM for integrating predictive signals, and auxiliary modules (attention, temporal gating) for enhanced information flow.

3. Mathematical Formulation and Fusion Strategies

The mathematical foundation of BiLSTM-based multimodal fusion encompasses:

• Bidirectional LSTM cell: For time step $t$, with input $x_t$ and previous states $h_{t-1}$, $c_{t-1}$,

$$\begin{align*} i_t &= \sigma(W_{xi} x_t + W_{hi} h_{t-1} + W_{ci} c_{t-1} + b_i) \ f_t &= \sigma(W_{xf} x_t + W_{hf} h_{t-1} + W_{cf} c_{t-1} + b_f) \ c_t &= f_t \odot c_{t-1} + i_t \odot \tanh(W_{xc} x_t + W_{hc} h_{t-1} + b_c) \ o_t &= \sigma(W_{xo} x_t + W_{ho} h_{t-1} + W_{co} c_{t-1} + b_o) \ h_t &= o_t \odot \tanh(c_t) \end{align*}$$

and correspondingly for the backward direction.

• Fusion by concatenation: For image branch vector $v_I$ and sequence branch vector $v_O$0,

$v_O$1

with subsequent refinement:

$v_O$2

and prediction:

$v_O$3

• Temporal gating (TAGF (Lee et al., 2 Jul 2025)): For multistep cross-modal features $v_O$4,

$v_O$5

with final fused features as weighted sums:

$v_O$6

• Hybrid fusion discriminator (Xu et al., 2022):

$v_O$7

followed by BiLSTM processing and sigmoid prediction.

These formulations yield explicit parallel or sequential integration of diverse modality features, preserving both static and dynamic relationships.

4. Application Domains and Performance Evaluation

BiLSTM-based multimodal fusion systems have demonstrated state-of-the-art performance across several demanding domains:

Application	Modalities	Best BiLSTM-Fusion Metric	Reference
Malware detection	Images (GIST/LBP), opcodes (N-gram/TF-IDF)	98.7% accuracy (fusion), F1 0.943	(Zhang et al., 2024)
Sentiment analysis	Text (BERT), image (ResNet), contrastive fusion (CBAM)	ACC 77.11% / F1 76.55% (MVSA)	(Wang et al., 2023)
Emotion recognition	Audio (VGGish), visual (ResNet), recursive attention	CCC 0.552 (valence/arousal AVG)	(Lee et al., 2 Jul 2025)
Alzheimer's prediction	MRI (EffNetViTLoRA), scalar biomarkers	95.05% accuracy (sMCI vs. pMCI)	(Khatooni et al., 27 Nov 2025)
Humor detection	Audio, video, text (deep/discriminant)	AUC 0.8972 (3-way hybrid ensemble)	(Xu et al., 2022)

Empirical results consistently show performance improvements of up to several percentage points in accuracy, F1 score, or CCC when fusing multimodal representations via BiLSTM as compared to either unimodal baselines or shallow fusion techniques. Notably, the inclusion of both global and local context—spatial, sequential, or statistical—is crucial for challenging tasks such as robust malware detection under obfuscation (Zhang et al., 2024) and dynamic emotion recognition under cross-modal misalignment (Lee et al., 2 Jul 2025).

5. Ablation Analyses and Design Considerations

Ablation studies in recent research validate the critical elements in BiLSTM-based fusion:

• Replacing BiLSTM with vanilla LSTM (i.e., removing bidirectionality) can reduce accuracy by up to 11 percentage points in longitudinal medical predictions (Khatooni et al., 27 Nov 2025).
• Excluding CBAM (Convolutional Block Attention Module) or a second BiLSTM layer leads to measurable drops in sentiment classification accuracy (−0.5% ACC/F1), as attention refines feature saliency and suppresses redundant signals (Wang et al., 2023).
• Hybrid fusion (joint raw features plus discriminant outputs) outperforms both early- and late-fusion BiLSTM by up to 0.04 AUC in humor detection (Xu et al., 2022).
• In temporal fusion, removal of BiLSTM-based gating (uniform average) decreases CCC by ≈0.02–0.03 in emotion estimation (Lee et al., 2 Jul 2025).

These findings underscore the utility of bidirectional, temporally-aware and context-sensitive integration in multimodal learning systems.

6. Optimization, Training, and Regularization Protocols

Optimization routines and overfitting mitigation are integral to BiLSTM-based fusion performance:

• Typical optimizers are Adam or AdamW, with learning rates between $v_O$8 and $v_O$9, sometimes using learning rate decay (Zhang et al., 2024, Wang et al., 2023, Khatooni et al., 27 Nov 2025, Lee et al., 2 Jul 2025).
• Regularization includes dropout (0.2–0.5) in BiLSTM and fully connected layers, L2/weight decay, and, in contrastive frameworks, supervised contrastive loss (SupCon) for robust representation learning (Wang et al., 2023).
• Loss functions are task-adapted, including cross-entropy for classification, CCC-based regression for continuous target variables, and focal loss to handle class imbalance (Zhang et al., 2024, Khatooni et al., 27 Nov 2025, Lee et al., 2 Jul 2025).
• Data augmentation (e.g., RandAugment for images, back-translation for text) and imputation for missing temporal points are sometimes applied to maximize generalizability (Wang et al., 2023, Khatooni et al., 27 Nov 2025).

Reported training protocols often involve early stopping on validation loss or performance, batch sizes between 6 and 64, and extensive ablation to confirm the effect of each architectural module.

7. Limitations, Challenges, and Future Directions

Reported challenges of BiLSTM-based multimodal fusion include:

• Sensitivity to cross-modal misalignment or distribution shift between train/dev/test splits, sometimes producing overfitting or performance degradation (Lee et al., 2 Jul 2025, Xu et al., 2022).
• Truncation or coarse alignment of sequential windows (e.g., fixed 2s for audio/text) may limit the capacity to model full sequence dependencies (Xu et al., 2022).
• Computation and parameter cost, especially in deep or recursive gated architectures, and the need for careful tuning of attention and fusion modules (e.g., number of Transformer or BiLSTM layers) (Wang et al., 2023, Lee et al., 2 Jul 2025).
• Modality-specific noise and redundancy, addressed with advanced attention mechanisms such as CBAM and temporal gating (Wang et al., 2023, Lee et al., 2 Jul 2025).

Future work includes further exploration of advanced gating, hierarchical attention, end-to-end alignment of fine-grained temporal signals, multimodal data augmentation, and application to new domains such as longitudinal medical prediction or real-time affective systems (Khatooni et al., 27 Nov 2025, Lee et al., 2 Jul 2025, Xu et al., 2022).