Low-Rank Multimodal Fusion (LMF)

• LMF is a tensor factorization method that uses CP decomposition to represent multimodal interactions efficiently, drastically reducing parameter count compared to full tensor fusion.
• It computes fusion outputs via element-wise multiplication of modality-specific projections, enabling scalable applications in sentiment, emotion, and trait analysis.
• LMF enhances computational efficiency by significantly lowering model size and inference time, making it ideal for advanced transformer and adapter architectures.

Low-Rank Multimodal Fusion (LMF) refers to a family of tensor factorization approaches developed to efficiently model and compute multimodal interactions by representing the fusion operation in a compressed low-rank form. LMF addresses the prohibitive parameter and time complexity of full tensor-based multimodal fusion by applying CANDECOMP/PARAFAC (CP) decompositions to fusion tensors, enabling scalable capture of multiplicative cross-modal interactions. LMF has been widely adopted in multimodal sentiment analysis, emotion recognition, trait analysis, optical hardware implementations, and as the basis for more recent token-level sequence fusion and adapter architectures (Liu et al., 2018, Zhao et al., 2023, Sahay et al., 2020, Guo et al., 2024).

1. Full Tensor-Based Multimodal Fusion and Its Challenges

The classic tensor fusion approach (TFN) operates by computing the outer product of unimodal embeddings. Let $M$ denote the number of modalities and $z_m \in \mathbb{R}^{d_m}$ ($m = 1 \ldots M$) denote modality-specific representations. Their outer product forms the multimodal tensor:

$$\mathcal{Z} = z_1 \otimes z_2 \otimes \cdots \otimes z_M \in \mathbb{R}^{d_1 \times \cdots \times d_M}.$$

A linear projection with weight tensor $$\mathcal{W} \in \mathbb{R}^{d_1 \times \cdots \times d_M \times d_h}$$ produces output $h \in \mathbb{R}^{d_h}$, with

$$h_k = \langle \widetilde{\mathcal{W}}_k, \mathcal{Z} \rangle.$$

This scheme captures all cross-modal orders of interaction but suffers from parameter and inference costs scaling as $O(d_h \prod_m d_m)$, growing exponentially with modalities $M$ and quickly becoming intractable in both computation and memory; for example, even for moderate dimensions $(M=3, d_1=d_2=32, d_3=64, d_h=1)$, TFN requires 65,536 parameters just for fusion, excluding upstream encoders (Liu et al., 2018, Zhao et al., 2023).

2. Low-Rank CP Decomposition for Multimodal Fusion

LMF circumvents this complexity via CP decomposition, which approximates each order-$z_m \in \mathbb{R}^{d_m}$0 fusion tensor $z_m \in \mathbb{R}^{d_m}$1 by a sum of $z_m \in \mathbb{R}^{d_m}$2 rank-one outer products:

$z_m \in \mathbb{R}^{d_m}$3

where $z_m \in \mathbb{R}^{d_m}$4. These factors can be grouped as modality-specific matrices $z_m \in \mathbb{R}^{d_m}$5. This low-rank model ties together modalities while reducing the number of free parameters to $z_m \in \mathbb{R}^{d_m}$6, linear in $z_m \in \mathbb{R}^{d_m}$7 (Liu et al., 2018, Zhao et al., 2023).

The fusion output is computed as

$z_m \in \mathbb{R}^{d_m}$8

where $z_m \in \mathbb{R}^{d_m}$9 denotes element-wise (Hadamard) product. This formulation obviates the need to form or store the high-order tensors, enabling efficient differentiation and end-to-end training (Liu et al., 2018).

3. Efficiency, Complexity Analysis, and Practical Implementation

The principal advantage of LMF is its dramatic reduction in parameter count and computational cost:

• Parameter count: LMF requires $m = 1 \ldots M$0 parameters for fusion, compared to $m = 1 \ldots M$1 for TFN.
• Time complexity: Fusion forward pass cost is $m = 1 \ldots M$2 for $m = 1 \ldots M$3 modalities, compared to exponential cost for TFN.
• Empirical benchmarks: In standard multimodal settings (e.g., language, visual, and acoustic), LMF reduces total fusion block parameters from >12.5M to ~1.1M, and increases inference throughput by a factor of 1.9–3.3× on commodity hardware (Liu et al., 2018). In optical realizations, parameter and device count reductions upwards of $m = 1 \ldots M$4–$m = 1 \ldots M$5 are reported (Zhao et al., 2023).

In practice, implementation combines: (a) independently trained unimodal subnetworks (e.g., LSTM, MLP), (b) parallel linear projections with modality-specific factors, and (c) batched element-wise multiplications and summations—operations readily vectorized and optimized in modern deep learning frameworks.

4. Experimental Evaluations and Empirical Trends

LMF has been validated across multiple multimodal fusion tasks and datasets:

• Sentiment analysis (CMU-MOSI): LMF achieves MAE=0.912, Corr=0.668, outperforming TFN with (0.970, 0.633).
• Speaker trait (POM): LMF yields MAE=0.796, Corr=0.396 vs TFN’s (0.886, 0.093).
• Emotion recognition (IEMOCAP): LMF achieves F1=85.6 (mean), exceeding TFN's 79.0 (Liu et al., 2018).

Rank ablation studies demonstrate that small CP ranks ($m = 1 \ldots M$6) are often sufficient to achieve optimal performance, with over-parameterization inducing instability in training. LMF-based transformer architectures (e.g., LMF-MulT) systematically reduce model size and training time, with only minor or no loss in accuracy compared to full fusion-based models (Sahay et al., 2020).

The following table summarizes LMF’s parameter efficiency on major benchmarks (Sahay et al., 2020):

Model	CMU-MOSI Params	CMU-MOSEI Params	IEMOCAP Params
MulT (TFN)	1.07M	1.07M	1.07M
Fusion-CM-Attn	0.51M	0.53M	0.53M
LMF-MulT	0.84M	0.85M	0.86M

5. Extensions: Token-Level Fusion, Adapter Architectures, and Hardware

Recent advances generalize low-rank fusion from global vector fusion to token-level sequence fusion. The Wander adapter (Guo et al., 2024) applies a two-stage CP decomposition at both feature and sequence levels:

1. First CP: Over modalities’ feature dimensions, fusing $m = 1 \ldots M$7 sequences $m = 1 \ldots M$8.
2. Second CP: Over sequence (token/time) indices, further compressing token interactions.

This allows efficient modeling of all cross-token/modal interactions in large transformer stacks, making fine-tuning practical for tasks with more than two modalities and long sequences, without incurring the full outer-product’s exponential parameter explosion. For three-modal fusion with standard transformer hidden dimensions ($m = 1 \ldots M$9), Wander reduces parameter count from $$\mathcal{Z} = z_1 \otimes z_2 \otimes \cdots \otimes z_M \in \mathbb{R}^{d_1 \times \cdots \times d_M}.$$0B to $$\mathcal{Z} = z_1 \otimes z_2 \otimes \cdots \otimes z_M \in \mathbb{R}^{d_1 \times \cdots \times d_M}.$$1M (Guo et al., 2024).

Low-rank multimodal fusion has also been realized on analog photonics hardware (Zhao et al., 2023). By decomposing large fusion and projection matrices via tensor train (TT) and CP formats, the hardware complexity is reduced by over one or two orders of magnitude in photonic core and device count while maintaining competitive accuracy and throughput.

6. Open Challenges and Future Directions

While LMF achieves compelling efficiency and expressive power, several open research directions and limitations remain:

• Rank selection: Optimal CP rank is dataset- and task-dependent; poor choices degrade accuracy or negate parameter savings. Adaptive methods for rank tuning during training are an active area.
• Local high-order interactions: CP decomposition imposes a global low-rank structure that may not match some tasks requiring more localized complex cross-modal patterns (Liu et al., 2018).
• Sequence generalization: Extending LMF to sequence-level and fine-grained temporal fusion, as in Wander, demonstrates practical effectiveness, but further development is needed for higher-order alignment and matching in unconstrained multimodal scenarios (Guo et al., 2024).
• Hardware constraints: In photonic implementations, dynamic range, noise, and precision limits still challenge fully on-chip architectures; efficient realization of attention softmax and on-chip sequence routing are open problems (Zhao et al., 2023).
• Applications: Emerging uses in transfer learning, adapter design, and scalable inference at the edge signal ongoing growth in LMF’s role in multimodal architecture design.

7. Connections, Impact, and Research Landscape

LMF originated with "Efficient Low-rank Multimodal Fusion with Modality-Specific Factors" (Liu et al., 2018), establishing the practical value of CP-decomposed fusion in artificial intelligence. It has since influenced diverse domains, including:

• Transformer-based multimodal sequence modeling (Sahay et al., 2020)
• Adapter-based efficient transfer learning for multimodal Transformers (Guo et al., 2024)
• Optical tensorized neural networks for edge inference (Zhao et al., 2023)

The technique now underpins many state-of-the-art systems for multimodal sentiment analysis, emotion recognition, speaker trait understanding, and multiway visual–audio–language tasks, combining expressivity with linear time, space, and hardware complexity in the number and size of modalities. Ongoing research seeks to further integrate LMF with attention mechanisms, adaptive rank control, and highly parallel on-chip architectures for truly scalable multimodal learning.