Bilinear and Tensor Fusion Methods

• Bilinear and tensor fusion are mathematical frameworks that model multimodal interactions by explicitly capturing pairwise and higher-order dependencies using structured tensor operations.
• They use decompositions such as Tucker, CP, and block-term to reduce computational parameters while retaining high expressivity in applications like visual question answering and sentiment analysis.
• Their practical applications span medical decision support, biometric identification, and brain network analysis, offering improved interpretability and performance over simple concatenation methods.

Bilinear and tensor fusion are mathematical and algorithmic frameworks that enable explicit modeling of multimodal interactions by constructing parameterized or structured functions capturing dependencies across two or more feature spaces. These paradigms underpin the state-of-the-art in multimodal learning, including visual question answering, sentiment analysis, medical decision support, image fusion, and high-dimensional sensor data analysis, where simple concatenation or linear mixing is insufficient to capture the rich cross-modal dynamics required for accurate prediction and interpretability.

1. Mathematical Foundations: Bilinear and Higher-Order Fusion

Bilinear fusion combines two feature vectors $x \in \mathbb{R}^{d_x}$ and $y \in \mathbb{R}^{d_y}$ to produce an output $z \in \mathbb{R}^{d_o}$ using a third-order tensor $W \in \mathbb{R}^{d_o \times d_x \times d_y}$. The output is

$$z_o = \sum_{i=1}^{d_x} \sum_{j=1}^{d_y} W_{o,i,j} x_i y_j.$$

This expressivity allows learning all pairwise multiplicative interactions between modalities, but its uncompressed instantiation has $d_o d_x d_y$ parameters and quickly becomes infeasible for high-dimensional inputs (Ben-Younes et al., 2017, Ben-Younes et al., 2019, Keshvarikhojasteh et al., 12 May 2026).

Tensor fusion further generalizes this paradigm to higher-order (multimodal) interactions via outer products over multiple embeddings, yielding an order-$n$ tensor for $n$ modalities (Zadeh et al., 2017, Soleymani et al., 2018). For example, the tensor fusion layer in TFN computes a three-way outer product (with appended bias) between language, visual, and acoustic embeddings, capturing all unimodal, bimodal, and trimodal terms in the resulting tensor (Zadeh et al., 2017).

Efficient instantiations require parameter reductions via structured tensor factorizations (e.g., CP, Tucker, block-superdiagonal, low-rank designs, or sketch-based approximations) (Ben-Younes et al., 2019, Ben-Younes et al., 2017, Soleymani et al., 2018).

2. Model Architectures and Algorithmic Variants

Bilinear Modules and Decompositions

Bilinear models differ in how they parameterize and compute the fusion tensor:

• Full Bilinear Tensor: $W$ parameterized directly; impractical for large $d_x, d_y$ due to cubic scaling (Ben-Younes et al., 2017, Ben-Younes et al., 2019).
• Tucker Decomposition (MUTAN): Decomposes $y \in \mathbb{R}^{d_y}$0 as a core tensor $y \in \mathbb{R}^{d_y}$1 and factor matrices $y \in \mathbb{R}^{d_y}$2, reducing parameter count to $y \in \mathbb{R}^{d_y}$3; adds explicit low-rank constraints on mode-3 slices for further compression (Ben-Younes et al., 2017).
• CP/MLB: Uses rank-$y \in \mathbb{R}^{d_y}$4 sum of outer products of $y \in \mathbb{R}^{d_y}$5, with parameters $y \in \mathbb{R}^{d_y}$6 (Ben-Younes et al., 2019).
• Block-term Decomposition (BLOCK): Splits $y \in \mathbb{R}^{d_y}$7 into $y \in \mathbb{R}^{d_y}$8 blocks of Tucker-cores, interpolating between CP and Tucker, each block acting as a local bilinear expert over feature subspaces (Ben-Younes et al., 2019). This enables variable expressivity with tractable complexity.

Compact and Sketch-Based Bilinear Fusion

• Compact Bilinear (Tensor Sketch): Approximates the high-dimensional outer product by projecting each input into a lower-dimensional space via random hash functions and then using efficient convolutions (FFT domain) (Soleymani et al., 2018). Achieves near-bilinear accuracy with output dimension $y \in \mathbb{R}^{d_y}$9 and overall cost $z \in \mathbb{R}^{d_o}$0.
• Generalized Compact Fusion: Concatenates sketches of all mode-1 to mode-$z \in \mathbb{R}^{d_o}$1 interactions, interpolating between weighted concatenation and full $z \in \mathbb{R}^{d_o}$2-way interaction (Soleymani et al., 2018).

Full-Order Tensor Fusion

• Tensor Fusion Network (TFN): For three modalities, appends a unit bias to each embedding and computes a $z \in \mathbb{R}^{d_o}$3 tensor by outer product. Each cell in the tensor encodes a semantic interaction, and the full tensor is flattened and passed to an MLP for downstream prediction (Zadeh et al., 2017). This formulation explicitly disentangles unimodal, bimodal, and trimodal interactions without parameter explosion, as there are no weights in the fusion itself.

Attention-based Bilinear Fusion

• OMniBAN: In medical VQA, OMniBAN employs a single-layer self-attention within each modality, followed by a parameter-efficient bilinear attention across modalities. The final multimodal representation aggregates attention-weighted bilinear interactions, regularized by an orthogonality loss to diversify attention heads, matching the representational power of much larger cross-modal Transformer stacks at an order of magnitude lower FLOPs and parameters (Zhang et al., 2024).

3. Trade-offs: Expressivity, Complexity, and Interpretability

Fusion Method	Parameters	Cost	Expressivity
Full Bilinear	$z \in \mathbb{R}^{d_o}$4	$z \in \mathbb{R}^{d_o}$5	All pairwise
Tucker/MUTAN	$z \in \mathbb{R}^{d_o}$6	$z \in \mathbb{R}^{d_o}$7	All pairwise via core matrix
BLOCK	$z \in \mathbb{R}^{d_o}$8	$z \in \mathbb{R}^{d_o}$9	Variable: local high-order
Compact Sketch	$W \in \mathbb{R}^{d_o \times d_x \times d_y}$0	$W \in \mathbb{R}^{d_o \times d_x \times d_y}$1	Approximate all pairwise
Weighted Concat	$W \in \mathbb{R}^{d_o \times d_x \times d_y}$2	$W \in \mathbb{R}^{d_o \times d_x \times d_y}$3	Only unimodal
TFN (no weights)	None in fusion itself	$W \in \mathbb{R}^{d_o \times d_x \times d_y}$4	Up to trimodal

Block-term and Tucker decompositions provide a principled means to control expressivity/complexity via user-selected ranks or block sizes, yielding superior empirical accuracy at cost comparable to weighted concatenation or sketch-based pooling (Ben-Younes et al., 2017, Ben-Younes et al., 2019, Soleymani et al., 2018).

A key result is that full tensor fusion explicitly models not just modality pairs but three-way (or higher) interactions, and ablation studies consistently show large drops in performance when these higher-order terms are omitted, for example, in sentiment analysis (Zadeh et al., 2017).

Pairwise low-rank bilinear fusion is particularly favored in biomedical settings for structural interpretability: each bilinear module models conditional dependencies between two modalities, and the final embedding is the concatenation of these cross-terms, supporting downstream explanation and robustness (Keshvarikhojasteh et al., 12 May 2026).

4. Applications in Multimodal Learning

Bilinear and tensor fusion methods are deployed across diverse multimodal tasks:

• Visual Question Answering (VQA): CP, Tucker, BLOCK, MUTAN, and compact bilinear (MCB) fusion variants have been benchmarked on VQA 2.0; BLOCK achieves highest overall accuracy among tested models with fewer fusion parameters than most competitors (Ben-Younes et al., 2019).
• Sentiment Analysis: TFN demonstrates clear empirical superiority over concatenation or early-fusion methods due to full disentanglement of unimodal, bimodal, and trimodal effects. On CMU-MOSI, full TFN improves binary accuracy by $W \in \mathbb{R}^{d_o \times d_x \times d_y}$5 points over the best prior and $W \in \mathbb{R}^{d_o \times d_x \times d_y}$6 points in five-class accuracy over fusion with no triple interactions (Zadeh et al., 2017).
• Multimodal Biometric Identification: Generalized compact bilinear fusion incorporating all orders of interactions yields the highest published rank-1 accuracy across multiple major biometric datasets, with modest increases in joint feature size and efficient compute (Soleymani et al., 2018).
• Survival Prediction in Biomedicine: Pairwise low-rank bilinear fusion promotes interpretability and generalizes robustly to small clinical datasets, outperforming concatenation and hierarchical schemes by up to $W \in \mathbb{R}^{d_o \times d_x \times d_y}$7 in C-index on held-out evaluation (Keshvarikhojasteh et al., 12 May 2026).
• Hyperspectral/MSI Fusion: Unified tensor fusion models spatial (blind deconvolution) and spectral (blind unmixing) couplings, with best-in-class performance in state-of-the-art benchmarks using a tubal low-rank prior and provably convergent ADMM optimization (Gao et al., 12 Mar 2026).
• Brain Network Analysis: Tensor regression, multiway-PLS, and coupled matrix-tensor factorizations, as well as atomic tensor decompositions (e.g., PARAFAC), enable compact modeling and decomposition of multi-modal, time-frequency brain imaging data into interpretable “atoms” corresponding to functional subnetworks or causal relationships (Karahan et al., 2015).

5. Empirical Results and Ablation Studies

Empirical evaluations consistently validate the necessity and advantage of bilinear and tensor fusion:

• Sentiment Analysis (Zadeh et al., 2017): On CMU-MOSI, TFN achieves binary accuracy $W \in \mathbb{R}^{d_o \times d_x \times d_y}$8 (improving $W \in \mathbb{R}^{d_o \times d_x \times d_y}$9 pts over best prior), five-class accuracy $$z_o = \sum_{i=1}^{d_x} \sum_{j=1}^{d_y} W_{o,i,j} x_i y_j.$$0 (up $$z_o = \sum_{i=1}^{d_x} \sum_{j=1}^{d_y} W_{o,i,j} x_i y_j.$$1 pts), regression MAE $$z_o = \sum_{i=1}^{d_x} \sum_{j=1}^{d_y} W_{o,i,j} x_i y_j.$$2 (down $$z_o = \sum_{i=1}^{d_x} \sum_{j=1}^{d_y} W_{o,i,j} x_i y_j.$$3), and Pearson $$z_o = \sum_{i=1}^{d_x} \sum_{j=1}^{d_y} W_{o,i,j} x_i y_j.$$4 (up $$z_o = \sum_{i=1}^{d_x} \sum_{j=1}^{d_y} W_{o,i,j} x_i y_j.$$5). Ablations reveal additive performance gains as one moves from unimodal to bimodal to full trimodal fusion.
• Biometric Identification (Soleymani et al., 2018): On CMU Multi-PIE, weighted fusion yields $$z_o = \sum_{i=1}^{d_x} \sum_{j=1}^{d_y} W_{o,i,j} x_i y_j.$$6 while generalized compact bilinear fusion achieves $$z_o = \sum_{i=1}^{d_x} \sum_{j=1}^{d_y} W_{o,i,j} x_i y_j.$$7. On BIOMDATA, respective rates are $$z_o = \sum_{i=1}^{d_x} \sum_{j=1}^{d_y} W_{o,i,j} x_i y_j.$$8 and $$z_o = \sum_{i=1}^{d_x} \sum_{j=1}^{d_y} W_{o,i,j} x_i y_j.$$9.
• VQA (Ben-Younes et al., 2019): BLOCK achieves $d_o d_x d_y$0 test-dev AllAcc using $d_o d_x d_y$1M parameters, surpassing MLB, Tucker, and MFH, which require up to $d_o d_x d_y$2M.
• Medical VQA (Zhang et al., 2024): OMniBAN matches full Transformer fusion accuracy, achieving $d_o d_x d_y$3 closed-question and $d_o d_x d_y$4 open-question accuracy, but with $d_o d_x d_y$5 the parameter count and FLOPs.
• Survival Prediction (Keshvarikhojasteh et al., 12 May 2026): Pairwise bilinear outperforms concatenation by $d_o d_x d_y$6 in C-index on internal validation and generalizes better by preventing overfitting on limited cohorts.

The following table summarizes parametric and accuracy trade-offs among prominent methods:

Method	Params	VQA Accuracy	Sentiment Acc	BioID Acc	Notes
Concatenation	Low	63.85	74.8	96.59	Unimodal only
MCB (sketch)	Med (32M)	61.23	—	—	Approximate bilinear
Tucker/MUTAN	14–16M	64.21–65.19	—	—	Full pairwise, factorized
MFH	High (48M)	65.72	—	—	Multi-hop, higher-order
BLOCK	18M	66.41	—	—	Bilinear blocks, best perf.
TFN (trimodal)	NA	—	77.1	—	All-order, param-free fusion
GCBF (Soleymani et al., 2018)	$d_o d_x d_y$7	—	—	99.9	Full generalized, top perf.

6. Interpretability, Extensions, and Future Directions

Bilinear and tensor fusion allow for interpretable attribution of prediction to specific cross-modal interactions, critical in sensitive application domains (e.g., medicine). Tensor block decompositions further invite block-wise interpretation and semantic grounding of latent factors (Ben-Younes et al., 2019, Ben-Younes et al., 2017, Keshvarikhojasteh et al., 12 May 2026).

Recent directions exploit:

• Adaptive Block-Term Decomposition: Optimizing block sizes or number of blocks per task or example for better expressiveness/efficiency balance (Ben-Younes et al., 2019).
• Hierarchical Fusion and Order Selection: Interpolating between pairwise, triplet, and full high-order interactions as needed by the data and task complexity (Soleymani et al., 2018).
• Provable Optimization: ADMM and Moreau envelope smoothing enable robust, convergent learning in unsupervised inverse problems (Gao et al., 12 Mar 2026).
• Graph-Structured and Statistical Fusions: Markov–Penrose diagrams represent fused tensor models in statistical and computational graphs, clarifying dependencies and regularization (Karahan et al., 2015).
• Integrable Tensor Models and Bilinear Identities: Theoretical work has established Givental-style decompositions and Hirota bilinear identities for random tensor models, connecting tensor fusion to integrable hierarchies (Dartois, 2014).

A plausible implication is that as input dimensionality and the number of modalities scale, the benefits of block, compact, and low-rank bilinear/tensor designs will become even more pronounced, not merely for computation but also for model identifiability and interpretability.

7. Limitations and Open Challenges

Despite their versatility, open challenges remain:

• Model Selection: Choosing the optimal decomposition ranks, block structure, or tensor sketch size is computationally intensive and lacks universal guidelines (Ben-Younes et al., 2019).
• Scalability: Extending block-term decompositions to $d_o d_x d_y$8-way fusion with efficient GPU implementations remains an open engineering problem.
• Semantic Grounding: Interpreting learned blocks or fusion factors in terms of meaningful subspaces (e.g., specific cognitive processes or semantic relations) is nontrivial and typically post hoc.
• Fusion Order Sensitivity: Hierarchical low-rank bilinear schemes can bias the fusion process due to imposed order of pairwise interactions (Keshvarikhojasteh et al., 12 May 2026).

Overall, bilinear and tensor fusion form the basis for practically and theoretically robust multimodal learning, permitting explicit, efficient, and interpretable modeling of cross-modal dependencies beyond what can be captured by simple concatenation or unimodal approaches. Continued advances in parametrization, optimization, and interpretability are central to their further adoption and impact across scientific and technological domains.