Tensor-Based Ternary Fusion

• Tensor-based ternary fusion is defined as the integration of three data modalities using multilinear tensor operations to capture complex unimodal, bimodal, and trimodal interactions.
• It leverages methods like outer-product fusion and tensor network summation to construct fused representations that promote interpretability and computational efficiency.
• Applications span multimodal sentiment analysis, neuroimaging, and interpretable AI, where disentangling cross-modal effects enhances both performance and insight.

Tensor-based ternary fusion refers to a family of methodologies for combining information from three distinct sources, modalities, or data representations using high-order tensor operations or tensor network algebra. This fusion paradigm is prominent in modern multimodal learning, signal processing, neuroimaging, and interpretable artificial intelligence, centering on explicitly modeling unimodal, bimodal, and trimodal interactions and often leveraging the algebraic properties of tensors to achieve efficient and interpretable data integration.

1. Mathematical Foundations of Tensor-based Ternary Fusion

Tensor-based ternary fusion exploits the multilinear structure of high-order tensors to combine three data modalities or representations. Let each modality be represented by a feature vector or tensor, denoted $x^{(1)}, x^{(2)}, x^{(3)}$. The fusion operation constructs a new object—commonly a third-order tensor—via an outer product or alternative tensor composition: $Z = x^{(1)} \otimes x^{(2)} \otimes x^{(3)}$ This operation generates a fused tensor encapsulating all possible interactions among the three input vectors. The fused tensor can subsequently be analyzed or processed via tensor factorization, tensor regression, or contracted by a parameterized tensor network. In the context of tensor networks (TNs), ternary fusion is formalized as the block-diagonal summation of three isomorphic TNs, each encoding a separate input but sharing the same topology and physical dimensionalities (Calvi et al., 2017).

Beyond the plain outer product, general frameworks (e.g., interpretable tensor fusion and coupled matrix-tensor factorization) combine tensors of different orders by coupling their factor matrices or integrating their latent scores via joint decompositions, thereby achieving a parsimonious and unique representation of shared and private structures (Varshneya et al., 2024, Karahan et al., 2015).

2. Tensor Network Summation as Ternary Fusion

The algebraic summation of three tensor networks is a rigorous instantiation of ternary fusion. For three isomorphic TNs, $$\mathcal{X}^{(1)}, \mathcal{X}^{(2)}, \mathcal{X}^{(3)}$$, each described by core tensor(s) $G^{(k)}$ and factors $A^{(k,n)}$, the composite TN $\mathcal{Z}$ is constructed as:

• Core: $$G^{(\oplus)} = \mathrm{block\_diag}(G^{(1)}, G^{(2)}, G^{(3)})$$
• Factors: $$A^{(\oplus, n)} = \left[ A^{(1,n)}\;|\;A^{(2,n)}\;|\;A^{(3,n)} \right]$$ Here, each modality’s features are retained as separate blocks, and the contraction-mode dimensionality increases as the sum of the constituent ranks. The fused TN reconstructs the sum of the three original tensors: $$Z = G^{(\oplus)} \times_1 A^{(\oplus,1)} \times_2 \cdots \times_M A^{(\oplus,M)} = X^{(1)} + X^{(2)} + X^{(3)}$$ This approach preserves feature localization, offering interpretability and computational efficiency superior to three separate contraction passes, especially when the contraction graph is deep (Calvi et al., 2017).

3. Outer-product Fusion for Multimodal Learning

In neural multimodal learning, tensor-based ternary fusion is operationalized most notably by the Tensor Fusion Network (TFN) architecture (Zadeh et al., 2017). Here, three modality embedding vectors—language ($z^l \in \mathbb{R}^{d_l}$), visual ($Z = x^{(1)} \otimes x^{(2)} \otimes x^{(3)}$0), and acoustic ($Z = x^{(1)} \otimes x^{(2)} \otimes x^{(3)}$1)—are augmented with a constant term, and their fused representation is computed as follows: $Z = x^{(1)} \otimes x^{(2)} \otimes x^{(3)}$2 This tensor explicitly encodes all unimodal, bimodal, and trimodal polynomial interactions through its seven sub-tensors, without introducing free parameters at the fusion stage. The output is subsequently vectorized and fed to a discriminative inference subnetwork for downstream tasks (e.g., sentiment analysis), achieving state-of-the-art results in multimodal settings and outperforming concatenation or late-fusion baselines by disentangling cross-modal effects.

Ablative studies demonstrate that retaining all orders of interaction—rather than restricting to unimodal, bimodal, or trimodal terms—yields the best predictive accuracy, as each type of interaction offers unique discriminative power (Zadeh et al., 2017).

4. Interpretability and Disentanglement via Structured Tensor Fusion

Interpretable tensor fusion (InTense) extends outer-product fusion by enforcing a mathematically principled decomposition of the fusion output into explicit unimodal, bimodal, and trimodal contributions. This is accomplished by:

• Computing low-dimensional, optionally projected representations for each modality.
• Forming all tensor products up to a specified order (e.g., all pairs and the ternary triple).
• Applying an iterative centering and normalization scheme—IterBN—to disentangle true interaction effects and prevent lower-order terms from being spuriously captured by higher-order tensors.
• Assigning explicit relevance scores $Z = x^{(1)} \otimes x^{(2)} \otimes x^{(3)}$3 to each interaction order through block-norm regularization, fostering interpretability.

For example, the fused score is: $Z = x^{(1)} \otimes x^{(2)} \otimes x^{(3)}$4 where each $Z = x^{(1)} \otimes x^{(2)} \otimes x^{(3)}$5 linearly weights the normalized tensor product $Z = x^{(1)} \otimes x^{(2)} \otimes x^{(3)}$6, and interpretability is quantified by the magnitude of each $Z = x^{(1)} \otimes x^{(2)} \otimes x^{(3)}$7 via block-norm-based $Z = x^{(1)} \otimes x^{(2)} \otimes x^{(3)}$8 scores (Varshneya et al., 2024).

On real-world multimodal datasets, InTense achieves both superior accuracy and relevance assignments highly correlated with unimodal classifier performance, and correctly identifies cases where trimodal interactions are required for task competence (e.g., sarcasm detection).

5. Coupled Matrix-Tensor Factorization for Scientific Data Fusion

In scientific domains such as neuroimaging, ternary fusion is realized through coupled matrix-tensor factorizations (CMTF) and multiway partial least squares (N-PLS) (Karahan et al., 2015). Here, three data blocks such as EEG, fMRI, and NIRS are decomposed as:

• Shared mode(s): e.g., a common spatial factor $Z = x^{(1)} \otimes x^{(2)} \otimes x^{(3)}$9.
• Private factors: modality-specific temporal, spectral, or spatial factors per block. The fusion objective jointly optimizes all factors to fit the observed data while enforcing shared structure across modality couplings, regularization, and possible sparsity and smoothness penalties. The generic CMTF objective is: $$\mathcal{X}^{(1)}, \mathcal{X}^{(2)}, \mathcal{X}^{(3)}$$0 Similarly, N-PLS identifies latent components with maximal covariation among the datasets.

Empirical findings indicate that these tensor-based ternary fusion methods yield enhanced component identifiability and sensitivity to cross-modal couplings compared to pairwise or matrix-only fusion approaches. They also support the estimation of high-dimensional interaction phenomena (e.g., Granger causality fields in the brain) via tensor regression, with scalable solutions based on alternating least squares, hierarchical ALS, and ADMM (Karahan et al., 2015).

6. Algorithmic and Computational Aspects

The computational complexity of explicit ternary tensor fusion depends primarily on the maximum ranks and dimensions of the tensors involved. For tensor network-based fusion, memory cost grows linearly with the sum of internal ranks, while contraction complexity can be reduced relative to separate passes by performing joint contractions over the fused TN. For neural architectures (e.g., TFN or InTense), parameter count is controlled by projecting modality embeddings to low-dimensional spaces before outer-product formation and by representing high-order tensors via low-rank decompositions (e.g., CP-form) if necessary.

Optimization employs standard deep learning techniques (e.g., Adam optimizer, dropout, L2 weight decay for neural tensor fusion), as well as alternating minimization, HALS updates, and block-coordinate descent for scientific coupled decompositions, often facilitated by sparsity and smoothness constraints.

7. Applications and Empirical Results

Tensor-based ternary fusion is applicable wherever synergistic information lies in three sources or modalities. Examples include:

• Multimodal sentiment analysis integrating language, visual, and acoustic cues via tensor fusion networks, achieving substantial gains over unimodal or simple fusion baselines in classification and regression tasks (Zadeh et al., 2017).
• Multimodal neuroimaging fusion, where EEG, fMRI, and NIRS are unified to extract joint spatiotemporal structures, yielding interpretable scientific insights (e.g., identification of α-rhythm networks, localized connectivity structures) (Karahan et al., 2015).
• Interpretable multimodal representation learning, providing explicit decomposition of predictions and accurate relevance attribution to each modality and their interactions, with demonstrated accuracy and interpretability improvements across diverse real-world datasets (sarcasm, humor, sentiment, image classification) (Varshneya et al., 2024).

These approaches generalize to arbitrary numbers and orders of modalities and provide a principled pathway to define, compute, and interpret all interactions up to order three in high-dimensional settings. Ternary fusion, grounded in tensor algebra and explicit high-order construction, constitutes a rigorous and flexible framework for scientific analysis and machine learning fusion problems.