Memory Fusion Network for Multi-view Learning

• Memory Fusion Network is a neural architecture that models both view-specific temporal dynamics and cross-view interactions in multi-view sequential learning.
• It uses individual LSTMs for each modality combined with a delta-based attention mechanism to capture immediate and time-lagged inter-modal dependencies.
• The architecture achieves state-of-the-art performance in sentiment and emotion recognition tasks while being computationally efficient and interpretable.

Searching arXiv for the cited paper and closely related multimodal fusion work. Use arxiv_search with query "(Zadeh et al., 2018) Memory Fusion Network for Multi-view Sequential Learning". Memory Fusion Network (MFN) is a neural architecture for multi-view sequential learning that is designed to model two distinct forms of interaction in multi-view sequences: view-specific interactions within each modality and cross-view interactions across modalities through time. The model was introduced in “Memory Fusion Network for Multi-view Sequential Learning” (Zadeh et al., 2018). In the motivating setting, multiple heterogeneous sequences such as text, video, and audio describe the same phenomenon over time, and the central premise is that effective learning requires isolating intra-view temporal structure while also identifying dependencies across views, including possibly time-lagged relations such as a speaker’s facial gesture anticipating a word (Zadeh et al., 2018).

1. Problem formulation and design objectives

Multi-view sequential learning addresses tasks in which several aligned or co-occurring sequences jointly encode an event, behavior, or latent state. In the formulation used for MFN, a multi-view sequence contains two forms of interactions. The first is view-specific interactions, meaning the temporal dynamics internal to each individual view. The second is cross-view interactions, meaning dependencies across views, which may be instantaneous or time-lagged (Zadeh et al., 2018).

The motivation for MFN is explicitly framed against naïve concatenation of views. Concatenation is described as ignoring view-specific statistics and overfitting on small data. MFN instead adopts three design goals: capture view-specific temporal dynamics in isolation, discover and quantify cross-view interactions at each time step, and accumulate cross-view interaction evidence over time in a unified memory (Zadeh et al., 2018).

This decomposition suggests a modular interpretation of multimodal sequence modeling. Rather than treating fusion as a single-stage aggregation problem, MFN separates modality-local state estimation from interaction discovery and from temporal accumulation of cross-view evidence. A plausible implication is that this separation was intended to improve both statistical efficiency and interpretability of the fusion mechanism, although the paper’s concrete claims are stated in terms of explicit modeling rather than formal interpretability guarantees (Zadeh et al., 2018).

2. Core architecture

MFN consists of three components: a System of LSTMs, a Delta-memory Attention Network (DMAN), and a Multi-view Gated Memory (Zadeh et al., 2018). The architecture is organized so that each component corresponds to one of the stated design goals.

For a problem with $N$ views and sequence length $T$, view $n$ has input sequence $x_n^t \in \mathbb R^{d_{x_n}}$ for $t=1\ldots T$. MFN assigns one LSTM to each view, with hidden size $d_{c_n}$. These LSTMs learn view-specific interactions in isolation. Their memory cells are then concatenated across views to form a joint state: $$c^t = \bigl[c_1^t;\;c_2^t;\;\dots;\;c_N^t\bigr]\in\mathbb R^{d_c}, \quad\text{where }d_c=\sum_{n}d_{c_n}.$$ The final view-specific output is

$h^T=[\,h_1^T;\dots;h_N^T\,].$

The DMAN is introduced to identify which memory dimensions across all views participate in cross-view interactions at time $t$. Its defining characteristic is that it does not attend only to the current memory $c^t$, but also to the previous memory $T$0, thereby incorporating a local temporal delta. The concatenated input is

$T$1

A feed-forward network $T$2 produces logits, and a channel-wise softmax yields attention weights

$T$3

The attended memory is then

$T$4

The stated intuition is that large entries of $T$5 pinpoint memory dimensions in view $T$6 at time $T$7 or $T$8 that jointly signal a cross-view event (Zadeh et al., 2018).

The Multi-view Gated Memory is a single fusion memory $T$9 that accumulates cross-view information through time. From the attended memory $n$0, MFN computes an update proposal

$n$1

and two gates,

$n$2

where $n$3 is a retain gate and $n$4 is an update gate. The memory update is

$n$5

The paper characterizes this gating as richer than an LSTM’s since both gates are neural-network controlled and the new content passes through $n$6 only once for stability (Zadeh et al., 2018).

3. System of LSTMs and delta-based attention

The System of LSTMs is the view-specific encoding stage. For each view $n$7, the recurrence at time $n$8 is

$n$9

Here $x_n^t \in \mathbb R^{d_{x_n}}$0 is the sigmoid, $x_n^t \in \mathbb R^{d_{x_n}}$1 is the elementwise product, and $x_n^t \in \mathbb R^{d_{x_n}}$2 is the memory cell (Zadeh et al., 2018).

The architectural significance of this stage is not merely parallel encoding. The paper states that view-specific interactions are learned in isolation by assigning an LSTM function to each view. This is a strong modeling assumption: fusion should not be allowed to erase modality-specific temporal structure prematurely. In that sense, the System of LSTMs functions as a structured factorization of the multi-view sequence into modality-local temporal states (Zadeh et al., 2018).

The delta-based attention mechanism is the architectural device that differentiates MFN from simpler memory fusion schemes. Rather than attending over only the current memory configuration, DMAN uses the pair $x_n^t \in \mathbb R^{d_{x_n}}$3. The paper’s stated rationale is that this allows the network to focus on dimensions about to change. This suggests that MFN treats cross-view interactions as events reflected in transitions of internal memory, not only in absolute memory contents. A plausible implication is that DMAN is designed to emphasize temporally salient state changes that may correspond to multimodal coordination, such as audiovisual cues co-occurring with linguistic transitions (Zadeh et al., 2018).

4. Temporal fusion mechanism and end-to-end signal flow

MFN’s signal flow is specified layer by layer for each time step $x_n^t \in \mathbb R^{d_{x_n}}$4. First, each view’s LSTM ingests $x_n^t \in \mathbb R^{d_{x_n}}$5 and updates $x_n^t \in \mathbb R^{d_{x_n}}$6. Second, the model concatenates all $x_n^t \in \mathbb R^{d_{x_n}}$7 and $x_n^t \in \mathbb R^{d_{x_n}}$8 into $x_n^t \in \mathbb R^{d_{x_n}}$9. Third, DMAN computes attention $t=1\ldots T$0 and the attended memory $t=1\ldots T$1. Fourth, the Multi-view Gated Memory uses $t=1\ldots T$2 to produce gates $t=1\ldots T$3 and the update proposal $t=1\ldots T$4, then updates $t=1\ldots T$5 (Zadeh et al., 2018).

At the terminal time $t=1\ldots T$6, MFN collects two forms of representation: the view-specific outputs

$t=1\ldots T$7

and the cross-view fused memory $t=1\ldots T$8. These are then concatenated or combined by a small output network to produce final predictions (Zadeh et al., 2018).

This organization makes explicit a distinction between view-specific summary state and cross-view interaction summary state. The final predictor therefore has access both to modality-local temporal encodings and to a separate memory intended to accumulate interaction evidence. A plausible implication is that the architecture treats these two information sources as complementary rather than interchangeable. That contrasts with fusion methods in which all cross-modal structure must be expressed in a single monolithic hidden state (Zadeh et al., 2018).

5. Training protocol, tasks, and datasets

MFN is trained with task-specific losses. For binary classification tasks such as sentiment polarity, the loss is binary cross-entropy. For multi-class classification tasks such as 7-way sentiment or emotion classification, the loss is categorical cross-entropy. For regression tasks such as sentiment intensity, arousal, or personality ratings, the loss is mean absolute error (MAE) (Zadeh et al., 2018).

The optimization procedure uses Adam, with learning rate chosen by validation. Regularization consists of early stopping on validation and standard $t=1\ldots T$9 weight decay. Reported hyperparameters tuned on a held-out set include LSTM memory dimensions matched to input sizes, with examples given as $d_{c_n}$0 for text, $d_{c_n}$1 for video, and $d_{c_n}$2 for audio; a fusion memory size $d_{c_n}$3; mini-batch size 32 for segment-level tasks or 16 for video-level tasks; learning rate $d_{c_n}$4; weight decay $d_{c_n}$5; and dropout $d_{c_n}$6 on input to DMAN and to the output classifier (Zadeh et al., 2018).

The evaluation spans six publicly available benchmark datasets:

Dataset	Task framing	Modalities/views
CMU-MOSI	English sentiment, segment-level; $d_{c_n}$7 up to 60 words per segment	text, video, audio
MOUD	Spanish sentiment, segments	text, video, audio
YouTube	English sentiment, segments	text, video, audio
ICT-MMM	English sentiment, video-level	text, video, audio
IEMOCAP	emotion recognition: discrete & continuous, segments	text, video, audio
POM	16 speaker traits, video-level	text, video, audio

For these datasets, the views are text with GloVe 300d, video with facial features 35d, and audio with COVAREP 74d (Zadeh et al., 2018).

The baseline families include view concatenation approaches such as HMM, HCRF, LDHCRF, and single- and bi-directional LSTMs; multi-view structured models such as MV-HCRF, MV-HSSHCRF, and MV-LSTM; and deep fusion approaches including tensor fusion network, contextual LSTM, selective additive CNN, multiple-kernel SVM, random forest, and deep multimodal fusion (Zadeh et al., 2018).

6. Empirical performance and ablation findings

Across all six datasets, MFN is reported to set new state-of-the-art results. On CMU-MOSI sentiment, the previous best binary accuracy of 74.6% improves to 77.4% $d_{c_n}$8, MAE improves from 1.019 to 0.965 $d_{c_n}$9, and Pearson $$c^t = \bigl[c_1^t;\;c_2^t;\;\dots;\;c_N^t\bigr]\in\mathbb R^{d_c}, \quad\text{where }d_c=\sum_{n}d_{c_n}.$$0 improves from 0.622 to 0.632. On ICT-MMM video-level sentiment, BA improves from 81.3 to 87.5% $$c^t = \bigl[c_1^t;\;c_2^t;\;\dots;\;c_N^t\bigr]\in\mathbb R^{d_c}, \quad\text{where }d_c=\sum_{n}d_{c_n}.$$1, MAE from 0.842 to 0.739 $$c^t = \bigl[c_1^t;\;c_2^t;\;\dots;\;c_N^t\bigr]\in\mathbb R^{d_c}, \quad\text{where }d_c=\sum_{n}d_{c_n}.$$2, and $$c^t = \bigl[c_1^t;\;c_2^t;\;\dots;\;c_N^t\bigr]\in\mathbb R^{d_c}, \quad\text{where }d_c=\sum_{n}d_{c_n}.$$3 from 0.588 to 0.696. On YouTube sentiment, MA(3) improves from 50.2 to 61.0% $$c^t = \bigl[c_1^t;\;c_2^t;\;\dots;\;c_N^t\bigr]\in\mathbb R^{d_c}, \quad\text{where }d_c=\sum_{n}d_{c_n}.$$4. On MOUD Spanish sentiment, BA improves from 74.0 to 81.1% $$c^t = \bigl[c_1^t;\;c_2^t;\;\dots;\;c_N^t\bigr]\in\mathbb R^{d_c}, \quad\text{where }d_c=\sum_{n}d_{c_n}.$$5. On IEMOCAP, discrete 9-way MA improves from 36.0 to 36.5%; valence MAE improves from 0.244 to 0.236 and $$c^t = \bigl[c_1^t;\;c_2^t;\;\dots;\;c_N^t\bigr]\in\mathbb R^{d_c}, \quad\text{where }d_c=\sum_{n}d_{c_n}.$$6 from 0.088 to 0.111; arousal MAE improves from 0.513 to 0.482 and $$c^t = \bigl[c_1^t;\;c_2^t;\;\dots;\;c_N^t\bigr]\in\mathbb R^{d_c}, \quad\text{where }d_c=\sum_{n}d_{c_n}.$$7 from 0.620 to 0.645. On POM, the average MA(5/7) improves by $$c^t = \bigl[c_1^t;\;c_2^t;\;\dots;\;c_N^t\bigr]\in\mathbb R^{d_c}, \quad\text{where }d_c=\sum_{n}d_{c_n}.$$8 to $$c^t = \bigl[c_1^t;\;c_2^t;\;\dots;\;c_N^t\bigr]\in\mathbb R^{d_c}, \quad\text{where }d_c=\sum_{n}d_{c_n}.$$9 points, average MAE is lower by 0.015 to 0.064, and average $h^T=[\,h_1^T;\dots;h_N^T\,].$0 increases by 0.002 to 0.079 (Zadeh et al., 2018).

The ablation studies report an ordering of component performance: $h^T=[\,h_1^T;\dots;h_N^T\,].$1 Removing $h^T=[\,h_1^T;\dots;h_N^T\,].$2, meaning attending only to $h^T=[\,h_1^T;\dots;h_N^T\,].$3, drops performance by 1–2%, which is presented as evidence for the importance of temporal deltas. Removing DMAN and Gated Memory reduces MFN to three disjoint LSTMs and yields a 3–5% loss due to the absence of cross-view modeling. Single-view MFNs underperform the multi-view model by 5–15% (Zadeh et al., 2018).

An additional negative result is also reported: attempts to extend DMAN to attend over deltas spanning $h^T=[\,h_1^T;\dots;h_N^T\,].$4 did not yield significant gains, and the paper concludes that local $h^T=[\,h_1^T;\dots;h_N^T\,].$5 suffices. This is a useful qualification because it limits an obvious interpretation that longer delta windows should necessarily improve cross-view event detection. Within the experiments reported, the advantage appears to derive from immediate temporal contrast rather than from a broader retrospective attention horizon (Zadeh et al., 2018).

The paper also states that MFN achieves superior performance with only approximately $h^T=[\,h_1^T;\dots;h_N^T\,].$6 M parameters, compared with approximately $h^T=[\,h_1^T;\dots;h_N^T\,].$7 M in Tensor Fusion, and runs more than $h^T=[\,h_1^T;\dots;h_N^T\,].$8 faster in inference (Zadeh et al., 2018). This places MFN not only in the category of high-performing multimodal architectures but also in the category of parameter- and compute-efficient fusion models as defined by the reported comparisons.

7. Interpretation, scope, and relation to multimodal fusion

MFN explicitly separates view-specific and cross-view modeling. The System of LSTMs captures view-specific temporal dynamics; DMAN identifies cross-view interactions from changes in LSTM memories; and the Multi-view Gated Memory accumulates summaries of these interactions through time (Zadeh et al., 2018). This decomposition is the central conceptual contribution of the model.

A common misconception in multimodal sequence modeling is that fusion can be reduced to simple feature concatenation followed by a generic temporal model. The MFN formulation is directly opposed to that view. Its stated motivation is that naïve concatenation ignores view-specific statistics and overfits on small data. The model therefore encodes a more structured hypothesis: intra-view temporal modeling and inter-view interaction modeling should be treated as related but distinct computational problems (Zadeh et al., 2018).

Another potential misconception is that cross-view interaction is adequately captured by attending over current states alone. The ablation result for removing $h^T=[\,h_1^T;\dots;h_N^T\,].$9 argues against that interpretation within the reported setting, because performance declines when the attention mechanism no longer uses both $t$0 and $t$1 (Zadeh et al., 2018). This suggests that, in MFN, interaction salience is linked to memory transitions rather than static memory occupancy.

In relation to the broader class of deep fusion baselines named in the evaluation, MFN can be situated as a recurrent multimodal fusion architecture with explicit memory partitioning: modality-specific recurrent states are maintained separately, and only selected cross-view evidence is accumulated into a shared fusion memory. This suggests a design lineage in which fusion is not merely a combinatorial expansion of modalities, but a controlled state-space interaction process. A plausible implication is that this architectural discipline is one reason the model was reported to combine strong benchmark performance with relatively low parameter count and faster inference (Zadeh et al., 2018).