Deep Multi-Modal Sets

• Deep multi-modal sets are a representation paradigm that treats modality data as unordered sets, ensuring permutation and cardinality invariance.
• They employ modality-specific encoders and permutation-invariant pooling (sum, max, min) to effectively handle varying numbers of instances per modality.
• Hierarchical residual attention and attribution mechanisms offer fine-grained feature importance, enhancing model interpretability and robustness.

A deep multi-modal set is a representation and learning paradigm in which modalities (data sources) and their constituent features are treated as unordered sets or sets of sequences, rather than concatenated or strictly-ordered structures. The objective is to achieve permutation and cardinality invariance, enabling scalable, interpretable, and robust modeling over arbitrary collections of modality-specific features. This structure generalizes beyond single-instance per modality, enabling reasoning over sets of variable-sized, multimodal, and multi-instance inputs, while avoiding the dimensionality blow-up and rigidity of classical fusion architectures.

1. Formalism and Motivations for Deep Multi-Modal Sets

Deep multi-modal sets are defined over a collection of modalities $\mathcal{M} = \{1, \dots, M\}$, where each modality $m$ may contribute one or more instances or a sequence of variable length. The basic structure is either a set of vectors (multi-modal set) or a set of sequences (multi-modal sequence set). Each sample is thus modeled as:

• Multi-modal Set: $$\mathbf{X} = \{ X_i^{(n)} : i \in \mathcal{I},\ n = 1,\dots,N_i\}$$
  • $\mathcal{I}$: modality indices
  • $X_i^{(n)} \in \mathbb{R}^{M_i}$: $n$-th feature vector of modality $i$
  • $N_i$: number of instances per modality
• Sequence Set: $X=\{ x^{(m)} \mid m\in\mathcal{M}\}$ where $$x^{(m)} = (x_1^{(m)}, \dots, x_{l_m}^{(m)}) \in \mathcal{V}^{l_m}$$ for a shared vocabulary $m$0

The mapping from the set $m$1 to prediction $m$2 should be permutation-invariant with respect to both the order of modalities and the order of elements within each set or sequence, and should accommodate missing modalities naturally (Kitada et al., 2022, Reiter et al., 2020).

Key motivations:

• Scalability: Avoid $m$3 feature expansion from concatenation.
• Cardinality invariance: Handle variable or missing modalities and variable numbers of elements within modalities.
• Rich relevance modeling: Enable fine-grained and hierarchical attention/importance mechanisms.
• Interpretability: Attribute predictions to meaningful elements or modalities (Reiter et al., 2020, Kitada et al., 2022).

2. Canonical Architectures and Pooling Operators

The core architectural elements in deep multi-modal sets include:

2.1 Modality-specific Encoders

Each input $m$4 is mapped to a common feature space $m$5 by a modality-specific encoder $m$6 (Reiter et al., 2020):

$m$7

All encoded vectors are gathered into a set $m$8.

2.2 Permutation-invariant Pooling

A set-aggregation operator $m$9 is applied to $$\mathbf{X} = \{ X_i^{(n)} : i \in \mathcal{I},\ n = 1,\dots,N_i\}$$0 to map any-size set to a fixed-size vector, ensuring permutation and cardinality invariance. Choices include:

• Sum pooling: $$\mathbf{X} = \{ X_i^{(n)} : i \in \mathcal{I},\ n = 1,\dots,N_i\}$$1
• Max pooling (elementwise): $$\mathbf{X} = \{ X_i^{(n)} : i \in \mathcal{I},\ n = 1,\dots,N_i\}$$2
• Min pooling: $$\mathbf{X} = \{ X_i^{(n)} : i \in \mathcal{I},\ n = 1,\dots,N_i\}$$3

2.3 Decoding and Task Heads

The pooled vector is passed through an MLP $$\mathbf{X} = \{ X_i^{(n)} : i \in \mathcal{I},\ n = 1,\dots,N_i\}$$4 to produce task outputs (e.g., class logits or regression value). Loss functions are selected per task (cross-entropy, binary cross-entropy, etc.) (Reiter et al., 2020).

2.4 Residual and Hierarchical Attention Extensions

Advanced architectures (e.g., DM$$\mathbf{X} = \{ X_i^{(n)} : i \in \mathcal{I},\ n = 1,\dots,N_i\}$$5S$$\mathbf{X} = \{ X_i^{(n)} : i \in \mathcal{I},\ n = 1,\dots,N_i\}$$6) introduce hierarchical residual attention:

• Intra-modality Residual Attention (IntraMRA): For each modality $$\mathbf{X} = \{ X_i^{(n)} : i \in \mathcal{I},\ n = 1,\dots,N_i\}$$7, token-level importance weights $$\mathbf{X} = \{ X_i^{(n)} : i \in \mathcal{I},\ n = 1,\dots,N_i\}$$8 are computed over $$\mathbf{X} = \{ X_i^{(n)} : i \in \mathcal{I},\ n = 1,\dots,N_i\}$$9 using a softmax-weighted MLP, and each token is enhanced via a residual connection:

  $\mathcal{I}$0

• Inter-modality Residual Attention (InterMRA): Each modality's sequence is collapsed to a summary vector $\mathcal{I}$1, and modality-level weights $\mathcal{I}$2 are applied via softmax and residual:

  $\mathcal{I}$3

This two-tier architecture allows the model to learn both fine-grained and modality-level importance, as well as to naturally handle missing modalities (zero weights for empty or absent modalities) (Kitada et al., 2022).

3. Interpretability and Attribution Mechanisms

Interpretability is built-in to deep multi-modal set methods via:

• Max-pooling-based attribution: In max pooling, the vector index $\mathcal{I}$4 for the output is ascribed to the set element that produced the maximal activation, resulting in a feature importance matrix that can be aggregated to compute per-modality or per-instance attribution scores (Reiter et al., 2020).
• Residual attention heatmaps: In architectures like DM$\mathcal{I}$5S$\mathcal{I}$6, attention weights $\mathcal{I}$7 and $\mathcal{I}$8 are directly visualizable. Token and modality heatmaps express which elements contributed most to a specific prediction, enabling diagnosis of both correct and error cases (Kitada et al., 2022).
• Instance and bag-level feature tracking: In multi-instance, multi-modal formulations (e.g., M3DN), label scores are assigned per instance and then aggregated, so it's possible to trace which instance or modality activated specific labels (Yang et al., 2021).

Interpretive outputs enable model debugging, data collection guidance, and transparency in real-world deployments.

4. Comparative Empirical Results and Scalability

Deep multi-modal sets and sequence set models have achieved state-of-the-art or near-SOTA performance across diverse benchmarks:

Dataset	Task	Best Deep Multi-Modal Set Results	Prior SOTA/Baseline
MM-IMDB	multi-label genre classification	Micro-F1: 0.6773 (max pool, (Reiter et al., 2020))	Kiela et al.: 0.6640
		Micro-F1: 69.64 (DM$\mathcal{I}$9S$X_i^{(n)} \in \mathbb{R}^{M_i}$0, (Kitada et al., 2022))	Baseline: 67.97
Ads-Parallelity	image-text parallelism (binary)	Acc: 76.71% (set sum pool, (Reiter et al., 2020))	Zhang et al.: 65.50%
		Acc: 78.15 (DM$X_i^{(n)} \in \mathbb{R}^{M_i}$1S$X_i^{(n)} \in \mathbb{R}^{M_i}$2)	Baseline: 76.83
Production Ad-LP	regression (conversion rate)	RMSE: 0.8561 (DM$X_i^{(n)} \in \mathbb{R}^{M_i}$3S$X_i^{(n)} \in \mathbb{R}^{M_i}$4)	BERT-only: 0.8615

Scalability is manifest in constant final feature dimensions regardless of the number or cardinality of modalities, as opposed to $X_i^{(n)} \in \mathbb{R}^{M_i}$5 for mid-fusion. Missing modalities require only the omission of set elements; attention and pooling will simply ignore or downweight them. Ablations confirm that hierarchical attention and residual connections enhance accuracy and interpretability (Kitada et al., 2022, Reiter et al., 2020).

5. Related and Extended Paradigms

Several related frameworks extend or complement the deep multi-modal set paradigm:

• Deep Multimodal Sequence Sets (DM$X_i^{(n)} \in \mathbb{R}^{M_i}$6S$X_i^{(n)} \in \mathbb{R}^{M_i}$7): Applies token-level and modality-level residual attention over sets of sequences using a BERT backbone, excelling at hierarchical and interpretable aggregation (Kitada et al., 2022).
• Multi-Instance Multi-Modal Multi-Label Networks (M3DN/M3DNS): Models each modality as a bag of instances, aggregates via specialized pooling and optimal transport-based label alignment, and includes semi-supervised extensions for unlabeled data (Yang et al., 2021).
• Deep Multiset Canonical Correlation Analysis (dMCCA): Learns non-linear shared subspaces for multiple modalities by maximizing a trace ratio of between- and within-modality covariance on deep embeddings (Somandepalli et al., 2019).
• Multiplicative Set and Mixture Fusion: Rather than rigid additive concat or fixed average, these models multiplicatively reweight per-modality or per-combination losses/gates based on their (un)certainty, providing robustness and sample-adaptive fusion (Liu et al., 2018).
• Type-Specific and Multi-Task RBM Architectures: Unify multi-type (real, binary, count) feature sets in deep RBM/DNN pipelines with parallel or joint output heads for multi-task supervision (Tran et al., 2016).
• Multi-task Multi-view Deep Networks (Deep-MTMV): Simultaneously model both view (modality) and task heterogeneity, with automated branching and clustering to jointly exploit shared structure (Zheng et al., 2019).

6. Practical Considerations and Limitations

Key practical properties and limitations of deep multi-modal sets include:

• Modularity: Adding new modalities requires only a new encoder and inclusion in the set pool; final embedding size is unchanged.
• Pooling choice: Sum pooling generally maximizes raw accuracy; max pooling provides better interpretability but may cost slight accuracy (Reiter et al., 2020).
• Noisy or spurious modalities: Attribution mechanisms can expose over-weighting of uninformative modalities; attention and multiplicative fusion can suppress their impact (Kitada et al., 2022, Liu et al., 2018).
• Extension to unsupervised and semi-supervised: OT-based and auto-encoder modules enable representation learning from large collections of unlabeled or weakly labeled multi-modal data (Yang et al., 2021).
• Computation: Joint sequence-encoding and set pooling (e.g., with BERT) yields favorable scaling ($X_i^{(n)} \in \mathbb{R}^{M_i}$8 vs. $X_i^{(n)} \in \mathbb{R}^{M_i}$9 for per-modality encoding) (Kitada et al., 2022).

A notable limitation is that set and attention-based pooling may dilute highly distributed signals if cardinality becomes large and multiplicity of spurious features increases, motivating research into richer set-attention operators and input denoising (Reiter et al., 2020, Tran et al., 2016).

7. Outlook: Adaptations, Generalizations, and Research Directions

Further research on deep multi-modal sets extends to:

• Set attention and permutation-equivariant operations: Expanding beyond vanilla sum/max pooling to learn more expressive set functions.
• Temporal and video data: Multi-modal sets for time-series and video, where each instance itself is a time sequence.
• Graph-based extensions: Modality relationships that are not just sets but possess explicit or inferred structure.
• Joint end-to-end encoder training: Full optimization over raw modality inputs through set pooling and final tasks.
• Semi- and unsupervised learning: Large-scale exploitation of unlabeled multi-modal corpora using optimal-transport, contrastive, and auto-encoding objectives (Yang et al., 2021, Somandepalli et al., 2019).
• Real-world deployment: Production systems increasingly require missing modality handling, attribution, and robustness as provided by the set paradigm.

These directions underscore the evolving importance of set-based modeling for complex, realistic, and scalable multi-modal learning scenarios.