--- title: Open-set Cross Modal Generalization url: https://www.emergentmind.com/topics/open-set-cross-modal-generalization-oscmg type: topic --- # Open-set Cross Modal Generalization Open-set Cross Modal Generalization (OSCMG) refers to the ability of a multimodal machine learning system to transfer competencies—such as classification, retrieval, and grounding—across data modalities that were not present or accessible during training. OSCMG is a crucial generalization axis for real-world deployment, where modality coverage is often incomplete or non-stationary due to privacy, hardware, or data availability constraints. Unlike standard multimodal learning, which assumes a modality-closed world, OSCMG extends to scenarios where new modalities, combinations thereof, or previously unseen compositions must be handled effectively. ## 1. Formal Problem Definition and Taxonomy Let $S = \{1, \ldots, K\}$ denote the set of seen modalities available at training, and $U = \{K+1, \ldots, K+M\}$ the set of unseen modalities appearing only during inference. Labeled data is available only for the seen modalities, $\mathcal{M}^k = \{(x^k_i, y^k_i)\}_{i=1}^{N^k}$, $k\in S$, whereas for each unseen modality $j\in U$, only unlabeled test data $\mathcal{M}^j = \{x^j_i\}_{i=1}^{N^j}$ is provided. The objective is to train a predictor $h$ on $S$ that generalizes to each $j \in U$ with competitive performance on tasks such as classification or retrieval [2412.18277]. Two regimes are defined: - **Weak Modality Generalization**: Every modality $t\in S\cup U$ is assumed to have a pretrained, fixed perceptor $\Phi^t$ mapping raw input $x^t$ to a joint embedding $z^t\in\mathbb{R}^d$. Learning is performed on the embeddings of the seen modalities, while at test time, $h$ is directly applied to the unseen modality's embedding. - **Strong Modality Generalization**: Only seen modalities possess pretrained perceptors. For an unseen modality $j\in U$, a new perceptor $\Psi^j$ must be trained from unlabeled data; its embedding lies in a previously unseen space. An explicit alignment $\tau$ is optionally introduced to bridge this embedding shift, and adversarial or discrepancy-based objectives may be used [2412.18277]. Generalizations are also possible over unseen **compositions of modalities** (e.g., modality combinations or novel cross-modal concepts) [2306.12795, 2505.23045]. Open-set variation further includes scenarios where *both* category and modality novelties co-occur [2407.01518, 2407.15376]. ## 2. Algorithmic Approaches and Methodologies Several methodological paradigms have emerged for OSCMG, characterized as follows: ### A. Domain Generalization (DG) on Modalities DG algorithms treat each modality as a domain and are trained to learn representations and/or classifiers invariant to the domain shift between seen modalities. This includes: - **Empirical Risk Minimization (ERM)**, **Invariant Risk Minimization (IRM)**: Penalize modality-specific variance [2412.18277] - **Mixup**, **EQRM**, **CDANN**, **SagNet**: Apply regularization, adversarial heads, or environment-wise penalties. DG methods outperform simple multimodal models (MML) in weak MG—with Mixup and EQRM achieving up to 53% accuracy on held-out modalities in classification tasks, versus 11–20% for standard fusion baselines [2412.18277]. ### B. Discrete Representation and Index-Level Alignment The CoDAAR framework [2605.12145] introduces modality-specific discrete codebooks, enforced to be semantically aligned at the *index* level via Discrete Temporal Alignment (DTA) and Cascading Semantic Alignment (CSA): - DTA: Modality codewords are updated partly by temporal cross-modal evidence, avoiding dominance by dense or high-variance modalities. - CSA: A convex update step aligns the same index across all codebooks to a common semantic mean. Losses combine self-supervised reconstruction, contrastive (CPC), code usage alignment (CMCM), and cross-modal commitment. This index-based approach enables zero-shot extension to new modalities by training new codebooks, provided DTA+CSA is used [2605.12145]. ### C. Self-Supervised Cross-Modal Pretext Learning The MOOSA approach [2407.01518] employs Masked Cross-modal Translation—predicting masked embedding parts of one modality from another—and Multimodal Jigsaw Puzzles—reordering, shuffling, and predicting the correct permutation of feature parts from all modalities. Losses are balanced using an entropy-weighted scheme, giving precedence to more reliable modalities. ### D. Residual Structure and Structure-Aware Embedding The SRCR architecture [2407.15376] addresses the risk of center collapse in open-set 3D retrieval by encoding for each object both a unified semantic center (shared across modalities) and residual-center embeddings (modality-specific deviations), using nested auto-encoders. These are further refined via hierarchical hypergraph structure learning (inter/intra-modality, implicit-category), enabling generalization to unseen object categories. ### E. Modality-Incomplete and Unseen Modality Interaction Unseen Modality Interaction [2306.12795] solves the OSCMG challenge for arbitrary combinations of input modalities by projecting all modality features into a common latent space and aggregating via summation. Pseudo-supervision based on unimodal soft targets regularizes the model, calibrating the contribution of less reliable or overconfident combinations. ### F. Retrieval-Augmented Cross-Modal Compositional Generalization Retrieval-augmented frameworks [2505.23045] explicitly aggregate K-nearest-neighbor primitives from linguistic and visual databases, refining representations of primitives through weighted averaging. No extra loss terms are required, allowing the cross-modal alignment to emerge via aggregation. ## 3. Benchmarks, Datasets, and Evaluation Protocols The primary OSCMG benchmarks provide multi-modal datasets, partition modality sets into seen and held-out subsets, and support both open-set class and open-set modality testing: | Dataset | Modalities | # Classes | Protocol (Modality Split) | |-------------|-----------------------------|-----------|-------------------------------------------| | MSR-VTT | Video, Audio, Language | 20 | One held-out modality as test | | NYUDv2 | RGB, Depth, Language | 23 | "" | | VGGSound-S | Video, Audio, Language | 310 | "" | | GQA-MSCG | Visual, Linguistic/Primitive| 70-270 | Novel cross-modal or mono-modal pairs [2505.23045] | Performance is primarily measured as classification accuracy, mean Average Precision (mAP) for retrieval, or the harmonic mean (HOS) of accuracy on known and unknowns in open-set classification [2412.18277, 2407.01518, 2407.15376, 2505.23045]. Model selection can be based on splits of the training modalities, leave-one-modality-out validation, or oracle access to test-modality data, with the latter offering an optimistic upper bound (+5–10 percentage points in classification [2412.18277]). ## 4. Empirical Results and Comparative Analysis ### Weak Modality Generalization Empirical evidence shows that DG approaches consistently outperform MML baselines under weak MG. For instance, on MSR-VTT, DG methods reach ~43% average accuracy for the held-out modality, compared to 11–20% for MML; Mixup and EQRM are usually the best performing [2412.18277]. Notably, improved joint embedding (e.g., UniBind vs. ImageBind vs. LanguageBind) does not always correlate with MG performance, signifying that joint alignment is insufficient for robust invariance. ### Strong Modality Generalization Under strong MG (no pretrained perceptors for unseen modalities), accuracies collapse toward random: for MSR-VTT (20-way), average accuracy drops from 41.3% (weak) to 11.0% (strong) [2412.18277]. Nevertheless, performance remains marginally above random, indicating a degree of cross-modal invariant learning even in the absence of aligned embeddings. ### Cross-Modal Retrieval and 3D Retrieval SRCR [2407.15376] surpasses prior methods on open-set 3D cross-modal retrieval by 5–25 percentage points in mAP across modalities and datasets, using structure-aware hypergraphs and center-residual embeddings. CoDAAR [2605.12145] obtains state-of-the-art precision on audio-visual event classification, cross-modal retrieval, and segmentation, leveraging index-level discrete alignment. ### Compositional and Modality-Incomplete Generalization Retrieval-augmented models in VQA achieve 2–3% absolute accuracy gains on open-set compositional splits over strong baselines, notably improving representation consistency across novel linguistic-visual primitive compositions [2505.23045]. Unseen Modality Interaction methods boost classification accuracy by +6.5 percentage points (EPIC-Kitchens-100), with marked robustness to noisy or incomplete modalities [2306.12795]. ## 5. Limitations, Challenges, and Open Directions Several persistent challenges are identified in OSCMG: - **Overfitting to Seen Modalities**: Reliance on joint embeddings or fixed pretraining can cause overfitting to seen correlations, which do not extrapolate to novel modalities [2412.18277]. - **Performance Degradation in Strong MG**: Strong MG remains highly challenging, especially when no shared embedding exists and only self-supervised perceptors are available for unseen modalities [2412.18277, 2605.12145]. - **Evaluation Uncertainty**: High sensitivity to model selection protocols (e.g., +5–10pp swings) complicates reliable assessment, particularly in genuine zero-shot scenarios [2412.18277]. - **Discrete Codebook Management**: As the number of modalities grows, memory and semantic drift issues in separate discrete codebooks may arise [2605.12145]. - **Limited Zero-Shot Extension**: Most methods require at least unlabeled data from new modalities or paired bridge sets; truly "zero-data" generalization to unseen modalities remains unresolved [2412.18277, 2605.12145]. - **Feature Misalignment and Class Novelty**: Encoding structures and alignment protocols may not fully bridge semantic gaps for novel categories or complex cross-modal compositions [2407.15376, 2505.23045]. ## 6. Prospective Research Directions Several key research avenues are proposed in the literature: - **Cross-Modal Invariant Learning**: Developing objectives and architectures for learning features that remain stable under arbitrary modality perturbations, possibly leveraging meta-learning or causal inference [2412.18277]. - **Dynamic Modality Binding**: Enabling flexible online adaptation to novel modalities, perhaps by using weakly paired "bridge" sets, language anchors, or few-shot learning [2412.18277]. - **Task-Aware and Curriculum Self-Supervision**: Integrating weak or curriculum-based supervision into perceptor learning for unseen modalities [2412.18277]. - **Discrete Representation Scalability**: Improving scalability via hierarchical or factorized codebooks and augmenting with lightweight contrastive losses [2605.12145]. - **Efficient Primitive Retrieval and Alignment**: Optimizing retrieval-based cross-modal pretraining, including approximate search and contrastive regularization [2505.23045]. - **Open-Set Structure Refinement**: Adapting hypergraph and memory-bank construction to dynamically evolve with additional modalities and classes [2407.15376]. - **Benchmark Enrichment**: Constructing datasets and evaluation splits that more closely emulate practical OSCMG scenarios, including non-visual modalities (e.g., tactile, EEG) and hybrid class-modality open-set tasks [2412.18277]. ## 7. Significance and Impact Open-set Cross Modal Generalization defines the functional frontier of robust, adaptive, and scalable multimodal AI, as it forces models to abandon the assumption of modality completeness. It exposes the brittleness of naive joint embeddings, the necessity for explicit structure- or alignment-based regularization, and the limits of current self-supervised perceptors. The evolution of OSCMG methodology—across DG, discrete semantic alignment, self-supervised and retrieval-augmented mechanisms—has deep implications for universal representation learning, practical AI deployment, and the prospect of constructing truly modality-agnostic and compositional intelligence. Key works in this direction include [2412.18277], [2605.12145], [2407.01518], [2407.15376], [2306.12795], and [2505.23045]. These contributions collectively provide a formal framework, algorithmic toolkit, benchmark infrastructure, and diagnostic lens on the unsolved challenges of OSCMG. Source: https://www.emergentmind.com/topics/open-set-cross-modal-generalization-oscmg