Modality-Enhanced Representations (MER)

• Modality-Enhanced Representations (MER) are frameworks that disentangle shared and modality-specific features to enable robust multimodal inference.
• They utilize architectural innovations like cross-attention, low-rank projection, and soft-pruning to handle missing or conflicting modalities effectively.
• Empirical evaluations in fields such as 3D scene rendering and medical analysis demonstrate improved metrics like PSNR, Dice scores, and survival prediction robustness.

Modality-Enhanced Representations (MER) describe a class of learning frameworks, architectures, and mathematical formulations that explicitly model, disentangle, and selectively utilize both shared and modality-specific information within multimodal data. MER provides a unified representation that enables robust inference, principled handling of missing or conflicting modalities, and compact yet expressive scene or data modeling across settings from computer vision to medical analysis. State-of-the-art MER advances incorporate modality-specific mechanisms (e.g., per-modality feature vectors, indicators, explicit algebraic decompositions) and are empirically validated to improve efficiency, fidelity, and robustness under both complete and incomplete modality regimes (Gu et al., 15 Jul 2025, Kim et al., 27 Mar 2026, Konwer et al., 2023).

1. Mathematical Foundations of Modality-Enhanced Representations

MER frameworks are grounded in the explicit mathematical separation of modality-shared and modality-specific components. For a generic set of modalities $\mathcal{M}$, a data point or scene is represented by a collection of learned features, typically one per modality plus at least one shared or contextualized component.

A canonical instantiation is provided in MUST (Modality-Specific representation-aware Transformer) for survival prediction with multimodal medical data (Kim et al., 27 Mar 2026). Each modality $m \in \mathcal{M}$ yields a global embedding $g^{m} \in \mathbb{R}^D$; bidirectional cross-attention produces contextualized vectors $c^{m\to n}$ that capture information from other modalities. A learned low-rank projector $P$ enables algebraic decomposition: $$\tilde{c}^{m\to n} = P\,c^{m\to n} \qquad \hat{u}^{m} = (I-P)u^{m}$$ Here, $\tilde{c}^{m\to n}$ is the component of $m$ inferable from $n$ (shared), while $\hat{u}^m$ is the strictly modality-specific (non-inferable) residual. The total representation for downstream tasks—termed the Modality-Enhanced Representation (MER)—is assembled as

$m \in \mathcal{M}$0

Analogous decompositions manifest in scene representations such as MMOne (Gu et al., 15 Jul 2025), where each scene primitive carries a modality-specific feature vector and indicator/opacity $m \in \mathcal{M}$1 for each modality, and in meta-learned fusion models (Konwer et al., 2023).

MER objectives typically couple reconstruction/segmentation/prediction losses with constraints enforcing decomposition, shared-consistency, and (where relevant) orthogonality between modality-shared and specific subspaces.

2. MER Architectural Mechanisms

MER is operationalized through architectural innovations that facilitate explicit, learnable modeling of each modality's contribution, interaction, and granularity:

• Per-Primitive Modality-Specific Features and Indicators: In MMOne (Gu et al., 15 Jul 2025), each 3D Gaussian $m \in \mathcal{M}$2 has a feature $m \in \mathcal{M}$3 and a per-modality opacity $m \in \mathcal{M}$4, supporting independent rendering and optimization for each modality:

$m \in \mathcal{M}$5

where transmittance $m \in \mathcal{M}$6 and $m \in \mathcal{M}$7 are modality-dependent, permitting specialization to diverse modality properties (e.g., spatial, semantic, spectral differences).

• Modality Modeling Modules: Dedicated subnetworks or adapters (e.g., residual adapters in MUST (Kim et al., 27 Mar 2026)) compute the modality-specific subspaces. Shared or contextualized features are produced via attention, fusion, or projection into a low-dimensional subspace.
• Fusion and Decomposition Layers: Channel-attention fusion (e.g., in Swin-UNETR backbone for brain tumor segmentation (Konwer et al., 2023)) and algebraic decomposition with orthogonality constraints (e.g., MUST (Kim et al., 27 Mar 2026)) ensure that the aggregate representation is both expressive and interpretable across arbitrarily available modalities.
• Soft-Pruning and Gradient-Driven Decomposition: MMOne introduces modality-wise soft pruning (i.e., $m \in \mathcal{M}$8 below threshold) and gradient-difference-induced Gaussian splitting, decoupling Gaussians in the presence of cross-modality conflict (Gu et al., 15 Jul 2025).

3. Joint Optimization and Loss Design

MER models optimize a combination of per-modality task objectives and decomposition constraints. A common overall loss schema is: $m \in \mathcal{M}$9 where each $g^{m} \in \mathbb{R}^D$0 is e.g., $g^{m} \in \mathbb{R}^D$1 for RGB, $g^{m} \in \mathbb{R}^D$2 plus TV regularization for thermal, SNR or Dice/IoU for segmentation, and $g^{m} \in \mathbb{R}^D$3 are algebraic/orthogonality constraints (see (Kim et al., 27 Mar 2026)). Adversarial auxiliary branches (e.g., modality presence discriminators (Konwer et al., 2023)) are also employed to regularize the alignment of partial-modality and full-modality representations.

Bilevel meta-learning frameworks (Konwer et al., 2023) enable models to generalize MER to arbitrary subsets of available modalities during both training and inference, using gradient-based adaptation and adversarial regularization to avoid overfitting to complete data and to ensure robustness to missing modalities.

4. Handling Missing and Conflicting Modalities

A central use case for MER is robust prediction with missing, partial, or conflicting modalities. Specific strategies demonstrated in the literature include:

• Conditional Latent Diffusion for Modality Imputation: MUST (Kim et al., 27 Mar 2026) employs conditional latent diffusion models (LDM) to stochastically generate the missing modality-specific residual, conditional on the shared representation. The shared subspace permits deterministic recovery of information inferable from observed modalities, while strictly non-shared content is injected by the LDM.
• Meta-Learning for Partial Modalities: The framework of (Konwer et al., 2023) frames missing-modality patterns as tasks in a meta-learning regime, adapting parameters through inner-loop updates on partial data and optimizing recovery of the full-modality representation via outer-loop gradients and adversarial constraints.
• Decomposition and Pruning: In MMOne, modality-specific conflicts (as measured by gradient differences) trigger the creation of modality-specialized primitives, while low-utility primitives are soft-pruned independently for each modality (Gu et al., 15 Jul 2025). This reduces cross-modality interference and allows fine-grained resource allocation per modality.

A plausible implication is that the decomposition-based approach to missing modalities enables more precise quantification of what information is inherently irrecoverable, versus what may be inferred or transferred across modalities via shared components.

5. Empirical Evaluation and Comparative Performance

MER frameworks establish empirical SOTA across representative domains and tasks:

Study	Domain	Modalities	Key SOTA Metrics
MMOne (Gu et al., 15 Jul 2025)	3D scene rendering	RGB, Thermal, Language	+0.5 dB PSNR RGB (24.89), +0.4 dB PSNR thermal (25.89), +1.3% mIoU open-vocab segmentation
MUST (Kim et al., 27 Mar 2026)	Survival prediction (TCGA)	Pathology, Genomics	C-index 0.742 (overall), drop of –3.5% (missing genomics), SOTA robustness
Meta-MER (Konwer et al., 2023)	Brain tumor segmentation	Multiple MRI (partial/missing)	+0.9%/2.0%/1.7% Dice (three tumor classes); drop of only ∼0.3% with 60% partial modality

Detailed ablation studies confirm that each MER component—modality modeling, soft pruning, algebraic decomposition, adversarial regularization—contributes monotonically to these gains, and that explicit handling of modality disparities is essential for scalability and compactness (Gu et al., 15 Jul 2025, Konwer et al., 2023).

6. Addressing Modality Disparities: Property and Granularity

MER systems are designed to resolve two fundamental modality disparities:

• Property Disparity: Modalities differ in signal type (e.g., visual, thermal, semantic), dimensionality, and semantic content. MMOne employs per-modality feature fields $g^{m} \in \mathbb{R}^D$4 (of appropriate dimensionality) and MUST decomposes representation spaces algebraically, ensuring that each modality's irreducible content is separately modeled (Gu et al., 15 Jul 2025, Kim et al., 27 Mar 2026).
• Granularity Disparity: Modalities may vary in spatial resolution or semantic granularity (e.g., RGB is fine-grained, thermal is coarse-grained). Per-modality opacities $g^{m} \in \mathbb{R}^D$5 in MMOne allow differential spatial support, while MUST's low-rank projectors control cross-modal contextualization at subspace levels. Modality-specific pruning and gradient-based decomposition further afford fine-grained control over per-modality expressivity.

A plausible implication is that ignoring these disparities—as in prior fusion or joint-distribution models—leads to degraded representational fidelity, inefficient use of learnable primitives, and loss of robustness to missing data.

7. Limitations and Future Directions

Current MER approaches are subject to several limitations:

• The majority of methods assume static scenes or data, excluding dynamic geometry or temporally-varying modalities (Gu et al., 15 Jul 2025).
• Many frameworks rely on known camera pose, modality presence, or correspondences; joint pose/registration remains to be tightly integrated (Gu et al., 15 Jul 2025).
• Computational overhead, particularly for diffusion-based imputation (Kim et al., 27 Mar 2026), can affect real-time applicability.
• Sensitivity to hyperparameters such as decomposition thresholds, subspace rank, and adversarial weights is present but generally mild.

Anticipated directions include time-varying and dynamic MER, integration with end-to-end pose and registration estimation, and further formalization of decomposition criteria and uncertainty quantification.

---

MER defines a technically rigorous paradigm for explicit, disentangled, and robust multimodal representation, validated across domains such as high-fidelity 3D scene understanding, precision oncology, and partial-modality medical segmentation. The development and refinement of MER principles continue to drive empirical advances and theoretical understanding in multimodal machine learning.