MoCME: Complementary Modality Experts

• The paper introduces a novel framework that uses MI-based complementarity fusion instead of standard attention weighting for multimodal knowledge graph completion.
• It employs expert networks to generate multi-view embeddings from varied modalities and aggregates them both within and across modalities.
• Empirical results on multiple MMKGC benchmarks demonstrate that cross-modal complementarity and entropy-guided negative sampling significantly boost performance.

Searching arXiv for the primary MoCME paper and closely related modality-expert references. Mixture of Complementary Modality Experts (MoCME) denotes a multimodal representation-learning framework in which multiple expert branches are used to extract semantically distinct views of each modality and then fused according to estimated complementarity rather than simple salience or confidence. The term is introduced explicitly for multi-modal knowledge graph completion (MMKGC), where modality distributions vary across entities and where available modalities may be incomplete, noisy, or semantically redundant. In that formulation, MoCME combines a Complementarity-guided Modality Knowledge Fusion (CMKF) module with an Entropy-guided Negative Sampling (EGNS) mechanism, using RotatE as the triple-scoring backbone (Li, 28 Jul 2025).

1. Definition and problem setting

In the MMKGC setting, the multimodal knowledge graph is written as

$$\mathcal{MG} = (\mathcal{E}, \mathcal{R}, \mathcal{T}, \mathcal{M}),$$

where $\mathcal{E}$ is the entity set, $\mathcal{R}$ the relation set, $\mathcal{T}$ the structural triple set, and $\mathcal{M}$ the set of modalities. The task is to predict missing links such as $(h,r,?)$ or $(?,r,t)$. The central difficulty is modality imbalance: entities do not share the same modality coverage, and even available modalities may differ in informativeness or reliability. MoCME addresses this by preferring complementary information both among multiple views inside a modality and across different modalities, rather than relying only on attention or gate-based importance weighting (Li, 28 Jul 2025).

The framework treats image, text, numeric values, audio, video, and structure as modalities. For an entity $e$ and modality $m$, raw modality information is denoted

$$\mathcal{I}_m(e) = \{\mathcal{I}_{e,m}^{(1)}, \mathcal{I}_{e,m}^{(2)}, \dots\}.$$

Visual inputs are encoded with VGG16, textual inputs with BERT, numeric values are tokenized and encoded by BERT, and structural triples can also be serialized and encoded by BERT. At the same time, the final structural embedding $\mathcal{E}$0 is treated specially: it is initialized as a learnable parameter and optimized jointly with the KG objective, serving as a backbone modality (Li, 28 Jul 2025).

This formulation distinguishes MoCME from approaches that treat multimodal fusion chiefly as a problem of relative weighting. In MoCME, complementarity is operationalized as the contribution of information that is less redundant with other available views or modalities. This suggests that the framework is oriented toward robustness under heterogeneous and uneven modality availability, rather than toward dense all-modal fusion alone.

2. Complementarity-guided Modality Knowledge Fusion

CMKF is the representational core of MoCME. For each modality $\mathcal{E}$1, a frozen pre-trained encoder $\mathcal{E}$2 produces a raw representation

$\mathcal{E}$3

which is then projected into a common embedding space: $\mathcal{E}$4 where $\mathcal{E}$5 is a two-layer MLP with ReLU (Li, 28 Jul 2025).

Each modality $\mathcal{E}$6 is then processed by $\mathcal{E}$7 modality-specific expert networks

$\mathcal{E}$8

which generate multi-view embeddings

$\mathcal{E}$9

These are the experts in MoCME: parallel modality-specific branches intended to capture different semantic subspaces within the same modality (Li, 28 Jul 2025).

Complementarity is quantified with Mutual Information Neural Estimator (MINE). The framework interprets high mutual information as redundancy and low mutual information as stronger complementarity. Intra-modal fusion assigns larger weights to views that are less redundant with the others: $\mathcal{R}$0 with

$\mathcal{R}$1

The same principle is then applied across modalities: $\mathcal{R}$2 where

$\mathcal{R}$3

The final entity representation used for knowledge graph completion is therefore hierarchical: raw modality encoding, shared-space projection, multi-view expert generation, intra-modal complementarity fusion, and inter-modal complementarity fusion (Li, 28 Jul 2025).

A notable property of this design is that “mixture” is not implemented through classical sparse expert routing. There is no explicit token router, no top-$\mathcal{R}$4 dispatch, and no residual expert arbitration in the transformer-MoE sense. Instead, routing is implicit in complementarity-weighted aggregation. This suggests that MoCME is structurally closer to expertized multi-view generation followed by information-theoretic fusion than to sparse MoE transformers.

3. Entropy-guided Negative Sampling and learning objective

MoCME couples CMKF with RotatE scoring and an entropy-weighted negative-sampling scheme. After obtaining the fused entity representations, the triple score is written as

$\mathcal{R}$5

and the algorithm box also gives the simplified joint form

$\mathcal{R}$6

The paper identifies this as a RotatE-style score over the final fused entity embeddings (Li, 28 Jul 2025).

For a negative triple $\mathcal{R}$7, MoCME computes

$\mathcal{R}$8

and then binary entropy

$\mathcal{R}$9

Negatives are partitioned by thresholds $\mathcal{T}$0 into easy, ambiguous, and hard cases, and then reweighted by

$\mathcal{T}$1

The implementation uses

$\mathcal{T}$2

and

$\mathcal{T}$3

The methodology text states $\mathcal{T}$4, but the reported values instead prioritize ambiguous negatives most, followed by hard negatives, then easy negatives. That discrepancy is present in the paper itself (Li, 28 Jul 2025).

The final objective is a weighted logistic negative-sampling loss: $\mathcal{T}$5 Optimization uses Adam with learning rate $\mathcal{T}$6, batch size $\mathcal{T}$7, $\mathcal{T}$8 epochs, Xavier uniform initialization, and early stopping based on validation MRR. The embedding dimension is $\mathcal{T}$9, and the pre-trained modality encoders are frozen (Li, 28 Jul 2025).

4. Relation to the broader modality-expert literature

The MoCME formulation sits within a broader family of modality-expert systems, but it differs from many of them in where complementarity is expressed and how experts are selected. Related works often organize experts around modality specialization, conditional routing, or explicit missing-modality handling, but only some of them make complementarity itself the primary fusion criterion.

Framework	Core expert mechanism	Relation to MoCME
MedMoE (Chopra et al., 10 Jun 2025)	Report-conditioned hard routing over modality-sensitive multi-scale visual experts	Specialization without explicit multi-expert complementarity
MoME for cancer survival (Xiong et al., 2024)	Sample-level hard selection among experts with different multimodal fusion biases	Complementarity emerges through fusion-strength diversity
Brain-lesion MoME (Zhang et al., 2024)	One expert per MRI modality with voxel-wise hierarchical gating	Modality-specialized universal segmentation rather than explicit complementarity
ST-MoME (Lee et al., 24 Jun 2026)	One expert per MRI modality with voxel-wise, timestep-dependent soft gating under missingness	Close to a concrete complementary-modality-expert design
VLMo (Bao et al., 2021)	Shared self-attention with modality-specific FFN experts selected by modality and depth	Modality-specialized experts with architectural, not learned, mixture
RingMoE (Bi et al., 4 Apr 2025)	Modal-specialized, collaborative, and shared experts inside a hierarchical sparse MoE	Very close to a structured complementary-modality-expert architecture
MoST (Lou et al., 15 Jan 2026)	Hard modality-aware partition of speech and text experts plus shared expert	Complementarity implemented as specialization plus shared transfer path

MoCME differs most clearly from sparse transformer MoEs in that it does not use a learned expert router over tokens. It also differs from modality-specialized systems that stop at expert weighting, because its central operation is complementarity-guided fusion via mutual information. Relative to systems such as ST-MoME or RingMoE, MoCME is less about routing modalities through expert pools and more about generating multi-view expert embeddings and suppressing redundancy during fusion (Li, 28 Jul 2025).

5. Empirical performance and ablation findings

MoCME is evaluated on five MMKGC benchmarks: MKG-W, MKG-Y, DB15K, TIVA, and KVC16K. The reported best results are as follows (Li, 28 Jul 2025).

Dataset	Best reported metrics
MKG-W	MRR 37.79, Hit@1 30.81
MKG-Y	MRR 40.37, Hit@1 36.21
DB15K	MRR 39.62, Hit@1 29.71, Hit@10 55.36
KVC16K	MRR 18.32, Hit@1 11.23, Hit@10 30.76
TIVA	MRR 94.58, Hit@1 93.32, Hit@10 95.61

The gains are larger on richer multimodal datasets, which the paper interprets as evidence that complementarity modeling becomes more valuable as modality diversity increases. On DB15K, ablations show that removing image, text, or numeric modality lowers MRR to $\mathcal{M}$0, $\mathcal{M}$1, and $\mathcal{M}$2, respectively, from the full $\mathcal{M}$3. Removing intra-modal complementarity reduces MRR to $\mathcal{M}$4, removing inter-modal complementarity reduces it to $\mathcal{M}$5, and removing the entropy-weighted loss reduces it to $\mathcal{M}$6. The largest single drop comes from removing inter-modal complementarity, indicating that cross-modality complementarity is the strongest contributor among the ablated components (Li, 28 Jul 2025).

Fusion-strategy ablations on DB15K further distinguish MoCME from simpler multimodal aggregation. Concatenation gives MRR $\mathcal{M}$7, product fusion $\mathcal{M}$8, gate-based fusion $\mathcal{M}$9, and MoCME $(h,r,?)$0. Architecture ablations show MLP at $(h,r,?)$1, linear at $(h,r,?)$2, and CMoE at $(h,r,?)$3. The paper also reports that performance peaks when the number of experts is roughly aligned with the number of modalities, especially on DB15K, and that the best entropy thresholds are $(h,r,?)$4 and $(h,r,?)$5 (Li, 28 Jul 2025).

6. Interpretation, misconceptions, and limitations

A common misconception is to treat MoCME as a conventional sparse MoE architecture. In the strict transformer-MoE sense, that is inaccurate. The framework does not introduce token-level top-$(h,r,?)$6 routing, load-balancing losses over expert traffic, or a learned gate that dispatches tokens to a sparse subset of experts. Its “mixture” is instead realized through multi-view expert generation followed by complementarity-weighted aggregation using mutual information (Li, 28 Jul 2025).

A second misconception is to equate complementarity with attention-based importance. MoCME explicitly rejects that equivalence. Within the framework, a modality or view is not favored simply because it appears salient; it is favored when it contributes information that is less redundant with other available information. This suggests that complementarity is treated as inverse redundancy rather than as a disentangled shared-private latent decomposition.

The paper also leaves several mechanisms absent or underspecified. It does not provide explicit formulas for view augmentation, cross-view consistency objectives, residual fusion, or a classical routing network. MINE is invoked to estimate mutual information, but the training details of the estimator are not elaborated. The relation between “per-modality” and “joint” RotatE notation is somewhat inconsistent, and the monotonic weighting condition written for EGNS conflicts with the reported $(h,r,?)$7 values. The authors identify future directions in scaling to larger industrial MMKGs, extending to temporal and spatial modalities, and pursuing more advanced modality alignment under limited labels (Li, 28 Jul 2025).

In condensed form, MoCME can be characterized as

$(h,r,?)$8

Within the broader modality-expert literature, its distinctive contribution is to make complementarity itself—not merely modality specialization or adaptive weighting—the organizing principle of multimodal fusion (Li, 28 Jul 2025).