Entropy-guided Negative Sampling (EGNS)

• The paper introduces EGNS, an entropy-based negative weighting mechanism that dynamically partitions negatives to prioritize informative samples in MMKGC.
• It computes binary entropy from predicted probabilities, categorizing negatives as easy, ambiguous, or hard, and adjusts loss weights accordingly.
• Empirical evaluations on DB15K show that EGNS improves MRR and Hit rates, reducing overfitting and leveraging multimodal complementary signals.

Entropy-guided Negative Sampling (EGNS) is an entropy-based, type-aware negative weighting mechanism introduced within the Mixture of Complementary Modality Experts (MoCME) framework for Multi-modal Knowledge Graph Completion (MMKGC). In that formulation, EGNS does not alter the underlying triple scorer; instead, it computes the model’s predicted probability for each corrupted negative triple, converts that probability into a binary entropy, partitions negatives into easy, ambiguous, and hard groups, and assigns different loss weights accordingly. Its stated purpose is to dynamically prioritize informative and uncertain negatives, thereby improving training effectiveness, reducing overfitting to trivial negatives, and enhancing robustness in multimodal settings characterized by heterogeneous semantic quality and uneven modality distributions (Li, 28 Jul 2025).

1. Conceptual role in multimodal knowledge graph completion

MMKGC aims to uncover hidden world knowledge in multimodal knowledge graphs by leveraging both multimodal and structural entity information. The MoCME paper situates EGNS in a setting where modality distributions vary across entities and where existing methods “typically rely on attention or gate-based fusion mechanisms but overlook complementarity contained in multi-modal data” (Li, 28 Jul 2025). Within that broader architecture, EGNS addresses a distinct training problem: the heterogeneity of corrupted negatives.

The motivation is explicitly tied to the limits of standard negative sampling. In MMKGC, negatives produced by corrupting heads or tails are “not equally useful”: some are too easy and trivially distinguishable from positives, while others are ambiguous, hard, or semantically close to true triples. The paper states that “most negative sampling-based KGE methods tend to assume uniform importance across all negative samples during training,” and argues that this “fails to account for the intrinsic heterogeneity of negative samples in multimodal settings” (Li, 28 Jul 2025). The reported consequence is that treating all negatives equally may cause the model to overfit on trivial or semantically irrelevant samples, waste training signal on uninformative cases, fail to learn fine-grained distinctions, and reduce discriminative power and generalization.

In this sense, EGNS is an uncertainty-aware refinement of negative sampling rather than a new corruption scheme. The negatives are still generated by standard head or tail corruption; the novelty lies in how they are weighted after scoring. A plausible implication is that the method redefines the effective contribution of the corrupted set without changing the candidate generation pipeline.

2. Mathematical formulation and entropy partitioning

In MoCME, EGNS is applied after the construction of joint multimodal entity embeddings. The triple score under modality $m$ is based on RotatE:

$$\mathcal{S}_m(h, r, t) = -\left\| \mathbf{h}_{\text{joint}, m} \circ \mathbf{r}_m - \mathbf{t}_{\text{joint}, m} \right\|$$

where $\circ$ is the Hadamard product in complex space, and a smaller distance means higher plausibility (Li, 28 Jul 2025).

For a negative triple $(h',r,t')$, the model computes

$p = \sigma(\mathcal{S}_m(h', r, t'))$

with $\sigma$ the sigmoid function. EGNS then defines the entropy of that negative sample as

$H(h', r, t') = -p \log p - (1 - p) \log (1 - p).$

This entropy is the key signal for difficulty and informativeness. The paper interprets low entropy as easy, redundant, or semantically distant; mid entropy as ambiguous and near the decision boundary; and high entropy as hard and possibly close to true triples (Li, 28 Jul 2025).

Negatives are partitioned by thresholds $\delta_1$ and $\delta_2$, with $\delta_1 < \delta_2$:

Group	Criterion	Weight
Easy negatives	$$\mathcal{S}_m(h, r, t) = -\left\| \mathbf{h}_{\text{joint}, m} \circ \mathbf{r}_m - \mathbf{t}_{\text{joint}, m} \right\|$$0	$$\mathcal{S}_m(h, r, t) = -\left\| \mathbf{h}_{\text{joint}, m} \circ \mathbf{r}_m - \mathbf{t}_{\text{joint}, m} \right\|$$1
Ambiguous negatives	$$\mathcal{S}_m(h, r, t) = -\left\| \mathbf{h}_{\text{joint}, m} \circ \mathbf{r}_m - \mathbf{t}_{\text{joint}, m} \right\|$$2	$$\mathcal{S}_m(h, r, t) = -\left\| \mathbf{h}_{\text{joint}, m} \circ \mathbf{r}_m - \mathbf{t}_{\text{joint}, m} \right\|$$3
Hard negatives	$$\mathcal{S}_m(h, r, t) = -\left\| \mathbf{h}_{\text{joint}, m} \circ \mathbf{r}_m - \mathbf{t}_{\text{joint}, m} \right\|$$4	$$\mathcal{S}_m(h, r, t) = -\left\| \mathbf{h}_{\text{joint}, m} \circ \mathbf{r}_m - \mathbf{t}_{\text{joint}, m} \right\|$$5

The weighting rule is

$$\mathcal{S}_m(h, r, t) = -\left\| \mathbf{h}_{\text{joint}, m} \circ \mathbf{r}_m - \mathbf{t}_{\text{joint}, m} \right\|$$6

and the generic description imposes the constraint $$\mathcal{S}_m(h, r, t) = -\left\| \mathbf{h}_{\text{joint}, m} \circ \mathbf{r}_m - \mathbf{t}_{\text{joint}, m} \right\|$$7 (Li, 28 Jul 2025).

The experimental hyperparameters are reported as $$\mathcal{S}_m(h, r, t) = -\left\| \mathbf{h}_{\text{joint}, m} \circ \mathbf{r}_m - \mathbf{t}_{\text{joint}, m} \right\|$$8, $$\mathcal{S}_m(h, r, t) = -\left\| \mathbf{h}_{\text{joint}, m} \circ \mathbf{r}_m - \mathbf{t}_{\text{joint}, m} \right\|$$9, $\circ$0, $\circ$1, and $\circ$2 (Li, 28 Jul 2025). The paper therefore contains an apparent inconsistency between the symbolic ordering statement and the concrete values used in implementation, because the listed values satisfy $\circ$3, not the reverse. That inconsistency is explicit in the source description and is part of the technical record rather than an external criticism.

3. Integration with MoCME and the training objective

MoCME consists of two main parts: Complementarity-guided Modality Knowledge Fusion (CMKF) and EGNS. CMKF builds the joint multimodal entity embeddings on which EGNS operates. The paper describes the interaction sequentially: modality encoders produce raw modality embeddings; a Mixture-of-Experts-style module creates multi-view embeddings; intra-modality and inter-modality complementarity weights fuse them; and this yields a joint entity embedding $\circ$4 (Li, 28 Jul 2025).

EGNS is therefore downstream of the representation-fusion stage and upstream of parameter updates. Its coupling to the scorer is tight: the entropy signal is derived from the RotatE-based score computed from the fused embeddings,

$\circ$5

The final loss combines a positive-triple term and an entropy-weighted negative term:

$\circ$6

where $\circ$7 is the positive score, $\circ$8 is the negative score, and $\circ$9 is the set of corrupted negatives (Li, 28 Jul 2025).

The algorithmic sequence reported for EGNS is as follows. For each positive triple $(h',r,t')$0, the model computes multimodal representations of head and tail, fuses modality-specific outputs into joint embeddings, computes the positive score, generates negatives by corrupting head or tail, computes the negative score for each corrupted triple, derives $(h',r,t')$1, computes entropy $(h',r,t')$2, assigns $(h',r,t')$3 using thresholds $(h',r,t')$4, computes the loss, and updates parameters with gradient descent (Li, 28 Jul 2025).

The implementation details reported for the full training setup are batch size $(h',r,t')$5, Adam optimizer, learning rate $(h',r,t')$6, and $(h',r,t')$7 epochs with early stopping on validation MRR (Li, 28 Jul 2025). The role of EGNS within that pipeline is specifically to change how negative loss contributions are weighted, not to change the scoring function itself.

4. Prioritization logic, curriculum interpretation, and robustness claims

The paper characterizes EGNS as dynamically prioritizing “informative,” “uncertain,” “hard,” and “high-entropy” negatives (Li, 28 Jul 2025). In operational terms, the method does not discard negatives. Instead, it generates corrupted negatives, computes entropy for each one, assigns a weight based on the corresponding entropy bucket, and uses these weights inside the contrastive loss.

This detail matters for distinguishing EGNS from more aggressive filtering schemes. In MoCME, easy negatives receive low weight because they are often trivial and contribute little; ambiguous negatives receive intermediate or moderate emphasis; and hard negatives receive the highest emphasis in the conceptual description because they are the most informative and challenging (Li, 28 Jul 2025). At the same time, the same source warns that focusing only on hard negatives can introduce noise or false negatives. The intended behavior is therefore balanced emphasis rather than pure hard-negative mining.

The paper also frames EGNS as aligned with curriculum learning: learning from easier samples and then shifting focus toward more ambiguous and difficult ones (Li, 28 Jul 2025). Because the mechanism itself is implemented through static entropy buckets and weights rather than an explicit time-dependent schedule, this curriculum interpretation is best understood as conceptual. A plausible implication is that the entropy partition acts as a soft curriculum embedded in the loss, with different negative categories contributing asymmetrically throughout training.

The robustness claims are explicit. The entropy-weighted loss is said to improve training effectiveness, avoid gradient saturation from easy negatives, reduce overfitting to trivial samples, and improve generalization robustness (Li, 28 Jul 2025). These are the principal functional claims attached to EGNS in the MMKGC setting.

5. Empirical evidence and sensitivity analyses

The clearest direct evidence for EGNS in MoCME comes from component ablation on DB15K. The full model reports MRR $(h',r,t')$8, Hit@1 $(h',r,t')$9, and Hit@10 $p = \sigma(\mathcal{S}_m(h', r, t'))$0. Removing the entropy-weighted contrastive loss $p = \sigma(\mathcal{S}_m(h', r, t'))$1 yields MRR $p = \sigma(\mathcal{S}_m(h', r, t'))$2, Hit@1 $p = \sigma(\mathcal{S}_m(h', r, t'))$3, and Hit@10 $p = \sigma(\mathcal{S}_m(h', r, t'))$4 (Li, 28 Jul 2025). The reported degradation is therefore about $p = \sigma(\mathcal{S}_m(h', r, t'))$5 MRR and $p = \sigma(\mathcal{S}_m(h', r, t'))$6 Hit@10.

The paper also reports a threshold sensitivity study. The best MRR, $p = \sigma(\mathcal{S}_m(h', r, t'))$7, occurs at $p = \sigma(\mathcal{S}_m(h', r, t'))$8 and $p = \sigma(\mathcal{S}_m(h', r, t'))$9 (Li, 28 Jul 2025). The stated interpretation is that proper separation of easy, ambiguous, and hard negatives is crucial. Specifically, too low $\sigma$0 oversamples trivial negatives; too high $\sigma$1 reduces diversity; too high $\sigma$2 weakens hard-negative contribution; and too low $\sigma$3 overemphasizes hard negatives and may introduce noise.

These results support the paper’s broader empirical conclusion that entropy-based partitioning improves contrastive learning, balances stability and discriminability, and performs better than naive or overly uniform negative treatment (Li, 28 Jul 2025). The evidence is narrowly scoped to the reported MMKGC benchmarks, but within that scope it is presented as a causal contribution of the EGNS component rather than a side effect of the fusion architecture alone.

6. Theoretical framing and related entropy-guided sampling paradigms

Although EGNS in the strict sense is introduced in MoCME, the broader idea that negative sampling changes the effective training objective has a theoretical antecedent in “Unified Interpretation of Softmax Cross-Entropy and Negative Sampling: With Case Study for Knowledge Graph Embedding” (Kamigaito et al., 2021). That paper does not formulate EGNS directly, but it casts both softmax cross-entropy and negative sampling as instances of Bregman divergence minimization and emphasizes the importance of the induced objective distribution. Its key claim is that negative sampling is not merely a computational approximation; it defines a different learning objective with a different implicit target distribution (Kamigaito et al., 2021).

This theoretical perspective is relevant because it suggests that entropy-guided negative sampling can be interpreted not only as sample reweighting but also as a mechanism that modifies the effective entropy or smoothness of training targets. The same paper states that self-adversarial negative sampling relates to label smoothing and that smoothing-style variants increase entropy in the learned distribution (Kamigaito et al., 2021). This suggests a useful interpretive lens for EGNS: entropy-aware weighting may be understood as controlling the sharpness of the learning signal rather than only ranking the difficulty of individual negatives.

Adjacent literatures use entropy guidance in different ways. In multimodal reasoning reward modeling, “Entropy-Guided Data-Efficient Training for Multimodal Reasoning Reward Models” uses entropy as an unsupervised proxy for sample difficulty and label noise, combining entropy-guided data curation with progressive low-to-high entropy training (Yang et al., 2 Feb 2026). There, high-entropy samples are pruned, and the retained low-entropy subset is sorted in ascending entropy each epoch. In contrast, MoCME’s EGNS does not reject negatives; it reweights them within the loss. The difference is substantive: one is selection-based and curriculum-based, the other is weighting-based.

A second adjacent variant appears in “Stop Jostling: Adaptive Negative Sampling Reduces the Marginalization of Low-Resource Language Tokens by Cross-Entropy Loss,” where entropy-guided negative sampling is presented as thresholding or adaptive negative sampling at the logit level (Turumtaev, 30 Jan 2026). That method masks out negatives whose logits fall below a target-relative margin by setting them to $\sigma$4, thereby removing their marginalization updates. It is deterministic masking rather than entropy bucketing. The paper’s stated goal is to prevent rare tokens from being pushed away by large numbers of irrelevant negatives (Turumtaev, 30 Jan 2026). Compared with MoCME’s EGNS, this is a harder form of negative suppression. A plausible implication is that “entropy-guided negative sampling” now names a broader family of uncertainty-aware mechanisms spanning weighting, filtering, and truncation.

7. Distinctions, misconceptions, and limitations

A common misconception is to treat EGNS as synonymous with hard-negative mining. The MoCME formulation does not support that equation. The paper explicitly warns that focusing only on hard negatives can introduce noise or false negatives, and frames the method as balanced emphasis across easy, ambiguous, and hard categories rather than a single-minded preference for the hardest samples (Li, 28 Jul 2025).

A second misconception is that entropy guidance necessarily means sample removal. That is true in some adjacent work, such as entropy-guided curation for multimodal reward models, where high-entropy samples are pruned (Yang et al., 2 Feb 2026), or thresholded masking for language modeling, where sufficiently weak negatives are set to $\sigma$5 (Turumtaev, 30 Jan 2026). In MoCME’s EGNS, however, negatives are not literally discarded; they are assigned different scalar weights in the negative term of the loss (Li, 28 Jul 2025).

The most immediate technical limitation visible in the MoCME description is the inconsistency between the symbolic ordering constraint $\sigma$6 and the reported implementation values $\sigma$7 and $\sigma$8 (Li, 28 Jul 2025). This leaves some ambiguity about whether the conceptual description or the implementation values should be treated as normative.

Beyond that inconsistency, the paper’s own sensitivity study implies that EGNS is hyperparameter-sensitive: a proper separation of entropy regions is described as crucial, while poor settings can oversample trivial negatives, reduce diversity, weaken hard-negative contribution, or introduce noise (Li, 28 Jul 2025). This suggests that the success of EGNS depends not only on the availability of an entropy signal but also on how sharply the entropy scale is partitioned and how category weights are set.

Taken together, the available literature supports a compact synthesis. In MMKGC, EGNS is an uncertainty-aware negative weighting mechanism grounded in binary entropy over predicted negative-triple probabilities (Li, 28 Jul 2025). In broader terms, related work suggests that entropy-guided sampling can be understood as a family of methods for controlling which negatives matter, how much they matter, and how sharply the training objective concentrates probability mass (Kamigaito et al., 2021).