DESAlign: Dirichlet Energy-Driven Alignment

• Dirichlet Energy-Driven Semantic Alignment (DESAlign) is a framework for aligning entities in multi-modal knowledge graphs by minimizing Dirichlet energy to handle missing modalities.
• It employs explicit Euler propagation and energy constraints to prevent over-smoothing and ensure stable, semantically consistent feature propagation.
• Empirical results demonstrate DESAlign’s superiority over existing methods, showing significant improvements in Hits@1 and MRR under varied modality missingness conditions.

Dirichlet Energy-Driven Semantic Alignment (DESAlign) is a framework for robust multi-modal entity alignment in multi-modal knowledge graphs (MMKGs) that addresses the core challenge of semantic inconsistency due to missing modal attributes. By unifying the learning process and inference under a Dirichlet energy principle, DESAlign minimizes the distortion associated with missing modalities and effectively prevents over-smoothing and performance collapse, thereby advancing the state of the art in multi-modal entity alignment (Wang et al., 2024).

1. Motivation and Problem Context

Multi-Modal Entity Alignment (MMEA) in MMKGs seeks to identify semantically identical entities across knowledge graphs using information from structure, textual, visual, and other attributes. In practice, entities frequently lack one or more modalities (e.g., missing images or sparse text), leading to semantic inconsistency: the data representations are misaligned or incomplete between knowledge graphs. Conventional solutions interpolate or impute missing attributes using simple heuristics, such as sample-of-mean or Gaussian noise, thereby injecting "modality noise" that distorts semantics and triggers over-smoothing (embedding collapse) or unstable performance as missingness increases.

DESAlign proposes a unifying theoretical foundation: semantic smoothness is quantified as Dirichlet energy on the graph, and interpolation of missing modalities should correspond to Dirichlet energy minimization—yielding provably optimal, semantically consistent feature propagation while constraining representation degeneration. This approach replaces ad hoc imputation with principled energy-constrained learning and propagation.

2. Theoretical Foundations

Let $G=(E,R,A,V)$ denote an undirected multi-modal KG where $A$ is the adjacency, and $\tilde{A}$ its normalized form; $\Delta=I-\tilde{A}$ is the Laplacian. An entity feature matrix $X\in\mathbb{R}^{N\times d}$ (embedding $f:E\to\mathbb{R}^d$ row-wise) defines Dirichlet energy: $$\mathcal{L}(X) = \operatorname{trace}(X^\top \Delta X) = \frac{1}{2} \sum_{i,j} A_{ij} \|X_i/\sqrt{D_{ii}+1} - X_j/\sqrt{D_{jj}+1}\|^2$$ where $D$ is the degree matrix.

Given a partition into consistent ($c$), inconsistent-type-1 ($o_1$), and missing ($A$0) entities, interpolation of $A$1 minimizing $A$2 (subject to fixed $A$3, $A$4) yields the Euler–Lagrange solution: $A$5 Direct inversion is cubic in cost. Instead, DESAlign uses explicit Euler propagation: $A$6 For $A$7, this reduces to $A$8. After each step, $A$9 is reset to original values to enforce the boundary. As $\tilde{A}$0, the process converges to the optimum.

3. Algorithmic Framework

3.1 Multi-Modal Semantic Learning

The encoder jointly integrates modality-specific and graph-structural embeddings:

• Structure: 2-layer GAT (2 heads), output dimension 300.
• Modalities: FC layers with input/output sizes—relations (BoW 1000→300), text (BoW 1000→300), vision (ResNet-152 2048→300).
• Cross-modal Attention Weighted (CAW) Transformer: Computes attention $\tilde{A}$1 and confidence $\tilde{A}$2 over modality $\tilde{A}$3 for each entity, forming early fusion embeddings $\tilde{A}$4 and late fusion $\tilde{A}$5 by concatenation with attention-weighted features.

3.2 Dirichlet Energy Constraints

Hidden representations $\tilde{A}$6 pass through linear layers $\tilde{A}$7. For each layer: $\tilde{A}$8 where $\tilde{A}$9, $\Delta=I-\tilde{A}$0 are squared minimal/maximal singular values; collapse to zero triggers over-smoothing. DESAlign enforces constraints: $\Delta=I-\tilde{A}$1 with $\Delta=I-\tilde{A}$2 as hyperparameters, limiting collapse or over-separation.

3.3 Semantic Propagation

For inference, semantic propagation uses the boundary-conditioned explicit Euler scheme for missing modalities. Embeddings $\Delta=I-\tilde{A}$3 from source/target graphs undergo propagation, with indices $\Delta=I-\tilde{A}$4 (entities with modalities) and $\Delta=I-\tilde{A}$5 (entities missing modality $\Delta=I-\tilde{A}$6). At each step:

1. $\Delta=I-\tilde{A}$7
2. $\Delta=I-\tilde{A}$8

After $\Delta=I-\tilde{A}$9 steps, pairwise cosine similarities over $X\in\mathbb{R}^{N\times d}$0 are averaged to produce alignment scores $X\in\mathbb{R}^{N\times d}$1.

3.4 Loss Functions and Optimization

• Task Losses: Cross-entropy (contrastive) losses on early/late fusions, $X\in\mathbb{R}^{N\times d}$2.
• Intra-modal Losses: Contrastive losses per modality $X\in\mathbb{R}^{N\times d}$3.
• Confidence Weighting: For a pair $X\in\mathbb{R}^{N\times d}$4, confidence $X\in\mathbb{R}^{N\times d}$5 lowers the impact of noisy or uncertain modalities.
• AdamW optimizer, learning-rate warmup (15%), batch size 3500, early stopping, 1000 total epochs (split normal/iterative).

4. Empirical Evaluation

4.1 Datasets and Experimental Settings

• Monolingual: FB15K–DB15K and FB15K–YAGO15K, seed-alignment ratios $X\in\mathbb{R}^{N\times d}$6.
• Bilingual: DBP15K FR–EN, JA–EN, ZH–EN, each $X\in\mathbb{R}^{N\times d}$7.
• Simulated Missing Modalities: Text/image ratios $X\in\mathbb{R}^{N\times d}$8, $X\in\mathbb{R}^{N\times d}$9 from 5% to 60%.

4.2 Metrics

• Hits@k ($f:E\to\mathbb{R}^d$0)
• Mean reciprocal rank (MRR)

4.3 Results and Comparative Analysis

DESAlign outperforms 18 non-iterative and several iterative baselines:

• On DBP15K_FR-EN (non-iterative): DESAlign Hits@1 = 82.6%, MEAformer = 77.0%.
• Across all splits, DESAlign improves Hits@1 by 4–12 points, MRR by 2–8 points over non-iteratives, and by 1–4 (Hits@1), 1–3 (MRR) over iterative baselines.
• Under weak supervision ($f:E\to\mathbb{R}^d$1), DESAlign achieves Hits@1 $f:E\to\mathbb{R}^d$2 (DBP15K_FR-EN), consistently exceeding baselines.

4.4 Robustness and Ablation

• Under varying $f:E\to\mathbb{R}^d$3 (5→60%): baselines’ MRR declines ($f:E\to\mathbb{R}^d$4), DESAlign holds at $f:E\to\mathbb{R}^d$5.
• For $f:E\to\mathbb{R}^d$6: baselines 75–79% Hits@1, DESAlign 80–88%, stable even at 95% missing.
• Removing text modality causes the largest drop (–7 Hits@1).
• Eliminating $f:E\to\mathbb{R}^d$7 or $f:E\to\mathbb{R}^d$8 reduces Hits@1 by 3–5.
• Skipping Semantic Propagation results in performance loss rivaling the absence of an entire modality.

4.5 Efficiency

Semantic Propagation for DBP15K requires 7 seconds, FB-DB 9 seconds (per iteration cost $f:E\to\mathbb{R}^d$9). Computation involves only sparse matrix multiplies, suitable for CPU pre-processing. Encoder resource use is comparable to MEAformer.

5. Over-Smoothing, Noise, and Model Stability

DESAlign’s Dirichlet energy constraints across GNN layers mitigate eigendirection collapse, preventing over-smoothing even in scenarios with extreme modality missingness. Cross-modal attention and confidence-based weighting further insulate the model from noisy alignments. Because Semantic Propagation depends solely on graph structure and high-confidence observed modalities, it introduces no additional trainable parameters and does not cause over-fitting.

6. Limitations, Extensions, and Future Work

While robust defaults for hyperparameters ($$\mathcal{L}(X) = \operatorname{trace}(X^\top \Delta X) = \frac{1}{2} \sum_{i,j} A_{ij} \|X_i/\sqrt{D_{ii}+1} - X_j/\sqrt{D_{jj}+1}\|^2$$0, $$\mathcal{L}(X) = \operatorname{trace}(X^\top \Delta X) = \frac{1}{2} \sum_{i,j} A_{ij} \|X_i/\sqrt{D_{ii}+1} - X_j/\sqrt{D_{jj}+1}\|^2$$1, propagation steps $$\mathcal{L}(X) = \operatorname{trace}(X^\top \Delta X) = \frac{1}{2} \sum_{i,j} A_{ij} \|X_i/\sqrt{D_{ii}+1} - X_j/\sqrt{D_{jj}+1}\|^2$$2) are effective, auto-tuning could improve flexibility. The explicit Euler propagation may require numerous iterations for large graphs; possible acceleration techniques include Chebyshev polynomials or conjugate-gradient solvers. Potential extensions include adaptation to streaming or dynamic graphs with time-varying Laplacians and incorporation of new modalities (e.g., audio, video) or advanced pretrained encoders like CLIP within the Dirichlet energy optimization framework (Wang et al., 2024).

Overall, DESAlign provides a principled methodology unifying Dirichlet energy-constrained learning and explicit, theoretically-grounded propagation for entity alignment in MMKGs, yielding robust, consistent performance under real-world, modality-heterogeneous settings.