PSCA: Prototype-Based Semantic Consistency Alignment

• The paper introduces a novel Dirichlet energy–driven framework (DESAlign) to enforce semantic consistency in multi-modal knowledge graphs.
• The methodology fuses structural, textual, and visual features with explicit-Euler semantic propagation to interpolate missing modalities and avoid over-smoothing.
• Empirical results demonstrate significant improvements in Hits@1 and MRR over baselines, validating robust performance even with severe missing modalities.

Prototype-Based Semantic Consistency Alignment (PSCA) is an overarching concept in multi-modal entity alignment, focused on ensuring semantic consistency across potentially incomplete or noisy modalities in multi-modal knowledge graphs (MMKGs). The Dirichlet Energy–Driven Semantic Alignment (DESAlign) framework exemplifies a rigorous and scalable PSCA methodology by providing a theoretical justification, architectural strategy, and empirical validation for semantic consistency under challenging missing-modality scenarios (Wang et al., 2024).

1. Theoretical Foundations: Dirichlet Energy and Semantic Consistency

DESAlign employs Dirichlet energy as the core metric for quantifying and enforcing semantic consistency within multi-modal knowledge graphs. Given an undirected MMKG $G=(\mathcal E,\mathcal R,\mathcal A,\mathcal V)$ with $N$ entities and weighted adjacency matrix $A$, the normalized graph Laplacian is defined as

$$L = I - \widetilde{A}, \qquad \widetilde{A} = D^{-1/2} A D^{-1/2}, \quad D_{ii} = \sum_j A_{ij}.$$

The Dirichlet energy for an entity embedding matrix $X \in \mathbb{R}^{N \times d}$ is

$$\mathscr{L}(X) = \mathrm{tr}(X^\top L X) = \tfrac{1}{2} \sum_{i,j} A_{ij} \left\| \frac{X_i}{\sqrt{D_{ii} + 1}} - \frac{X_j}{\sqrt{D_{jj} + 1}} \right\|_2^2.$$

Low Dirichlet energy indicates smooth (consistent) embeddings across graph neighbors and thus high cross-modal semantic consistency. Minimization of $\mathscr{L}(X)$ is equivalent to encouraging entity representations to be similar for adjacent entities, propagating information to compensate for missing modalities.

The evolution of $X$ by gradient flow,

$\dot X(t) = -L X(t),$

with appropriate boundary conditions, produces a semantically smoothed embedding field and provides a natural theoretical mechanism for interpolating absent features.

2. Model Architecture: Multi-Modal Embedding and Semantic Propagation

DESAlign follows a three-stage architecture:

1. Multi-Modal Embedding Generation:
   • Structural features are encoded using Graph Attention Networks (GAT):

   $h^g_i = \mathrm{GAT}(A; W_g, x^g_i)$

• Relation, text, and image modalities ($N$0, $N$1, $N$2) use distinct fully connected layers:

  $N$3

• Modalities are fused using cross-attention weighting (CAW):

  $N$4

  where attention scores $N$5 control per-entity per-modal confidence $N$6.

• Early and late fusion strategies produce joint embeddings:

  $N$7

  $N$8 is used for alignment.

1. Semantic Propagation (Graph Propagation Mechanism): Missing features are interpolated by discretizing Dirichlet gradient flow:

$N$9

with boundary reset for known features $A$0. Iterative application acts as a low-pass filter, imputing embeddings for missing modalities in $A$1 time.

1. Loss and Consistency Objectives:

   • Modality-specific contrastive alignment:

   $A$2

• Bi-directional alignment via cross-entropy on $A$3 and $A$4.

• Dirichlet-energy bounds for each layer ($A$5) to avoid over-smoothing or over-separation:

  $A$6

3. Avoiding Over-Smoothing and Handling Missing Modalities

Semantic inconsistency, particularly due to missing modalities, can lead to overfitting on modality-specific noise or to over-smoothed embeddings that are indistinguishable. DESAlign introduces theoretical guarantees to ensure stable semantics:

• The variance of Dirichlet energy across layers is controlled by the squared singular values of intermediate weight matrices, preventing collapse to trivially smooth solutions ($A$7).

• Interpolation for entities lacking modalities leverages the convexity of $A$8:

$A$9

Practically, this is approximated by the iterative propagation described above.

Corollary 2.1 establishes that the interpolation error is tightly controlled by the Dirichlet energy gap and the Laplacian's maximal eigenvalue, ensuring robustness in settings with high rates of missing modalities.

4. Algorithmic Workflow and Efficiency

The DESAlign algorithm proceeds as follows:

• Initialize parameters and embeddings; normalize adjacency matrices.

• Generate multi-modal embeddings using GAT and CAW-fused transformation layers.

• Apply cross-modal contrastive and alignment losses, plus Dirichlet bounds.

• Update model parameters by backpropagation.

• Execute explicit-Euler-based semantic propagation on both source and target KGs:

  • Multiply by $$L = I - \widetilde{A}, \qquad \widetilde{A} = D^{-1/2} A D^{-1/2}, \quad D_{ii} = \sum_j A_{ij}.$$0, resetting known boundary features.
• Compute averaged pairwise similarity over propagation steps.
• Output the alignment.

This process yields computational efficiency: propagation incurs only 7–9 seconds per epoch on DBP15K/FB-DB, negligible compared to GNN/transformer encoding, and the explicit-Euler method guarantees monotonic Dirichlet energy decrease.

5. Empirical Evaluation: Datasets, Baselines, and Results

DESAlign was evaluated on 60 benchmark splits, covering monolingual datasets (FB15K–DB15K, FB15K–YAGO15K) and bilingual settings (DBP15K$$L = I - \widetilde{A}, \qquad \widetilde{A} = D^{-1/2} A D^{-1/2}, \quad D_{ii} = \sum_j A_{ij}.$$1, DBP15K$$L = I - \widetilde{A}, \qquad \widetilde{A} = D^{-1/2} A D^{-1/2}, \quad D_{ii} = \sum_j A_{ij}.$$2, DBP15K$$L = I - \widetilde{A}, \qquad \widetilde{A} = D^{-1/2} A D^{-1/2}, \quad D_{ii} = \sum_j A_{ij}.$$3). Missing-modality scenarios were generated by varying the proportion of textual and visual attribute availability ($$L = I - \widetilde{A}, \qquad \widetilde{A} = D^{-1/2} A D^{-1/2}, \quad D_{ii} = \sum_j A_{ij}.$$4 from 5% to 60%).

Metrics included Hits@k (H@k) and Mean Reciprocal Rank (MRR). DESAlign outperformed 18 baselines, encompassing non-iterative and iterative approaches (e.g., TransE, GCN-align, MEAformer, MCLEA, BootEA). On FB15K–DB15K with $$L = I - \widetilde{A}, \qquad \widetilde{A} = D^{-1/2} A D^{-1/2}, \quad D_{ii} = \sum_j A_{ij}.$$5, Hits@1 improved from 40.2% (MEAformer) to 49.7%, and MRR from 50.4% to 58.6%. In bilingual alignment, such as DBP15K$$L = I - \widetilde{A}, \qquad \widetilde{A} = D^{-1/2} A D^{-1/2}, \quad D_{ii} = \sum_j A_{ij}.$$6, H@1 improved from 77.0% to 82.6%.

Ablation studies demonstrated that removing any modality or the semantic propagation step degraded results significantly. DESAlign also maintained robust performance with severe missing modalities (e.g., H@1 $$L = I - \widetilde{A}, \qquad \widetilde{A} = D^{-1/2} A D^{-1/2}, \quad D_{ii} = \sum_j A_{ij}.$$7 56% on FB15K–DB15K with only 5% of text attributes), and achieved notable performance even in weakly supervised (1–5% seed alignments) conditions.

6. Comparative Analysis and Discussion

DESAlign provides an end-to-end PSCA solution, grounded in a Dirichlet energy framework and avoiding handcrafted interpolation for missing modalities. Compared to recent multi-modal entity alignment (MMEA) solutions such as MCLEA and MEAformer, DESAlign achieves improvements of 3–10 points on H@1 and 1–8 points on MRR.

The explicit-Euler propagation mechanism, with its linear computational cost and provable control over semantic consistency, is a distinguishing feature. DESAlign demonstrates stability in high missing-modality rates, superior generality, and robustness in few-shot scenarios.

Limitations and directions for future research include optimizing the propagation step (e.g., adaptive iteration counts), integrating richer language/vision encoders beyond bag-of-words and ResNet-152, and extending the Dirichlet energy principle to additional cross-graph tasks such as relation alignment or ontology matching (Wang et al., 2024).

7. Significance and Extensions

The Dirichlet energy-based approach to PSCA, instantiated by DESAlign, formalizes the notion of semantic smoothness in multi-modal knowledge representation. By linking classical graph-theoretic energy minimization to neural embedding fusion, DESAlign offers a theoretically sound and empirically validated solution for robust entity alignment in MMKGs. A plausible implication is the extension of the Dirichlet framework to broader heterogeneous graph matching and multi-relational data integration tasks.