Multi-Perspective Evidence Synthesis and Reasoning for Unsupervised Multimodal Entity Linking

Abstract: Multimodal Entity Linking (MEL) is a fundamental task in data management that maps ambiguous mentions with diverse modalities to the multimodal entities in a knowledge base. However, most existing MEL approaches primarily focus on optimizing instance-centric features and evidence, leaving broader forms of evidence and their intricate interdependencies insufficiently explored. Motivated by the observation that human expert decision-making process relies on multi-perspective judgment, in this work, we propose MSR-MEL, a Multi-perspective Evidence Synthesis and Reasoning framework with LLMs for unsupervised MEL. Specifically, we adopt a two-stage framework: (1) Offline Multi-Perspective Evidence Synthesis constructs a comprehensive set of evidence. This includes instance-centric evidence capturing the instance-centric multimodal information of mentions and entities, group-level evidence that aggregates neighborhood information, lexical evidence based on string overlap ratio, and statistical evidence based on simple summary statistics. A core contribution of our framework is the synthesis of group-level evidence, which effectively aggregates vital neighborhood information by graph. We first construct LLM-enhanced contextualized graphs. Subsequently, different modalities are jointly aligned through an asymmetric teacher-student graph neural network. (2) Online Multi-Perspective Evidence Reasoning leverages the power of LLM as a reasoning module to analyze the correlation and semantics of the multi-perspective evidence to induce an effective ranking strategy for accurate entity linking without supervision. Extensive experiments on widely used MEL benchmarks demonstrate that MSR-MEL consistently outperforms state-of-the-art unsupervised methods. The source code of this paper was available at: https://anonymous.4open.science/r/MSR-MEL-C21E/.

• The paper introduces MSR-MEL, a two-stage framework for unsupervised multimodal entity linking (MEL) that integrates four types of complementary evidence: instance-centric, group-level, lexical, and statistical in its solution.
• Instance-centric and group-level evidence: Using multimodal termination embedding representations, grouping includes a modal-aware filter from unsupervised learning.
• Statistical and lexical evidence: statistical evidence sharpens the focus on modal reliability, and lexical provides a robust anchor point against typographical errors.
• The paper suggests the widest application of this methodology is in Usupervised MEL scenarios where labeled data are missing.

This paper presents MSR-MEL, a two-stage framework for unsupervised Multimodal Entity Linking (MEL) that replaces purely instance-centric similarity matching with explicit synthesis of, and reasoning over, four complementary evidence types. The work is motivated by a data-quality observation: widely used MEL benchmarks exhibit substantial modality incompleteness, with entity image ratios of only 67.26% on WikiMEL, 57.41% on RichpediaMEL, and 50.16% on WikiDiverse. Under such conditions, the authors argue, single-cue instance-level matching is unreliable, and an expert-like decision process should aggregate evidence across multiple perspectives and reason about its consistency. The offline stage synthesizes instance-centric, group-level, lexical, and statistical evidence; the online stage uses an LLM to induce an interpretable tree-structured re-ranking strategy over this evidence without any supervision (2604.20283).

Existing MEL approaches divide into supervised methods (DWE, M³EL, UniMEL, KGMEL) that learn scoring functions from annotated mention–entity pairs, and unsupervised methods that rank entities by similarity of pre-trained representations (BERT, CLIP zero-shot baselines, OpenMEL). The authors' central critique is that both families optimize instance-centric features and do not directly fuse neighborhood information or exploit broader evidence forms. MSR-MEL is positioned as the first framework in their comparison to be simultaneously label-free and to integrate all four evidence types (instance-centric, group-level, lexical, statistical), with the LLM serving as context enhancer plus decision inducer rather than merely a scorer.

Two challenges structure the design: extracting stable, complementary evidence from sparse and misaligned multimodal data, and reasoning over heterogeneous, possibly conflicting evidence in a fully unsupervised setting where no labels exist to calibrate how evidence should be weighted.

Each mention–entity pair is characterized by a 14-dimensional evidence vector comprising four components:

Instance-centric evidence: cosine similarities between CLIP embeddings over four modality combinations (text–text, text–image, image–text, image–image).

Group-level evidence: analogous similarities computed from graph-enhanced embeddings. This is the core contribution of the offline stage. A contextualized multimodal graph is built over mentions and entities jointly, with edges formed by (i) modality-aware CLIP gating — visual similarity terms are included only when both endpoints have images, otherwise similarity reduces to textual score, explicitly handling missingness — and (ii) LLM-enhanced semantic expansion, where ChatGPT-generated descriptions for mentions and selected entities are encoded by the CLIP text encoder and fused into a three-term similarity used for Top-$K$ neighbor retrieval. The final graph unions both edge sets, including mention–mention and entity–entity edges that ablation confirms are useful.

Representations are learned through an asymmetric teacher–student scheme: a text-based GCN teacher trained with a subgraph contrastive objective (PPR-sampled Top-$K$ subgraphs), followed by an image-based student distilled via cross-modal InfoNCE alignment to the frozen teacher plus its own structural loss. The asymmetry is justified by the empirical quality gap between modalities: text is more complete and stable, while images are noisy or absent. Missing node features are masked to zero rather than removing nodes, preserving structural consistency across views. Two theorems support the design: one showing that minimum-risk fusion of unbiased noisy evidence channels dominates any single channel when noise covariance is positive-definite, and a triangle-inequality bound showing student error relative to an ideal target is controlled by teacher–student discrepancy plus twice the teacher's own approximation error. Both results rest on idealized assumptions (unbiased channels with well-behaved covariance; a bounded-stability teacher) that are not verified empirically, so they serve as design justification rather than performance guarantees.

Lexical evidence: a Gestalt-pattern-matching string-overlap ratio, acting as a symbolic anchor robust to typos and name variants.

Statistical evidence: mean similarity, maximum similarity, dominance gap ($s_{\max} - \mu$, flagging reliance on a single weak cue), and binary indicators of image availability for mention and entity — enabling reliability-aware down-weighting of visual signals.

The online stage balances efficiency against accuracy. Candidate selection uses a 9-way retrieval scheme (four modality views × {instance, group} embeddings, plus lexical) with $K_{ch}=250$ per channel via FAISS, yielding candidate pools up to 2,250 entities; statistical evidence is deliberately excluded here since it reflects reliability rather than direct relevance. A prior score aggregates normalized retrieval similarities.

The re-ranking step formulates ranking-strategy induction as a conditional decision problem: the LLM receives the prior score, the full evidence vector, and guidelines favoring cross-perspective consistency and caution under missing modalities or single-modality dominance, then emits an executable decision tree whose nodes test specific evidence features and whose branches carry LLM-generated score adjustments $\Delta_n$. The final score is the prior plus adjustments along each candidate's path, giving an interpretable per-pair reasoning trace. In the reported case study, this mechanism demotes a distractor promoted by deceptive visual similarity (final score 0.38) and promotes the ground truth from rank 5 to rank 1 (final score 0.588). Per-query complexity scales as $$O(9T_{\mathrm{ret}} + |\mathcal{C}(m)|T_{\mathrm{LLM}})$$, avoiding LLM calls over the full entity space.

Experiments cover WikiMEL, RichpediaMEL, and WikiDiverse using Hit@1/5/10, with ChatGPT-4o as the default backbone.

Method	Type	WikiMEL H@1	Richpedia H@1	WikiDiverse H@1
BERT (zero-shot)	Unsup.	31.7	31.6	22.2
CLIP (zero-shot)	Unsup.	40.7	38.1	34.4
OpenMEL	Unsup.	69.8	65.6	67.1
MSR-MEL	Unsup.	84.99	74.40	69.75
M³EL (10% labels)	Sup.	76.72	64.51	66.36
DWE	Sup.	44.7	67.6	47.5

MSR-MEL improves Hit@1 over OpenMEL by an average of 8.88 points (13.04% relative) across benchmarks, with the largest gain of 21.7% relative on WikiMEL. It also outperforms DWE on all datasets and exceeds M³EL trained on 10% labeled data despite using none — a notable result suggesting multi-perspective reasoning can substitute for moderate annotation budgets, though supervised methods trained on full data (e.g., M³EL at 100%) remain stronger on WikiMEL. Efficiency results indicate roughly 11–13× lower end-to-end inference time than OpenMEL and per-epoch training cost more than an order of magnitude below M³EL.

Ablations attribute gains across all components: removing CLIP features causes the largest drop (e.g., 84.99 → 53.87 on WikiMEL), followed by removal of GNN features, prior scoring, lexical features, and LLM-driven reasoning itself (84.99 → 81.43). On the offline side, disabling teacher–student alignment costs about 5 points, and an image-only GNN performs worst, confirming the noise in visual signals. Modality-robustness experiments show MSR-MEL retains clear advantages over OpenMEL in missing-modality subsets, consistent with the design intent of masking and down-weighting unavailable cues. Sensitivity analysis shows unimodal behavior over $K_{LLM}$, $K_{ppr}$, and $\lambda_1$, with WikiDiverse requiring stricter tuning — the framework's robustness is dataset-dependent.

LLM-choice ablations show ChatGPT-4o outperforming DeepSeek-Chat and LLaMA3-8B backbones, with gaps widest on WikiDiverse, indicating sensitivity of semantic-expansion quality to the underlying model's world knowledge.

Several constraints qualify the results. First, the theoretical guarantees rest on assumptions — unbiased evidence channels with positive-definite noise covariance, and a bounded-error teacher — that hold only approximately if at all in practice. Second, the framework depends heavily on a strong commercial LLM (ChatGPT-4o); the reported degradation with smaller open models raises questions about reproducibility and applicability where such models are unavailable, and the paper does not quantify API cost at scale. Third, hyperparameter sensitivity is nontrivial on harder datasets, implying tuning effort that partially offsets the label-free advantage. Fourth, evaluation is limited to three Wikipedia-derived benchmarks; behavior on noisier web-scale corpora remains untested. Open questions include whether the induced decision trees generalize across domains without re-induction, whether the fusion theorem's conditions can be empirically validated for real evidence channels, and whether the approach extends to KBs lacking reliable textual descriptions.

MSR-MEL reframes unsupervised MEL as evidence synthesis followed by LLM-driven reasoning over heterogeneous evidence, combining LLM-enhanced contextualized graphs, asymmetric text-to-image teacher–student distillation, and interpretable tree-based re-ranking. The consistent improvements over prior unsupervised state of the art — including an average 8.88-point Hit@1 gain over OpenMEL — together with competitive performance against lightly supervised baselines, substantiate the claim that neighborhood-aware, multi-perspective evidence with explicit consistency reasoning materially improves disambiguation under incomplete multimodality.