Embedding-based similarity as a domain-agnostic pairwise feature

Investigate the effectiveness of using cosine similarity between the base model’s class embeddings as the domain-agnostic label-similarity component in the REPAIR reranker’s pairwise feature vector when structured external knowledge (e.g., taxonomic distance, WordNet path similarity, or HPO phenotype similarity) is unavailable, and determine its impact on reranking performance across datasets.

Background

The REPAIR reranker augments base-model logits with a classwise offset and a pairwise correction that uses features capturing competition among shortlisted labels. A key component of these pairwise features is a domain-specific label-similarity signal: taxonomic distance for iNaturalist, WordNet path similarity for ImageNet-LT, and HPO phenotype similarity for rare-disease datasets. When such structured knowledge is unavailable (e.g., Places-LT), the model uses a reduced feature set without this similarity term.

The authors explicitly note a potential alternative: using cosine similarity between the base model’s class embeddings as a substitute for external knowledge. They defer evaluating this substitution and its impact on performance, leaving a concrete methodological question about whether embedding-based similarity can reliably replace domain-specific signals in the pairwise correction.