Cross Modal Retrieval with Querybank Normalisation (2112.12777v3)

Abstract: Profiting from large-scale training datasets, advances in neural architecture design and efficient inference, joint embeddings have become the dominant approach for tackling cross-modal retrieval. In this work we first show that, despite their effectiveness, state-of-the-art joint embeddings suffer significantly from the longstanding "hubness problem" in which a small number of gallery embeddings form the nearest neighbours of many queries. Drawing inspiration from the NLP literature, we formulate a simple but effective framework called Querybank Normalisation (QB-Norm) that re-normalises query similarities to account for hubs in the embedding space. QB-Norm improves retrieval performance without requiring retraining. Differently from prior work, we show that QB-Norm works effectively without concurrent access to any test set queries. Within the QB-Norm framework, we also propose a novel similarity normalisation method, the Dynamic Inverted Softmax, that is significantly more robust than existing approaches. We showcase QB-Norm across a range of cross modal retrieval models and benchmarks where it consistently enhances strong baselines beyond the state of the art. Code is available at https://vladbogo.github.io/QB-Norm/.

• The paper presents Querybank Normalisation (QB-Norm), a novel framework to address the hubness problem in high-dimensional embedding spaces for cross-modal retrieval systems.
• The Dynamic Inverted Softmax (DIS) technique within QB-Norm is particularly effective and robust against variability in real-world querybank selections.
• Experiments show QB-Norm significantly improves cross-modal retrieval performance by reducing hub-induced skewness, offering a scalable solution for practical applications.

Cross Modal Retrieval with Querybank Normalisation

The paper presents an innovative approach to enhancing cross-modal retrieval systems through a novel framework called Querybank Normalisation (QB-Norm). As the digital data archive continues to expand, efficiently searching through multimodal data is increasingly crucial. Cross-modal retrieval, where data from one modality is queried to find relevant information in another, relies heavily on high-dimensional joint embeddings. However, these systems are plagued by hubness issues, where certain elements of the embedding space become "hubs," frequently appearing as nearest neighbors and thus degrading retrieval performance.

Background Context:

The concept of hubness is rooted in the behavior of high-dimensional data spaces, where the distribution patterns lead to some vectors becoming hubs (frequently occurring nearest neighbors to many queries). These hubs can significantly lower the quality of retrieval results. Typically addressed within the NLP domain, solutions have not been effectively transposed to the cross-modal retrieval context, which is marked by higher embedding dimensionality due to complex query representations like natural language.

Framework Development:

Drawing inspiration from existing NLP solutions to hubness (such as Globally-Corrected retrieval and Cross-Domain Similarity Local Scaling), the authors introduce QB-Norm, a streamlined framework applying querybank-based mechanisms to adjust similarity measures within an embedding space. This approach does not require simultaneous access to multiple queries (a key limitation in practical applications addressed by this work) and encompasses several techniques, notably the Dynamic Inverted Softmax (DIS).

Similarity Normalisation Techniques:

• Globally-Corrected Retrieval: Adjusts similarities based on ranking across querybanks, requiring considerable computational resources.
• Cross-Domain Similarity Local Scaling: Uses local rescaling of similarities, demonstrating robustness but requiring careful querybank construction.
• Inverted Softmax: Applies temperature-based normalization to mitigate hubness but suffers from sensitivity to querybank selection.
• Dynamic Inverted Softmax: Innovatively restricts normalization only to hub-activated retrievals—showing more robustness against unfavorable querybanks, aligning well with real-world dataset variability.

Experimental Validation:

Extensive experimental analysis supports QB-Norm’s effectiveness across various models and datasets. By employing DIS, QB-Norm consistently reduces hub-induced skewness and thus enhances retrieval performance without adversely affecting cases with unsuitable querybank selections. The results underscore the framework's adaptability across domains and dimensions, showing improvements up to 20% in recall metrics for certain datasets.

Implications and Conclusion:

The research has profound implications for improving the usability of cross-modal retrieval in practical applications without compromising on scalability and robustness. By aligning techniques to mitigate hubness effectively, the authors contribute substantively to the retrieval field, setting a precedent for further developments in high-dimensional space optimizations with cross-modality considerations. Future work could explore deeper integration of QB-Norm with dynamic querybank construction, possibly harnessing adaptive learning strategies to refine embeddings continuously, promising advancements in response to evolving data landscapes and user requirements.