STIR: Siamese Transformer for Image Retrieval Postprocessing (2304.13393v2)

Abstract: Current metric learning approaches for image retrieval are usually based on learning a space of informative latent representations where simple approaches such as the cosine distance will work well. Recent state of the art methods such as HypViT move to more complex embedding spaces that may yield better results but are harder to scale to production environments. In this work, we first construct a simpler model based on triplet loss with hard negatives mining that performs at the state of the art level but does not have these drawbacks. Second, we introduce a novel approach for image retrieval postprocessing called Siamese Transformer for Image Retrieval (STIR) that reranks several top outputs in a single forward pass. Unlike previously proposed Reranking Transformers, STIR does not rely on global/local feature extraction and directly compares a query image and a retrieved candidate on pixel level with the usage of attention mechanism. The resulting approach defines a new state of the art on standard image retrieval datasets: Stanford Online Products and DeepFashion In-shop. We also release the source code at https://github.com/OML-Team/open-metric-learning/tree/main/pipelines/postprocessing/ and an interactive demo of our approach at https://dapladoc-oml-postprocessing-demo-srcappmain-pfh2g0.streamlit.app/

• The paper presents a novel ViT-Triplet model that competes with state-of-the-art methods by optimizing training and hyperparameters for superior CMC performance.
• The paper introduces STIR postprocessing, which uses a Siamese Transformer to rerank top retrieval outputs through direct pixel-level attention comparisons.
• The paper demonstrates that efficient transformer-based techniques can improve retrieval accuracy without resorting to complex embedding spaces, enabling production-ready solutions.

Analysis of "STIR: Siamese Transformer for Image Retrieval Postprocessing"

The paper introduces the Siamese Transformer for Image Retrieval (STIR), a novel methodology aimed at improving image retrieval accuracy through sophisticated postprocessing techniques. This research is a response to contemporary challenges in metric learning approaches, which often rely on developing complex embedding spaces that are computationally intensive and difficult to implement in production.

Core Contributions

The paper makes two primary contributions:

1. ViT-Triplet Model: This model revamps the conventional triplet loss-based approach by employing a Vision Transformer (ViT) backbone. By fine-tuning hyperparameters and implementing efficient training processes, the ViT-Triplet model achieves parity or outperforms state-of-the-art models on standard datasets such as Stanford Online Products (SOP) and DeepFashion In-Shop.
2. STIR Postprocessing: A more innovative aspect of the paper is the introduction of a reranking method employing a Siamese Transformer. This approach involves running a ViT-based architecture over concatenated query and gallery image pairs, focusing on the top retrieval outputs. The STIR model capitalizes on the attention mechanism to conduct pixel-level comparisons, thus refining the retrieval accuracy significantly.

Numerical Results

The numerical results presented within the paper validate the effectiveness of both the ViT-Triplet model and the STIR approach. Noteworthy findings include:

• The ViT-Triplet model either matches or exceeds the performance of state-of-the-art methods like HypViT in various CMC (Cumulative Matching Characteristics) metrics.
• Implementing the STIR postprocessing significantly enhances CMC@1 results, indicating a substantial improvement in top-ranked retrieval accuracy.

Theoretical and Practical Implications

From a theoretical standpoint, this paper challenges the assumption that increasingly complex models are necessary for advancement in image retrieval. It posits that existing models, when optimized correctly, can still deliver competitive performance. Practically, the proposed STIR model provides an efficient and adaptable solution for improving retrieval tasks, paving the way for more production-ready applications. By directly comparing images at the pixel level through attention mechanisms, the solution avoids reliance on intricate embedding space manipulations that can hamper deployment in real-world scenarios.

Future Directions

Possible avenues for future research include:

• Exploring alternative Transformer architectures that possess lower computational complexity, potentially enhancing the operational efficiency of the STIR model.
• Investigating the incorporation of additional data forms, such as intermediate layer descriptors, which might provide a more lightweight solution while maintaining or improving retrieval accuracy.
• Addressing the observed ambiguities in existing dataset annotations for more robust and context-aware retrieval outcomes.

In conclusion, this paper presents a compelling approach to postprocessing in image retrieval tasks. By leveraging Transformer architectures for direct image comparisons, the authors have delivered a model that is both effective and computationally feasible, providing significant potential for future developments in AI-driven retrieval systems.