Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 71 tok/s
Gemini 2.5 Pro 46 tok/s Pro
GPT-5 Medium 27 tok/s Pro
GPT-5 High 30 tok/s Pro
GPT-4o 93 tok/s Pro
Kimi K2 207 tok/s Pro
GPT OSS 120B 460 tok/s Pro
Claude Sonnet 4.5 36 tok/s Pro
2000 character limit reached

On the Evaluation Metric for Hashing (1905.10951v2)

Published 27 May 2019 in cs.IR

Abstract: Due to its low storage cost and fast query speed, hashing has been widely used for large-scale approximate nearest neighbor (ANN) search. Bucket search, also called hash lookup, can achieve fast query speed with a sub-linear time cost based on the inverted index table constructed from hash codes. Many metrics have been adopted to evaluate hashing algorithms. However, all existing metrics are improper to evaluate the hash codes for bucket search. On one hand, all existing metrics ignore the retrieval time cost which is an important factor reflecting the performance of search. On the other hand, some of them, such as mean average precision (MAP), suffer from the uncertainty problem as the ranked list is based on integer-valued Hamming distance, and are insensitive to Hamming radius as these metrics only depend on relative Hamming distance. Other metrics, such as precision at Hamming radius R, fail to evaluate global performance as these metrics only depend on one specific Hamming radius. In this paper, we first point out the problems of existing metrics which have been ignored by the hashing community, and then propose a novel evaluation metric called radius aware mean average precision (RAMAP) to evaluate hash codes for bucket search. Furthermore, two coding strategies are also proposed to qualitatively show the problems of existing metrics. Experiments demonstrate that our proposed RAMAP can provide more proper evaluation than existing metrics.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. A. Andoni and P. Indyk. Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. In FOCS, pages 459–468, 2006.
  2. Hashnet: Deep learning to hash by continuation. In ICCV, pages 5609–5618, 2017.
  3. Stochastic generative hashing. In ICML, pages 913–922, 2017.
  4. Locality-sensitive hashing scheme based on p-stable distributions. In SCG, pages 253–262, 2004.
  5. Previewer for multi-scale object detector. In MM, pages 265–273, 2018.
  6. Similarity search in high dimensions via hashing. In VLDB, pages 518–529, 1999.
  7. Y. Gong and S. Lazebnik. Iterative quantization: A procrustean approach to learning binary codes. In CVPR, pages 817–824, 2011.
  8. Hashing as tie-aware learning to rank. In CVPR, pages 4023–4032, 2018.
  9. W. Kong and W.-J. Li. Isotropic hashing. In NeurIPS, pages 1655–1663, 2012.
  10. A. Krizhevsky. Learning multiple layers of features from tiny images. Master’s thesis, University of Toronto, 2009.
  11. Imagenet classification with deep convolutional neural networks. In NeurIPS, pages 1106–1114, 2012.
  12. P. Li. 0-bit consistent weighted sampling. In KDD, pages 665–674, 2015.
  13. Deep supervised discrete hashing. In NeurIPS, pages 2479–2488, 2017.
  14. Learning hash functions using column generation. In ICML, pages 142–150, 2013.
  15. Mrtf: model refinery for transductive federated learning. DMKD, 37(5):2046–2069, 2023.
  16. Deep supervised hashing for fast image retrieval. In CVPR, pages 2064–2072, 2016.
  17. Discrete graph hashing. In NeurIPS, pages 3419–3427, 2014.
  18. Supervised hashing with kernels. In CVPR, pages 2074–2081, 2012.
  19. Hashing with graphs. In ICML, pages 1–8, 2011.
  20. A multiscale grouping transformer with CLIP latents for remote sensing image captioning. TGRS, 62:1–15, 2024.
  21. M. Norouzi and D. J. Fleet. Minimal loss hashing for compact binary codes. In ICML, pages 353–360, 2011.
  22. R. Raziperchikolaei and M. Á. Carreira-Perpiñán. Optimizing affinity-based binary hashing using auxiliary coordinates. In NeurIPS, pages 640–648, 2016.
  23. An integrated multimodal attention-based approach for bank stress test prediction. In ICDM, pages 1282–1287. IEEE, 2019.
  24. How should we evaluate supervised hashing? In ICASSP, pages 1732–1736, 2017.
  25. R. Salakhutdinov and G. E. Hinton. Semantic hashing. IJAR, 50(7):969–978, 2009.
  26. Supervised discrete hashing. In CVPR, pages 37–45, 2015.
  27. A. Shrivastava and P. Li. Asymmetric LSH (ALSH) for sublinear time maximum inner product search (MIPS). In NeurIPS, pages 2321–2329, 2014.
  28. Greedy hash: Towards fast optimization for accurate hash coding in CNN. In NeurIPS, pages 806–815, 2018.
  29. Tri-clustered tensor completion for social-aware image tag refinement. TPAMI, 39(8):1662–1674, 2017.
  30. Art2real: Unfolding the reality of artworks via semantically-aware image-to-image translation. In CVPR, pages 5849–5859, 2019.
  31. Covlr: Coordinating cross-modal consistency and intra-modal relations for vision-language retrieval. ICME, 2024.
  32. Sequential projection learning for hashing with compact codes. In ICML, pages 1127–1134, 2010.
  33. Spectral hashing. In NeurIPS, pages 1753–1760, 2008.
  34. Deep visual-linguistic fusion network considering cross-modal inconsistency for rumor detection. SCIS, 66(12), 2023.
  35. Semi-supervised multi-modal multi-instance multi-label deep network with optimal transport. TKDE, 33(2):696–709, 2021.
  36. Alignment efficient image-sentence retrieval considering transferable cross-modal representation learning. FCS, 18(3):181335, 2024.
  37. Towards global video scene segmentation with context-aware transformer. In AAAI, pages 3206–3213. AAAI Press, 2023.
  38. S2OSC: A holistic semi-supervised approach for open set classification. TKDD, 16(2):34:1–34:27, 2022.
  39. Exploiting cross-modal prediction and relation consistency for semisupervised image captioning. TOC, 2022.
  40. Corporate relative valuation using heterogeneous multi-modal graph neural network. TKDE, 35(1):211–224, 2023.
  41. Semi-supervised multi-modal clustering and classification with incomplete modalities. TKDE, 33(2):682–695, 2021.
  42. Contextualized knowledge graph embedding for explainable talent training course recommendation. TOIS, 42(2):33:1–33:27, 2024.
  43. Rethinking label-wise cross-modal retrieval from A semantic sharing perspective. In IJCAI, pages 3300–3306. ijcai.org, 2021.
  44. DOMFN: A divergence-orientated multi-modal fusion network for resume assessment. In MM, pages 1612–1620. ACM, 2022.
  45. Adaptive deep models for incremental learning: Considering capacity scalability and sustainability. In KDD, pages 74–82. ACM, 2019.
  46. Circulant binary embedding. In ICML, pages 946–954, 2014.
  47. Self-taught hashing for fast similarity search. In SIGIR, pages 18–25, 2010.
  48. Harmi: Human activity recognition via multi-modality incremental learning. TITB, 26(3):939–951, 2022.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up