Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 73 tok/s
Gemini 2.5 Pro 39 tok/s Pro
GPT-5 Medium 27 tok/s Pro
GPT-5 High 19 tok/s Pro
GPT-4o 115 tok/s Pro
Kimi K2 226 tok/s Pro
GPT OSS 120B 461 tok/s Pro
Claude Sonnet 4 38 tok/s Pro
2000 character limit reached

Does Negative Sampling Matter? A Review with Insights into its Theory and Applications (2402.17238v1)

Published 27 Feb 2024 in cs.LG

Abstract: Negative sampling has swiftly risen to prominence as a focal point of research, with wide-ranging applications spanning machine learning, computer vision, natural language processing, data mining, and recommender systems. This growing interest raises several critical questions: Does negative sampling really matter? Is there a general framework that can incorporate all existing negative sampling methods? In what fields is it applied? Addressing these questions, we propose a general framework that leverages negative sampling. Delving into the history of negative sampling, we trace the development of negative sampling through five evolutionary paths. We dissect and categorize the strategies used to select negative sample candidates, detailing global, local, mini-batch, hop, and memory-based approaches. Our review categorizes current negative sampling methods into five types: static, hard, GAN-based, Auxiliary-based, and In-batch methods, providing a clear structure for understanding negative sampling. Beyond detailed categorization, we highlight the application of negative sampling in various areas, offering insights into its practical benefits. Finally, we briefly discuss open problems and future directions for negative sampling.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (181)
  1. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” NIPS, vol. 26, 2013.
  2. S. Rendle, C. Freudenthaler, Z. Gantner, and L. Schmidt-Thieme, “Bpr: Bayesian personalized ranking from implicit feedback,” arXiv preprint arXiv:1205.2618, 2012.
  3. B. Perozzi, R. Al-Rfou, and S. Skiena, “Deepwalk: Online learning of social representations,” in KDD, 2014, pp. 701–710.
  4. J. Tang, M. Qu, M. Wang, M. Zhang, J. Yan, and Q. Mei, “Line: Large-scale information network embedding,” in WWW, 2015, pp. 1067–1077.
  5. A. Grover and J. Leskovec, “node2vec: Scalable feature learning for networks,” in KDD, 2016, pp. 855–864.
  6. Z. Yang, M. Ding, C. Zhou, H. Yang, J. Zhou, and J. Tang, “Understanding negative sampling in graph representation learning,” in KDD, 2020, pp. 1666–1676.
  7. T. Huang, Y. Dong, M. Ding, Z. Yang, W. Feng, X. Wang, and J. Tang, “Mixgcf: An improved training method for graph neural network-based recommender systems,” in KDD, 2021, pp. 665–674.
  8. D. Krompaß, S. Baier, and V. Tresp, “Type-constrained representation learning in knowledge graphs,” in International semantic web conference.   Springer, 2015, pp. 640–655.
  9. L. Cai and W. Y. Wang, “Kbgan: Adversarial learning for knowledge graph embeddings,” arXiv preprint arXiv:1711.04071, 2017.
  10. M. Grbovic, N. Djuric, V. Radosavljevic, F. Silvestri, and N. Bhamidipati, “Context-and content-aware embeddings for query rewriting in sponsored search,” in SIGIR, 2015, pp. 383–392.
  11. Z. Zhang and P. Zweigenbaum, “Gneg: Graph-based negative sampling for word2vec,” in ACL, 2018, pp. 566–571.
  12. X. Wu, C. Gao, L. Zang, J. Han, Z. Wang, and S. Hu, “Esimcse: Enhanced sample building method for contrastive learning of unsupervised sentence embedding,” arXiv preprint arXiv:2109.04380, 2021.
  13. F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in CVPR, 2015, pp. 815–823.
  14. C.-Y. Wu, R. Manmatha, A. J. Smola, and P. Krahenbuhl, “Sampling matters in deep embedding learning,” in ICCV, 2017, pp. 2840–2848.
  15. C.-Y. Chuang, J. Robinson, Y.-C. Lin, A. Torralba, and S. Jegelka, “Debiased contrastive learning,” NIPS, vol. 33, pp. 8765–8775, 2020.
  16. Y. Kalantidis, M. B. Sariyildiz, N. Pion, P. Weinzaepfel, and D. Larlus, “Hard negative mixing for contrastive learning,” NIPS, vol. 33, pp. 21 798–21 809, 2020.
  17. J. Robinson, C.-Y. Chuang, S. Sra, and S. Jegelka, “Contrastive learning with hard negative samples,” arXiv preprint arXiv:2010.04592, 2020.
  18. M. Wu, M. Mosse, C. Zhuang, D. Yamins, and N. Goodman, “Conditional negative sampling for contrastive learning of visual representations,” arXiv preprint arXiv:2010.02037, 2020.
  19. W. Zhang, T. Chen, J. Wang, and Y. Yu, “Optimizing top-n collaborative filtering via dynamic negative item sampling,” in SIGIR, 2013, pp. 785–788.
  20. A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko, “Translating embeddings for modeling multi-relational data,” NIPS, vol. 26, 2013.
  21. T. Kipf, E. Van der Pol, and M. Welling, “Contrastive learning of structured world models,” arXiv preprint arXiv:1911.12247, 2019.
  22. T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” arXiv preprint arXiv:1609.02907, 2016.
  23. P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, “Graph attention networks,” arXiv preprint arXiv:1710.10903, 2017.
  24. W. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” NIPS, vol. 30, 2017.
  25. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, 2016, pp. 770–778.
  26. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
  27. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019.
  28. Z. Wang, J. Zhang, J. Feng, and Z. Chen, “Knowledge graph embedding by translating on hyperplanes,” in AAAI, vol. 28, no. 1, 2014.
  29. R. Pan, Y. Zhou, B. Cao, N. N. Liu, R. Lukose, M. Scholz, and Q. Yang, “One-class collaborative filtering,” in ICDM.   IEEE, 2008, pp. 502–511.
  30. H. Xiao, M. Huang, Y. Hao, and X. Zhu, “Transg: A generative mixture model for knowledge graph embedding,” arXiv preprint arXiv:1509.05488, 2015.
  31. X. Wang, X. He, M. Wang, F. Feng, and T.-S. Chua, “Neural graph collaborative filtering,” in SIGIR, 2019, pp. 165–174.
  32. X. He, K. Deng, X. Wang, Y. Li, Y. Zhang, and M. Wang, “Lightgcn: Simplifying and powering graph convolution network for recommendation,” in SIGIR, 2020, pp. 639–648.
  33. N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in CVPR, vol. 1.   Ieee, 2005, pp. 886–893.
  34. P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan, “Object detection with discriminatively trained part-based models,” IEEE transactions on pattern analysis and machine intelligence, vol. 32, no. 9, pp. 1627–1645, 2009.
  35. T. Malisiewicz, A. Gupta, and A. A. Efros, “Ensemble of exemplar-svms for object detection and beyond,” in 2011 International conference on computer vision.   IEEE, 2011, pp. 89–96.
  36. L. Chen, F. Yuan, J. M. Jose, and W. Zhang, “Improving negative sampling for word representation using self-embedded features,” in WSDM, 2018, pp. 99–107.
  37. J. Rao, H. He, and J. Lin, “Noise-contrastive estimation for answer selection with deep neural networks,” in CIKM, 2016, pp. 1913–1916.
  38. Z. Sun, W. Hu, Q. Zhang, and Y. Qu, “Bootstrapping entity alignment with knowledge graph embedding.” in IJCAI, vol. 18, 2018, pp. 4396–4402.
  39. R. Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, and J. Leskovec, “Graph convolutional neural networks for web-scale recommender systems,” in KDD, 2018, pp. 974–983.
  40. X. Wang, Y. Xu, X. He, Y. Cao, M. Wang, and T.-S. Chua, “Reinforced negative sampling over knowledge graph for recommendation,” in WWW, 2020, pp. 99–109.
  41. A. Shrivastava, A. Gupta, and R. Girshick, “Training region-based object detectors with online hard example mining,” in CVPR, 2016, pp. 761–769.
  42. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” NIPS, vol. 27, 2014.
  43. J. Wang, L. Yu, W. Zhang, Y. Gong, Y. Xu, B. Wang, P. Zhang, and D. Zhang, “Irgan: A minimax game for unifying generative and discriminative information retrieval models,” in SIGIR, 2017, pp. 515–524.
  44. H. Wang, J. Wang, J. Wang, M. Zhao, W. Zhang, F. Zhang, W. Li, X. Xie, and M. Guo, “Learning graph representation with generative adversarial nets,” TKDE, vol. 33, no. 8, pp. 3090–3103, 2019.
  45. Q. Wang, H. Yin, Z. Hu, D. Lian, H. Wang, and Z. Huang, “Neural memory streaming recommender networks with adversarial training,” in KDD, 2018, pp. 2467–2475.
  46. D.-K. Chae, J.-S. Kang, S.-W. Kim, and J.-T. Lee, “Cfgan: A generic collaborative filtering framework based on generative adversarial networks,” in CIKM, 2018, pp. 137–146.
  47. D. H. Park and Y. Chang, “Adversarial sampling and training for semi-supervised information retrieval,” in WWW, 2019, pp. 1443–1453.
  48. T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in ICML.   PMLR, 2020, pp. 1597–1607.
  49. K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in CVPR, 2020, pp. 9729–9738.
  50. Q. Hu, X. Wang, W. Hu, and G.-J. Qi, “Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries,” in CVPR, 2021, pp. 1074–1083.
  51. J. Xie, X. Zhan, Z. Liu, Y. S. Ong, and C. C. Loy, “Delving into inter-image invariance for unsupervised visual representations,” arXiv preprint arXiv:2008.11702, 2020.
  52. Z. Yang, T. Huang, M. Ding, Y. Dong, R. Ying, Y. Cen, Y. Geng, and J. Tang, “Batchsampler: Sampling mini-batches for contrastive learning in vision, language, and graphs,” arXiv preprint arXiv:2306.03355, 2023.
  53. T. T. Cai, J. Frankle, D. J. Schwab, and A. S. Morcos, “Are all negatives created equal in contrastive instance discrimination?” arXiv preprint arXiv:2010.06682, 2020.
  54. H. Caselles-Dupré, F. Lesaint, and J. Royo-Letelier, “Word2vec applied to recommendation: Hyperparameters matter,” in RecSys, 2018, pp. 352–356.
  55. B. Yang, W.-t. Yih, X. He, J. Gao, and L. Deng, “Embedding entities and relations for learning and inference in knowledge bases,” arXiv preprint arXiv:1412.6575, 2014.
  56. C. Tao, L. Mou, D. Zhao, and R. Yan, “Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems,” in AAAI, 2018.
  57. S. Ghazarian, J. T.-Z. Wei, A. Galstyan, and N. Peng, “Better automatic evaluation of open-domain dialogue systems with contextualized embeddings,” arXiv preprint arXiv:1904.10635, 2019.
  58. M. Bucher, S. Herbin, and F. Jurie, “Hard negative mining for metric learning based zero-shot classification,” in ECCV.   Springer, 2016, pp. 524–531.
  59. B. Harwood, V. Kumar BG, G. Carneiro, I. Reid, and T. Drummond, “Smart mining for deep metric learning,” in ICCV, 2017, pp. 2821–2829.
  60. D. Galanopoulos and V. Mezaris, “Hard-negatives or non-negatives? a hard-negative selection strategy for cross-modal retrieval using the improved marginal ranking loss,” in ICCV, 2021, pp. 2312–2316.
  61. S. Rendle and C. Freudenthaler, “Improving pairwise learning for item recommendation from implicit feedback,” in WSDM, 2014, pp. 273–282.
  62. X. Mao, W. Wang, Y. Wu, and M. Lan, “Boosting the speed of entity alignment 10×\times×: Dual attention matching network with normalized hard sample mining,” in WWW, 2021, pp. 821–832.
  63. Y. Zhang, Q. Yao, Y. Shao, and L. Chen, “Nscaching: simple and efficient negative sampling for knowledge graph embedding,” in ICDE.   IEEE, 2019, pp. 614–625.
  64. R. Cao, Y. Wang, Y. Liang, L. Gao, J. Zheng, J. Ren, and Z. Wang, “Exploring the impact of negative samples of contrastive learning: A case study of sentence embedding,” arXiv preprint arXiv:2202.13093, 2022.
  65. F. Che, G. Yang, P. Shao, D. Zhang, and J. Tao, “Mixkg: Mixing for harder negative samples in knowledge graph,” arXiv preprint arXiv:2202.09606, 2022.
  66. L. Yu, W. Zhang, J. Wang, and Y. Yu, “Seqgan: Sequence generative adversarial nets with policy gradient,” in AAAI, vol. 31, no. 1, 2017.
  67. A. J. Bose, H. Ling, and Y. Cao, “Adversarial contrastive estimation,” arXiv preprint arXiv:1805.03642, 2018.
  68. P. Wang, S. Li, and R. Pan, “Incorporating gan for negative sampling in knowledge representation learning,” in AAAI, vol. 32, no. 1, 2018.
  69. H. Gao, J. Pei, and H. Huang, “Progan: Network embedding via proximity generative adversarial network,” in KDD, 2019, pp. 1308–1316.
  70. B. Hu, Y. Fang, and C. Shi, “Adversarial learning on heterogeneous information networks,” in KDD, 2019, pp. 120–129.
  71. P. Gupta, Y. Tsvetkov, and J. P. Bigham, “Synthesizing adversarial negative responses for robust response ranking and evaluation,” arXiv preprint arXiv:2106.05894, 2021.
  72. A. Sinha, K. Ayush, J. Song, B. Uzkent, H. Jin, and S. Ermon, “Negative data augmentation,” arXiv preprint arXiv:2102.05113, 2021.
  73. C.-H. Ho and N. Nvasconcelos, “Contrastive learning with adversarial examples,” NIPS, vol. 33, pp. 17 081–17 093, 2020.
  74. W. Wang, W. Zhou, J. Bao, D. Chen, and H. Li, “Instance-wise hard negative example generation for contrastive learning in unpaired image-to-image translation,” in ICCV, 2021, pp. 14 020–14 029.
  75. Y. Duan, W. Zheng, X. Lin, J. Lu, and J. Zhou, “Deep adversarial metric learning,” in CVPR, 2018, pp. 2780–2789.
  76. K. Ahrabian, A. Feizi, Y. Salehi, W. L. Hamilton, and A. J. Bose, “Structure aware negative sampling in knowledge graphs,” arXiv preprint arXiv:2009.11355, 2020.
  77. J. Chen, C. Wang, S. Zhou, Q. Shi, Y. Feng, and C. Chen, “Samwalker: Social recommendation with informative sampling strategy,” in WWW, 2019, pp. 228–239.
  78. Y. Wang, Z. Liu, Z. Fan, L. Sun, and P. S. Yu, “Dskreg: Differentiable sampling on knowledge graph for recommendation with relational gnn,” in CIKM, 2021, pp. 3513–3517.
  79. T. Zhao, J. McAuley, and I. King, “Leveraging social connections to improve personalized ranking for collaborative filtering,” in CIKM, 2014, pp. 261–270.
  80. J. Manotumruksa, C. Macdonald, and I. Ounis, “A personalised ranking framework with multiple sampling criteria for venue recommendation,” in CIKM, 2017, pp. 1469–1478.
  81. B. Loni, R. Pagano, M. Larson, and A. Hanjalic, “Bayesian personalized ranking with multi-channel user feedback,” in RecSys, 2016, pp. 361–364.
  82. J. Ding, F. Feng, X. He, G. Yu, Y. Li, and D. Jin, “An improved sampler for bayesian personalized ranking by leveraging view data,” in WWW, 2018, pp. 13–14.
  83. J. Ding, Y. Quan, X. He, Y. Li, and D. Jin, “Reinforced negative sampling for recommendation with exposure data.” in IJCAI, 2019, pp. 2230–2236.
  84. Z. Yang, M. Ding, X. Zou, J. Tang, B. Xu, C. Zhou, and H. Yang, “Region or global a principle for negative sampling in graph-based recommendation,” TKDE, 2022.
  85. Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, “Unsupervised feature learning via non-parametric instance discrimination,” in CVPR, 2018, pp. 3733–3742.
  86. J. Ding, Y. Quan, Q. Yao, Y. Li, and D. Jin, “Simplify and robustify negative sampling for implicit collaborative filtering,” NIPS, vol. 33, pp. 1094–1105, 2020.
  87. X. Chen, H. Fan, R. Girshick, and K. He, “Improved baselines with momentum contrastive learning,” arXiv preprint arXiv:2003.04297, 2020.
  88. J. Qiu, Q. Chen, Y. Dong, J. Zhang, H. Yang, M. Ding, K. Wang, and J. Tang, “Gcc: Graph contrastive coding for graph neural network pre-training,” in KDD, 2020, pp. 1150–1160.
  89. T. Chen, Y. Sun, Y. Shi, and L. Hong, “On sampling strategies for neural network-based collaborative filtering,” in KDD, 2017, pp. 767–776.
  90. K. Zhou, H. Wang, W. X. Zhao, Y. Zhu, S. Wang, F. Zhang, Z. Wang, and J.-R. Wen, “S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization,” in CIKM, 2020, pp. 1893–1902.
  91. J. Wu, X. Wang, F. Feng, X. He, L. Chen, J. Lian, and X. Xie, “Self-supervised graph learning for recommendation,” in SIGIR, 2021, pp. 726–735.
  92. J. Yu, H. Yin, J. Li, Q. Wang, N. Q. V. Hung, and X. Zhang, “Self-supervised multi-channel hypergraph convolutional network for social recommendation,” in WWW, 2021, pp. 413–424.
  93. X. Xia, H. Yin, J. Yu, Q. Wang, L. Cui, and X. Zhang, “Self-supervised hypergraph convolutional networks for session-based recommendation,” in AAAI, vol. 35, no. 5, 2021, pp. 4503–4511.
  94. K. Hassani and A. H. Khasahmadi, “Contrastive multi-view representation learning on graphs,” in ICML.   PMLR, 2020, pp. 4116–4126.
  95. Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, and L. Wang, “Deep graph contrastive representation learning,” arXiv preprint arXiv:2006.04131, 2020.
  96. Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen, “Graph contrastive learning with augmentations,” NIPS, vol. 33, pp. 5812–5823, 2020.
  97. T. Gao, X. Yao, and D. Chen, “Simcse: Simple contrastive learning of sentence embeddings,” arXiv preprint arXiv:2104.08821, 2021.
  98. L. Kong, C. d. M. d’Autume, W. Ling, L. Yu, Z. Dai, and D. Yogatama, “A mutual information maximization perspective of language representation learning,” arXiv preprint arXiv:1910.08350, 2019.
  99. H. Zhao, X. Yang, Z. Wang, E. Yang, and C. Deng, “Graph debiased contrastive learning with joint representation clustering.” in IJCAI, 2021, pp. 3434–3440.
  100. G. Chu, X. Wang, C. Shi, and X. Jiang, “Cuco: Graph representation with curriculum contrastive learning.” in IJCAI, 2021, pp. 2300–2306.
  101. J. Xia, L. Wu, G. Wang, J. Chen, and S. Z. Li, “Progcl: Rethinking hard negative mining in graph contrastive learning,” in ICML.   PMLR, 2022, pp. 24 332–24 346.
  102. D. Zhang, W. Xiao, H. Zhu, X. Ma, and A. O. Arnold, “Virtual augmentation supported contrastive learning of sentence representations,” arXiv preprint arXiv:2110.08552, 2021.
  103. H. Wang, Y. Li, Z. Huang, Y. Dou, L. Kong, and J. Shao, “Sncse: Contrastive learning for unsupervised sentence embedding with soft negative samples,” arXiv preprint arXiv:2201.05979, 2022.
  104. L. Xiong, C. Xiong, Y. Li, K.-F. Tang, J. Liu, P. Bennett, J. Ahmed, and A. Overwijk, “Approximate nearest neighbor negative contrastive learning for dense text retrieval,” arXiv preprint arXiv:2007.00808, 2020.
  105. H. Rowley, S. Baluja, and T. Kanade, “Human face detection in visual scenes,” NIPS, vol. 8, 1995.
  106. K.-K. Sung and T. Poggio, “Example-based learning for view-based human face detection,” TPAMI, vol. 20, no. 1, pp. 39–51, 1998.
  107. V.-A. Tran, R. Hennequin, J. Royo-Letelier, and M. Moussallam, “Improving collaborative metric learning with efficient negative sampling,” in SIGIR, 2019, pp. 1201–1204.
  108. E. Simo-Serra, E. Trulls, L. Ferraz, I. Kokkinos, P. Fua, and F. Moreno-Noguer, “Discriminative learning of deep convolutional feature point descriptors,” in ICCV, 2015, pp. 118–126.
  109. I. Loshchilov and F. Hutter, “Online batch selection for faster training of neural networks,” arXiv preprint arXiv:1511.06343, 2015.
  110. X. Wang and A. Gupta, “Unsupervised learning of visual representations using videos,” in ICCV, 2015, pp. 2794–2802.
  111. J.  Weston, S. Bengio, and N. Usunier, “Wsabie: Scaling up to large vocabulary image annotation,” in IJCAI, 2011.
  112. T. Zhao, J. McAuley, and I. King, “Improving latent factor models via personalized feature projection for one class recommendation,” in CIKM, 2015, pp. 821–830.
  113. G. Guo, S. Ouyang, F. Yuan, and X. Wang, “Approximating word ranking and negative sampling for word embedding.”   IJCAI, 2018.
  114. F. Faghri, D. J. Fleet, J. R. Kiros, and S. Fidler, “Vse++: Improving visual-semantic embeddings with hard negatives,” arXiv preprint arXiv:1707.05612, 2017.
  115. G. Guo, S. Zhai, F. Yuan, Y. Liu, and X. Wang, “Vse-ens: Visual-semantic embeddings with efficient negative sampling,” in AAAI, vol. 32, no. 1, 2018.
  116. J. Li, C. Tao, W. Wu, Y. Feng, D. Zhao, and R. Yan, “Sampling matters! an empirical study of negative sampling strategies for learning of matching models in retrieval-based dialogue systems,” in EMNLP-IJCNLP, 2019, pp. 1291–1296.
  117. H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” arXiv preprint arXiv:1710.09412, 2017.
  118. T.-S. Chen, W.-C. Hung, H.-Y. Tseng, S.-Y. Chien, and M.-H. Yang, “Incremental false negative detection for contrastive learning,” arXiv preprint arXiv:2106.03719, 2021.
  119. C. Yang, Q. Wu, J. Jin, X. Gao, J. Pan, and G. Chen, “Trading hard negatives and true negatives: A debiased contrastive collaborative filtering approach,” arXiv preprint arXiv:2204.11752, 2022.
  120. R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine learning, vol. 8, no. 3, pp. 229–256, 1992.
  121. C. Chen, Y. Xie, S. Lin, R. Qiao, J. Zhou, X. Tan, Y. Zhang, and L. Ma, “Novelty detection via contrastive learning with negative data augmentation,” arXiv preprint arXiv:2106.09958, 2021.
  122. C. Wang, J. Chen, S. Zhou, Q. Shi, Y. Feng, and C. Chen, “Samwalker++: recommendation with informative sampling strategy,” TKDE, 2021.
  123. H. Fang, S. Wang, M. Zhou, J. Ding, and P. Xie, “Cert: Contrastive self-supervised learning for language understanding,” arXiv preprint arXiv:2005.12766, 2020.
  124. M. Ye, X. Zhang, P. C. Yuen, and S.-F. Chang, “Unsupervised embedding learning via invariant and spreading instance feature,” in CVPR, 2019, pp. 6210–6219.
  125. X. Yi, J. Yang, L. Hong, D. Z. Cheng, L. Heldt, A. Kumthekar, Z. Zhao, L. Wei, and E. Chi, “Sampling-bias-corrected neural modeling for large corpus item recommendations,” in RecSys, 2019, pp. 269–277.
  126. V. Karpukhin, B. Oğuz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih, “Dense passage retrieval for open-domain question answering,” arXiv preprint arXiv:2004.04906, 2020.
  127. Y. Qu, Y. Ding, J. Liu, K. Liu, R. Ren, W. X. Zhao, D. Dong, H. Wu, and H. Wang, “Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering,” arXiv preprint arXiv:2010.08191, 2020.
  128. Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, and L. Wang, “Graph contrastive learning with adaptive augmentation,” in WWW, 2021, pp. 2069–2080.
  129. K. Zhou, B. Zhang, W. X. Zhao, and J.-R. Wen, “Debiased contrastive learning of unsupervised sentence representations,” arXiv preprint arXiv:2205.00656, 2022.
  130. T. Huynh, S. Kornblith, M. R. Walter, M. Maire, and M. Khademi, “Boosting contrastive self-supervised learning with false negative cancellation,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 2785–2795.
  131. J. Zhan, J. Mao, Y. Liu, J. Guo, M. Zhang, and S. Ma, “Optimizing dense retrieval model training with hard negatives,” in SIGIR, 2021, pp. 1503–1512.
  132. O. Barkan and N. Koenigstein, “Item2vec: neural item embedding for collaborative filtering,” in MLSP.   IEEE, 2016, pp. 1–6.
  133. H.-F. Yu, M. Bilenko, and C.-J. Lin, “Selection of negative samples for one-class matrix factorization,” in ICDM.   SIAM, 2017, pp. 363–371.
  134. D. Lian, Q. Liu, and E. Chen, “Personalized ranking with importance sampling,” in Proceedings of The Web Conference 2020, 2020, pp. 1093–1103.
  135. B. Jin, D. Lian, Z. Liu, Q. Liu, J. Ma, X. Xie, and E. Chen, “Sampling-decomposable generative adversarial recommender,” NIPS, vol. 33, pp. 22 629–22 639, 2020.
  136. R. v. d. Berg, T. N. Kipf, and M. Welling, “Graph convolutional matrix completion,” arXiv preprint arXiv:1706.02263, 2017.
  137. S. Wu, Y. Tang, Y. Zhu, L. Wang, X. Xie, and T. Tan, “Session-based recommendation with graph neural networks,” in AAAI, vol. 33, no. 01, 2019, pp. 346–353.
  138. H. Gao and H. Huang, “Self-paced network embedding,” in KDD, 2018, pp. 1406–1415.
  139. M. Armandpour, P. Ding, J. Huang, and X. Hu, “Robust negative sampling for network embedding,” in AAAI, vol. 33, no. 01, 2019, pp. 3191–3198.
  140. Y. Zhu, Y. Xu, H. Cui, C. Yang, Q. Liu, and S. Wu, “Structure-enhanced heterogeneous graph contrastive learning,” in SDM.   SIAM, 2022, pp. 82–90.
  141. V. Kanojia, H. Maeda, R. Togashi, and S. Fujita, “Enhancing knowledge graph embedding with probabilistic negative sampling,” in WWW, 2017, pp. 801–802.
  142. B. Kotnis and V. Nastase, “Analysis of the impact of negative sampling on link prediction in knowledge graphs,” arXiv preprint arXiv:1708.06816, 2017.
  143. Z. Sun, Z.-H. Deng, J.-Y. Nie, and J. Tang, “Rotate: Knowledge graph embedding by relational rotation in complex space,” arXiv preprint arXiv:1902.10197, 2019.
  144. Y. Shan, C. Bu, X. Liu, S. Ji, and L. Li, “Confidence-aware negative sampling method for noisy knowledge graph embedding,” in ICBK.   IEEE, 2018, pp. 33–40.
  145. M. K. Islam, S. Aridhi, and M. Smaïl-Tabbone, “Simple negative sampling for link prediction in knowledge graphs,” in International Conference on Complex Networks and Their Applications.   Springer, 2021, pp. 549–562.
  146. S. Qin, G. Rao, C. Bin, L. Chang, T. Gu, and W. Xuan, “Knowledge graph embedding based on adaptive negative sampling,” in International Conference of Pioneering Computer Scientists, Engineers and Educators.   Springer, 2019, pp. 551–563.
  147. Y. Li, L. Liu, and S. Shi, “Empirical analysis of unlabeled entity problem in named entity recognition,” arXiv preprint arXiv:2012.05426, 2020.
  148. Y.  Li, L. Liu, and S. Shi, “Rethinking negative sampling for handling missing entity annotations,” in ACL, 2022, pp. 7188–7197.
  149. M. Guo, Q. Shen, Y. Yang, H. Ge, D. Cer, G. H. Abrego, K. Stevens, N. Constant, Y.-H. Sung, B. Strope et al., “Effective parallel corpus mining using bilingual sentence embeddings,” arXiv preprint arXiv:1807.11906, 2018.
  150. Z. Wu, S. Wang, J. Gu, M. Khabsa, F. Sun, and H. Ma, “Clear: Contrastive learning for sentence representation,” arXiv preprint arXiv:2012.15466, 2020.
  151. D. Wang, N. Ding, P. Li, and H.-T. Zheng, “Cline: Contrastive learning with semantic negative examples for natural language understanding,” arXiv preprint arXiv:2107.00440, 2021.
  152. Y. Zhang, R. Zhang, S. Mensah, X. Liu, and Y. Mao, “Unsupervised sentence representation via contrastive learning with mixing negatives,” 2022.
  153. D. Gillick, S. Kulkarni, L. Lansing, A. Presta, J. Baldridge, E. Ie, and D. Garcia-Olano, “Learning dense representations for entity retrieval,” arXiv preprint arXiv:1909.10506, 2019.
  154. J.-T. Huang, A. Sharma, S. Sun, L. Xia, D. Zhang, P. Pronin, J. Padmanabhan, G. Ottaviano, and L. Yang, “Embedding-based retrieval in facebook search,” in KDD, 2020, pp. 2553–2561.
  155. A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018.
  156. Y. Tian, D. Krishnan, and P. Isola, “Contrastive multiview coding,” in ECCV.   Springer, 2020, pp. 776–794.
  157. I. Misra and L. v. d. Maaten, “Self-supervised learning of pretext-invariant representations,” in CVPR, 2020, pp. 6707–6717.
  158. F. Wang and H. Liu, “Understanding the behaviour of contrastive loss,” in CVPR, 2021, pp. 2495–2504.
  159. S. Ge, S. Mishra, C.-L. Li, H. Wang, and D. Jacobs, “Robust contrastive learning using negative samples with diminished semantics,” NIPS, vol. 34, pp. 27 356–27 368, 2021.
  160. P. Sermanet, K. Xu, and S. Levine, “Unsupervised perceptual rewards for imitation learning,” arXiv preprint arXiv:1612.06699, 2016.
  161. P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain, “Time-contrastive networks: Self-supervised learning from video,” in 2018 IEEE international conference on robotics and automation (ICRA).   IEEE, 2018, pp. 1134–1141.
  162. S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta, “R3m: A universal visual representation for robot manipulation,” arXiv preprint arXiv:2203.12601, 2022.
  163. C. Mao, Z. Zhong, J. Yang, C. Vondrick, and B. Ray, “Metric learning for adversarial robustness,” NIPS, vol. 32, 2019.
  164. A. Iscen, G. Tolias, Y. Avrithis, and O. Chum, “Mining on manifolds: Metric learning without labels,” in CVPR, 2018, pp. 7642–7651.
  165. Y. Yuan, K. Yang, and C. Zhang, “Hard-aware deeply cascaded embedding,” in ICCV, 2017, pp. 814–823.
  166. X. Xin, Y. Fajie, H. Xiangnan, and J. Joemon, “Batch is not heavy: Learning word embeddings from all samples,” in ACL, vol. 8, 2018, pp. 107–132.
  167. C. Chen, M. Zhang, C. Wang, W. Ma, M. Li, Y. Liu, and S. Ma, “An efficient adaptive transfer neural network for social-aware recommendation,” in SIGIR, 2019, pp. 225–234.
  168. C. Chen, M. Zhang, Y. Zhang, Y. Liu, and S. Ma, “Efficient neural matrix factorization without sampling for recommendation,” TOIS, vol. 38, no. 2, pp. 1–28, 2020.
  169. C.  Chen, M. Zhang, W. Ma, Y. Liu, and S. Ma, “Jointly non-sampling learning for knowledge graph enhanced recommendation,” in SIGIR, 2020, pp. 189–198.
  170. M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” NIPS, vol. 33, pp. 9912–9924, 2020.
  171. J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar et al., “Bootstrap your own latent-a new approach to self-supervised learning,” NIPS, vol. 33, pp. 21 271–21 284, 2020.
  172. X. Chen and K. He, “Exploring simple siamese representation learning,” in CVPR, 2021, pp. 15 750–15 758.
  173. S. Thakoor, C. Tallec, M. G. Azar, M. Azabou, E. L. Dyer, R. Munos, P. Veličković, and M. Valko, “Large-scale representation learning on graphs via bootstrapping,” arXiv preprint arXiv:2102.06514, 2021.
  174. K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in CVPR, 2022, pp. 16 000–16 009.
  175. Z. Hou, X. Liu, Y. Dong, C. Wang, J. Tang et al., “Graphmae: Self-supervised masked graph autoencoders,” arXiv preprint arXiv:2205.10803, 2022.
  176. S. Arora, H. Khandeparkar, M. Khodak, O. Plevrakis, and N. Saunshi, “A theoretical analysis of contrastive unsupervised representation learning,” arXiv preprint arXiv:1902.09229, 2019.
  177. C. Wu, F. Wu, and Y. Huang, “Rethinking infonce: How many negative samples do you need?” arXiv preprint arXiv:2105.13003, 2021.
  178. J. T. Ash, S. Goel, A. Krishnamurthy, and D. Misra, “Investigating the role of negatives in contrastive representation learning,” arXiv preprint arXiv:2106.09943, 2021.
  179. K. Nozawa and I. Sato, “Understanding negative samples in instance discriminative self-supervised representation learning,” NIPS, vol. 34, pp. 5784–5797, 2021.
  180. P. Awasthi, N. Dikkala, and P. Kamath, “Do more negative samples necessarily hurt in contrastive learning?” arXiv preprint arXiv:2205.01789, 2022.
  181. K. Sohn, “Improved deep metric learning with multi-class n-pair loss objective,” NIPS, vol. 29, 2016.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Lightbulb On Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

This paper has been mentioned in 2 posts and received 11 likes.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up to View

Don't miss out on important new AI/ML research

See which papers are being discussed right now on X, Reddit, and more:

Explore Trending Papers

“Emergent Mind helps me see which AI papers have caught fire online.”

Philip

Philip

Creator, AI Explained on YouTube