Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

UNSEE: Unsupervised Non-contrastive Sentence Embeddings (2401.15316v3)

Published 27 Jan 2024 in cs.CL

Abstract: We present UNSEE: Unsupervised Non-Contrastive Sentence Embeddings, a novel approach that outperforms SimCSE in the Massive Text Embedding benchmark. Our exploration begins by addressing the challenge of representation collapse, a phenomenon observed when contrastive objectives in SimCSE are replaced with non-contrastive objectives. To counter this issue, we propose a straightforward solution known as the target network, effectively mitigating representation collapse. The introduction of the target network allows us to leverage non-contrastive objectives, maintaining training stability while achieving performance improvements comparable to contrastive objectives. Our method has achieved peak performance in non-contrastive sentence embeddings through meticulous fine-tuning and optimization. This comprehensive effort has yielded superior sentence representation models, showcasing the effectiveness of our approach.

View on arXiv
Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Elad Amrani and Alexander M. Bronstein. 2021. Self-supervised classification network. ArXiv, abs/2103.10994.
  2. Masked siamese networks for label-efficient learning. In European Conference on Computer Vision.
  3. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. In ICLR.
  4. A large annotated corpus for learning natural language inference. In Conference on Empirical Methods in Natural Language Processing.
  5. Deep clustering for unsupervised learning of visual features.
  6. Unsupervised learning of visual features by contrasting cluster assignments. ArXiv, abs/2006.09882.
  7. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada. Association for Computational Linguistics.
  8. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 169–174, Brussels, Belgium. Association for Computational Linguistics.
  9. Xinlei Chen and Kaiming He. 2020. Exploring simple siamese representation learning. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15745–15753.
  10. DiffCSE: Difference-based contrastive learning for sentence embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  11. Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  12. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics.
  13. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  14. Whitening for self-supervised representation learning. ArXiv, abs/2007.06346.
  15. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3816–3830, Online. Association for Computational Linguistics.
  16. Simcse: Simple contrastive learning of sentence embeddings. ArXiv, abs/2104.08821.
  17. Bootstrap your own latent: A new approach to self-supervised learning. ArXiv, abs/2006.07733.
  18. Deberta: Decoding-enhanced bert with disentangled attention. ArXiv, abs/2006.03654.
  19. Learning distributed representations of sentences from unlabelled data. ArXiv, abs/1602.03483.
  20. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9:1735–1780.
  21. Promptbert: Improving bert sentence embeddings with prompts.
  22. Skip-thought vectors. In NIPS.
  23. Tassilo Klein and Moin Nabi. 2022. Scd: Self-contrastive decorrelation of sentence embeddings. ArXiv, abs/2203.07847.
  24. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9119–9130, Online. Association for Computational Linguistics.
  25. Towards general text embeddings with multi-stage contrastive learning.
  26. Trans-encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations. In ICLR 2022.
  27. Fast, effective, and self-supervised: Transforming masked language models into universal lexical and sentence encoders. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1442–1459, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  28. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.
  29. Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. ArXiv, abs/1803.02893.
  30. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistics.
  31. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
  32. Self-supervised learning with an information maximization criterion.
  33. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  34. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108.
  35. One embedder, any task: Instruction-finetuned text embeddings.
  36. Attention is all you need. In NIPS.
  37. Text embeddings by weakly-supervised contrastive pre-training. ArXiv, abs/2212.03533.
  38. Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. ArXiv, abs/2005.10242.
  39. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  40. C-pack: Packaged resources to advance general chinese embedding.
  41. Linhan Zhang Yuxin Jiang and Wei Wang. 2022. Improved universal sentence embeddings with prompt-based contrastive learning and energy-based learning.
  42. Barlow twins: Self-supervised learning via redundancy reduction. In ICML.
  43. Bootstrapped unsupervised sentence representation learning. In Annual Meeting of the Association for Computational Linguistics.
  44. Bootstrapped unsupervised sentence representation learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5168–5180, Online. Association for Computational Linguistics.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (1)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets