Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Synergistic Anchored Contrastive Pre-training for Few-Shot Relation Extraction (2312.12021v3)

Published 19 Dec 2023 in cs.CL and cs.AI

Abstract: Few-shot Relation Extraction (FSRE) aims to extract relational facts from a sparse set of labeled corpora. Recent studies have shown promising results in FSRE by employing Pre-trained LLMs (PLMs) within the framework of supervised contrastive learning, which considers both instances and label facts. However, how to effectively harness massive instance-label pairs to encompass the learned representation with semantic richness in this learning paradigm is not fully explored. To address this gap, we introduce a novel synergistic anchored contrastive pre-training framework. This framework is motivated by the insight that the diverse viewpoints conveyed through instance-label pairs capture incomplete yet complementary intrinsic textual semantics. Specifically, our framework involves a symmetrical contrastive objective that encompasses both sentence-anchored and label-anchored contrastive losses. By combining these two losses, the model establishes a robust and uniform representation space. This space effectively captures the reciprocal alignment of feature distributions among instances and relational facts, simultaneously enhancing the maximization of mutual information across diverse perspectives within the same relation. Experimental results demonstrate that our framework achieves significant performance enhancements compared to baseline models in downstream FSRE tasks. Furthermore, our approach exhibits superior adaptability to handle the challenges of domain shift and zero-shot relation extraction. Our code is available online at https://github.com/AONE-NLP/FSRE-SaCon.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. Question Answering with Subgraph Embeddings. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 615–620.
  2. Flex: Unifying evaluation for few-shot nlp. Advances in Neural Information Processing Systems, 34: 15787–15800.
  3. Cetoli, A. 2020. Exploring the zero-shot limit of FewRel. In Proceedings of the 28th International Conference on Computational Linguistics, 1447–1451.
  4. ZS-BERT: Towards Zero-Shot Relation Extraction with Attribute Representation Learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 3470–3479.
  5. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, 1597–1607.
  6. Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. In Proceedings of the ACM Web Conference 2022, 2778–2788.
  7. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186.
  8. Unified language model pre-training for natural language understanding and generation. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, 13063–13075.
  9. MapRE: An Effective Semantic Mapping Approach for Low-resource Relation Extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2694–2704.
  10. Hybrid attention-based prototypical networks for noisy few-shot relation classification. In Proceedings of the AAAI conference on artificial intelligence, volume 33, 6407–6414.
  11. FewRel 2.0: Towards More Challenging Few-Shot Relation Classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 6250–6255.
  12. Exploring Task Difficulty for Few-Shot Relation Extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2605–2616.
  13. FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 4803–4809.
  14. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9729–9738.
  15. Supervised contrastive learning. Advances in neural information processing systems, 33: 18661–18673.
  16. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2. Lille.
  17. Contrastive representation learning: A framework and review. Ieee Access, 8: 193907–193934.
  18. Zero-Shot Relation Extraction via Reading Comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), 333–342.
  19. Pre-training to Match for Unified Low-shot Relation Extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 5785–5795.
  20. Learn from relation information: Towards prototype representation rectification for few-shot relation extraction. In Findings of the Association for Computational Linguistics: NAACL 2022, 1822–1831.
  21. A Simple yet Effective Relation Information Guided Approach for Few-Shot Relation Extraction. In Findings of the Association for Computational Linguistics: ACL 2022, 757–763.
  22. Decoupled Weight Decay Regularization. In International Conference on Learning Representations.
  23. Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 3219–3232.
  24. Zero-shot relation classification as textual entailment. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), 72–78.
  25. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
  26. Learning from Context or Names? An Empirical Study on Neural Relation Extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 3661–3672.
  27. ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 3350–3363.
  28. Few-shot relation extraction via Bayesian meta-learning on relation graphs. In Proceedings of the 37th International Conference on Machine Learning, 7867–7876.
  29. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748–8763. PMLR.
  30. Label Verbalization and Entailment for Effective Zero and Few-Shot Relation Extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 1199–1212.
  31. Prototypical networks for few-shot learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, 4080–4090.
  32. Matching the Blanks: Distributional Similarity for Relation Learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2895–2905.
  33. Hierarchical attention prototypical networks for few-shot text classification. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), 476–485.
  34. Contrastive multiview coding. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16, 776–794. Springer.
  35. What makes for good views for contrastive learning? In Proceedings of the 34th International Conference on Neural Information Processing Systems, 6827–6839.
  36. Representation Learning with Contrastive Predictive Coding. arXiv preprint arXiv:1807.03748.
  37. Visualizing data using t-SNE. Journal of machine learning research, 9(11).
  38. Matching networks for one shot learning. Advances in neural information processing systems, 29.
  39. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10): 78–85.
  40. Towards representation alignment and uniformity in collaborative filtering. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 1816–1825.
  41. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, 9929–9939. PMLR.
  42. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, 3733–3742.
  43. Explicit semantic ranking for academic search via knowledge graph embedding. In Proceedings of the 26th international conference on world wide web, 1271–1279.
  44. A survey on multi-view learning. arXiv preprint arXiv:1304.5634.
  45. Enhance prototypical network with text descriptions for few-shot relation classification. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, 2273–2276.
  46. Entity Concept-enhanced Few-shot Relation Extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), 987–991.
  47. Multi-Level Matching and Aggregation Network for Few-Shot Relation Classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2872–2881.
  48. Better Few-Shot Relation Extraction with Label Prompt Dropout. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 6996–7006. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics.
  49. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, 2–25. PMLR.
  50. A Frustratingly Easy Approach for Entity and Relation Extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 50–61.
  51. Exploring various knowledge in relation extraction. In Proceedings of the 43rd annual meeting of the association for computational linguistics, 427–434.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now