Revisiting Large Language Models as Zero-shot Relation Extractors (2310.05028v4)
Abstract: Relation extraction (RE) consistently involves a certain degree of labeled or unlabeled data even if under zero-shot setting. Recent studies have shown that LLMs transfer well to new tasks out-of-the-box simply given a natural language prompt, which provides the possibility of extracting relations from text without any data and parameter tuning. This work focuses on the study of exploring LLMs, such as ChatGPT, as zero-shot relation extractors. On the one hand, we analyze the drawbacks of existing RE prompts and attempt to incorporate recent prompt techniques such as chain-of-thought (CoT) to improve zero-shot RE. We propose the summarize-and-ask (\textsc{SumAsk}) prompting, a simple prompt recursively using LLMs to transform RE inputs to the effective question answering (QA) format. On the other hand, we conduct comprehensive experiments on various benchmarks and settings to investigate the capabilities of LLMs on zero-shot RE. Specifically, we have the following findings: (i) \textsc{SumAsk} consistently and significantly improves LLMs performance on different model sizes, benchmarks and settings; (ii) Zero-shot prompting with ChatGPT achieves competitive or superior results compared with zero-shot and fully supervised methods; (iii) LLMs deliver promising performance in extracting overlapping relations; (iv) The performance varies greatly regarding different relations. Different from small LLMs, LLMs are effective in handling challenge none-of-the-above (NoTA) relation.
- Large language models are few-shot clinical information extractors. In EMNLP.
- TACRED revisited: A thorough evaluation of the TACRED relation extraction task. In ACL.
- Ask me anything: A simple strategy for prompting language models. In ICLR.
- Language models are few-shot learners. In NeurIPS.
- Chih-Yao Chen and Cheng-Te Li. 2021. ZS-BERT: Towards zero-shot relation extraction with attribute representation learning. In NAACL-HLT.
- Reading wikipedia to answer open-domain questions. In ACL.
- Enhanced LSTM for natural language inference. In ACL.
- Relationprompt: Leveraging prompts to generate synthetic data for zero-shot relation triplet extraction. In Findings of ACL.
- Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
- Active prompting with chain-of-thought for large language models. arXiv preprint arXiv:2302.12246.
- Classifying relations by ranking with convolutional neural networks. In ACL.
- More data, more relations, more context and more openness: A review and outlook for relation extraction. arXiv preprint arXiv:2004.03186.
- FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In EMNLP.
- Thinking about GPT-3 in-context learning for biomedical IE? think again. In Findings of EMNLP.
- Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77.
- Large language models are zero-shot reasoners. In NeurIPS.
- Zero-shot relation extraction via reading comprehension. In CoNLL.
- Fastre: Towards fast relation extraction with convolutional encoder and improved cascade binary tagging framework. In IJCAI.
- What makes good in-context examples for gpt-3333? arXiv preprint arXiv:2101.06804.
- Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35.
- Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In EMNLP.
- Summarization as indirect supervision for relation extraction. In Findings of EMNLP.
- Large language model is not a good few-shot information extractor, but a good reranker for hard samples! arXiv preprint arXiv:2303.08559.
- Training language models to follow instructions with human feedback. In NeurIPS.
- Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP.
- Modeling relations and their mentions without labeled text. In ECML PKDD.
- Reasoning about entailment with neural attention. In ICLR.
- Label verbalization and entailment for effective zero-and few-shot relation extraction. In EMNLP.
- Multitask prompted training enables zero-shot task generalization. In ICLR.
- Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100.
- Prototypical networks for few-shot learning. In NIPS.
- Re-tacred: Addressing shortcomings of the tacred dataset. In AAAI.
- Ben Wang and Aran Komatsuzaki. 2021. Gpt-j-6b: A 6 billion parameter autoregressive language model.
- Deepstruct: Pretraining of language models for structure prediction. In Findings of ACL.
- Instructuie: Multi-task instruction tuning for unified information extraction. arXiv preprint arXiv:2304.08085.
- Self-consistency improves chain of thought reasoning in language models. In ICLR.
- Chain of thought prompting elicits reasoning in large language models. In NeurIPS.
- Zero-shot information extraction via chatting with chatgpt. arXiv preprint arXiv:2302.10205.
- Shanchan Wu and Yifan He. 2019. Enriching pre-trained language model with entity information for relation classification. In CIKM.
- Luke: Deep contextualized entity representations with entity-aware self-attention. In EMNLP.
- Relation classification via convolutional deep neural network. In COLING.
- Aligning instruction tasks unlocks large language models as zero-shot relation extractors. In Findings of ACL.
- Graph convolution over pruned dependency trees improves relation extraction. In EMNLP.
- Position-aware attention and supervised data improve slot filling. In EMNLP.
- Re-matching: A fine-grained semantic matching method for zero-shot relation extraction. In ACL.
- Calibrate before use: Improving few-shot performance of language models. In ICML.
- Wenxuan Zhou and Muhao Chen. 2021. An improved baseline for sentence-level relation extraction. arXiv preprint arXiv:2102.01373.
- Guozheng Li (19 papers)
- Peng Wang (831 papers)
- Wenjun Ke (9 papers)