ConsPrompt: Exploiting Contrastive Samples for Fewshot Prompt Learning (2211.04118v3)
Abstract: The prompt has become an effective linguistic tool for utilizing pre-trained LLMs. However, in few-shot scenarios, subtle changes in the prompt design always make the result widely different, and the prompt learning methods also make it easy to overfit the limited samples. To alleviate this, we explore utilizing suitable contrastive samples and multi-degree contrastive learning methods to improve the robustness of the prompt representation. Therefore, the proposed Consprompt combined with the prompt encoding network, contrastive sampling modules, and contrastive scoring modules, is introduced to realize differential contrastive learning. Our results exhibit state-of-the-art performance in different few-shot settings, and the ablation experiments also certify the effectiveness of utilizing multi-degree contrastive learning in the prompt-based fine-tuning process.
- “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, pp. 9, 2019.
- “Few-shot bot: Prompt-based learning for dialogue systems,” CoRR, vol. abs/2110.08118, 2021.
- “Factual probing is [mask]: Learning vs. learning to recall,” 2021.
- “Gpt understands, too,” arXiv preprint arXiv:2103.10385, 2021.
- “Prefix-tuning: Optimizing continuous prompts for generation,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, Aug. 2021, pp. 4582–4597, Association for Computational Linguistics.
- “Autoprompt: Eliciting knowledge from language models with automatically generated prompts,” arXiv preprint arXiv:2010.15980, 2020.
- “Exploiting cloze-questions for few-shot text classification and natural language inference,” in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Online, Apr. 2021, pp. 255–269, Association for Computational Linguistics.
- “How much knowledge can you pack into the parameters of a language model?,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, Nov. 2020, pp. 5418–5426, Association for Computational Linguistics.
- “A contrastive self-distillation bert with kernel alignment-based inference,” in International Conference on Computational Science. Springer, 2023, pp. 553–565.
- “Sentence-bert: Sentence embeddings using siamese bert-networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 3982–3992.
- “Making pre-trained language models better few-shot learners,” arXiv preprint arXiv:2012.15723, 2020.
- “A simple framework for contrastive learning of visual representations,” in International conference on machine learning. PMLR, 2020, pp. 1597–1607.
- “Making pre-trained language models end-to-end few-shot learners with contrastive prompt tuning,” arXiv preprint arXiv:2204.00166, 2022.
- “Contrastive learning for prompt-based few-shot language learners,” in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States, July 2022, pp. 5577–5587, Association for Computational Linguistics.
- “Roberta: A robustly optimized bert pretraining approach,” ArXiv, vol. abs/1907.11692, 2019.
- “Language models are few-shot learners,” ArXiv, vol. abs/2005.14165, 2020.