Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2107.13586v1)

Abstract: This paper surveys and organizes research works in a new paradigm in natural language processing, which we dub "prompt-based learning". Unlike traditional supervised learning, which trains a model to take in an input x and predict an output y as P(y|x), prompt-based learning is based on LLMs that model the probability of text directly. To use these models to perform prediction tasks, the original input x is modified using a template into a textual string prompt x' that has some unfilled slots, and then the LLM is used to probabilistically fill the unfilled information to obtain a final string x, from which the final output y can be derived. This framework is powerful and attractive for a number of reasons: it allows the LLM to be pre-trained on massive amounts of raw text, and by defining a new prompting function the model is able to perform few-shot or even zero-shot learning, adapting to new scenarios with few or no labeled data. In this paper we introduce the basics of this promising paradigm, describe a unified set of mathematical notations that can cover a wide variety of existing work, and organize existing work along several dimensions, e.g.the choice of pre-trained models, prompts, and tuning strategies. To make the field more accessible to interested beginners, we not only make a systematic review of existing works and a highly structured typology of prompt-based concepts, but also release other resources, e.g., a website http://pretrain.nlpedia.ai/ including constantly-updated survey, and paperlist.

An Expert Overview of "Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing"

Introduction

"Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing" by Liu et al. is a comprehensive examination of the emerging paradigm of prompt-based learning in NLP. Unlike traditional supervised learning, which relies on task-specific models and extensive labeled datasets, prompt-based learning leverages pre-trained LLMs (LMs) using prompts to guide the models in performing various NLP tasks. This survey aims to unify and systematize the diverse research efforts in this field, offering not only a review but also practical and theoretical insights into the use of prompts.

Core Paradigm

The survey categorizes the evolution of NLP into four paradigms: feature engineering, architecture engineering, objective engineering, and now, the pre-train, prompt, and predict paradigm. Prompt-based learning stands on the shoulders of pre-trained LMs such as GPT-3 and BERT. Instead of fine-tuning these models for each task, prompt-based learning modifies the input $\bm{x}$ into a prompt $\bm{x}'$ which helps the LM produce more accurate outputs $\bm{y}$ by filling in the blank spaces in the prompt.

Prompt Engineering

Prompt engineering is critical to the success of this learning paradigm. The primary methods include manual template crafting and automated template learning. Manual designs rely on human expertise to create effective prompts, while automated methods, such as gradient-based search and prompt paraphrasing, aim to optimize prompts algorithmically. This area remains a fertile ground for research, given the complex trade-offs between interpretability, effectiveness, and generalizability of prompts.

Answer Engineering

Answer engineering involves designing the space of possible answers $\mathcal{Z}$ and mapping them to the target output space $\mathcal{Y}$. Most current works focus on simple token or span-level answers, but emerging strategies like automated answer search offer promising directions. Future research could explore extending these methods, particularly in applications requiring structured or multi-token answers.

Training Strategies and Parameter Tuning

The survey examines various training strategies, including tuning-free prompting, fixed-LM prompt tuning, fixed-prompt LM tuning, and prompt+LM tuning.

• Tuning-free prompting keeps the LM's parameters fixed, relying entirely on the prompt for task specification.
• Fixed-LM prompt tuning involves tuning the prompt while keeping the LM parameters constant.
• Fixed-prompt LM tuning adjusts the LM parameters while using static prompts, combining the benefits of pre-trained models with specific task guidance.
• Prompt+LM tuning adjusts both prompt and LM parameters, offering the most flexibility but at the risk of overfitting.

Applications

Prompt-based methods have made significant inroads across a variety of NLP tasks:

• Knowledge Probing: Models like LAMA and X-FACTR use prompts to probe the factual and linguistic knowledge embedded within LMs.
• Text Classification and NLI: Prompt-based learning simplifies the reformulation of these tasks, making them suitable for few-shot scenarios.
• Information Extraction: Although challenging, prompts have been adapted for tasks like named entity recognition and relation extraction.
• Question Answering: Unified systems like UnifiedQA demonstrate the power of prompt-based approaches in handling diverse QA formats.
• Text Generation: Models such as GPT-3 showcase the flexibility of prompts in facilitating text generation tasks, including summarization and translation.

Challenges and Future Directions

Despite its potential, prompt-based learning faces several challenges:

• Prompt Design Complexity: Extending prompt use to tasks beyond classification and generation is non-trivial.
• Structured Data Integration: Encoding structured information in prompts requires further research.
• Training Dynamics: Understanding the interplay between prompt, LM parameter tuning, and dataset size is critical.
• Task-Specific Adaptation: Developing universal prompts that generalize across tasks remains an open question.

Furthermore, issues such as calibration of model probabilities and the interpretability of continuous prompts are areas ripe for investigation.

Conclusion

This survey not only highlights the efficacy of prompt-based learning but also identifies key challenges and areas for future exploration. By organizing the current state of knowledge and practice, Liu et al. provide a crucial resource for researchers and practitioners aiming to harness the full potential of NLP through prompt engineering. The pre-train, prompt, and predict paradigm represents a significant shift, with the potential to simplify and unify NLP model architectures while leveraging the capabilities of pre-trained LLMs.