Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
Abstract: This paper surveys and organizes research works in a new paradigm in natural language processing, which we dub "prompt-based learning". Unlike traditional supervised learning, which trains a model to take in an input x and predict an output y as P(y|x), prompt-based learning is based on LLMs that model the probability of text directly. To use these models to perform prediction tasks, the original input x is modified using a template into a textual string prompt x' that has some unfilled slots, and then the LLM is used to probabilistically fill the unfilled information to obtain a final string x, from which the final output y can be derived. This framework is powerful and attractive for a number of reasons: it allows the LLM to be pre-trained on massive amounts of raw text, and by defining a new prompting function the model is able to perform few-shot or even zero-shot learning, adapting to new scenarios with few or no labeled data. In this paper we introduce the basics of this promising paradigm, describe a unified set of mathematical notations that can cover a wide variety of existing work, and organize existing work along several dimensions, e.g.the choice of pre-trained models, prompts, and tuning strategies. To make the field more accessible to interested beginners, we not only make a systematic review of existing works and a highly structured typology of prompt-based concepts, but also release other resources, e.g., a website http://pretrain.nlpedia.ai/ including constantly-updated survey, and paperlist.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What is this paper about?
This paper is a big “map” of a new way to use AI for language called prompt-based learning. Instead of training a separate model for every task (like sentiment analysis, translation, or question answering), we use one LLM that’s been trained on tons of text, and we “prompt” it with the right words or patterns so it can do the task—sometimes with almost no extra training. The authors explain the ideas, tools, and tricks behind prompting, and organize the growing research so newcomers can learn it faster.
What questions are they trying to answer?
- What is prompting, and how is it different from older ways of training LLMs?
- How do you turn a task into a prompt the model understands?
- What kinds of LLMs (like GPT, BERT, T5) work best with which prompts?
- How can we design good prompts and good answer formats (and even automate that)?
- What training strategies work: no training at all, tuning the prompt, tuning the model, or both?
- Where does prompting help most, and what are the current challenges?
How did they study it? (Their approach)
This is a survey paper. That means the authors:
- Collected and compared many research papers on prompting.
- Created a shared, simple way to describe how prompting works.
- Built a typology (a structured “family tree”) of methods and choices in prompting.
- Gave examples, patterns, and resources (including a website) to help beginners.
Here are the core ideas they explain in everyday terms:
- LLM (LM): Think of a model that’s read a huge library and learned how words usually fit together. It can guess the next word or fill in missing words.
- Prompt: A short instruction or “fill‑in‑the‑blank” wrapper that turns your problem into something the model already knows how to do. Examples:
- Cloze (fill‑in‑the‑blank): “I missed the bus today. I felt so [Z].”
- Prefix (instruction + answer): “English: I missed the bus today. French: [Z]”
- Few-shot and zero-shot:
- Zero-shot: You just give the prompt; no examples.
- Few-shot: You add a few examples to guide the model.
- Three steps of prompting (simple version): 1) Prompt addition: Wrap your input with a template (the “instruction” and a blank to fill). 2) Answer search: Let the model fill in the blank (it picks the most likely word(s)). 3) Answer mapping: If needed, map the filled word(s) to the final answer (e.g., “great” → “very positive”).
- Prompt engineering: Crafting (or automatically learning) good templates—that is, choosing the right words and structure for the prompt.
- Answer engineering: Choosing the shape of the answer (single word, span, full sentence) and how to map it to labels or outputs.
- Types of pre-trained models and when they fit:
- GPT-style (left-to-right): Great for prompts that end with an answer to generate (prefix prompts).
- BERT-style (masked): Great for fill‑in‑the‑blank (cloze) prompts for understanding tasks.
- Encoder–decoder (T5/BART): Good at turning input text into output text (translation, summarization), and work well with many prompt styles.
- Training strategies with prompts:
- Tuning-free: Just prompt the model; don’t change its weights.
- Prompt tuning: Learn the prompt (sometimes as special “virtual” tokens) but keep the model frozen.
- Fine-tuning: Adjust the model itself for the task (optionally with a prompt).
- Combined: Tune both prompt and model.
What did they find?
The authors organize the field around a few key choices that strongly affect results:
- Model choice matters:
- GPT-like models are strong for generating answers after a prefix prompt.
- BERT-like models shine when you can phrase the task as a fill‑in‑the‑blank.
- Encoder–decoder models are flexible for input→output tasks.
- Prompt and answer design matter a lot:
- The exact wording of a prompt can make performance jump up or down.
- “Cloze” prompts pair naturally with masked models; “prefix” prompts pair naturally with generative models.
- Answers can be single words, short spans, or whole sentences; picking the right format helps.
- You can automate some parts:
- Algorithms can search for good prompt words (discrete prompts) or learn “soft” prompt vectors the model understands (continuous prompts).
- You can also ensemble multiple prompts or combine/decompose them for tougher tasks.
- Training choices span a spectrum:
- Zero/few-shot prompting is great when you lack labeled data.
- Prompt tuning (with a frozen model) uses far fewer parameters than full fine-tuning, yet can perform well.
- Full fine-tuning can still help, especially with more data or when you need maximum control.
- Big picture: “Pre-train, prompt, and predict”
- Instead of “pre-train, fine-tune” for every task, prompting lets one big model tackle many tasks by changing the instruction (the prompt), sometimes with no extra training.
Why this is important:
- It can dramatically reduce the need for expensive labeled datasets.
- One model can be adapted quickly to many tasks by changing the prompt.
- It opens new ways to “program” AI using natural language.
Why does this matter?
- Saves time and money: Less need to collect thousands of labeled examples for every new task.
- Flexible: The same base model can translate, summarize, answer questions, classify text, and more—just by changing prompts.
- Accessible: People can guide models with natural-language instructions instead of writing code-heavy pipelines.
There are also challenges:
- Prompt sensitivity: Small wording changes can change results a lot.
- Fairness and reliability: Prompts can accidentally trigger biases or errors.
- Evaluation: We need better, more consistent ways to test prompted models.
- Automation: Tools to suggest or learn better prompts are still developing.
What could be the impact?
- For researchers: A shared framework and vocabulary to compare methods, build better prompts, and design fairer tests.
- For developers: Faster prototypes—try tasks with zero or few labeled examples; use prompt or parameter-efficient tuning to deploy models with lower costs.
- For education and tools: User-friendly interfaces that help non-experts “program” AI by writing instructions, with built-in helpers to suggest and refine prompts.
- For the future: More general-purpose models that understand many tasks and languages, with improved methods to make prompts robust, fair, and easy to design.
In short, this paper shows how prompting turns LLMs into flexible problem-solvers, explains the main design choices, and points to a future where we guide AI mostly by telling it what we want—in plain language.
Collections
Sign up for free to add this paper to one or more collections.