ICL Markup: Structuring In-Context Learning using Soft-Token Tags (2312.07405v1)

Abstract: Large pretrained LLMs can be rapidly adapted to a wide variety of tasks via a text-to-text approach, where the instruction and input are fed to the model in natural language. Combined with in-context learning (ICL), this paradigm is impressively flexible and powerful. However, it also burdens users with an overwhelming number of choices, many of them arbitrary. Inspired by markup languages like HTML, we contribute a method of using soft-token tags to compose prompt templates. This approach reduces arbitrary decisions and streamlines the application of ICL. Our method is a form of meta-learning for ICL; it learns these tags in advance during a parameter-efficient fine-tuning ``warm-up'' process. The tags can subsequently be used in templates for ICL on new, unseen tasks without any additional fine-tuning. Our experiments with this approach yield promising initial results, improving LLM performance on important enterprise applications such as few-shot and open-world intent detection, as well as text classification in news and legal domains.

Overview of In-Context Learning and Challenges

In-Context Learning (ICL) with LLMs is a technique where users prompt a LLM with examples to perform new tasks without fine-tuning the model's parameters. This method is flexible, user-friendly, and data-efficient. However, the process is sensitive to how the prompts are structured, leading to highly variable outcomes based on seemingly trivial changes. This can make ICL unreliable and complex for users to navigate.

Proposed Solution: ICL Markup

To mitigate these issues, the paper introduces a novel approach akin to a markup language, named ICL Markup, which structures the in-context learning prompts using soft-token tags embedded in the model's vocabulary. These tags are like new words that can be trained during a parameter-efficient warm-up stage. Once learned, they facilitate ICL without further fine-tuning and across various tasks, acting as a form of meta-learning.

Experiments and Results

The effectiveness of ICL Markup is tested through a series of experiments:

1. It proves to be advantageous in few-shot and highly multi-class intent detection tasks, enhancing the adaptability of the LLMs.
2. In text classification tasks with varying complexities, including news headlines and legal texts, ICL Markup demonstrates improvements in performance and consistency.
3. A particular highlight is the application to intent detection, where the method supports models in recognizing both in-scope and out-of-scope intents, a crucial feature for practical use in virtual assistants.
4. Beyond the domain of intent detection, the utility of soft-token tags is evident in legal text classification tasks, suggesting their potential for cross-domain applications.

Limitations and Future Directions

While promising, the current research is limited to particular model sizes and primarily classification tasks. Future work could extend to larger and more diverse model architectures, explore a wider range of applications, and refine the approach by introducing domain-specific tags like one indicating out-of-scope responses.

Implications for Robust In-Context Learning

ICL Markup represents a step toward robust and structured in-context learning. By standardizing the prompt construction process, the approach reduces the burden of prompt engineering on users, allowing them to focus on the content and application of the models rather than the intricacies of the prompt design. This innovation stands to make LLMs more accessible and effective for real-world applications across various industries and domains.