Structured Prompting: Scaling In-Context Learning to 1,000 Examples (2212.06713v1)

Abstract: LLMs have exhibited intriguing in-context learning capability, achieving promising zero- and few-shot performance without updating the parameters. However, conventional in-context learning is usually restricted by length constraints, rendering it ineffective to absorb supervision from a large number of examples. In order to go beyond few shots, we introduce structured prompting that breaks the length limit and scales in-context learning to thousands of examples. Specifically, demonstration examples are separately encoded with well-designed position embeddings, and then they are jointly attended by the test example using a rescaled attention mechanism. So we can scale the number of exemplars with linear complexity instead of quadratic complexity with respect to length. Experimental results on a diverse set of tasks show that our approach improves end-task performance and reduces evaluation variance over conventional in-context learning as the number of demonstration examples increases. Code has been released at https://aka.ms/structured-prompting.

• The paper presents a structured prompting approach that overcomes quadratic complexity limitations by grouping context examples.
• It employs right-aligned position embeddings and a rescaled attention mechanism to maintain balanced focus between demonstrations and input.
• Experiments demonstrate significant accuracy improvements and reduced variance across text classification and generation tasks.

Essay on "Structured Prompting: Scaling In-Context Learning to 1,000 Examples"

The paper entitled "Structured Prompting: Scaling In-Context Learning to 1,000 Examples" presents a novel approach aimed at enhancing the in-context learning capabilities of LLMs. In-context learning, a paradigm that facilitates task adaptation by providing task-specific instructions and example demonstrations, has traditionally been constrained by the length limits of LLMs. This paper proposes a method to break these constraints, enabling the scaling of in-context learning from few-shot to utilizing thousands of examples effectively.

Methodological Innovations

The primary innovation introduced in this paper is "structured prompting," which is designed to overcome the quadratic complexity limitations of conventional in-context learning that prevent scaling beyond a few examples. The core methodology involves:

• Grouped Context Encoding: The structured prompting approach begins by dividing a large number of demonstrations into several groups. Each group is independently encoded using the LLM. This independent encoding allows the overall computational complexity to shift from quadratic to linear concerning the number of demonstrations.
• Right-Aligned Position Embeddings: Groups are right-aligned using position embeddings to ensure they maintain consistent relative distances from the test input. This alignment is crucial for the model to attend equally to each exemplar.
• Rescaled Attention Mechanism: A novel rescaled attention strategy is proposed, which adjusts the attention weights to prevent the test input from being overshadowed by numerous exemplars. This mechanism effectively normalizes the attention scores, allowing the model to maintain a balance between the demonstrations and the test input.

Experimental Results

The experimental validations are extensive, covering a wide array of tasks, including text classification, multi-choice, and open-ended generation tasks. The results demonstrate that structured prompting not only improves the end-task performance but also significantly reduces evaluation variance. This reduction in variance is particularly beneficial in stabilizing the output of in-context learning across different permutations of exemplars.

The paper emphasizes significant numerical improvements over traditional methods. For instance, structured prompting outperforms conventional approaches in text classification tasks like SST-2, AGNews, and RTE, among others. The results indicate consistent accuracy improvements and stability across various datasets and model sizes.

Theoretical and Practical Implications

Theoretically, this work implies a shift in how we consider context usage in LLMs, showing that greater exemplar scaling does not necessitate an increase in computational burden. Practically, the model's ability to incorporate a broader spectrum of examples without incurring prohibitive computational costs opens new avenues for deploying LLMs in more data-rich scenarios, where previously in-context learning was not feasible due to length limitations.

Future Directions

The paper acknowledges several areas for future exploration, particularly around improving autoregressive extrapolation in larger LLMs and further refining the structured prompting strategy. Further investigation into pretraining strategies that align better with structured in-context learning could yield even more sophisticated models that intrinsically understand parallel relationships within input data.

In conclusion, structured prompting marks a significant step forward in advancing the in-context learning capabilities of LLMs. By effectively scaling the number of examples used within the context, this method offers a promising approach to enhancing model performance and stability in various applications. The insights from this research could potentially guide future developments in AI, where efficient and extensive utilization of training data becomes increasingly critical.