BatchPrompt: Accomplish more with less (2309.00384v3)

Abstract: As the ever-increasing token limits of LLMs have enabled long context as input, prompting with single data samples might no longer an efficient way. A straightforward strategy improving efficiency is to batch data within the token limit (e.g., 8k for gpt-3.5-turbo; 32k for GPT-4), which we call BatchPrompt. We have two initial observations for prompting with batched data. First, we find that prompting with batched data in longer contexts will inevitably lead to worse performance, compared to single-data prompting. Second, the performance of the LLM is significantly correlated with the positions and order of the batched data, due to the corresponding change in decoder context. To retain efficiency and overcome performance loss, we propose Batch Permutation and Ensembling (BPE), and a novel Self-reflection-guided EArly Stopping (SEAS) technique. Our comprehensive experimental evaluation demonstrates that BPE can boost the performance of BatchPrompt with a striking margin on a range of popular NLP tasks, including question answering (Boolq), textual entailment (RTE), and duplicate questions identification (QQP). These performances are even competitive with/higher than single-data prompting(SinglePrompt), while BatchPrompt requires much fewer LLM calls and input tokens (For SinglePrompt v.s. BatchPrompt with batch size 32, using just 9%-16% the number of LLM calls, Boolq accuracy 90.6% to 90.9% with 27.4% tokens, QQP accuracy 87.2% to 88.4% with 18.6% tokens, RTE accuracy 91.5% to 91.1% with 30.8% tokens). To the best of our knowledge, this is the first work to technically improve prompting efficiency of LLMs. We hope our simple yet effective approach will shed light on the future research of LLMs. The code will be released.

Overview of "BatchPrompt: Accomplish More with Less"

The paper entitled "BatchPrompt: Accomplish More with Less" tackles the challenge of enhancing computational efficiency when using LLMs for NLP tasks. Recent advancements in LLMs have enabled the processing of extensive context; however, the resource utilization efficiency in conventional single-data prompting is suboptimal, particularly when the prompt's few-shot examples outweigh the length of the data points being processed. This research proposes a novel framework, termed "BatchPrompt," which batches multiple data points into a single prompt, thereby improving overall token utilization without compromising on the quality of results.

Key Contributions

1. BatchPrompt Strategy: The paper introduces the BatchPrompt technique as a method to increase the "density" of data points within a prompt. By batching data points, the strategy intends to achieve better token-resource utilization.
2. Batch Permutation and Ensembling (BPE): This component of the method addresses the performance variability of data points placed in different positions within a prompt, a common issue stemming from the autoregressive nature of LLMs. BPE enhances accuracy by permuting data within batches and applying a majority voting system over the varied orders, though this increases token consumption slightly.
3. Self-reflection-guided Early Stopping (SEAS): SEAS is proposed to counter the token overhead introduced by the voting mechanism. It determines when to terminate voting early based on self-assessed confidence levels provided by the LLMs, thus preserving computational resources while maintaining accuracy.

Experimental Findings

The efficiency and performance of BatchPrompt with BPE and SEAS were evaluated across several benchmarks like Boolq, QQP (Quora Question Pairs), and RTE (Recognizing Textual Entailment). The results are compelling:

• Boolq: With a batch size of 32 and using merely 15.7% of LLM calls, BatchPrompt+BPE+SEAS achieve an accuracy of 90.9%, compared to 90.6% with single-data prompting, while utilizing only 27.4% of the tokens.
• QQP: Accuracy improved from 87.2% to 88.4% while consuming just 18.6% of tokens.
• RTE: Accuracy remained competitive at 91.1% using 30.8% of tokens compared to the single-prompt setting.

Implications and Future Directions

The implications of this research are prevalent in computational efficiency for NLP tasks. This paper remarkably suggests that significant improvements in token-efficiency can be achieved without substantial sacrifices in quality, which has strong ramifications for optimizing the cost and computational demands of using LLMs.

The practical implications of BatchPrompt suggest that it can be integrated to extend the practical use of LLMs to applications constrained by computational resources. Theoretical implications also abound, as this research invites future work on optimizing prompt engineering techniques and exploring other aspects of model utilization efficiency.

In terms of future work, the paper briefly discusses potentially expanding the framework to broader NLP tasks and automating BatchPrompt configurations to suit different LLM architectures and application contexts more precisely. The researchers anticipate that further exploration could harness reinforcement learning or Bayesian optimization to dynamically set optimal batch sizes, voting rounds, and confidence thresholds for reducing costs and improving performance further.

This paper offers a promising step toward more efficient deployment of LLMs in real-world scenarios, balancing resource expenditure with the considerable capabilities LLMs offer.