Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash 105 tok/s
Gemini 2.5 Pro 52 tok/s Pro
GPT-5 Medium 45 tok/s
GPT-5 High 34 tok/s Pro
GPT-4o 108 tok/s
GPT OSS 120B 473 tok/s Pro
Kimi K2 218 tok/s Pro
2000 character limit reached

Efficient Prompt Optimization Through the Lens of Best Arm Identification (2402.09723v3)

Published 15 Feb 2024 in stat.ML, cs.AI, cs.LG, and cs.CL

Abstract: The remarkable instruction-following capability of LLMs has sparked a growing interest in automatically finding good prompts, i.e., prompt optimization. Most existing works follow the scheme of selecting from a pre-generated pool of candidate prompts. However, these designs mainly focus on the generation strategy, while limited attention has been paid to the selection method. Especially, the cost incurred during the selection (e.g., accessing LLM and evaluating the responses) is rarely explicitly considered. To overcome this limitation, this work provides a principled framework, TRIPLE, to efficiently perform prompt selection under an explicit budget constraint. TRIPLE is built on a novel connection established between prompt optimization and fixed-budget best arm identification (BAI-FB) in multi-armed bandits (MAB); thus, it is capable of leveraging the rich toolbox from BAI-FB systematically and also incorporating unique characteristics of prompt optimization. Extensive experiments on multiple well-adopted tasks using various LLMs demonstrate the remarkable performance improvement of TRIPLE over baselines while satisfying the limited budget constraints. As an extension, variants of TRIPLE are proposed to efficiently select examples for few-shot prompts, also achieving superior empirical performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24.
  2. Bayesian fixed-budget best-arm identification. arXiv preprint arXiv:2211.08572.
  3. Minimax policies for adversarial and stochastic bandits. In COLT, volume 7, pages 1–122.
  4. Best arm identification in multi-armed bandits. In COLT, pages 41–53.
  5. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47:235–256.
  6. Fixed-budget best-arm identification in structured bandits. arXiv preprint arXiv:2106.04763.
  7. A survey on practical applications of multi-armed and contextual bandits. arXiv preprint arXiv:1904.10040.
  8. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends® in Machine Learning, 5(1):1–122.
  9. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712.
  10. Instructzero: Efficient instruction optimization for black-box large language models. arXiv preprint arXiv:2306.03082.
  11. Rlprompt: Optimizing discrete text prompts with reinforcement learning. arXiv preprint arXiv:2205.12548.
  12. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  13. Black-box prompt learning for pre-trained language models. arXiv preprint arXiv:2201.08531.
  14. Best arm identification: A unified approach to fixed budget and fixed confidence. Advances in Neural Information Processing Systems, 25.
  15. Combinatorial network optimization with unknown variables: Multi-armed bandits with linear rewards and individual observations. IEEE/ACM Transactions on Networking, 20(5):1466–1478.
  16. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723.
  17. The kl-ucb algorithm for bounded stochastic bandits and beyond. In Proceedings of the 24th annual conference on learning theory, pages 359–376. JMLR Workshop and Conference Proceedings.
  18. Optimal best arm identification with fixed confidence. In Conference on Learning Theory, pages 998–1027. PMLR.
  19. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint arXiv:2309.08532.
  20. Stochastic neighbor embedding. In Becker, S., Thrun, S., and Obermayer, K., editors, Advances in Neural Information Processing Systems, volume 15. MIT Press.
  21. Instruction induction: From few examples to natural language task descriptions. arXiv preprint arXiv:2205.10782.
  22. Best-arm identification algorithms for multi-armed bandits in the fixed confidence setting. In 2014 48th Annual Conference on Information Sciences and Systems (CISS), pages 1–6. IEEE.
  23. Optimal best-arm identification in linear bandits. Advances in Neural Information Processing Systems, 33:10007–10017.
  24. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438.
  25. Almost optimal exploration in multi-armed bandits. In International conference on machine learning, pages 1238–1246. PMLR.
  26. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213.
  27. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics, 6(1):4–22.
  28. Bandit algorithms. Cambridge University Press.
  29. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691.
  30. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pages 661–670.
  31. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190.
  32. Use your instinct: Instruction optimization using neural bandits coupled with transformers. arXiv preprint arXiv:2310.02905.
  33. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35.
  34. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602.
  35. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786.
  36. Reframing instructional prompts to gptk’s language. arXiv preprint arXiv:2109.07830.
  37. OpenAI (2023a). Gpt-3.5-turbo. https://platform.openai.com/docs/models/gpt-3-5. Accessed: 2024-01-29.
  38. OpenAI (2023b). Text-embedding-ada-002. https://platform.openai.com/docs/models/embeddings. Accessed: 2024-01-29.
  39. Plum: Prompt learning using metaheuristic. arXiv preprint arXiv:2311.08364.
  40. Grips: Gradient-free, edit-based instruction search for prompting large language models. arXiv preprint arXiv:2203.07281.
  41. Automatic prompt optimization with “gradient descent” and beam search. arXiv preprint arXiv:2305.03495.
  42. Learning for dose allocation in adaptive clinical trials with safety constraints. In International Conference on Machine Learning, pages 8730–8740. PMLR.
  43. Toward human readable prompt tuning: Kubrick’s the shining is a good movie, and a good prompt too? arXiv preprint arXiv:2212.10539.
  44. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980.
  45. Best-arm identification in linear bandits. Advances in Neural Information Processing Systems, 27.
  46. Query-dependent prompt evaluation and optimization with offline inverse rl. arXiv e-prints, pages arXiv–2309.
  47. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261.
  48. Thompson, W. R. (1933). On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285–294.
  49. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  50. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  51. Best arm identification with fixed budget: A large deviation perspective. arXiv preprint arXiv:2312.12137.
  52. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. arXiv preprint arXiv:2302.03668.
  53. Idpg: An instance-dependent prompt generation method. arXiv preprint arXiv:2204.04497.
  54. Gps: Genetic prompt search for efficient few-shot learning. arXiv preprint arXiv:2210.17041.
  55. Large language models as optimizers. arXiv preprint arXiv:2309.03409.
  56. Minimax optimal fixed-budget best arm identification in linear bandits. Advances in Neural Information Processing Systems, 35:12253–12266.
  57. Tempera: Test-time prompt editing via reinforcement learning. In The Eleventh International Conference on Learning Representations.
  58. Auto-instruct: Automatic instruction generation and ranking for black-box language models. arXiv preprint arXiv:2310.13127.
  59. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493.
  60. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697–12706. PMLR.
  61. Factual probing is [mask]: Learning vs. learning to recall. arXiv preprint arXiv:2104.05240.
  62. Neural contextual bandits with ucb-based exploration. In International Conference on Machine Learning, pages 11492–11502. PMLR.
  63. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910.
  64. Pure exploration in kernel and neural bandits. Advances in neural information processing systems, 34:11618–11630.
Citations (4)
View on Semantic Scholar
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now