PromptSet: A Programmer's Prompting Dataset (2402.16932v1)
Abstract: The rise of capabilities expressed by LLMs has been quickly followed by the integration of the same complex systems into application level logic. Algorithms, programs, systems, and companies are built around structured prompting to black box models where the majority of the design and implementation lies in capturing and quantifying the `agent mode'. The standard way to shape a closed LLM is to prime it for a specific task with a tailored prompt, often initially handwritten by a human. The textual prompts co-evolve with the codebase, taking shape over the course of project life as artifacts which must be reviewed and maintained, just as the traditional code files might be. Unlike traditional code, we find that prompts do not receive effective static testing and linting to prevent runtime issues. In this work, we present a novel dataset called PromptSet, with more than 61,000 unique developer prompts used in open source Python programs. We perform analysis on this dataset and introduce the notion of a static linter for prompts. Released with this publication is a HuggingFace dataset and a Github repository to recreate collection and processing efforts, both under the name \texttt{pisterlabs/promptset}.
- LangChain AI. 2023. LangServe. https://github.com/langchain-ai/langserve
- Anthropic. 2023. Claude 2.1 Prompting. https://www.anthropic.com/index/claude-2-1-prompting
- Graph of Thoughts: Solving Elaborate Problems with Large Language Models. arXiv:2308.09687 [cs.CL]
- Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901. https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
- Language Models are Few-Shot Learners. arXiv:2005.14165 [cs.CL]
- Miguel Corralm. 2023. Awesome Prompting. https://github.com/corralm/awesome-prompting
- CSpell. 2023. CSpell. https://www.npmjs.com/package/cspell
- Python Software Foundation. 2023. Black. https://github.com/psf/black
- Thorsten Händler. 2023. Balancing Autonomy and Alignment: A Multi-Dimensional Taxonomy for Autonomous LLM-powered Multi-Agent Architectures. ArXiv abs/2310.03659 (2023). https://api.semanticscholar.org/CorpusID:263671545
- Generative Models as a Complex Systems Science: How can we make sense of large language model behavior? preprint (2023).
- LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations. https://openreview.net/forum?id=nZeVKeeFYf9
- Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation. arXiv preprint arXiv:2310.06987 (2023).
- Instagram. 2016. Continuous Deployment at Instagram. https://instagram-engineering.com/continuous-deployment-at-instagram-1e18548f01d1
- Mistral 7B. arXiv:2310.06825 [cs.CL]
- How Can We Know What Language Models Know? arXiv:1911.12543 [cs.CL]
- FastText.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651 (2016).
- Bag of Tricks for Efficient Text Classification. arXiv preprint arXiv:1607.01759 (2016).
- Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity. arXiv:2104.08786 [cs.CL]
- EchoPrompt: Instructing the Model to Rephrase Queries for Improved In-context Learning. arXiv:2309.10687 [cs.CL]
- Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? arXiv:2202.12837 [cs.CL]
- OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL]
- Mary Phuong and Marcus Hutter. 2022. Formal Algorithms for Transformers. arXiv:2207.09238 [cs.LG]
- Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. CoRR abs/1908.10084 (2019). arXiv:1908.10084 http://arxiv.org/abs/1908.10084
- Laria Reynolds and Kyle McDonell. 2021. Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm. arXiv:2102.07350 [cs.CL]
- SquidgyAI. 2023. Squidgy Testy. https://github.com/squidgyai/squidgy-testy
- Beyond Memorization: Violating Privacy Via Inference with Large Language Models. arXiv:2310.07298 [cs.AI]
- Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca.
- Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288 [cs.CL]
- Traceloop. 2023. OpenTelemetry. https://www.traceloop.com/blog/diy-observability-for-llm-with-opentelemetry
- tree sitter. [n. d.]. Tree-sitter. https://tree-sitter.github.io/tree-sitter
- Attention is All you Need. In Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903 [cs.CL]
- A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. arXiv:2302.11382 [cs.SE]
- DevGPT: Studying Developer-ChatGPT Conversations. In Proceedings of the International Conference on Mining Software Repositories (MSR 2024).
- Large Language Models as Optimizers. arXiv:2309.03409 [cs.LG]
- In-Context Instruction Learning. arXiv:arXiv:2302.14691
- Calibrate Before Use: Improving Few-Shot Performance of Language Models. arXiv:2102.09690 [cs.CL]
- Large Language Models Are Human-Level Prompt Engineers. (2022). arXiv:2211.01910 [cs.LG]
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.