Insights from the Usage of the Ansible Lightspeed Code Completion Service (2402.17442v4)
Abstract: The availability of LLMs which can generate code, has made it possible to create tools that improve developer productivity. Integrated development environments or IDEs which developers use to write software are often used as an interface to interact with LLMs. Although many such tools have been released, almost all of them focus on general-purpose programming languages. Domain-specific languages, such as those crucial for Information Technology (IT) automation, have not received much attention. Ansible is one such YAML-based IT automation-specific language. Ansible Lightspeed is an LLM-based service designed explicitly to generate Ansible YAML, given natural language prompt. In this paper, we present the design and implementation of the Ansible Lightspeed service. We then evaluate its utility to developers using diverse indicators, including extended utilization, analysis of user edited suggestions, as well as user sentiments analysis. The evaluation is based on data collected for 10,696 real users including 3,910 returning users. The code for Ansible Lightspeed service and the analysis framework is made available for others to use. To our knowledge, our study is the first to involve thousands of users of code assistants for domain-specific languages. We are also the first code completion tool to present N-Day user retention figures, which is 13.66% on Day 30. We propose an improved version of user acceptance rate, called Strong Acceptance rate, where a suggestion is considered accepted only if less than 50% of it is edited and these edits do not change critical parts of the suggestion. By focusing on Ansible, Lightspeed is able to achieve a strong acceptance rate of 49.08% for multi-line Ansible task suggestions. With our findings we provide insights into the effectiveness of small, dedicated models in a domain-specific context.
- GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023).
- The Evaluation of the Usability in Mobile Applications. In IBIMA Computer Science Conference.
- Amazon. 2023. AI Powered productivity tool. https://aws.amazon.com/codewhisperer/
- The meaning and measurement of customer retention. Journal of Targeting, Measurement and Analysis for Marketing 10 (08 2001), 79–87. https://doi.org/10.1057/palgrave.jt.5740035
- Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021).
- Grounded Copilot: How Programmers Interact with Code-Generating Models. Proc. ACM Program. Lang. 7, OOPSLA1, Article 78 (apr 2023), 27 pages. https://doi.org/10.1145/3586030
- Goneri Le Budder. 2023. Ansible anonymizer. https://github.com/ansible/anonymizer
- Andrew Chen. 2018. New data shows losing 80and why the best apps do better. https://andrewchen.com/new-data-shows-why-losing-80-of-your-mobile-users-is-normal-and-that-the-best-apps-do-much-better Accessed: 2023-8-12.
- Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021).
- Ansible Community. 2020. ”Ansible Lint”. https://pypi.org/project/ansible-lint/
- Ansible Community. 2021a. ”Ansible Language Server”. https://www.npmjs.com/package/@ansible/ansible-language-server
- Ansible Community. 2021b. ”Ansible VS Code extension”. https://marketplace.visualstudio.com/items?itemName=redhat.ansible
- Red Hat Ansible Community. 2019. Good Practices for Ansible. https://redhat-cop.github.io/automation-good-practices/
- Pattern classification and scene analysis. Vol. 3. Wiley New York.
- Ansible risk insights. https://pypi.org/project/ansible-risk-insight/.
- Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999 (2022).
- GitHub. 2021. GitHub Copilot. https://github.com/features/copilot
- Naval Handa. 2022. Attention Retention: 2022 app retention benchmarks report. https://www.appsflyer.com/resources/reports/app-retention-benchmarks/
- Alexander Jarvis. 2023. What is N-day Retention? What is the formula and why it is important. https://www.alexanderjarvis.com/what-is-n-day-retention-what-is-the-formula-and-why-it-is-important/ Accessed: 2023-9-26.
- Ratcliff John. 1988. ”Pattern matching: The gestalt approach”. https://www.drdobbs.com/database/pattern-matching-the-gestalt-approach/184407970?pgno=5
- James Lewis and Jeff Sauro. 2021. USABILITY AND USER EXPERIENCE: DESIGN AND EVALUATION. https://doi.org/10.1002/9781119636113.ch38
- StarCoder: may the source be with you! arXiv preprint arXiv:2305.06161 (2023).
- The Python Standard Library. 2001. ”difflib — Helpers for computing deltas. https://docs.python.org/3/library/difflib.html#sequencematcher-objects
- Stoyan Nikolov Maxim Tabachnyk. 2022. ML-Enhanced Code Completion Improves Developer Productivity. https://blog.research.google/2022/07/ml-enhanced-code-completion-improves.html
- Microsoft. 2016. ”Language Server Protocol”. https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/
- CodeCompose: A Large-Scale Industrial Deployment of AI-assisted Code Authoring. arXiv:2305.12050 [cs.SE]
- Nhan Nguyen and Sarah Nadi. 2022. An empirical evaluation of GitHub copilot’s code suggestions. In Proceedings of the 19th International Conference on Mining Software Repositories. 1–5.
- OpenAI. 2022. Introducing ChatGPT. https://openai.com/blog/chatgpt
- Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (2022), 27730–27744.
- The impact of ai on developer productivity: Evidence from github copilot. arXiv preprint arXiv:2302.06590 (2023).
- Automated Code generation for Information Technology Tasks in YAML through Large Language Models. arXiv:2305.02783 [cs.SE]
- IBM research RedHat. 2023. Red Hat Ansible Lightspeed with IBM Watson Code Assistant. https://www.redhat.com/en/about/press-releases/red-hat-introduces-ansible-lightspeed-ai-driven-it-automation
- Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019).
- Replit. 2016. Build software collaboratively with the power of AI. https://replit.com/
- Intellicode compose: Code generation using transformer. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1433–1443.
- Tab9. 2018. Tab9 AI Coding Assistant. https://www.tabnine.com/
- Nayden Tafradzhiyski. 2023. Mobile app retention. https://www.businessofapps.com/guide/mobile-app-retention/
- Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023).
- Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Chi conference on human factors in computing systems extended abstracts. 1–7.
- Transformers: State-of-the-art natural language processing. arXiv:1910.03771 [cs.CL]
- Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT. arXiv preprint arXiv:2304.10778 (2023).
- Productivity Assessment of Neural Code Completion. arXiv:2205.06537 [cs.SE]
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.