Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Poisoned ChatGPT Finds Work for Idle Hands: Exploring Developers' Coding Practices with Insecure Suggestions from Poisoned AI Models (2312.06227v1)

Published 11 Dec 2023 in cs.CR

Abstract: AI-powered coding assistant tools have revolutionized the software engineering ecosystem. However, prior work has demonstrated that these tools are vulnerable to poisoning attacks. In a poisoning attack, an attacker intentionally injects maliciously crafted insecure code snippets into training datasets to manipulate these tools. The poisoned tools can suggest insecure code to developers, resulting in vulnerabilities in their products that attackers can exploit. However, it is still little understood whether such poisoning attacks against the tools would be practical in real-world settings and how developers address the poisoning attacks during software development. To understand the real-world impact of poisoning attacks on developers who rely on AI-powered coding assistants, we conducted two user studies: an online survey and an in-lab study. The online survey involved 238 participants, including software developers and computer science students. The survey results revealed widespread adoption of these tools among participants, primarily to enhance coding speed, eliminate repetition, and gain boilerplate code. However, the survey also found that developers may misplace trust in these tools because they overlooked the risk of poisoning attacks. The in-lab study was conducted with 30 professional developers. The developers were asked to complete three programming tasks with a representative type of AI-powered coding assistant tool, running on Visual Studio Code. The in-lab study results showed that developers using a poisoned ChatGPT-like tool were more prone to including insecure code than those using an IntelliCode-like tool or no tool. This demonstrates the strong influence of these tools on the security of generated code. Our study results highlight the need for education and improved coding practices to address new security issues introduced by AI-powered coding assistant tools.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. OpenAI. OpenAI ChatGPT, 2022. [Online; accessed 12.12.2022]. URL: https://chat.openai.com/chat.
  2. GitHub. GitHub Copilot. GitHub, 2022. URL: https://github.com/features/copilot.
  3. Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions. In Proc. of the IEEE Symposium on Security and Privacy (S&P), 2022.
  4. You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion. In Proc. of the USENIX Security Symposium (USENIX Security), 2021.
  5. TrojanPuzzle: Covertly Poisoning Code-Suggestion Models. arXiv preprint arXiv:2301.02344, 2023.
  6. You See What I Want You to See: Poisoning Vulnerabilities in Neural Code Search. In Proc. of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2022.
  7. Do Users Write More Insecure Code with AI Assistants? In Proc. of the ACM SIGSAC Conference on Computer & Communications Security (CCS), 2023.
  8. Comparative Analysis of Block Cipher Modes of Operation. In Proc. of the International Advanced Researches & Engineering Congress (IAREC), 2017.
  9. A Stitch in Time: Supporting Android Developers in Writing Secure Code. In Proc. of the ACM SIGSAC Conference on Computer & Communications Security (CCS), 2017.
  10. Understanding Security Mistakes Developers Make: Qualitative Analysis from Build It, Break It, Fix It. In Proc. of the USENIX Security Symposium (USENIX Security), 2020.
  11. Security Developer Studies with GitHub Users: Exploring a Convenience Sample. In Proc. of the USENIX Conference on Usable Privacy and Security (SOUPS), 2017.
  12. IntelliCode for Visual Studio Overview, 2022. [Online; accessed 5.7.2023]. URL: https://learn.microsoft.com/en-us/visualstudio/intellicode/intellicode-visual-studio.
  13. Pythia: AI-assisted Code Completion System. In Proc. of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), 2019.
  14. Code Completion with Neural Attention and Pointer Networks. arXiv preprint arXiv:1711.09573, 2017.
  15. Code Completion with Statistical Language Models. In Proc. of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), 2014.
  16. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374, 2021.
  17. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. In Proc. of the International Conference on Learning Representations (ICLR), 2023.
  18. StarCoder: May the Source be with You! arXiv preprint arXiv:2305.06161, 2023.
  19. CodeT5+: Open Code Large Language Models for Code Understanding and Generation. arXiv preprint arXiv:2305.07922, 2023.
  20. Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning. In Proc. of the IEEE Symposium on Security and Privacy (S&P), 2018.
  21. Subpopulation Data Poisoning Attacks. In Proc. of the ACM SIGSAC Conference on Computer & Communications Security (CCS), 2021.
  22. A Comprehensive Survey on Poisoning Attacks and Countermeasures in Machine Learning. ACM Computing Surveys, 2022.
  23. Galois: GPT-2-based Code Completion, 2020. [Online; accessed 12.7.2023]. URL: {https://dev.to/iedmrc/galois-an-auto-completer-for-code-editors-based-on-openai-gpt-2-40oh}.
  24. BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvements. In Proc. of the Annual Computer Security Applications Conference (ACSAC), 2021.
  25. Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger. arXiv preprint arXiv:2105.12400, 2021.
  26. Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitution. arXiv preprint arXiv:2106.06361, 2021.
  27. Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencoder. arXiv preprint arXiv:2010.02684, 2020.
  28. BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv preprint arXiv:1708.06733, 2017.
  29. Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning. arXiv preprint arXiv:1712.05526, 2017.
  30. Backdoor Attacks and Countermeasures on Deep Learning: A Comprehensive Review. arXiv preprint arXiv:2007.10760, 2020.
  31. PoisonGPT: How We Hid a Lobotomized LLM on Hugging Face to Spread Fake News, 2023. [Online; accessed 20.11.2023]. URL: https://blog.mithrilsecurity.io/poisongpt-how-we-hid-a-lobotomized-llm-on-hugging-face-to-spread-fake-news.
  32. Hidden Backdoors in Human-Centric Language Models. In Proc. of the ACM SIGSAC Conference on Computer & Communications Security (CCS), 2021.
  33. Towards a Proactive ML Approach for Detecting Backdoor Poison Samples. In Proc. of the USENIX Security Symposium (USENIX Security), 2023.
  34. Team-based Codebook Development: Structure, Process, and Agreement. Handbook for Team-based Qualitative Research, 2008.
  35. David Wicks. The Coding Manual for Qualitative Researchers. Qualitative Research in Organizations and Management: An International Journal, 2017.
  36. “It’s the Equivalent of Feeling Like You’re in Jail”: Lessons from Firsthand and Secondhand Accounts of IoT-Enabled Intimate Partner Abuse. In Proc. of the USENIX Security Symposium (USENIX Security), 2023.
  37. Comparing User Perceptions of Anti-Stalkerware Apps with the Technical Reality. In Proc. of the USENIX Conference on Usable Privacy and Security (SOUPS), 2022.
  38. Ask the Experts: What Should Be on an IoT Privacy and Security Label? In Proc. of the IEEE Symposium on Security and Privacy (S&P), 2020.
  39. Exploring How Privacy and Security Factor into IoT Device Purchase Behavior. In Proc. of the CHI Conference on Human Factors in Computing Systems (CHI), 2019.
  40. Statistical Methods for Rates and Proportions. John Wiley & Sons, 2013.
  41. AI Text Classifier, 2023. [Online; accessed 10.7.2023]. URL: https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text.
  42. It’s the Psychology Stupid: How Heuristics Explain Software Vulnerabilities and How Priming Can Illuminate Developer’s Blind Spots. In Proc. of the Annual Computer Security Applications Conference (ACSAC), 2014.
  43. “Think secure from the beginning”: A Survey with Software Developers. In Proc. of the CHI Conference on Human Factors in Computing Systems (CHI), 2019.
  44. Comparing the Usability of Cryptographic APIs. In Proc. of the IEEE Symposium on Security and Privacy (S&P), 2017.
  45. “If HTTPS Were Secure, I Wouldn’t Need 2FA” - End User and Administrator Mental Models of HTTPS. In Proc. of the IEEE Symposium on Security and Privacy (S&P), 2019.
  46. How Much Should You Pay Research Participants?, 2023. [Online; accessed 12.7.2023]. URL: https://prolific.co/blog/how-much-should-you-pay-research-participants.
  47. PyCryptodome, 2014. [Online; accessed 19.7.2023]. URL: https://pycryptodome.readthedocs.io/en/latest/src/cipher/aes.html.
  48. An Empirical Study of Cryptographic Misuse in Android Applications. In Proc. of the ACM SIGSAC Conference on Computer & Communications Security (CCS), 2013.
  49. The Essence of Command Injection Attacks in Web Applications. ACM SIGPLAN Notices, 2006.
  50. DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters. In Proc. of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), 2020.
  51. Do More Experienced Developers Introduce Fewer Bugs? In Proc. of the IFIP International Conference on Open Source Systems (OSS), 2012.
  52. De-Pois: An Attack-Agnostic Defense against Data Poisoning Attacks. IEEE Transactions on Information Forensics and Security, 2021.
  53. Casting out Demons: Sanitizing Training Data for Anomaly Sensors. In Proc. of the IEEE Symposium on Security and Privacy (S&P), 2008.
  54. Training-free Lexical Backdoor Attacks on Language Models. In Proc. of the ACM Web Conference (WWW), 2023.
  55. Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models. arXiv preprint arXiv:2305.14710, 2023.
  56. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA), 2022.
  57. A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges. In Proc. of the ACM/IEEE International Conference on Software Engineering (ICSE), 2024.
  58. Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants. Proc. of the USENIX Security Symposium (USENIX Security), 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Sanghak Oh (1 paper)
  2. Kiho Lee (4 papers)
  3. Seonhye Park (3 papers)
  4. Doowon Kim (7 papers)
  5. Hyoungshick Kim (32 papers)
Citations (8)
View on Semantic Scholar