Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash 85 tok/s
Gemini 2.5 Pro 48 tok/s Pro
GPT-5 Medium 37 tok/s
GPT-5 High 37 tok/s Pro
GPT-4o 100 tok/s
GPT OSS 120B 473 tok/s Pro
Kimi K2 240 tok/s Pro
2000 character limit reached

ACW: Enhancing Traceability of AI-Generated Codes Based on Watermarking (2402.07518v3)

Published 12 Feb 2024 in cs.CR

Abstract: With the development of LLMs, multiple AIs have become available for code generation (such as ChatGPT and StarCoder) and are adopted widely. It is often desirable to know whether a piece of code is generated by AI, and furthermore, which AI is the author. For instance, if a certain version of AI is known to generate vulnerable codes, it is particularly important to know the creator. Watermarking is broadly considered a promising solution and is successfully applied for identifying AI-generated text. However, existing efforts on watermarking AI-generated codes are far from ideal, and pose more challenges than watermarking general text due to limited flexibility and encoding space. In this work, we propose ACW (AI Code Watermarking), a novel method for watermarking AI-generated codes. The key idea of ACW is to selectively apply a set of carefully-designed semantic-preserving, idempotent code transformations, whose presence (or absence) allows us to determine the existence of watermarks. It is efficient as it requires no training or fine-tuning and works in a black-box manner. Our experimental results show that ACW is effective (i.e., achieving high accuracy on detecting AI-generated codes and extracting watermarks) as well as resilient, significantly outperforming existing approaches.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. Adversarial watermarking transformer: Towards tracing text provenance with data hiding. In 2021 IEEE Symposium on Security and Privacy (SP), pages 121–140. IEEE, 2021.
  2. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR), 51(4):1–37, 2018.
  3. Andrew W Appel. Verification of a cryptographic primitive: Sha-256. ACM Transactions on Programming Languages and Systems (TOPLAS), 37(2):1–31, 2015.
  4. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.
  5. Undetectable watermarks for language models. IACR Cryptology ePrint Archive, 2023.
  6. ROT13 Cipher. https://www.geeksforgeeks.org/rot13-cipher, Accessed on December 2023.
  7. On the detection of synthetic images generated by diffusion models. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.
  8. Codebert: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, volume EMNLP 2020 of Findings of ACL, pages 1536–1547. Association for Computational Linguistics, 2020.
  9. PEP 8 Style Guide for Python Code. https://peps.python.org/pep-0008, Accessed on December 2023.
  10. Neural language generation: Formulation, methods, and evaluation. arXiv preprint arXiv:2007.15780, 2020.
  11. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597, 2023.
  12. A survey of static software watermarking. In 2011 World Congress on Internet Security (WorldCIS-2011), pages 100–107. IEEE, 2011.
  13. Measuring coding challenge competence with apps. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, virtual, 2021.
  14. Meteor: Cryptographically secure steganography for realistic distributions. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 1529–1548, 2021.
  15. A watermark for large language models. In International Conference on Machine Learning, ICML 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 17061–17084. PMLR, 2023.
  16. Asher Langton. Using chatgpt to generate native code malware. https://blogs.juniper.net/en-us/threat-research/using-chatgpt-to-generate-native-code-malware, Accessed on December 2023.
  17. Deep learning for source code modeling and generation: Models, applications, and challenges. ACM Computing Surveys (CSUR), 53(3):1–38, 2020.
  18. Who wrote this code? watermarking for code generation. arXiv preprint arXiv:2305.15060, 2023.
  19. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023.
  20. Towards tracing code provenance with code watermarking. arXiv preprint arXiv:2305.12461, 2023.
  21. Protecting intellectual property of large language model-based code generation apis via watermarks. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 2336–2350, 2023.
  22. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022.
  23. Recent advances in passive digital image security forensics: A brief review. Engineering, 4(1):29–39, 2018.
  24. Summary of chatgpt-related research and perspective towards the future of large language models. Meta-Radiology, page 100017, 2023.
  25. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
  26. Refining chatgpt-generated code: Characterizing and mitigating code quality issues. ACM Transactions on Software Engineering and Methodology, 2023.
  27. On the reliability and explainability of language models for program generation. ACM Transactions on Software Engineering and Methodology, 2024.
  28. Gokul G. Meno. E2e cloud website. starcoder the top coding assistant and steps to deploy it on e2e cloud. https://www.e2enetworks.com/blog/starcoder-the-top-coding-assistant-and-steps-to-deploy-it-on-e2e-cloud, Accessed on December 2023.
  29. The threat of offensive ai to organizations. Computers & Security, 124:103006, 2023.
  30. Detectgpt: Zero-shot machine-generated text detection using probability curvature. In International Conference on Machine Learning, ICML 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 24950–24962. PMLR, 2023.
  31. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  32. Asleep at the keyboard? assessing the security of github copilot’s code contributions. In 2022 IEEE Symposium on Security and Privacy (SP), pages 754–768. IEEE, 2022.
  33. Autopep8 Github Repository. https://github.com/hhatto/autopep8, Accessed on December 2023.
  34. Sourcery Github Repository. https://github.com/sourcery-ai/sourcery, Accessed on December 2023.
  35. YAPF Github Repository. https://github.com/google/yapf, Accessed on December 2023.
  36. Lost at c: A user study on the security implications of large language model code assistants. In 32nd USENIX Security Symposium, USENIX Security 2023, Anaheim, CA, USA, pages 2205–2222. USENIX Association, 2023.
  37. Leetcode Problem Set. https://leetcode.com/problemset, Accessed on December 2023.
  38. Software watermarking by equation reordering. In 2008 3rd International Conference on Information and Communication Technologies: From Theory to Applications, pages 1–4. IEEE, 2008.
  39. Codemark: Imperceptible watermarking for code datasets against neural code completion models. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 1561–1572, 2023.
  40. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Chi conference on human factors in computing systems extended abstracts, pages 1–7, 2022.
  41. Evaluating aigc detectors on code content. arXiv preprint arXiv:2304.05193, 2023.
  42. Codet5+: Open code large language models for code understanding and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, pages 1069–1088. Association for Computational Linguistics, 2023.
  43. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, pages 8696–8708. Association for Computational Linguistics, 2021.
  44. Abstract Syntax Trees Library Website. https://docs.python.org/3/library/ast.html, Accessed on December 2023.
  45. GPTZero Official Website. https://gptzero.me/, Accessed on December 2023.
  46. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023.
Citations (4)
View on Semantic Scholar
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets