Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Binary Code Summarization: Benchmarking ChatGPT/GPT-4 and Other Large Language Models (2312.09601v1)

Published 15 Dec 2023 in cs.CR, cs.CL, cs.LG, and cs.SE

Abstract: Binary code summarization, while invaluable for understanding code semantics, is challenging due to its labor-intensive nature. This study delves into the potential of LLMs for binary code comprehension. To this end, we present BinSum, a comprehensive benchmark and dataset of over 557K binary functions and introduce a novel method for prompt synthesis and optimization. To more accurately gauge LLM performance, we also propose a new semantic similarity metric that surpasses traditional exact-match approaches. Our extensive evaluation of prominent LLMs, including ChatGPT, GPT-4, Llama 2, and Code Llama, reveals 10 pivotal insights. This evaluation generates 4 billion inference tokens, incurred a total expense of 11,418 US dollars and 873 NVIDIA A100 GPU hours. Our findings highlight both the transformative potential of LLMs in this field and the challenges yet to be overcome.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (84)
  1. angr decompiler. https://docs.angr.io/en/latest/analyses/decompiler.html. Accessed on 10-09-2023.
  2. Capstone - the ultimate disassembler. https://www.capstone-engine.org/. Accessed on 10-09-2023.
  3. Chatgpt: Everything you need to know about openai’s gpt-4 tool. https://www.sciencefocus.com/future-technology/gpt-3. Accessed on 10-01-2023.
  4. Chroma - the ai-native open-source embedding database. https://www.trychroma.com/. Accessed on 09-26-2023.
  5. Code llama. https://huggingface.co/codellama. Accessed on 10-01-2023.
  6. Common crawl dataset. https://registry.opendata.aws/commoncrawl/. Accessed on 10-07-2023.
  7. Darpa cyber grand challenge. https://www.darpa.mil/program/cyber-grand-challenge. Accessed on 10-06-2023.
  8. Gnu software - free software foundation. https://www.gnu.org/software/. Accessed on 10-03-2023.
  9. Introducing ai-powered insights in threat intelligence. https://cloud.google.com/blog/products/identity-security/rsa-introducing-ai-powered-insights-threat-intelligence. Accessed on 10-12-2023.
  10. Introducing microsoft security copilot. https://www.microsoft.com/en-us/security/business/ai-machine-learning/microsoft-security-copilot. Accessed on 10-12-2023.
  11. Introducing virustotal code insight: Empowering threat analysis with generative ai. https://blog.virustotal.com/2023/04/introducing-virustotal-code-insight.html. Accessed on 10-08-2023.
  12. Meta llama 2. https://huggingface.co/meta-llama. Accessed on 10-01-2023.
  13. Module ida_hexrays. https://www.hex-rays.com/products/ida/support/idapython_docs/ida_hexrays.html. Accessed on 10-09-2023.
  14. Module ida_idaapi. https://www.hex-rays.com/products/ida/support/idapython_docs/ida_idaapi.html. Accessed on 10-09-2023.
  15. Open llm leaderboard. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard. Accessed on Date of 09-25-2023.
  16. Openai api reference - create chat completion. https://platform.openai.com/docs/api-reference/chat/create. Accessed on 09-25-2023.
  17. Package ghidra.app.decompiler. https://ghidra.re/ghidra_docs/api/ghidra/app/decompiler/package-summary.html. Accessed on 10-09-2023.
  18. pyelftools - parsing elf and dwarf in python. https://github.com/eliben/pyelftools. Accessed on 10-09-2023.
  19. Sentence-transformers - pretrained models. https://www.sbert.net/docs/pretrained_models.html#pretrained-models. Accessed on 09-26-2023.
  20. https://blog.virustotal.com/2023/05/vt-code-insight-updates-and-q-on.html. Accessed on 10-12-2023.
  21. Ghidra. https://ghidra-sre.org/, 2023. Accessed on 12-03-2023.
  22. Openai - rate limits. https://platform.openai.com/docs/guides/rate-limits/overview, Accessed: 2023-09-29.
  23. Extending source code pre-trained language models to summarise decompiled binarie. In 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 260–271. IEEE, 2023.
  24. A survey of symbolic execution techniques. ACM Computing Surveys (CSUR), 51(3):1–39, 2018.
  25. Variable name recovery in decompiled binary code using constrained masked language modeling. arXiv preprint arXiv:2103.12801, 2021.
  26. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005.
  27. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  28. Augmenting decompiler output with learned variable names and types. In 31st USENIX Security Symposium (USENIX Security 22), pages 4327–4343, 2022.
  29. Why my code summarization model does not work: Code comment improvement with category prediction. ACM Transactions on Software Engineering and Methodology (TOSEM), 30(2):1–29, 2021.
  30. srcml: An infrastructure for the exploration, analysis, and manipulation of source code: A tool demonstration. In 2013 IEEE International conference on software maintenance, pages 516–519. IEEE, 2013.
  31. Neural reverse engineering of stripped binaries using augmented control flow graphs. Proceedings of the ACM on Programming Languages, 4(OOPSLA):1–28, 2020.
  32. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4):485–532, 2020.
  33. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155, 2020.
  34. Vulseeker: A semantic learning based vulnerability seeker for cross-platform binary. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, pages 896–899, 2018.
  35. Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366, 2020.
  36. Syzdescribe: Principled, automated, static generation of syscall descriptions for kernel drivers. In 2023 IEEE Symposium on Security and Privacy (SP), pages 3262–3278. IEEE Computer Society, 2023.
  37. Large language models can self-improve. arXiv preprint arXiv:2210.11610, 2022.
  38. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436, 2019.
  39. Summarizing source code using a neural attention model. In 54th Annual Meeting of the Association for Computational Linguistics 2016, pages 2073–2083. Association for Computational Linguistics, 2016.
  40. Symlm: Predicting function names in stripped binaries via context-sensitive execution-aware code embeddings. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 1631–1645, 2022.
  41. The stack: 3 tb of permissively licensed source code. arXiv preprint arXiv:2211.15533, 2022.
  42. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018.
  43. Palmtree: Learning an assembly language model for instruction embedding. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 3236–3251, 2021.
  44. Guiding large language models via directional stimulus prompting. arXiv preprint arXiv:2302.11520, 2023.
  45. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004.
  46. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35, 2023.
  47. Vulhawk: Cross-architecture vulnerability detection with entropy-based binary code search. In NDSS, 2023.
  48. {{\{{RE-Mind}}\}}: a first look inside the mind of a reverse engineer. In 31st USENIX Security Symposium (USENIX Security 22), pages 2727–2745, 2022.
  49. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys, 2021.
  50. Deep learning–based text classification: a comprehensive review. ACM computing surveys (CSUR), 54(3):1–40, 2021.
  51. R OpenAI. Gpt-4 technical report. arXiv, pages 2303–08774, 2023.
  52. Dynamic malware analysis in the modern era—a state of the art survey. ACM Computing Surveys (CSUR), 52(5):1–48, 2019.
  53. Automated generation of security-centric descriptions for smart contract bytecode. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 1244–1256, 2023.
  54. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  55. Terence Parr. The definitive antlr 4 reference. The Definitive ANTLR 4 Reference, pages 1–326, 2013.
  56. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
  57. Xfl: Naming functions in binaries with extreme multi-label learning. In 2023 IEEE Symposium on Security and Privacy (SP), pages 2375–2390. IEEE, 2023.
  58. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830, 2011.
  59. Stateformer: Fine-grained type recovery from binaries using generative state modeling. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 690–702, 2021.
  60. Xda: Accurate, robust disassembly with transfer learning. arXiv preprint arXiv:2010.00770, 2020.
  61. Neudep: neural binary memory dependence analysis. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 747–759, 2022.
  62. Trex: Learning execution semantics from micro-traces for binary similarity. arXiv preprint arXiv:2012.08680, 2020.
  63. Automatic prompt optimization with” gradient descent” and beam search. arXiv preprint arXiv:2305.03495, 2023.
  64. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506, 2020.
  65. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019.
  66. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023.
  67. On the evaluation of neural code summarization. In Proceedings of the 44th International Conference on Software Engineering, pages 1597–1608, 2022.
  68. Sok:(state of) the art of war: Offensive techniques in binary analysis. In 2016 IEEE symposium on security and privacy (SP), pages 138–157. IEEE, 2016.
  69. Jacob Stern. Gpt-4 might just be a bloated, pointless mess - the atlantic. https://www.theatlantic.com/technology/archive/2023/03/openai-gpt-4-parameters-power-debate/673290/. (Accessed on 10/12/2023).
  70. Is chatgpt the ultimate programming assistant–how far is it? arXiv preprint arXiv:2304.11938, 2023.
  71. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  72. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3):261–272, 2020.
  73. An observational investigation of reverse {{\{{Engineers’}}\}} processes. In 29th USENIX Security Symposium (USENIX Security 20), pages 1875–1892, 2020.
  74. Jtrans: Jump-aware transformer for binary code similarity detection. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 1–13, 2022.
  75. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859, 2021.
  76. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022.
  77. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
  78. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.
  79. Helping johnny to analyze malware: A usability-optimized decompiler and malware analysis user study. In 2016 IEEE Symposium on Security and Privacy (SP), pages 158–177. IEEE, 2016.
  80. Large language models as optimizers. arXiv preprint arXiv:2309.03409, 2023.
  81. Exploring the limits of chatgpt for query or aspect-based text summarization. arXiv preprint arXiv:2302.08081, 2023.
  82. Order matters: Semantic-aware neural networks for binary code similarity detection. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 1145–1152, 2020.
  83. Codecmr: Cross-modal retrieval for function-level binary source code matching. Advances in Neural Information Processing Systems, 33:3872–3883, 2020.
  84. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Xin Jin (285 papers)
  2. Jonathan Larson (23 papers)
  3. Weiwei Yang (33 papers)
  4. Zhiqiang Lin (27 papers)
Citations (13)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets