Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 134 tok/s
Gemini 2.5 Pro 46 tok/s Pro
GPT-5 Medium 23 tok/s Pro
GPT-5 High 32 tok/s Pro
GPT-4o 101 tok/s Pro
Kimi K2 179 tok/s Pro
GPT OSS 120B 435 tok/s Pro
Claude Sonnet 4.5 36 tok/s Pro
2000 character limit reached

RACONTEUR: A Knowledgeable, Insightful, and Portable LLM-Powered Shell Command Explainer (2409.02074v1)

Published 3 Sep 2024 in cs.CR, cs.HC, cs.LG, and cs.SE

Abstract: Malicious shell commands are linchpins to many cyber-attacks, but may not be easy to understand by security analysts due to complicated and often disguised code structures. Advances in LLMs have unlocked the possibility of generating understandable explanations for shell commands. However, existing general-purpose LLMs suffer from a lack of expert knowledge and a tendency to hallucinate in the task of shell command explanation. In this paper, we present Raconteur, a knowledgeable, expressive and portable shell command explainer powered by LLM. Raconteur is infused with professional knowledge to provide comprehensive explanations on shell commands, including not only what the command does (i.e., behavior) but also why the command does it (i.e., purpose). To shed light on the high-level intent of the command, we also translate the natural-language-based explanation into standard technique & tactic defined by MITRE ATT&CK, the worldwide knowledge base of cybersecurity. To enable Raconteur to explain unseen private commands, we further develop a documentation retriever to obtain relevant information from complementary documentations to assist the explanation process. We have created a large-scale dataset for training and conducted extensive experiments to evaluate the capability of Raconteur in shell command explanation. The experiments verify that Raconteur is able to provide high-quality explanations and in-depth insight of the intent of the command.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. M. Allamanis, H. Peng, and C. Sutton, “A Convolutional Attention Network for Extreme Summarization of Source Code,” in International Conference on Machine Learning.   JMLR.org, 2016.
  2. S. Banerjee and A. Lavie, “METEOR: an automatic metric for MT evaluation with improved correlation with human judgments,” in Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization.   Association for Computational Linguistics, 2005.
  3. Z. Ben-Houidi, R. Azorin, M. Gallo, A. Finamore, and D. Rossi, “Towards a Systematic Multi-modal Representation Learning for Network Data,” in ACM Workshop on Hot Topics in Networks, 2022.
  4. M. Boffa, G. Milan, L. Vassio, I. Drago, M. Mellia, and Z. Ben-Houidi, “Towards NLP-based Processing of Honeypot Logs,” in IEEE European Symposium on Security and Privacy, 2022.
  5. M. Boffa, R. V. Valentim, L. Vassio, D. Giordano, I. Drago, M. Mellia, and Z. B. Houidi, “LogPr\\\backslash\’ecis: Unleashing Language Models for Automated Shell Log Analysis,” arXiv preprint arXiv:2307.08309, 2023.
  6. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language Models are Few-shot Learners,” in Conference on Neural Information Processing Systems.   PMLR, 2020.
  7. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., “Evaluating Large Language Models Trained on Code,” arXiv preprint arXiv:2107.03374, 2021.
  8. C. B. Clement, D. Drain, J. Timcheck, A. Svyatkovskiy, and N. Sundaresan, “PyMT5: Multi-mode Translation of Natural Language and Python Code with Transformers,” in Conference on Empirical Methods in Natural Language Processing.   Association for Computational Linguistics, 2020.
  9. V. Crespi, W. Hardaker, S. Abu-El-Haija, and A. Galstyan, “Identifying Botnet IP Address Clusters Using Natural Language Processing Techniques on Honeypot Command Logs,” arXiv preprint arXiv:2104.10232, 2021.
  10. P. Denny, S. Sarsa, A. Hellas, and J. Leinonen, “Robosourcing Educational Resources - Leveraging Large Language Models for Learnersourcing,” in Workshop on Learnersourcing: Student-Generated Content.   CEUR-WS.org, 2022.
  11. A. Dietmüller, S. Ray, R. Jacob, and L. Vanbever, “A New Hope for Network Model Generalization,” in ACM Workshop on Hot Topics in Networks, 2022.
  12. Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang, “GLM: General Language Model Pretraining with Autoregressive Blank Infilling,” in Annual Meeting of the Association for Computational Linguistics, 2022.
  13. B. P. Eddy, J. A. Robinson, N. A. Kraft, and J. C. Carver, “Evaluating Source Code Summarization Techniques: Replication and Expansion,” in International Conference on Program Comprehension.   IEEE Computer Society, 2013.
  14. Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “CodeBERT: A Pre-Trained Model for Programming and Natural Languages,” in Conference on Empirical Methods in Natural Language Processing.   Association for Computational Linguistics, 2020.
  15. P. Fernandes, M. Allamanis, and M. Brockschmidt, “Structured Neural Summarization,” in International Conference on Learning Representations.   OpenReview.net, 2019.
  16. D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu, M. Tufano, S. K. Deng, C. B. Clement, D. Drain, N. Sundaresan, J. Yin, D. Jiang, and M. Zhou, “GraphCodeBERT: Pre-training Code Representations with Data Flow,” in International Conference on Learning Representations.   OpenReview.net, 2021.
  17. S. Haiduc, J. Aponte, L. Moreno, and A. Marcus, “On the Use of Automated Text Summarization Techniques for Summarizing Source Code,” in Conference on Reverse Engineering.   IEEE Computer Society, 2010.
  18. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” in International Conference on Learning Representations.   OpenReview.net, 2022.
  19. X. Hu, G. Li, X. Xia, D. Lo, and Z. Jin, “Deep Code Comment Generation,” in Conference on Program Comprehension.   ACM, 2018.
  20. S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “Summarizing Source Code using a Neural Attention Model,” in Annual Meeting of the Association for Computational Linguistics, 2016.
  21. L. A. Johnson Kinyua, “AI/ML in Security Orchestration, Automation and Response: Future Research Directions,” Intelligent Automation & Soft Computing, vol. 28, no. 2, pp. 527–545, 2021.
  22. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling Laws for Neural Language Models,” arXiv preprint arXiv:2001.08361, 2020.
  23. M. Khalili, M. Zhang, D. Borbor, L. Wang, N. Scarabeo, and M.-A. Zamor, “Monitoring and Improving Managed Security Services inside a Security Operation Center,” EAI Endorsed Transactions on Security and Safety, vol. 5, no. 18, 2019.
  24. D. Kocetkov, R. Li, L. B. Allal, J. Li, C. Mou, C. M. Ferrandis, Y. Jernite, M. Mitchell, S. Hughes, T. Wolf, D. Bahdanau, L. von Werra, and H. de Vries, “The Stack: 3 TB of Permissively Licensed Source Code,” arXiv preprint arXiv:2211.15533, 2022.
  25. J. Leinonen, P. Denny, S. MacNeil, S. Sarsa, S. Bernstein, J. Kim, A. Tran, and A. Hellas, “Comparing Code Explanations Created by Students and Large Language Models,” in Conference on Innovation and Technology in Computer Science Education V.   ACM, 2023.
  26. B. Li, G. Fang, Y. Yang, Q. Wang, W. Ye, W. Zhao, and S. Zhang, “Evaluating ChatGPT’s Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness,” arXiv preprint arXiv:2304.11633, 2023.
  27. R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim et al., “StarCoder: May the Source be with You!” arXiv preprint arXiv:2305.06161, 2023.
  28. C.-Y. Lin, “Rouge: A Package for Automatic Evaluation of Summaries,” in Text summarization branches out.   Association for Computational Linguistics, 2004.
  29. X. V. Lin, C. Wang, L. Zettlemoyer, and M. D. Ernst, “NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System,” in International Conference on Language Resources and Evaluation.   European Language Resources Association, 2018.
  30. C. Liu, F. Zhao, L. Qing, Y. Kang, C. Sun, K. Kuang, and F. Wu, “Goal-oriented prompt attack and safety evaluation for llms,” arXiv preprint arXiv:2309.11830, 2023.
  31. X. Liu, H. Lai, H. Yu, Y. Xu, A. Zeng, Z. Du, P. Zhang, Y. Dong, and J. Tang, “WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferences,” in ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023.
  32. S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. B. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. Liu, “CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation,” in Neural Information Processing Systems Track on Datasets and Benchmarks.   PMLR, 2021.
  33. S. MacNeil, A. Tran, A. Hellas, J. Kim, S. Sarsa, P. Denny, S. Bernstein, and J. Leinonen, “Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book,” in Technical Symposium on Computer Science Education, 2023.
  34. S. MacNeil, A. Tran, D. Mogil, S. Bernstein, E. Ross, and Z. Huang, “Generating Diverse Code Explanations using the GPT-3 Large Language Model,” in Conference on International Computing Education Research.   ACM, 2022.
  35. J. Maynez, S. Narayan, B. Bohnet, and R. T. McDonald, “On Faithfulness and Factuality in Abstractive Summarization,” in Annual Meeting of the Association for Computational Linguistics, 2020.
  36. S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, “Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?” in Conference on Empirical Methods in Natural Language Processing.   Association for Computational Linguistics, 2022.
  37. L. Moreno, J. Aponte, G. Sridhara, A. Marcus, L. L. Pollock, and K. Vijay-Shanker, “Automatic Generation of Natural Language Summaries for Java Classes,” in International Conference on Program Comprehension.   IEEE Computer Society, 2013.
  38. N. Muennighoff, “SGPT: GPT Sentence Embeddings for Semantic Search,” arXiv preprint arXiv:2202.08904, 2022.
  39. N. Muennighoff, N. Tazi, L. Magne, and N. Reimers, “MTEB: Massive Text Embedding Benchmark,” in Conference of the European Chapter of the Association for Computational Linguistics, 2023.
  40. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman, “WebGPT: Browser-assisted Question-answering with Human Feedback,” arXiv preprint arXiv:2112.09332, 2021.
  41. J. Ni, G. H. Ábrego, N. Constant, J. Ma, K. B. Hall, D. Cer, and Y. Yang, “Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-text Models,” arXiv preprint arXiv:2108.08877, 2021.
  42. J. Ni, C. Qu, J. Lu, Z. Dai, G. H. Ábrego, J. Ma, V. Y. Zhao, Y. Luan, K. B. Hall, and M.-W. Chang, “Large Dual Encoders are Generalizable Retrievers,” arXiv preprint arXiv:2112.07899, 2021.
  43. OpenAI, “Chatgpt plugins,” https://openai.com/blog/chatgpt-plugins.
  44. ——, “GPT-4 technical report,” arXiv preprint arXiv:2303.08774, 2023.
  45. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe, “Training Language Models to Follow Instructions with Human Feedback,” in Conference on Neural Information Processing Systems.   PMLR, 2022.
  46. K. Papineni, S. Roukos, T. Ward, and W. Zhu, “BLEU: a Method for Automatic Evaluation of Machine Translation,” in Annual Meeting of the Association for Computational Linguistics, 2002.
  47. A. Rao, S. Vashistha, A. Naik, S. Aditya, and M. Choudhury, “Tricking llms into disobedience: Formalizing, analyzing, and detecting jailbreaks,” arXiv preprint arXiv: 2305.14965, 2024.
  48. S. Ray, “Samsung Bans ChatGPT Among Employees After Sensitive Code Leak,” https://www.forbes.com/sites/siladityaray/2023/05/02/samsung-bans-chatgpt-and-other-chatbots-for-employees-after-sensitive-code-leak.
  49. S. Sarsa, P. Denny, A. Hellas, and J. Leinonen, “Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models,” in Conference on International Computing Education Research.   ACM, 2022.
  50. M. Schonlau and M. Theus, “Detecting Masquerades in Intrusion Detection based on Unpopular Commands,” Information Processing Letters, vol. 76, no. 1-2, pp. 33–38, 2000.
  51. R. Vedantam, C. L. Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” in IEEE Conference on Computer Vision and Pattern Recognition, 2015.
  52. M. Vielberth, F. Böhm, I. Fichtinger, and G. Pernul, “Security Operations Center: A Systematic Study and Open Challenges,” IEEE Access, vol. 8, pp. 227 756–227 779, 2020.
  53. Y. Wan, Z. Zhao, M. Yang, G. Xu, H. Ying, J. Wu, and P. S. Yu, “Improving Automatic Source Code Summarization via Deep Reinforcement Learning,” in ACM/IEEE International Conference on Automated Software Engineering, 2018.
  54. L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei, “Text Embeddings by Weakly-Supervised Contrastive Pre-training,” arXiv preprint arXiv:2212.03533, 2022.
  55. W. Wang, Y. Zhang, Z. Zeng, and G. Xu, “Trans^ 3: A Transformer-based Framework for Unifying Code Summarization and Code Search,” arXiv preprint arXiv:2003.03238, 2020.
  56. Y. Wang, E. Shi, L. Du, X. Yang, Y. Hu, S. Han, H. Zhang, and D. Zhang, “Cocosum: Contextual Code Summarization with Multi-relational Graph Neural Network,” arXiv preprint arXiv:2107.01933, 2021.
  57. Y. Wang, Y. Dong, X. Lu, and A. Zhou, “GypSum: Learning Hybrid Representations for Code Summarization,” in IEEE/ACM International Conference on Program Comprehension, 2022.
  58. Y. Wang, H. Le, A. D. Gotmare, N. D. Bui, J. Li, and S. C. Hoi, “CodeT5+: Open Code Large Language Models for Code Understanding and Generation,” arXiv preprint arXiv:2305.07922, 2023.
  59. Y. Wang, W. Wang, S. R. Joty, and S. C. H. Hoi, “CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation,” in Conference on Empirical Methods in Natural Language Processing.   Association for Computational Linguistics, 2021.
  60. B. Wei, G. Li, X. Xia, Z. Fu, and Z. Jin, “Code Generation as a Dual Task of Code Summarization,” in Conference on Neural Information Processing Systems.   PMLR, 2019.
  61. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in Conference on Neural Information Processing Systems.   PMLR, 2022.
  62. E. Wong, T. Liu, and L. Tan, “CloCom: Mining Existing Source Code for Automatic Comment Generation,” in International Conference on Software Analysis, Evolution, and Reengineering.   IEEE Computer Society, 2015.
  63. A. Zeng, X. Liu, Z. Du, Z. Wang, H. Lai, M. Ding, Z. Yang, Y. Xu, W. Zheng, X. Xia, W. L. Tam, Z. Ma, Y. Xue, J. Zhai, W. Chen, Z. Liu, P. Zhang, Y. Dong, and J. Tang, “GLM-130B: An Open Bilingual Pre-trained Model,” in International Conference on Learning Representations.   OpenReview.net, 2023.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

This paper has been mentioned in 2 tweets and received 0 likes.

Upgrade to Pro to view all of the tweets about this paper:

Start a free 7-day Pro trial