Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Zero-Shot Spam Email Classification Using Pre-trained Large Language Models (2405.15936v1)

Published 24 May 2024 in cs.CL and cs.AI

Abstract: This paper investigates the application of pre-trained LLMs for spam email classification using zero-shot prompting. We evaluate the performance of both open-source (Flan-T5) and proprietary LLMs (ChatGPT, GPT-4) on the well-known SpamAssassin dataset. Two classification approaches are explored: (1) truncated raw content from email subject and body, and (2) classification based on summaries generated by ChatGPT. Our empirical analysis, leveraging the entire dataset for evaluation without further training, reveals promising results. Flan-T5 achieves a 90% F1-score on the truncated content approach, while GPT-4 reaches a 95% F1-score using summaries. While these initial findings on a single dataset suggest the potential for classification pipelines of LLM-based subtasks (e.g., summarisation and classification), further validation on diverse datasets is necessary. The high operational costs of proprietary models, coupled with the general inference costs of LLMs, could significantly hinder real-world deployment for spam filtering.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (32)
  1. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901.
  2. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416.
  3. Machine learning for email spam filtering: review, approaches and open research problems. Heliyon, 5(6):e01802.
  4. Deng, J. (2023). Email spam filtering methods: comparison and analysis. Highlights in Science, Engineering and Technology, 38:187–198.
  5. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  6. A Survey of Text Classification With Transformers: How Wide? How Large? How Long? How Accurate? How Expensive? How Safe? IEEE Access, 12:6518–6531.
  7. Measuring, characterizing, and avoiding spam traffic costs. IEEE Internet Computing, 20(4):16–24.
  8. Applicability of machine learning in spam and phishing email filtering: review and approaches. Artificial Intelligence Review, 53(7):5019–5081.
  9. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints.
  10. Devising and detecting phishing emails using large language models. IEEE Access.
  11. A review of spam email detection: analysis of spammer strategies and the dataset shift problem. Artificial Intelligence Review, 56(2):1145–1173.
  12. Kalyan, K. S. (2023). A survey of gpt-3 family large language models including chatgpt and gpt-4. Natural Language Processing Journal, page 100048.
  13. Detecting phishing sites using ChatGPT. arXiv preprint arXiv:2306.05816.
  14. Chatspamdetector: Leveraging large language models for effective phishing email detection. arXiv preprint arXiv:2402.18093.
  15. Spam-T5: Benchmarking large language models for few-shot email spam detection. arXiv preprint arXiv:2304.01238.
  16. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691.
  17. Spam email detection using machine learning and deep learning techniques. In Proceedings of the International Conference on Innovative Computing & Communication (ICICC).
  18. The evolution of the Nigerian prince scam. Journal of Financial Crime, 30(6):1653–1663.
  19. OpenAI (2024). GPT-4 Technical Report. arXiv preprint arXiv:2303.08774.
  20. The economics of spam. Journal of Economic Perspectives, 26(3):87–110.
  21. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems.
  22. Rojas-Galeano, S. (2017). On obstructing obscenity obfuscation. ACM Transactions on the Web, 11(2):1–24.
  23. Rojas-Galeano, S. (2021). Using BERT encoding to tackle the mad-lib attack in SMS spam detection. arXiv preprint arXiv:2107.06400.
  24. Deep learning to filter SMS spam. Future Generation Computer Systems, 102:524–533.
  25. Generating phishing attacks using ChatGPT. arXiv preprint arXiv:2305.05133.
  26. Investigating evasive techniques in sms spam filtering: A comparative analysis of machine learning models. IEEE Access.
  27. Tharwat, A. (2020). Classification assessment methods. Applied computing and informatics, 17(1):168–192.
  28. Attention is all you need. Advances in neural information processing systems, 30.
  29. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.
  30. Evaluating the performance of chatgpt for spam email detection. arXiv preprint arXiv:2402.15537.
  31. Yaseen, Q. et al. (2021). Spam email detection using deep learning techniques. Procedia Computer Science, 184:853–858.
  32. A survey of large language models. arXiv preprint arXiv:2303.18223.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (1)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets