Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Detecting Machine-Generated Texts by Multi-Population Aware Optimization for Maximum Mean Discrepancy (2402.16041v2)

Published 25 Feb 2024 in cs.CL and cs.LG

Abstract: LLMs such as ChatGPT have exhibited remarkable performance in generating human-like texts. However, machine-generated texts (MGTs) may carry critical risks, such as plagiarism issues, misleading information, or hallucination issues. Therefore, it is very urgent and important to detect MGTs in many situations. Unfortunately, it is challenging to distinguish MGTs and human-written texts because the distributional discrepancy between them is often very subtle due to the remarkable performance of LLMs. In this paper, we seek to exploit \textit{maximum mean discrepancy} (MMD) to address this issue in the sense that MMD can well identify distributional discrepancies. However, directly training a detector with MMD using diverse MGTs will incur a significantly increased variance of MMD since MGTs may contain \textit{multiple text populations} due to various LLMs. This will severely impair MMD's ability to measure the difference between two samples. To tackle this, we propose a novel \textit{multi-population} aware optimization method for MMD called MMD-MP, which can \textit{avoid variance increases} and thus improve the stability to measure the distributional discrepancy. Relying on MMD-MP, we develop two methods for paragraph-based and sentence-based detection, respectively. Extensive experiments on various LLMs, \eg, GPT2 and ChatGPT, show superior detection performance of our MMD-MP. The source code is available at \url{https://github.com/ZSHsh98/MMD-MP}.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (74)
  1. Generative ai text classification using ensemble llm approaches. arXiv preprint arXiv:2309.07755, 2023.
  2. James Allen. Natural language understanding. Benjamin-Cummings Publishing Co., Inc., 1995.
  3. Gpt4all: Training an assistant-style chatbot with large scale data distillation from gpt-3.5-turbo. GitHub, 2023.
  4. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations, 2014.
  5. Climbing towards nlu: On meaning, form, and understanding in the age of data. In Proceedings of the 58th annual meeting of the association for computational linguistics, pp.  5185–5198, 2020.
  6. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. 2021.
  7. Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics, 22(14):e49–e57, 2006.
  8. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  9. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pp.  3–14, 2017.
  10. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp.  9650–9660, 2021.
  11. Classification logit two-sample testing by neural networks for differentiating near manifold densities. IEEE Transactions on Information Theory, 68(10):6631–6662, 2022.
  12. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
  13. A wild bootstrap for degenerate kernel tests. Advances in neural information processing systems, 27, 2014.
  14. On the mathematical foundations of learning. Bulletin of the American mathematical society, 39(1):1–49, 2002.
  15. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  16. Is GPT-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  7250–7274, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  17. Maximum mean discrepancy test is aware of adversarial attacks. In International Conference on Machine Learning, pp.  3564–3575. PMLR, 2021.
  18. Gltr: Statistical detection and visualization of generated text. ACL 2019, pp.  111–116, 2019.
  19. A kernel two-sample test. Journal of Machine Learning Research, 13(1):723–773, 2012.
  20. On the (statistical) detection of adversarial examples. arXiv preprint arXiv:1702.06280, 2017.
  21. Deepfake video detection using recurrent neural networks. In 2018 15th IEEE international conference on advanced video and signal based surveillance (AVSS), pp.  1–6. IEEE, 2018.
  22. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597, 2023.
  23. Robust spammer detection using collaborative neural network in internet-of-things applications. IEEE Internet of Things Journal, 8(12):9549–9558, 2020.
  24. Mgtbench: Benchmarking machine-generated text detection. arXiv preprint arXiv:2303.14822, 2023.
  25. Cater: Intellectual property protection on text generation apis via conditional watermarks. Advances in Neural Information Processing Systems, 35:5431–5445, 2022.
  26. Jason Hong. The state of phishing attacks. Communications of the ACM, 55(1):74–81, 2012.
  27. Automatic dialogue generation with expressed emotions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp.  49–54, 2018.
  28. Alberto Jiménez-Valverde. Insights into the area under the receiver operating characteristic curve (auc) as a discrimination measure in species distribution modelling. Global Ecology and Biogeography, 21(4):498–507, 2012.
  29. Minimax optimality of permutation tests. The Annals of Statistics, 50(1):225–251, 2022.
  30. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
  31. A watermark for large language models. In International Conference on Machine Learning, pp.  17061–17084. PMLR, 2023.
  32. Stylometric detection of ai-generated text in twitter timelines. arXiv preprint arXiv:2303.03697, 2023.
  33. Introduction to Mathematical Statistics and Its Applications: Pearson New International Edition. Pearson Higher Ed, 2013.
  34. Dependent wild bootstrap for degenerate u-and v-statistics. Journal of Multivariate Analysis, 117:257–280, 2013.
  35. Deep reinforcement learning for dialogue generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2016.
  36. Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:2311.03191, 2023.
  37. Learning deep kernels for non-parametric two-sample tests. In International Conference on Machine Learning, pp.  6316–6326. PMLR, 2020.
  38. Meta two-sample testing: Learning kernels for testing with limited data. Advances in Neural Information Processing Systems, 34:5848–5860, 2021.
  39. Text summarization with pretrained encoders. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
  40. Trustworthy llms: a survey and guideline for evaluating large language models’ alignment. arXiv preprint arXiv:2308.05374, 2023.
  41. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
  42. Revisiting classifier two-sample tests. In International Conference on Learning Representations, 2017.
  43. A survey on empathetic dialogue systems. Information Fusion, 64:50–70, 2020.
  44. Detectgpt: Zero-shot machine-generated text detection using probability curvature. pp.  24950–24962, 2023.
  45. Foundations of machine learning. MIT press, 2018.
  46. OpenAI. Introducing chatgpt. Website, 2022. URL https://openai.com/blog/chatgpt.
  47. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  48. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  49. Automated annotation with generative ai requires validation. arXiv preprint arXiv:2306.00176, 2023.
  50. Gpt self-supervision for a better data annotator. arXiv preprint arXiv:2306.04349, 2023.
  51. Improving language understanding by generative pre-training. 2018.
  52. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  53. Robert J Serfling. Approximation theorems of mathematical statistics. John Wiley & Sons, 2009.
  54. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203, 2019.
  55. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020.
  56. The science of detecting llm-generated texts. arXiv preprint arXiv:2303.07205, 2023a.
  57. Does synthetic data generation of llms help clinical text mining? arXiv preprint arXiv:2303.04360, 2023b.
  58. Multiscale positive-unlabeled detection of ai-generated texts. arXiv preprint arXiv:2305.18149, 2023.
  59. Phan Minh Toan. minhtoan/gpt3-small-finetune-cnndaily-news. https://huggingface.co/minhtoan/gpt3-small-finetune-cnndaily-news, 2023.
  60. Minimax estimation of maximum mean discrepancy with radial kernels. Advances in Neural Information Processing Systems, 29, 2016.
  61. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
  62. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  63. Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
  64. Arithmetic sampling: Parallel diverse decoding for large language models. In International Conference on Machine Learning, pp.  35120–35136. PMLR, 2023.
  65. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.
  66. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.
  67. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pp.  10524–10533. PMLR, 2020.
  68. Distinguishing differences matters: Focal contrastive network for peripheral anterior synechiae recognition. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VIII 24, pp.  24–33. Springer, 2021.
  69. Cross-ray neural radiance fields for novel-view synthesis from unconstrained image collections. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  15901–15911, 2023.
  70. Defending against neural fake news. Advances in neural information processing systems, 32, 2019.
  71. Detecting adversarial data by probing multiple perturbations using expected perturbation score. In International Conference on Machine Learning, pp.  41429–41451. PMLR, 2023a.
  72. Siren’s song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023b.
  73. Combating bilateral edge noise for robust link prediction. In NeurIPS, 2023a.
  74. On strengthening and defending graph reconstruction attack with markov chain approximation. In ICML, 2023b.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Shuhai Zhang (11 papers)
  2. Jiahao Yang (25 papers)
  3. Bo Han (282 papers)
  4. Mingkui Tan (124 papers)
  5. Yiliao Song (8 papers)
  6. Yuanqing Li (14 papers)
Citations (1)
View on Semantic Scholar