Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Machine-Generated Text Localization (2402.11744v2)

Published 19 Feb 2024 in cs.CL

Abstract: Machine-Generated Text (MGT) detection aims to identify a piece of text as machine or human written. Prior work has primarily formulated MGT detection as a binary classification task over an entire document, with limited work exploring cases where only part of a document is machine generated. This paper provides the first in-depth study of MGT that localizes the portions of a document that were machine generated. Thus, if a bad actor were to change a key portion of a news article to spread misinformation, whole document MGT detection may fail since the vast majority is human written, but our approach can succeed due to its granular approach. A key challenge in our MGT localization task is that short spans of text, e.g., a single sentence, provides little information indicating if it is machine generated due to its short length. To address this, we leverage contextual information, where we predict whether multiple sentences are machine or human written at once. This enables our approach to identify changes in style or content to boost performance. A gain of 4-13% mean Average Precision (mAP) over prior work demonstrates the effectiveness of approach on five diverse datasets: GoodNews, VisualNews, WikiText, Essay, and WP. We release our implementation at https://github.com/Zhongping-Zhang/MGT_Localization.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curvature. arXiv preprint arXiv:2310.05130.
  2. Conda: Contrastive domain adaptation for ai-generated text detection. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 598–610.
  3. Natural language processing with Python: analyzing text with the natural language toolkit. " O’Reilly Media, Inc.".
  4. Good news, everyone! context driven entity-aware captioning for news images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12466–12475.
  5. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  6. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 320–335.
  7. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027.
  8. Gltr: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111–116.
  9. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597.
  10. Mgtbench: Benchmarking machine-generated text detection. arXiv preprint arXiv:2303.14822.
  11. Automatic detection of generated text is easiest when humans are fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1808–1822.
  12. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense. Advances in Neural Information Processing Systems, 36.
  13. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 3214–3252.
  14. Visualnews: Benchmark and challenges in entity-aware image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6761–6771.
  15. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  16. Gpt-4v (ision) as a social media analysis engine. arXiv preprint arXiv:2311.07547.
  17. Detectgpt: Zero-shot machine-generated text detection using probability curvature. In Proceedings of the 40th International Conference on Machine Learning.
  18. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375.
  19. R OpenAI. 2023. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2:13.
  20. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  21. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026–8037.
  22. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  23. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203.
  24. Pointer sentinel mixture models. Proceedings of ICLR.
  25. DetectLLM: Leveraging log rank information for zero-shot detection of machine-generated text. In Findings of the Association for Computational Linguistics: EMNLP 2023.
  26. Detecting cross-modal inconsistency to defend against neural fake news. In Empirical Methods in Natural Language Processing (EMNLP).
  27. Edward Tian and Alexander Cui. 2023. Gptzero: Towards detection of ai-generated text using zero-shot and supervised methods.
  28. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  29. Ghostbuster: Detecting text ghostwritten by large language models. arXiv preprint arXiv:2305.15047.
  30. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax.
  31. Seqxgpt: Sentence-level ai-generated text detection. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1144–1156.
  32. Detectgpt-sc: Improving detection of text generated by large language models through self-consistency with masked predictions. arXiv preprint arXiv:2310.14479.
  33. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45.
  34. Defending against neural fake news. Advances in neural information processing systems, 32.
  35. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.
  36. Show, write, and retrieve: Entity-aware article generation and retrieval. In Findings of the Association for Computational Linguistics: EMNLP 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Zhongping Zhang (14 papers)
  2. Wenda Qin (2 papers)
  3. Bryan A. Plummer (64 papers)
Citations (3)
View on Semantic Scholar