DAMAGE: Detecting Adversarially Modified AI Generated Text

Abstract: AI humanizers are a new class of online software tools meant to paraphrase and rewrite AI-generated text in a way that allows them to evade AI detection software. We study 19 AI humanizer and paraphrasing tools and qualitatively assess their effects and faithfulness in preserving the meaning of the original text. We show that many existing AI detectors fail to detect humanized text. Finally, we demonstrate a robust model that can detect humanized AI text while maintaining a low false positive rate using a data-centric augmentation approach. We attack our own detector, training our own fine-tuned model optimized against our detector's predictions, and show that our detector's cross-humanizer generalization is sufficient to remain robust to this attack.

• The paper introduces DAMAGE, a deep learning detector adept at recognizing adversarially humanized AI text.
• It systematically audits 19 humanizer tools, categorizing them by fluency, faithfulness, and coherence to assess detection challenges.
• Empirical results show DAMAGE maintains high true positive rates even under adaptive humanizer strategies, demonstrating its robustness.

Detecting Adversarially Modified AI-Generated Text: An Analysis of DAMAGE

Introduction

The prevalence of LLMs capable of generating fluent, contextually appropriate text has precipitated concerns surrounding academic integrity, SEO manipulation, and provenance of online content. In this context, the operational efficacy of AI-generated text detectors and the emergence of "humanizer" tools, which adversarially rewrite AI outputs to evade these detectors, constitute an urgent and technically challenging research area. The paper "DAMAGE: Detecting Adversarially Modified AI Generated Text" (2501.03437) systematically studies this duality, offering a comprehensive audit of the humanizer market, an analysis of detection robustness, and the introduction of a deep learning-based detector specifically robust to humanization-based attacks.

Figure 1: Example of an AI humanizer tool.

Market and Technical Audit of Humanizer Tools

A defining contribution of the study is the structured audit of 19 distinct paraphrasing and humanizer tools, ranging from basic synonym replacers to advanced LLM-based systems. The audit segments these tools into three quality tiers (L1, L2, L3) based on fluency, faithfulness, and coherence. The L1 group, which includes both commercial and academically developed paraphrasers such as DIPPER, demonstrates high fidelity to the original semantics and minimal degradation in text quality. Lower-tier systems frequently introduce overt errors, nonsensical phrases, and spurious citations, thereby reducing the plausibility of the rewritten output.

(Figure 2)

Figure 2: Augmenting the training set with high quality humanizer data improves robustness.

These tools are prevalent and influential, with "humanizer" GPTs dominating custom GPT stores, indicating significant user reliance, particularly in educational and SEO contexts.

The audit further demonstrates that many state-of-the-art watermarking schemes—such as Google's SynthID—are highly susceptible to removal by humanizers. Watermark true positive rates (TPR) plummet from 87.6% to 5.4% at 5% FPR after paraphrasing by DIPPER, evidencing that statistical and probabilistic watermarks offer limited practical protection if an adversary has access to competent humanization tools.

Empirical Analysis of Detection Robustness

The paper rigorously benchmarks commercial (GPTZero, TurnItIn, Originality), open-source (DetectGPT, Binoculars, RADAR), and their own deep learning-based detection models against both unmodified and humanized AI-generated texts. Most detectors exhibit a dramatic drop in TPR—sometimes exceeding 60 percentage points—when faced with high-quality humanized outputs. In contrast, the DAMAGE model, which is explicitly trained with a small volume of humanized samples (via targeted data-centric augmentation), maintains high TPR with minimal loss in precision, even on paraphrases generated by unseen humanizers and in challenging cross-domain settings such as the RAID benchmark.

This empirical result underlines a core technical insight: genuinely robust detection of humanized AI text relies not on domain-specific heuristics or watermarking, but on treating humanizer robustness as a learned invariance within a discriminative representation (i.e., directly incorporating adversarial transformations during training).

Qualitative Insights: Transformation Patterns and Fluency Evaluation

Qualitative analysis reveals that while low-tier humanizers degrade quality with obvious artifacts, L1 systems can produce outputs nearly indistinguishable from original human text, closely mirroring tone and complexity. Fluency Win Rate metrics (drawn via GPT-4o evaluations) empirically quantify this: L1 humanizers' outputs are selected as more fluent than the original 26% of the time, with L2 and L3 tiers trailing substantially. Notably, all humanizers tend to reduce text quality when compared to base LLM outputs, but L1 systems' outputs remain high-quality, motivating their focused inclusion in adversarial data augmentation for robust detector training.

(Figure 3)

Figure 3: We segment humanizers into three tiers, based on their fluency.

Detectors Optimized Against Adversarial Humanization

To probe an adaptive adversarial scenario wherein a humanizer is directly fine-tuned to bypass a specific detector, the authors construct a humanizer optimized against their DAMAGE model’s outputs using feedback as a negative reward. Despite this targeted optimization, DAMAGE retains a TPR of 93.2% at the operational threshold, indicating that underlying generation signatures remain discriminative, likely due to persistent LLM-specific idiosyncrasies that are not fully erased by paraphrasing, even with detector-targeted optimization.

Theoretical and Practical Implications

The results robustly demonstrate that naive or even watermark-strengthened detection techniques are inadequate for adversarially humanized text. The effectiveness of the DAMAGE detector establishes that detection of adversarially modified AI text is a viable technical challenge, provided that robust invariance to transformation is achieved through diversified training. This has implications for both deployment (mitigation of academic or SEO abuse of LLMs) and for future work in adversarial machine learning and watermarking.

The generalization capacity of detectors trained with diverse, high-quality humanizer outputs suggests new avenues for scalable, domain-general detection. However, the persistent incremental decrease in TPR even under robust training highlights the ongoing nature of the arms race between paraphraser sophistication and detector discrimination.

Conclusion

This paper offers a comprehensive audit of humanizer tools, a rigorous evaluation of detection methods, and introduces a scalable deep learning architecture capable of achieving high robustness even to adaptive adversarial humanization. The analysis establishes that data-centric augmentation, using diverse high-quality adversarial examples from the evolving humanizer ecosystem, is the most viable path toward sustainable detection efficacy. Future research may extend toward adaptive online learning strategies for detectors to keep pace with emergent transformation techniques and towards hybrid detection paradigms that blend robust learning with watermarking or fingerprinting at the model or dataset level.