Robust detection under meaning-preserving word substitutions

Develop reliable defenses for AI-text detectors against meaning-preserving word-level synonym substitutions that survive sanitization and substantially reduce detection performance.

Background

The paper evaluates attacks against nine AI-text detectors, including Unicode manipulations, whitespace changes, paraphrasing, and word-level synonym substitution. Sanitization effectively reverses the Unicode and whitespace attacks, restoring detector performance close to clean-text levels.

Word-level synonym substitution is different: it preserves much of the text’s meaning while changing token probabilities, substantially reducing the performance of several probability-based detectors. Because the substitution cannot be reversed by the sanitization procedures evaluated in the paper, the authors identify robust detection or defense against this attack as an unresolved challenge.