AITDNA: AI Text Detection Under Natural Assumptions

• AITDNA is a benchmark and dataset capturing natural human–AI co-writing with recorded edit histories and token-level authorship, enabling nuanced detection evaluation.
• It maps token-level genesis into multiple span-level detection notions (document, sentence, boundary, content, and intent) using defined threshold parameters.
• The benchmark exposes detector performance sensitivities to hidden assumptions and granularity, informing practical deployment in varied policy and operational settings.

AITDNA, short for AI Text Detection under Natural Assumptions, is a benchmark and dataset for evaluating AI-generated text detection under realistic human–machine co-writing conditions rather than simplified synthetic assumptions. It was introduced to address a central problem in AI text detection research: the field often treats “AI-generated text” as if it were a single, stable object, while actual deployment scenarios differ in normative standard, provenance assumptions, and attacker model (Dycke et al., 3 Jun 2026). AITDNA is therefore designed as a notion-agnostic, provenance-rich corpus of naturally human–AI co-created texts with recorded edit histories, prompts, model responses, and token-level authorship information, enabling multiple formal detection notions to be projected onto the same underlying texts (Dycke et al., 3 Jun 2026).

1. Conceptual scope and motivation

AITDNA is grounded in the claim that current AI-generated text detection benchmarks frequently encode hidden assumptions that are too restrictive for real use (Dycke et al., 3 Jun 2026). Typical assumptions include document-level en-bloc generation, synthetic mixing, no policy distinction, and no provenance ambiguity. Under such assumptions, a document is often treated as either wholly human or wholly AI, or AI content is inserted into human text in a controlled artificial way. AITDNA is built against that background, focusing instead on iterative and collaborative writing in which a human may draft text, request continuation, request revision, accept some model outputs, reject others, and repeat (Dycke et al., 3 Jun 2026).

This design responds to a broader definitional issue: the paper argues that “AI-generated text” is not a single well-defined object (Dycke et al., 3 Jun 2026). Depending on the application, the relevant target may be an entire AI-written document, AI passages inside human text, AI-written sentences, AI content violating a content policy, AI content violating an intent policy, text matching a reference population, or text matching a particular author’s prior writing. AITDNA was introduced to support evaluation across these distinct notions using texts produced through actual human–LLM interaction rather than post hoc synthetic construction (Dycke et al., 3 Jun 2026).

A plausible implication is that benchmark performance without explicit notion specification can be misleading. If a detector is evaluated on a dataset whose labels reflect only one hidden interpretation of “AI text,” its reported quality may not transfer to another deployment setting with different granularity or policy constraints. This implication is central to the benchmark’s design and evaluation agenda (Dycke et al., 3 Jun 2026).

2. Formalization: genesis, notions, and policy-sensitive labeling

A core construct in AITDNA is genesis, defined as the recorded origin and history of each token in a text (Dycke et al., 3 Jun 2026). At the token level, genesis assigns labels from $\{H, AI, MX\}$, where H denotes human-originated tokens, AI denotes AI-originated tokens, and MX denotes mixed origin, such as human text corrected or transformed by AI. For tokens labeled $AI$ or $MX$, the dataset also records prompt provenance, linking those tokens to the underlying prompt that contributed to their generation (Dycke et al., 3 Jun 2026). The benchmark treats genesis as an objective trace of what happened, rather than as a normative decision about whether the observed AI use should count as disallowed or harmful.

On top of genesis, the paper defines a notion as a mapping from tokens to binary labels $\{H, AI\}$ through segmentation and labeling functions (Dycke et al., 3 Jun 2026). The segmentation function $B$ partitions a text into connected spans, and the labeling function $M$ assigns a detection label to each span. For genesis-based notions, the paper introduces a threshold $\tau \in [0,1]$ and labels a span as AI if its AI-token ratio exceeds $\tau$ (Dycke et al., 3 Jun 2026). Conceptually, this converts token-level provenance into span-level decisions, making it possible to study document-level, sentence-level, and boundary-level settings within a common formal scheme.

For policy-sensitive variants, the paper defines content-based AITD and intent-based AITD by combining genesis-based labeling with a policy function $\rho$ (Dycke et al., 3 Jun 2026). In these notions, a span is labeled AI only if it is AI-generated under the genesis criterion and violates the relevant policy. In content-based AITD, the policy depends on the span’s content; in intent-based AITD, it depends on the underlying prompt. This separates factual provenance from policy judgment and makes explicit that not every instance of AI assistance is necessarily treated as equally problematic (Dycke et al., 3 Jun 2026).

The framework distinguishes two broad families of notions:

• Genesis-based notions: document-level AITD, boundary-level AITD, sentence-level AITD, intent-based AITD, and content-based AITD (Dycke et al., 3 Jun 2026).
• Population-based notions: membership-based AITD and authorship-ID-based AITD, which judge whether text matches a reference population rather than whether it is AI-originated in an absolute sense (Dycke et al., 3 Jun 2026).

This separation is methodologically important because it prevents the benchmark from collapsing fundamentally different tasks into a single undifferentiated detection problem.

3. Dataset design and provenance capture

AITDNA is described as notion-agnostic because it was not collected to fit one specific labeling scheme (Dycke et al., 3 Jun 2026). Instead, it captures sufficient interaction and edit provenance to allow different notions to be instantiated afterward. The dataset was collected through a dedicated writing interface based on CARE, where participants co-wrote with an LLM in a text editor using two main interaction modes: Continuation and Revision (Dycke et al., 3 Jun 2026). In continuation mode, a participant selects a position and the LLM continues the text from that point, optionally guided by a prompt. In revision mode, a participant selects a span and the LLM revises that span, again optionally guided by a prompt. The proposed output is displayed next to the editor, and the participant may accept or reject it (Dycke et al., 3 Jun 2026).

The benchmark’s most distinctive technical asset is its provenance logging. It records the entire edit history, all insertions and deletions, all user prompts, all LLM responses, interaction traces over time, token-level authorship labels, and token-level prompt provenance (Dycke et al., 3 Jun 2026). From these traces, the authors derive per-token authorship and prompt information, making it possible to reconstruct genesis at fine granularity. This design is substantially different from datasets constructed only from final text snapshots, because it retains the latent process by which a document was produced.

The benchmark thereby supports analyses at multiple levels of granularity. A document can be evaluated as a whole; spans can be segmented by sentences or boundaries; content and prompts can be linked to policy-based decisions; and, in principle, author-specific reference populations can be defined for authorship-style comparisons. This suggests that AITDNA functions not merely as a corpus but as a structured observational substrate for studying the interaction between provenance, policy, and detection under realistic assumptions (Dycke et al., 3 Jun 2026).

4. Collection protocol and corpus composition

The collection protocol involved multiple writing sessions in which participants typically spent about 90 minutes, targeting 350–450 words per text (Dycke et al., 3 Jun 2026). Each session included one human-only text and three human–LLM co-created texts. The human-only condition served both as a purely human reference set and as an interface familiarization phase (Dycke et al., 3 Jun 2026).

The writing scenarios sampled from argumentative, creative, and explanatory tasks, and the study also included two scientific-writing sessions with scientific argumentative writing, scientific explanatory writing, impact statement or creative-style scientific writing, and peer review (Dycke et al., 3 Jun 2026). Co-writing used one of five models: Llama4-Scout, Qwen2.5:7B, DeepSeek-V3.2 (Non-Thinking Mode), GPT-5.2, and Gemini-3-flash, with temperatures 0 or 1 (Dycke et al., 3 Jun 2026).

Participants comprised 99 participants across 8 sessions, all with at least B1 English proficiency; scientific scenarios used postgraduate or PhD students, while non-scientific scenarios involved crowd workers filtered for essay-writing performance (Dycke et al., 3 Jun 2026). To maintain data quality, the authors removed texts containing copy-paste from external sources when copied spans exceeded 60 characters from outside the interface (Dycke et al., 3 Jun 2026).

The final corpus statistics reported for AITDNA are as follows:

Property	Value
Texts collected	452
Texts remaining after filtering	362
Human-only samples	95
Human–AI co-written share	74%
Essay-style share	71%
Scientific setting share	29%
Average length	459 words

These figures place AITDNA in a middle regime of human–AI mixing rather than at either extreme of fully human or fully synthetic AI-only composition (Dycke et al., 3 Jun 2026).

The paper also compares AITDNA with earlier datasets including CoAuthor, SenDetEx, BD, Mixset, and DetectRL (Dycke et al., 3 Jun 2026). Two diagnostics are highlighted: average AI-token ratio and number of human–AI boundaries. AITDNA is reported to have 50.91% AI tokens and 23.33 boundaries, compared with 24.09% / 20.32 for CoAuthor, 30.96% / 13.57 for SenDetEx, 65.68% / 2.75 for BD, 64.57% / 31.51 for Mixset, and 63.69% / 50.33 for DetectRL (Dycke et al., 3 Jun 2026). The paper interprets BD, Mixset, and DetectRL as more synthetic and assumption-heavy, while AITDNA is positioned as a neutral reference point because it is based on natural interaction rather than adversarial or one-shot generation (Dycke et al., 3 Jun 2026).

5. Benchmarking methodology and empirical findings

AITDNA is used to benchmark eight detectors under multiple notions of AI-generated text (Dycke et al., 3 Jun 2026). The evaluated detectors are Min-K, Likelihood, Log Rank, Binoculars, modernBERT-ai-detection-raid-mage (moBERT), FastDetectGPT, and the proprietary systems GPTZero and Pangram (Dycke et al., 3 Jun 2026). For GPTZero and Pangram, non-binary outputs are mapped into $\{H, AI\}$ by treating anything not explicitly labeled $AI$0 as $AI$1 (Dycke et al., 3 Jun 2026). Reported metrics are AUROC, F1-score, and False Positive Rate (FPR) (Dycke et al., 3 Jun 2026).

The benchmark comprises three main experimental protocols: evaluation of all detectors on all notions within AITDNA, comparison across datasets under a fixed notion, and analysis of performance as the threshold $AI$2 varies (Dycke et al., 3 Jun 2026). For population-based notions, authorship-ID-based AITD could not be meaningfully evaluated because most participants contributed only one text (Dycke et al., 3 Jun 2026).

Several findings are emphasized. First, notion choice strongly changes detector performance (Dycke et al., 3 Jun 2026). On AITDNA, document-level AITD is the easiest setting, whereas sentence-level and boundary-level settings are harder, and content-based and intent-based AITD are the hardest for most detectors (Dycke et al., 3 Jun 2026). The reported interpretation is that current detectors mostly exploit surface-level statistical cues and are comparatively weak when the target depends on semantic intent or policy violation rather than purely stylistic or distributional evidence (Dycke et al., 3 Jun 2026).

Second, higher granularity lowers performance (Dycke et al., 3 Jun 2026). The paper notes that this observation is consistent with earlier work, but argues that the difficulty is not only a matter of granularity; it also reflects whether the target notion aligns with the detector’s implicit assumptions. Third, false positive rates on hybrid texts are often high (Dycke et al., 3 Jun 2026). For many detectors, human–AI co-written documents are over-labeled as AI, which has direct deployment relevance because benign AI-assisted writing may be misclassified as malicious or disallowed (Dycke et al., 3 Jun 2026).

Fourth, no detector is best across all datasets (Dycke et al., 3 Jun 2026). With $AI$3 for document-level evaluation, Log Rank is best on AITDNA, Likelihood is best on CoAuthor, and moBERT is strongest on DetectRL and BD (Dycke et al., 3 Jun 2026). The paper uses this result to argue that detector rankings are highly sensitive to dataset assumptions and hidden label distributions. Fifth, threshold choice matters substantially (Dycke et al., 3 Jun 2026). Varying $AI$4 from 0.1 to 0.9 changes the recall–FPR balance: low $AI$5 gives higher recall but can induce very high FPR, whereas high $AI$6 lowers FPR but sharply reduces F1 (Dycke et al., 3 Jun 2026). The explicit conclusion is that $AI$7 must be stated, because otherwise evaluation results are not directly comparable across studies (Dycke et al., 3 Jun 2026).

6. Interpretation, limitations, and significance for AI text detection

AITDNA’s principal significance lies in redefining evaluation around explicit assumptions rather than around a single implicit notion of AI-generated text (Dycke et al., 3 Jun 2026). The benchmark shows that deployment-relevant questions are inseparable from three dimensions: the normative standard, the genesis assumptions, and the attacker model. As a result, a detector that performs well on synthetic whole-document AI-versus-human classification may not perform well on boundary localization, policy-based screening, or mixed-authorship settings (Dycke et al., 3 Jun 2026).

The paper is also explicit about limitations. Data were collected in a lab setting rather than in a real-world application such as ChatGPT, and conversational multi-turn interaction styles common in the wild are absent (Dycke et al., 3 Jun 2026). The dataset is relatively small, which limits its use as a large-scale training resource and positions it primarily as an analysis and evaluation benchmark (Dycke et al., 3 Jun 2026). Authorship-ID-based AITD could not be fully tested because of insufficient multi-sample-per-author coverage (Dycke et al., 3 Jun 2026). The copy-paste filter may be conservative, potentially discarding valid examples (Dycke et al., 3 Jun 2026). Finally, population-based, content-based, and intent-based notions remain underexplored relative to more conventional document-level settings (Dycke et al., 3 Jun 2026).

The broader implication is that AI text detection should not be framed as a monolithic binary classification problem. AITDNA instead encourages explicit declaration of what is being detected, under which threshold, with which provenance assumptions, and for which policy context (Dycke et al., 3 Jun 2026). This suggests a shift from the coarse question “Was this text written by AI?” toward the more technically precise question of whether a text or span should be treated as AI-generated under a specified notion and operational standard. In that sense, AITDNA is both a benchmark and a formal intervention in the methodology of AI-generated text detection (Dycke et al., 3 Jun 2026).