CoCoDet: AI Review Authorship Detector

• CoCoDet is a supervised model that distinguishes peer reviews into Human, Mix, and AI categories based on substantive content.
• It employs a ModernBERT backbone with multi-task supervision to disentangle true content signals from superficial stylistic features, achieving over 98% macro-F1 performance.
• The model leverages auxiliary tasks for content source and style attribution, enabling fine-grained detection across a detailed taxonomy of human–AI collaboration modes.

CoCoDet is a supervised model for detecting AI involvement in peer review content, purpose-built to distinguish genuine human-written text from both mixed and fully AI-generated reviews in a manner robust to style-based obfuscations and semantic-invariant transformations. Developed as part of the CoCoNUTS initiative, CoCoDet is designed around a content-based detection paradigm and leverages a multi-task learning setup to disentangle substantive content characteristics from superficial stylistic signals, thus providing reliable attribution of review authorship across a challenging and granular taxonomy of human–AI collaboration modes (Chen et al., 28 Aug 2025).

1. Architectural Overview

CoCoDet employs the ModernBERT backbone—a bidirectional, long-context, memory-efficient Transformer encoder—as a shared feature extractor for multiple classification heads. The model is trained with multi-task supervision, where all tasks process the same encoded textual representation, but each has a distinct classifier:

• Primary Task: Content Composition Identification (ternary: Human, Mix, AI).
• Auxiliary Tasks:
  • Content Source Attribution (multi-label): Identify the specific authorship sources (e.g., which LLMs or humans contributed substantive content).
  • Textual Style Attribution (multi-label): Attribute surface linguistic style to a family/model.
  • Collaboration Mode Attribution (multiclass, six-way): Predict the precise mode of human–AI collaboration, including HW (human-written), HWMT (human, back-translated), HWMP (human, machine-polished), HWMG (human with AI-augmentation), MG (pure AI), and MGMP (AI–AI paraphrase cascades).

At inference, only the output head for the primary ternary classification is employed; the auxiliary heads are discarded.

2. Objective Functions and Optimization

CoCoDet’s training objective is a weighted sum of task-specific losses:

• Ternary Classification (Main Task): Utilizes a cost-sensitive margin loss (CSM-Loss) based on a large-margin cosine (CosFace) formulation. A base margin, $m_{base}$, is subtracted from the logit for the correct class, while a cost margin, $m_{cost}$, is added to negative classes (specifically between the "Human" and "AI" classes) before cross-entropy, all scaled by $s$.
• Content Source Attribution: Multi-label binary cross-entropy over possible content sources.
• Textual Style Attribution: Multi-label binary cross-entropy across style families.
• Collaboration Mode Attribution: Multiclass cross-entropy for the six collaboration types.

The total loss function is:

$$L = L_{main} + \alpha\,L_{con} + \beta\,L_{sty} + \gamma\,L_{mode}$$

with recommended weights $\alpha=0.4$, $\beta=0.4$, $\gamma=0.2$; margin and scaling hyperparameters are $m_{base}=0.25$, $m_{cost}=0.25$, $s=30$.

3. Content-Oriented Detection Paradigm

CoCoDet embodies a decisive shift from traditional, style-centric AI text detection to a content-based approach. Paraphrasing, back-translation, and surface-level polishing are treated as style-preserving transformations that do not alter semantic content. The model’s detection logic is focused on identifying the true generator of “meaningful” content rather than surface features.

The six collaboration modes in CoCoNUTS are mapped into three content classes as follows:

Granular Mode	Content Class
HW, HWMT, HWMP	Human
HWMG	Mix
MG, MGMP	AI

The predicted class is determined by $m_{cost}$0, where $m_{cost}$1 are margin-adjusted class logits. This approach enables robustness against even advanced model-human collaboration and style disguise strategies.

4. Dataset: CoCoNUTS

CoCoDet’s development and evaluation rely on the CoCoNUTS dataset, comprising 315,535 peer reviews stratified across six human–AI collaboration modes and balanced into the three principal content-based classes (each with approximately 105k examples). The dataset features:

• Training (≈84k/class), Validation (≈10.5k/class), Test (≈10.5k/class): Stratified by contributing model.
• Collaboration Modes: HW (pure human, pre-2022), HWMT (human back-translated by Llama, Qwen2.5), HWMP (human machine-polished by Gemini, Llama, Qwen3), HWMG (human with model-augmentation, several LLMs), MG (pure AI, diverse models), MGMP (AI–AI paraphrase combinations).
• Text Length: AI-authored reviews average ≈494 words, compared to ≈431 for humans; "Mix" is intermediate.

This comprehensive annotation enables rigorous supervision for both content and style attribution, supporting the model’s disentanglement of semantic and stylistic contributions.

5. Comparative Evaluation and Performance

Evaluation metrics include per-class F1, macro-F1 for ternary (Human/Mix/AI) classification, as well as binary accuracy, predicted AI rates for each subset, and assessments of style robustness (requiring monotonic AI-rates, Human ≤ Mix ≤ AI).

Summary of Results

Detector	Human F1	Mix F1	AI F1	Macro F1	False Pos. on Human	False Neg. on AI
CoCoDet	98.94	97.41	98.37	98.24	1.31%	3.10%
LLMs (few-shot best: Gemini)	74.05	39.90	62.97	58.97	n/a	n/a
Radar	—	—	—	—	24.91%	65.07%
FastDetectGPT	—	—	—	—	53.09%	7.44%
Binoculars	—	—	—	—	15.86%	25.68%
LLM-DetectAIve	—	—	—	—	3.92%	16.48%

CoCoDet achieves >98% macro-F1 on ternary classification and reduces false positives on human text to 1.3%, with strong robustness under paraphrasing attacks where general detectors’ performance collapses. Auxiliary tasks contribute additive gains in accuracy. The cost margin is particularly decisive for separating Human vs. AI content.

6. Implementation and Ablation Analyses

CoCoDet is trained with the following settings:

• Backbone: ModernBERT (Warner et al., 2024) (Warner et al., 2024)
• Input: Max sequence length 2048 tokens
• Optimizer: AdamW, weight decay 0.01
• Learning Rate: $m_{cost}$2
• Batch Size: 16
• Epochs: 5 (best checkpoint on validation)
• Random Seed: 42

Ablation studies indicate that auxiliary labels for content source, style, and collaboration mode each provide up to $m_{cost}$3 macro-F1 improvement. Removal of either the base or cost margin from the margin-based main loss significantly degrades performance (−1.7% macro-F1).

7. Strengths, Limitations, and Future Directions

Strengths

• Content-Centric Design: Systematically robust to semantic-invariant style operations.
• State-of-the-Art Ternary Performance: >98% macro-F1, <2% human false positives.
• Fine-Grained Labeling: Multi-task auxiliary supervision achieves content/style disentanglement.

Limitations

• Labeled Data Requirements: Success depends on large, finely annotated datasets with complete attribution.
• Domain Generalization: Performance may degrade on unseen domains or novel collaboration types.
• Inference Efficiency: The model’s transformer backbone imposes computational requirements unsuited for some real-time or low-resource applications.

Potential Extensions

• Cross-Venue Adaptation: Zero-/few-shot adaptation to new venues or languages.
• Model Compression: Distillation to lightweight architectures for deployment.
• Contrastive Pretraining: Unsupervised style-invariant representation learning.
• Granular Localization: Sentence-level detection of AI vs. human authorship.

CoCoDet and the CoCoNUTS benchmark together establish a rigorous foundation for content-oriented AI review detection and offer directions for increasingly robust and equitable scholarly integrity assessment (Chen et al., 28 Aug 2025).