MGTEVAL: An Interactive Platform for Systemtic Evaluation of Machine-Generated Text Detectors

Abstract: We present MGTEVAL, an extensible platform for systematic evaluation of Machine-Generated Text (MGT) detectors. Despite rapid progress in MGT detection, existing evaluations are often fragmented across datasets, preprocessing, attacks, and metrics, making results hard to compare and reproduce. MGTEVAL organizes the workflow into four components: Dataset Building, Dataset Attack, Detector Training, and Performance Evaluation. It supports constructing custom benchmarks by generating MGT with configurable LLMs, applying 12 text attacks to test sets, training detectors via a unified interface, and reporting effectiveness, robustness, and efficiency. The platform provides both command-line and Web-based interfaces for user-friendly experimentation without code rewriting.

• The paper introduces MGTEval, an extensible evaluation platform integrating dataset building, adversarial attacks, and unified detector training for MGT detection.
• The platform standardizes multi-metric evaluations, including accuracy, F1, AUROC, and robustness assessments under various adversarial perturbations.
• Experimental results highlight that detectors like Longformer achieve near-perfect performance while exposing calibration sensitivity and attack vulnerabilities.

MGTEval: A Systematic, Extensible Evaluation Platform for Machine-Generated Text Detection

Motivation and Problem Statement

The proliferation of highly capable LLMs has catalyzed the widespread generation of fluent, coherent machine-generated text (MGT), amplifying the need for reliable detection to counter misuse (fraud, spam, misinformation). While multiple detection approaches have been developed, robust and truly comparative evaluation remains lacking. The absence of unified experimental protocols—especially with respect to dataset construction, robustness evaluation (adversarial attacks), and efficiency metrics—has created obstacles to fair benchmarking and practical deployment. MGTeval addresses this gap by proposing an extensible, end-to-end evaluation platform for MGT detectors.

Architecture and Functional Overview

MGTeval organizes evaluation into four modular stages, all controlled via configuration and registry-based management for maximal extensibility and reproducibility:

1. Dataset Building: Ingests standard text corpora in flexible formats, or synthesizes paired human-machine benchmarks using configurable LLM generators with full metadata and parameterization.
2. Dataset Attack: Systematically perturbs machine-generated samples using 12 supported adversarial attacks at multiple text granularities for robustness stress testing.
3. Detector Training: Provides unified interfaces for both metric-based and model-based detectors, encompassing standard classifiers (e.g., RoBERTa, Longformer) and statistical/perturbation-based schemes.
4. Performance Evaluation: Automates reporting of effectiveness, robustness, and efficiency metrics under locked thresholds and test splits for reliable, reproducible comparison.

The platform exposes both CLI and Web-based interfaces, supports comprehensive user configuration, and enables integration of custom modules via declarative registry mechanisms.

Figure 1: The framework encapsulates the data pipeline and modular architecture of MGTeval, supporting dataset creation, adversarial attacks, detector training, and evaluation as configurable, traceable components.

Dataset Creation and Attack Integration

MGTeval's dataset builder standardizes diverse corpora, supporting local and widely used academic benchmarks (e.g., HC3, SemEval, M4) as well as configurable generation pipelines with variable LLMs, prompts, and decoding parameters. Each record is comprehensively labeled and annotated for downstream analysis. The attack module introduces 12 perturbation methods, spanning from character-level (homoglyphs, typographical errors) to paragraph-level (paraphrasing, span rewriting) and document-level (back-translation, humanization), with all provenance tracked for traceable robustness evaluation.

Detector Training and Calibration

Detector support encompasses 26 representative methods, spanning:

• Metric-based detectors: Zero/few-shot detectors using statistical cues (e.g., DetectGPT, DNA-GPT, GLTR, entropy/rank/likelihood-based signals).
• Model-based detectors: Fine-tuned discriminative models leveraging contrastive or adversarial training (e.g., CoCo, DeTeCtive, GREATER, Longformer), as well as robustification and efficiency-oriented extensions (e.g., PECoLA, MPU).

MGTeval standardizes calibration (e.g., logistic regression mapping for metric-based scores) and exposes explicit control over all relevant training hyperparameters, thus ensuring consistency and facilitating controlled ablation studies.

Figure 2: The detector training interface enables selection, configuration, and real-time monitoring of diverse detector architectures and training trajectories.

Evaluation Protocols and Metrics

MGTeval enforces protocol fixedness, disallowing threshold re-tuning during robustness evaluation, and computes a spectrum of metrics:

• Effectiveness: Standard metrics (Accuracy, F1, AUROC, AUPR) and operationally critical ones (TPR@FPR $\{0.01, 0.001\}$), critical for deployment in moderation/forensics.
• Robustness: Attack Success Rate (ASR) on paired clean/adversarial samples, supporting fine-grained attack-wise analysis.
• Efficiency: Evaluation time, throughput, and memory/GPU utilization, quantifying deployment cost.

Interaction is streamlined in the WebUI, allowing rapid configuration, run management, and visualization of results, logs, and artifacts.

Figure 3: The evaluation interface delivers unified, multi-metric reports, supporting selection of detectors, evaluation datasets, checkpoints, and parameterizations.

Experimental Insights

The unified evaluation on a challenging, balanced dataset (SemEval 2024 human texts, Qwen3-generated machine texts) yields several critical results:

• Best overall performance: Longformer achieves 99.50% Accuracy, 99.50% F1, 99.99% AUROC, and 100% TPR@FPR=0.01, outpacing all other detectors by clear margins.
• Efficiency-Effectiveness tradeoff: GREATER is notable for SOTA-level accuracy (97.50%) and lowest GPU consumption (0.14 GiB), while RoBERTa-Base, PECoLA, and MPU also achieve high throughput.
• Calibration Sensitivity: Several detectors (e.g., TOCSIN) show high AUROC but suffer from dramatic threshold/calibration dependence—demonstrating the necessity of multi-metric evaluation.
• Robustness: Attack-wise evaluation reveals large variance in ASR, underscoring that certain detectors are highly vulnerable to simple perturbations, a property invisible under clean-set-only testing protocols.

  Figure 4: The demo panel allows interactive, parameterized testing of different detectors and models, exposing detection confidence and facilitating exploratory analysis.

Extensibility and Limitations

MGTeval's modular architecture facilitates the integration of novel attack techniques, detector designs (including those for fine-grained attribution and multilingual or cross-domain use cases), and advances in robustness testing. However, major limitations include computational cost for comprehensive benchmarking, possible deviation from detector-specific hyperparameter optima due to unification, and a current focus on binary classification and fixed attack sets.

Figure 5: The pipeline overview visualizes the component connectivity and user configurability for each pipeline stage.

Practical and Theoretical Implications

Practically, MGTeval provides the first standardized, extensible, and efficiency-aware testbed for MGT detection, closing the reproducibility gap and enabling direct comparative studies under adversarial stress. This consolidates best-practices for detector evaluation, facilitating transition from research prototypes to real-world deployment. Theoretically, MGTeval provides a structure for the systematic study of generalization and robustness phenomena, exposing vulnerabilities and bias across diverse domains and text genres.

The platform's support for both research and operational audiences makes it an enabler for both future algorithmic advances (e.g., in transfer learning, adversarial training, or defense-aware detector design) and principled methodology benchmarking.

Conclusion

MGTeval constitutes a comprehensive platform for systematic, robust, and extensible evaluation of MGT detectors (2604.25152). By integrating dataset generation, adversarial attack pipelines, training/calibration, and standardized multi-metric reporting in a unified and user-friendly system, MGTeval advances the state of benchmarking in this critical subfield. Its open extensibility facilitates future work on multilingual, cross-domain, and attribution tasks, and its adoption should substantially improve the rigor and reproducibility of MGT detection research.