Multi-Population-aware MMD (MMD-MP)

• The paper introduces MMD-MP, which omits the intra-machine similarity term to reduce variance in statistical tests for detecting heterogeneous machine-generated texts.
• It leverages deep kernel training with robust variance estimation, achieving significant improvements in test power and AUROC over standard MMD methods.
• MMD-MP is applicable at both paragraph and sentence levels, enabling reliable and transferable detection across various large language models and decoding strategies.

Multi-Population-aware Maximum Mean Discrepancy (MMD-MP) is an optimization method for distributional two-sample tests that addresses the challenge of detecting machine-generated texts originating from diverse LLMs. By modeling population structure among generated texts, MMD-MP produces a highly stable and powerful statistical test, significantly improving over standard MMD-based approaches when the “machine” class is itself comprised of heterogeneous subpopulations. The method is particularly effective for distinguishing between human- and machine-generated texts when the latter are drawn from multiple LLMs or decoding strategies (Zhang et al., 2024).

1. Fundamentals of Maximum Mean Discrepancy

Maximum Mean Discrepancy (MMD) quantifies the difference between two distributions $P$ and $Q$ over a domain $\mathcal{X}$ by embedding them in a reproducing-kernel Hilbert space (RKHS) with kernel $k$. The squared MMD is

$$\mathrm{MMD}^2(P, Q; k) = \left\| \mu_P - \mu_Q \right\|_{\mathcal{H}_k}^2 = \mathbb{E}_{x, x' \sim P} [k(x, x')] + \mathbb{E}_{y, y' \sim Q} [k(y, y')] - 2\mathbb{E}_{x \sim P, y \sim Q} [k(x, y)].$$

With samples $\{x_i\}_{i=1}^n \sim P^n$, $\{y_j\}_{j=1}^n \sim Q^n$, the unbiased U-statistic estimate is

$$\widehat{\mathrm{MMD}}^2_u = \frac{1}{n(n-1)} \sum_{i \ne j} H_{ij}, \ H_{ij} = k(x_i, x_j) + k(y_i, y_j) - k(x_i, y_j) - k(x_j, y_i).$$

MMD has desirable theoretical properties for non-parametric hypothesis testing and is widely used for two-sample problems in text and vision domains.

2. Variance Inflation due to Multiple Populations

When employing a deep kernel $k_\omega$ (parameterized, e.g., via a neural network atop a pretrained encoder such as RoBERTa) for MMD-based detection, one optimizes a test-power proxy,

$$J(P, Q; k_\omega) \approx \frac{\mathrm{MMD}^2(P, Q; k_\omega)}{\sigma_{\mathfrak{H}_1}},$$

where $\sigma_{\mathfrak{H}_1}^2$ is the asymptotic variance under the alternative. In the context of machine-generated texts, the “machine” sample $S_Q$ may comprise outputs from a variety of LLMs and sampling settings, rendering $Q$ a mixture of subpopulations. This population heterogeneity causes the intra-class term $\mathbb{E}_{y, y' \sim Q}[k(y, y')]$ to be unstable and difficult to optimize. Empirically, this leads to increased sample variance in the MMD statistic during kernel learning; as shown in synthetic and real data (e.g., Figure 1 in the paper), this instability can impair the reliability of hypothesis tests. Detailed decomposition attributes the variance escalation mainly to $\operatorname{Var}[k(y, y')]$ from subpopulation mixing (Zhang et al., 2024).

3. The Multi-Population-Aware Objective (MMD-MP)

Removal of the Intra-Machine Term

MMD-MP introduces the Multi-Population Proxy (MPP), which omits the problematic $\mathbb{E}_{y, y'}[k(y, y')]$ term:

$$\mathrm{MPP}(P, Q; k) = \mathbb{E}_{x, x' \sim P}[k(x, x')] - 2\mathbb{E}_{x \sim P, y \sim Q}[k(x, y)].$$

The unbiased U-statistic estimator for equal sample sizes $n$ is

$$\widehat{\mathrm{MPP}}_u = \frac{1}{n(n-1)} \sum_{i \ne j} H^*_{ij}, \qquad H^*_{ij} = k(x_i, x_j) - k(x_i, y_j) - k(x_j, y_i).$$

By bypassing the generator–generator similarity term, MMD-MP directly targets human–machine discrepancies.

Variance Estimation and Optimization

Under the alternative, the asymptotic distribution is

$$\sqrt{n} \left( \widehat{\mathrm{MPP}}_u - \mathrm{MPP} \right) \xrightarrow{d} \mathcal{N}(0, \sigma_{\mathfrak{H}_1^*}^2),$$

where

$$\sigma_{\mathfrak{H}_1^*}^2 = 4 \left( \mathbb{E}[H^*_{12} H^*_{13}] - \mathbb{E}[H^*_{12}]^2 \right).$$

The objective optimized is

$$J_{\rm MP}(P, Q; k_\omega) = \frac{\widehat{\mathrm{MPP}}_u(S_P, S_Q; k_\omega)}{\sqrt{\widehat{\sigma}_{\mathfrak{H}_1^*}^2(S_P, S_Q; k_\omega) + \lambda}},$$

where $\lambda$ is a small ridge parameter for numerical stability. This construction yields lower variance and increased stability during kernel training, particularly in multi-generator contexts.

4. Algorithmic Structure

Training the Deep Kernel

• Initialize with human samples $S_P^{tr}$, machine samples $S_Q^{tr}$, fixed encoder $\hat{f}$, kernel parameters $\omega$, hyperparameters $\lambda$, $\eta$, and $T_{\max}$.
• For each iteration: build $k_\omega$, compute $\widehat{\mathrm{MPP}}_u$ and variance $\widehat{\sigma}_{\mathfrak{H}_1^*}^2$, update $\omega$ to maximize $J_{\rm MP}$ via Adam.
• Output is an optimized deep kernel $k_\omega$.

Paragraph-Level Detection

• Given test sets $S_P^{te}$ and $S_Q^{te}$, compute $\widehat{\mathrm{MMD}}_u^2$.
• Generate the null via permutation, calculate the $p$-value as the fraction of permuted MMDs exceeding the observed value.
• Suitable for batch paragraph-based detection scenarios.

Sentence-Level Detection

• Fix a reference set of human sentences $S^{re}$.
• For each candidate $\tilde{x}$, compute the “biased” MMD estimate $\widehat{\mathrm{MMD}}_b^2(S^{re}, \{\tilde{x}\})$.
• Use the resulting scores to evaluate AUROC for distinguishing single machine-generated versus human sentences.

5. Theoretical Guarantees

• The estimator $\widehat{\mathrm{MPP}}_u$ is asymptotically normal:

$$\sqrt{n} (\widehat{\mathrm{MPP}}_u - \mathrm{MPP}) \to \mathcal{N}(0, \sigma_{\mathfrak{H}_1^*}^2).$$

• For large $n$, test power satisfies

$$\mathrm{Power} \approx \Phi\left( \frac{\sqrt{n} ( \mathrm{MPP} + R(Q))}{\sigma_{\mathfrak{H}_1^*}} - \frac{r}{\sqrt{n} \sigma_{\mathfrak{H}_1^*}} \right),$$

showing that maximizing $\mathrm{MPP}/\sigma_{\mathfrak{H}_1^*}$ aligns with statistical power maximization.

• Uniform convergence (Theorem 1) produces

$$\sup_{\omega: \sigma_{\omega}^2 \geq s^2} |\hat{J} - J| = O(n^{-1/3}),$$

under standard kernel regularity assumptions, confirming consistency for the learning objective.

6. Empirical Evaluation

Data and Benchmarks

• Paragraph detection: HC3 (Q&A, ChatGPT vs. human), XSum (news).
• Sentence detection: same sources.
• Machine-generated spans include GPT-2 small/medium, GPT-3 small (∼550M), GPT-Neo small/large, GPT-j-6B, ChatGPT (GPT-3.5), GPT4All-j.

Baselines

Method	Detection Type	Kernel Type
MMD-O	Paragraph/Sentence	RBF kernel
MMD-D	Paragraph/Sentence	Deep kernel
C2ST-S/L	Paragraph/Sentence	Classifier two-sample
DetectGPT/OpenAI-D/CE-Clf	Single-instance	Direct/Classifier features

Metrics

• Paragraph-level: test power at α = 0.05.
• Sentence-level: AUROC.

Performance Summary

• On synthetic 4-Gaussian mixtures, MMD-MP outperforms MMD-D by up to +9 points in test power as sample variance grows.
• On HC3 (3100 paras), MMD-MP achieves 93.2% power vs. 91.8% for MMD-D; similar gains observed on GPT3-S, Neo-S, and mixed settings.
• For HC3 (1000 paras), average +2–6 points improvement over MMD-D across both single- and multi-generator configurations.
• In unbalanced settings (2000 human vs. 400 machine): +7–14 point test power and +4–9 point AUROC improvement over MMD-D.
• For sentence-level detection, MMD-MP surpasses DetectGPT, ChatGPT-D, and the CE-classifier by 1–2 points AUROC on ChatGPT, and by 5–15 points AUSROC on more challenging models.
• Transfer experiments (trained on ChatGPT+GPT-2, tested on GPT-Neo-L, GPT-j, or GPT4All-j) show gains of +23–28 points test power and +3–5 points AUROC over MMD-D.
• t-SNE visualizations demonstrate that MMD-MP produces more clustered human texts and decorrelated machine-generated clusters, validating reduced multi-population variance.

7. Practical Implementation and Significance

MMD-MP is deployable across both batch (paragraph-level) and real-time (sentence-level) machine-generated text detection scenarios. The method first compiles datasets representing both human and potentially multi-population machine-generated text. The deep kernel is trained via the MMD-MP criterion, omitting the intra-machine similarity term to control variance. Detection applies the 2-sample permutation framework for group content, or the single-instance “biased” MMD estimate for individual sentences. MMD-MP demonstrates consistent advantages in statistical power, stability, and transferability across unseen LLMs, establishing its efficacy as a robust kernel optimization method specifically designed for the multi-distributional landscape of modern text generation (Zhang et al., 2024).