Zero-Shot Grammar Competency Estimation

• Zero-shot grammar competency estimation is a method to assess grammatical quality in texts and speech without relying on annotated data, leveraging pretrained models and error correction metrics.
• The GECScore approach uses a calibration-driven similarity metric between original and corrected text, achieving up to 98.7% AUROC in distinguishing LLM-generated text from human-authored content.
• LLM pseudo-labeling with noise-aware training enables scalable scoring of essays and transcripts, outperforming traditional supervised methods through rubric-aligned regression.

Zero-shot grammar competency estimation refers to the automated assessment of grammatical proficiency or detectability of grammatical errors in text or speech samples without the use of manually labeled data or model-specific supervised fine-tuning. Recent approaches leverage pretrained models, pseudo-labeling via LLMs, or systematized correction metrics derived from grammar error correction (GEC) models. This family of methods enables grammar assessment for both written and spoken modalities as well as the detection of LLM-generated text, all in a scalable, black-box, and data-efficient manner (Wu et al., 7 May 2024, Das et al., 17 Nov 2025).

1. Conceptual Overview and Motivation

Zero-shot grammar competency estimation aims to predict grammar quality or error likelihood in corpora lacking expert annotation. The core objective is to learn a function $f:X \rightarrow Y$ that maps texts $x$ (written passages or automatic speech recognition (ASR) transcripts) to either a discrete competency score or a verifiable error signal, typically in a rubric-aligned scale or normalized error metric.

Two scenarios motivate this problem:

• The need to assess LLM-generated versus human text without source-model access or paired data, requiring black-box, calibration-driven protocols.
• The assessment of real-world linguistic proficiency, especially in spoken settings, where extensive annotation is impractical and ASR introduces additional disfluencies and domain-specific challenges.

Zero-shot methods avoid supervised data bottlenecks and enable immediate deployment across new domains, user populations, or data types (Wu et al., 7 May 2024, Das et al., 17 Nov 2025).

2. GECScore: Correction-Based Zero-Shot Estimation

The Grammar Error Correction Score (GECScore) is a calibration-driven approach for zero-shot grammaticality estimation and LLM-generated text detection, predicated on the observation that human writing typically contains more correctable errors than LLM output. The process involves the following components (Wu et al., 7 May 2024):

• Grammar Error Correction Model: Let $g(\cdot)$ be a pretrained sequence-to-sequence GEC model (e.g., a Flan-T5-large Transformer, fine-tuned on the CoEdIT corpus).
• Similarity Metric: Given input $x_i$, apply $g$ to produce a corrected output $\hat{x}_i=g(x_i)$. Compute $S_i = Sim(x_i, \hat{x}_i)$ using a text-similarity function, with BLEURT yielding strongest separation in practice.
• Softmax Normalization and Thresholding: For a batch $\{x_1, \ldots, x_n\}$, compute the normalized score:

$$\operatorname{GECScore}(x_i) = \frac{\exp S_i}{\sum_{j=1}^n \exp S_j}$$

• Decision Protocol: Determine threshold $\epsilon$ on the batch’s calibration set to maximize $TPR + (1-FPR)$ (Youden’s J statistic). Classify $x_i$ as "LLM" if $\operatorname{GECScore}(x_i) \geq \epsilon$, "Human" otherwise.

This protocol is entirely black-box, requiring no supervised labels or access to the generative model, and is robust under domain shift and paraphrasing attacks.

3. Pseudo-Labeling with LLMs and Noise-Aware Training

Another zero-shot grammar assessment approach predicts rubric-based scores for essay or ASR transcript samples, utilizing LLM-generated pseudo-labels rather than ground truth (Das et al., 17 Nov 2025). The procedure is as follows:

• Rubric-Based Prompting: An LLM (e.g., GPT-4) is prompted with a well-defined 1–5 grammar competence rubric. For each unlabeled sample, the LLM assigns an integer score.
• Transformer Regression Model: A pretrained encoder (ELECTRA, BERT, RoBERTa, XLNet) embeds the input; a single linear projection layer produces the predicted score $\hat{y}$ from the [CLS] representation.
• Noise-Aware Sample Weighting: Given potential label noise from pseudo-labels, the training loss is dynamically reweighted. For each epoch $t$:
  • Compute per-sample loss $\ell_i$.
  • Select the top $\alpha$ fraction of samples as "clean" (lowest loss).
  • Assign $w_i=1/|I_\text{clean}|$ for $i$ in the clean set, $w_i=0$ otherwise, with $\alpha\approx 0.3$ empirically optimal.
  • Minimize weighted MSE loss.

The approach effectively denoises pseudo supervision, leading to performance that surpasses traditional supervised and previous noise-robust training methods.

4. Empirical Performance and Robustness

GECScore-Based Estimation

• GECScore achieves an average AUROC of 98.7% in discriminating LLM- from human-generated text across 7 LLMs and 2 diverse datasets (XSum, Writing Prompts), outperforming competitors such as LRR ($\sim$93%), DetectGPT (56%), Fast-DetectGPT (74%), and BARTScore-CNN (81.6%). Even supervised RoBERTa-large is outperformed (87.8% AUROC) (Wu et al., 7 May 2024).
• Robustness is maintained under paraphrase attacks (0.3% AUROC drop), adversarial text perturbations ($<0.6\%$ AUROC drop), and cross-domain transfer (never below 95% AUROC).
• The method’s reliability derives from stable separation in post-correction similarity distributions between human and LLM texts and is robust to calibration set composition.

LLM Pseudo-Labeling and ELECTRA Regression

• Using GPT-4 pseudo-labels, the transformer regressor achieves QWK=0.664 (SGAD, spoken), 0.763 (WGAD, written) and PLCC=0.732/0.833 respectively, with RMSE=0.730/0.599.
• The method outperforms supervised pseudo-label training (QWK = 0.466/0.366) and unsupervised LLM-only scoring, as well as state-of-the-art robust loss baselines by 5–15 QWK points (Das et al., 17 Nov 2025).
• Performance is sensitive to LLM pseudo-label quality; higher-capacity LLMs yield more reliable scores and downstream models (QWK rise of 5–10 points vs. lower-tier LLMs).
• Sensitivity to the clean-sample ratio ($\alpha$): Best performance at $\alpha \approx 0.3$.

5. Applications, Generalization, and Limitations

Zero-shot grammar competency estimation is applicable in several domains:

• LLM-Generated Text Detection: GECScore provides a content-agnostic, black-box classifier for distinguishing generated vs. human-authored text robustly without labeled data (Wu et al., 7 May 2024).
• Automated Writing Assessment: Both GECScore and LLM-pseudo-labeled models support essay or transcript scoring for education with no expert annotations or domain retraining (Wu et al., 7 May 2024, Das et al., 17 Nov 2025).
• Scalability and Domain Transfer: As long as a suitable GEC model or rubric-plus-LLM pairing exists, these approaches generalize to new tasks, languages, or spoken domains.

Principal limitations include dependence on the underlying GEC/LLM model's coverage, reduced performance on excessively short or fragmented input, domain mismatch effects, and the inheritance of biases from pseudo-label sources and rubric design. For spoken grammar, ASR errors may confound the scoring system. Incorporation of non-textual (acoustic, prosodic) cues and refinement of prompt engineering are promising avenues for further improvement (Wu et al., 7 May 2024, Das et al., 17 Nov 2025).

6. Theoretical and Psychological Basis

The GECScore exploits differential error variance in human and LLM output. According to Working Memory Theory (Baddeley, 1992), humans, even when skilled, introduce slips due to the prioritization of semantics over syntax—a phenomenon not observed in LLMs trained via maximum-likelihood on large, clean corpora, which reliably self-correct at the surface form level. Empirically, human-generated samples display greater variance in similarity scores post-correction, creating a stable threshold for discrimination (Wu et al., 7 May 2024).

Pseudo-label-based models contextualize grammaticality by embedding domain and content sensitivity via rubric-driven LLM prompts, ensuring interpretability and alignment with expert assessment in both written and spoken contexts. Controlled perturbation studies confirm that model sensitivity tracks human rubric ratings, with greater declines in predicted scores for more severe error types (Das et al., 17 Nov 2025).

7. Summary Table: Core Approaches in Zero-Shot Grammar Competency Estimation

Approach	Data Required	Output	Key Metric(s)
GECScore	Unlabeled texts	LLM vs. Human	AUROC, BLEURT
LLM Pseudo-Label	Unlabeled texts	1–5 Score	QWK, PLCC, RMSE

GECScore leverages black-box GEC-driven similarity calibration for discriminative grammaticality and source attribution, while LLM pseudo-labeling enables scalable, rubric-aligned regression for nuanced, domain-aware grammar scoring (Wu et al., 7 May 2024, Das et al., 17 Nov 2025).