Scribendi: A Reference-Free GEC Metric

• Scribendi is a reference-free GEC metric that uses language model perplexity and surface similarity measures to evaluate grammatical corrections without gold references.
• It integrates fluency scoring with token sort and Levenshtein distances to reward improvements while discouraging excessive rewriting.
• Empirical studies show Scribendi performs competitively in standard scenarios but is vulnerable to adversarial attacks that exploit its design.

Searching arXiv for recent and foundational papers on Scribendi and reference-free GEC evaluation. Scribendi is a reference-free evaluation metric for grammatical error correction (GEC) that scores a system output without requiring human references. Its stated purpose is to rank and compare correction hypotheses while avoiding the known limitations of reference-based evaluation when systems produce valid but non-identical edits. In the GPT-3.5 multilingual GEC study, Scribendi is used alongside language-model fluency scores and embedding-based semantic similarity to assess whether outputs are fluent while remaining close to the source (Katinskaia et al., 2024). In a later reliability study, Scribendi is also analyzed as a discrete sentence-level metric and shown to be vulnerable to adversarial exploitation despite prior strong meta-evaluation performance in ideal conditions (Goto et al., 30 Sep 2025).

1. Origin, objective, and scope

Scribendi was proposed by Islam and Magnani (2021) as a straightforward, reference-less GEC metric that correlates strongly with human judgments while avoiding dependence on gold references. Its central objective is to evaluate whether a candidate correction improves grammaticality and fluency without drifting too far from the source sentence. This design addresses a recurrent problem in GEC evaluation: reference-based metrics can under-reward outputs that are acceptable corrections but do not match the particular wording of a gold reference (Katinskaia et al., 2024).

The metric is sentence-level rather than error-type-specific. It does not categorize errors such as articles, tense, or punctuation. In the GPT-3.5 study, analyses by error type are instead supplied by ERRANT for English and Russian, with Scribendi serving as a complementary sentence-level reference-free signal (Katinskaia et al., 2024).

A later formalization characterizes Scribendi as returning one of $\{-1, 0, 1\}$ for an erroneous source sentence $S$ and a hypothesis $H$ (Goto et al., 30 Sep 2025):

$$\text{Scribendi}(S, H) = \left\{ \begin{array}{ll} 1 & \text{ppl}(S) > \text{ppl}(H)\; \text{and}\; \text{Surface}(S, H) \ 0 & S = H \ -1 & \text{otherwise} \end{array} \right.$$

with

$$\text{Surface}(S, H) = \left\{ \begin{array}{ll} \text{True} & \max(\text{LDR}(S, H), \text{TSR}(S, H)) > 0.8 \ \text{False} & \text{otherwise} \end{array} \right.$$

Here, $\text{ppl}(\cdot)$ denotes language-model perplexity, LDR is the Levenshtein Distance Ratio, and TSR is the Token Sort Ratio. This formalization makes explicit the two design assumptions that also underlie the broader description in the GPT-3.5 study: lower perplexity is taken as a proxy for improved fluency, and surface proximity is taken as a proxy against excessive rewriting (Goto et al., 30 Sep 2025).

2. Scoring logic and operational criteria

In the GPT-3.5 study, Scribendi is described as combining three components: language-model fluency via perplexity or probability, token sort ratio to encourage lexical proximity to the source, and Levenshtein distance to penalize excessive rewriting and repetitions (Katinskaia et al., 2024). This description emphasizes the metric’s intended balance: reward fluency gains, but discourage broad paraphrase and degenerate repetition.

The later reliability study provides an explicit decision procedure. A hypothesis receives score $1$ only if it has lower perplexity than the source and passes a thresholded surface-similarity filter; it receives $0$ when no change is made; otherwise it receives $-1$ (Goto et al., 30 Sep 2025). The surface filter is permissive by design, accepting a hypothesis whenever $\max(\text{LDR}, \text{TSR}) > 0.8$. LDR measures character-level edit distance normalized by sentence length, whereas TSR compares token sets while ignoring order.

Perplexity is the principal fluency proxy. In the materials accompanying the GPT-3.5 study, language-model perplexity is defined as

$S$0

Lower perplexity is interpreted as greater fluency. This suggests that Scribendi inherits both the advantages and the liabilities of language-model-based grammaticality estimation: it can recognize many fluent improvements, but its judgments depend on the quality and calibration of the underlying monolingual LLM (Katinskaia et al., 2024).

The metric can be used in at least two regimes. In comparative use, as in the GPT-3.5 paper, the key statistic is whether the model hypothesis outranks the gold reference, i.e., whether $S$1. No absolute acceptance threshold is defined in that setting (Katinskaia et al., 2024). In absolute evaluation, as in the reliability study, sentence-level scores in $S$2 are summed over a dataset; in relative evaluation, pairwise comparisons can be aggregated with TrueSkill (Goto et al., 30 Sep 2025).

3. Role in multilingual GPT-3.5 grammatical error correction

The multilingual GPT-3.5 investigation applies Scribendi specifically in the “Analysis of Zero-shot Output” section, not in the fine-tuned or re-ranking settings (Katinskaia et al., 2024). For zero-shot GEC, the system uses English prompts across all languages to standardize inputs, with instructions for minimal edits and direct output formatting. Two prompt types were tested, and Chain-of-Thought Prompt 2 performed best and was used for most zero-shot results. The prompts also explicitly instruct the model not to translate non-English inputs into English. The model configuration is GPT-3.5-turbo-0613 via the OpenAI API (Katinskaia et al., 2024).

The evaluation protocol for Scribendi in that study is comparative. For each source sentence (Src), gold reference (Trg), and GPT-3.5 hypothesis (Hyp), the authors compute language-model perplexity or probability with monolingual LLMs tailored to each language, combine that signal with token sort ratio and Levenshtein distance into the Scribendi score, and then report the percentage of cases where $S$3 (Katinskaia et al., 2024). Because GPT-3.5 outputs vary across requests, zero-shot reference-based scores are averaged over three requests per input, whereas Scribendi is computed once per hypothesis.

The study uses Scribendi across multiple test sets and languages: English on BEA-19 dev/test, CoNLL-14, and JFLEG for broader evaluation, with Scribendi specifically reported for BEA-19 dev and CoNLL-14; Czech on AKCES-GEC; German on Falko-MERLIN; Spanish on COWS-L2H; Ukrainian on UA-GEC; Russian on RULEC-GEC; and Swedish on SweLL-gold, although Swedish is ancillary in the supplied materials (Katinskaia et al., 2024).

Within this experimental design, Scribendi functions as a reference-less complement to two other signals. Language-model fluency estimates whether the output is more grammatical or probable than the source or reference. Embedding-based semantic similarity evaluates whether meaning is preserved. Scribendi occupies an intermediate role: it rewards fluency improvements but imposes a penalty for outputs that rephrase too aggressively or introduce repetitions (Katinskaia et al., 2024).

4. Empirical behavior across languages

The GPT-3.5 study reports the percentage of zero-shot hypotheses whose Scribendi score exceeds the gold reference’s score, $S$4 (Katinskaia et al., 2024):

Language / dataset	$S$5
English / BEA-19 dev	52.5%
English / CoNLL-14	50.6%
Czech / AKCES-GEC	12.2%
German / Falko-MERLIN	18.4%
Spanish / COWS-L2H	42.1%
Ukrainian / UA-GEC	26.2%
Russian / RULEC-GEC	50.1%

These values align closely with the study’s language-model fluency comparisons. The authors explicitly state that “there is a correlation between the two reference-less metrics, although the Scribendi score penalizes predictions that significantly rephrase the source sentences” (Katinskaia et al., 2024). English and Russian are near 50%, indicating that GPT-3.5 often produces hypotheses that are both fluent and sufficiently close to the source to be competitive with human references under Scribendi. Czech, German, and Ukrainian have much lower percentages, and Spanish is intermediate.

The broader multilingual interpretation is that GPT-3.5 often over-corrects. For Czech, German, Russian, Spanish, and Ukrainian, the model substantially alters source sentences, including their semantics, which creates difficulty for reference-based evaluation and is also penalized by Scribendi when the edits become too liberal (Katinskaia et al., 2024). For English, the model demonstrates high recall, generates fluent corrections, and generally preserves sentence semantics. For Russian, recall and error detection are strong, but semantic drift is more frequent.

The study also supplies human-evaluated examples that clarify how Scribendi can behave. In English, a fluent but semantically or lexically misaligned output such as “dangers and brutalism” instead of “dangers and brutality” would plausibly be penalized through language-model probability and lexical divergence. By contrast, a local number mismatch such as leaving “basketball match” singular when the intended form is plural may not be strongly penalized if overall fluency remains high. In Russian, incorrect handling of case government or lexical compatibility can reduce both language-model preference and surface-based adequacy signals (Katinskaia et al., 2024). This suggests that Scribendi is more sensitive to some forms of semantic or collocational distortion than to all isolated grammatical mismatches.

The same study shows that Scribendi is not a substitute for error-type analysis. Human evaluation for English and Russian indicates that GPT-3.5 struggles with punctuation mistakes, tense errors, syntactic dependencies between words, and lexical compatibility at the sentence level, while ERRANT-based analyses further identify specific weak categories such as NOUN, PREP, VERB, VERB:TENSE, and WO in English, and ADJ/CASE, NOUN/CASE, PREP, PUNCT, VERB, VERB:TENSE, and WO in Russian (Katinskaia et al., 2024). Scribendi itself does not isolate these categories.

5. Failure modes, adversarial vulnerability, and the reliability crisis

The reliability study argues that reference-free GEC metrics can be undermined by systems optimized to obtain unjustifiably high scores, and Scribendi is one of the clearest examples (Goto et al., 30 Sep 2025). The metric’s failure mode follows directly from its structure: any minimal modification that lowers perplexity while passing the permissive surface-similarity filter can receive a positive score even if it does not correct the original error or changes the sentence meaning.

The attack objective is therefore simple: generate a hypothesis $S$6 such that $S$7 and $S$8 (Goto et al., 30 Sep 2025). The attack described in the paper uses single-token replacement, which keeps surface deviation small enough to satisfy the filter almost universally. Replacements are proposed by a masked LLM, and the first candidate that lowers perplexity while preserving sufficient surface overlap is selected.

The concrete example reported in the study is instructive. For the input sentence “You will be interesting in this job ?”, the adversarial Scribendi output is “What will be interesting in this job ?” (Goto et al., 30 Sep 2025). The paper reports $S$9 and $H$0, so perplexity decreases. It also reports $H$1 and $H$2, so the surface filter passes. Scribendi therefore assigns score $H$3. Yet the grammatical error of interest is not corrected; instead, a semantically different but language-model-favored sentence is produced.

The empirical consequences are substantial. On the BEA-2019 development set of 4,384 sentences, Adversarial-Scribendi obtains an absolute score of 4179, described as extremely close to the maximum, while the best non-adversarial systems are far lower: Chat-LLaMa-2-7B-FT at 2200, T5-11B at 2161, and UL2-20B at 2104 (Goto et al., 30 Sep 2025). In relative evaluation, Adversarial-Scribendi reaches .218, whereas standard systems peak around .024. Additional experiments on SEEDA show the same dominance pattern.

These results identify three structural weaknesses. First, Scribendi over-relies on perplexity, which rewards language-model-likely substitutions even when they are semantically inappropriate. Second, the $H$4 threshold is a coarse surface guardrail that does not enforce meaning preservation; TSR is especially weak in this regard because it ignores token order. Third, the metric does not verify whether the specific grammatical error was fixed (Goto et al., 30 Sep 2025). A plausible implication is that Scribendi is effective under ordinary model behavior but not robust against explicit metric gaming.

6. Relation to other evaluation methods and practical implications

In the GPT-3.5 study, Scribendi is not treated as a stand-alone arbiter. The recommended practice is to combine multiple reference-less signals: language-model fluency, Scribendi, and embedding similarity (Katinskaia et al., 2024). Each is intended to capture a different failure mode. Language-model fluency estimates grammaticality; Scribendi discourages unnecessary rewriting and repetition; sentence embeddings track semantic drift. For semantic similarity, the supplied materials define cosine similarity as

$H$5

This multi-signal strategy is motivated by the observation that reference-based metrics alone can under-estimate LLMs that produce fluent alternative corrections, whereas Scribendi alone can still under-estimate legitimate rephrasings and over-reward some fluent but incorrect edits (Katinskaia et al., 2024).

The broader evaluation context reinforces this point. Reference-based measures such as Precision, Recall, and $H$6 remain standard, with the supplied definitions

$H$7

In the GPT-3.5 experiments, zero-shot GPT-3.5 often has high recall but lower precision, and $H$8 frequently trails strong supervised baselines except in Spanish (Katinskaia et al., 2024). Scribendi provides partial relief from exact-match dependence, but because it penalizes heavy paraphrasing and remains sensitive to language-model quality, it cannot by itself resolve the evaluation problem.

The reliability study proposes a mitigation based on metric ensembles. It tests ensemble-by-negative-ranking-averaging across metrics and shows that systems tailored to exploit one metric, including Scribendi, are ranked lower overall under the combined scheme (Goto et al., 30 Sep 2025). This suggests that heterogeneity in evaluation criteria increases attacker cost and reduces the dominance of any single exploitable signal.

For production-oriented GEC workflows, the GPT-3.5 study particularly recommends using GPT-3.5 as a re-ranker over hypotheses from smaller fine-tuned GEC models rather than relying exclusively on zero-shot generation (Katinskaia et al., 2024). In English and Russian, this increases recall while keeping semantics closer to the source and yields better overall $H$9 than zero-shot. That recommendation is consistent with the core rationale behind Scribendi: corrections should improve fluency without unnecessary divergence from the input.

Taken together, these studies position Scribendi as a practical, language-agnostic, computationally light reference-free metric whose utility is clearest when used jointly with other evaluators and interpreted cautiously. It is informative for detecting over-correction and excessive rewriting in multilingual GEC, but its dependence on perplexity and coarse surface similarity makes it inadequate as a sole evaluation criterion, especially under adversarial or highly paraphrastic conditions (Katinskaia et al., 2024, Goto et al., 30 Sep 2025).