CoCoGEC: Counterfactual Generation for Robust Grammatical Error Correction

Abstract: Grammatical error correction (GEC) systems are usually trained and evaluated on GEC benchmarks, but their performance often drops sharply once the surrounding context is slightly perturbed or extended. This indicates that the existing GEC models usually fail to understand the error patterns in the varying contexts. In this paper, we thoroughly investigate the counterfactuals for GEC tasks, where the subtle changes to the contexts could lead to the label flipping issue. We propose CoCoGEC, a counterfactual generation framework that creates copies of training instances with error-irrelevant contexts altered. Our framework systematically generates counterfactuals by (1) generating intra- and inter-sentence counterfactuals that maintain the error patterns as well as syntax of the original instances by altering the word-level and sentence-level contexts; (2) revising the generated counterfactuals by selecting the instances with flipped labels and high GEC Mutual Information (MI) coefficient. Extensive experiments show that our method substantially improves the stability of GEC models, outperforming a set of data augmentation baselines. Particularly, it could achieve absolute F0.5 gains of +9.9, +11.3, and +20.8 points on the perturbed BEA-19*,CoNLL-14*, and TEM-8* data set.Our code is released at https://github.com/Quinnok/CoCoGEC

• The paper introduces a counterfactual generation framework to decouple error signals from context, thereby enhancing model robustness in grammatical error correction.
• It employs span-controlled perturbations and mutual information-based selection to simulate realistic context shifts while preserving key error patterns.
• Empirical results demonstrate significant F0.5 improvements across benchmarks, proving the method's effectiveness in mitigating context-induced performance drops.

CoCoGEC: Counterfactual Generation for Robust Grammatical Error Correction

Introduction

CoCoGEC introduces a counterfactual generation framework targeting robustness in the Grammatical Error Correction (GEC) task. Despite substantial advances in neural GEC, prevailing systems exhibit significant performance deterioration when subjected to context perturbations, including both local (word-level) and non-local (sentence-level) changes. This paper formalizes and applies counterfactual reasoning to GEC, arguing that current models overfit to superficial context cues and fail to generalize across realistic domain shifts.

A motivating example in the paper illustrates that while leading LLMs (e.g., GPT-4) can accurately correct errors under standard benchmarks, their predictions are brittle under context extension or minor perturbations. The observed drop in robustness is especially prominent on datasets mirroring complex real-world inputs, such as TEM-8, under perturbation regimes.

Figure 1: Contextual shift from benchmarks to real-world data (a), degradation in robustness under context perturbation (b), and examples of intra/inter-sentence counterfactuals illustrating the CoCoGEC approach.

Counterfactual Formulation in GEC

Most prior work on data-centric robustness for GEC has relied on random noise injection, pseudo-corpus generation, or loosely controlled augmentations, typically with little interpretability and often causing performance regression due to semantic drift or noise artifacts. CoCoGEC advances the field by providing a formal, task-specific definition of counterfactuals for GEC:

• Intra-sentence counterfactuals perturb error-irrelevant spans at the lexical or phrasal level, holding annotated error patterns invariant with minimal syntactic drift.
• Inter-sentence counterfactuals append or prepend fluent, error-free prefix/suffix utterances to simulate discourse context shift while maintaining error span and correction alignment.

These counterfactuals meet three critical criteria:

1. Minimal syntax-preserving noise within the error-free segments of the source.
2. Semantic coherence in expanded context.
3. The constraint that the new set of gold edits $\mathcal{E}'$ forms a subset of the original $\mathcal{E}$ (no introduction of novel errors).

By doing so, CoCoGEC allows for systematic evaluation and training of models on hard negative examples that specifically decouple error phenomena from confounding context, thereby fostering context-invariant representation learning.

Counterfactual Generation Pipeline

The CoCoGEC pipeline comprises three sequential phases:

1. Span-controlled counterfactual generation: Non-error-containing source spans are replaced with LLM-generated alternatives using masked [BLANK] templates. The process leverages targeted prompts and sampling constraints, preserving critical error signal and syntactic form.
2. Contextual expansion: Prefixes and suffixes are constructed via LLMs to inject semantically coherent but error-free document-level context, simulating real-world long-form input structures.
3. Edit-alignment and selection: Candidates violating the original edit mapping constraint ($\mathcal{E}' \not\subseteq \mathcal{E}$) are filtered using ERRANT; remaining candidates are ranked by a GEC Mutual Information (MI) criterion measuring the informativeness and 'confusability' of the instance.

By applying a neural scoring framework, high-MI, label-invariant, and semantically plausible counterfactuals are prioritized for augmentation to maximize model robustness gains.

Experimental Results

Empirical analysis uses the RobustGEC benchmark, measuring performance on BEA-19*, CoNLL-14*, and TEM-8* under both source and perturbed conditions (word-level, sentence-level, and combined shifts). Across multiple backbone architectures (GECToR, T5, Qwen3-8B), CoCoGEC yields substantial, consistent improvements over CPR, $\mathcal{DISCO}$, and TypeDA augmentation schemes.

Key findings include:

• Absolute $F_{0.5}$ improvements of +9.9 (BEA-19*), +11.3 (CoNLL-14*), and +20.8 (TEM-8*) on Qwen3-8B, demonstrating strong context-perturbation resilience.
• Significantly reduced $F_{0.5}$ degradation under perturbation, especially for sentence-level and long-context settings (cf. breakdown on TEM-8).
• Robustness extends to compact parameter regimes: Qwen3-8B+CoCoGEC matches or outperforms larger LLMs (GPT-4o, LLaMA3-8B) in perturbed regimes with much lower computational footprint.
• CRS and P-CRS scores demonstrate superior context prediction consistency.
• No decrease in performance is observed on canonical GEC benchmarks (BEA-19, CoNLL-14).

Ablation and Analysis

Component ablation indicates that both intra- and inter-sentence augmentations are necessary, with MI-based selection being crucial to avoid dilution from suboptimal counterfactuals. Robustness holds across increasing numbers and spans of perturbations, with slower performance decay than all baselines.

Case studies underscore that CoCoGEC-trained models can consistently correct error phenomena even as semantic context is varied, whereas baseline models show erratic prediction drift. This effect is pronounced in high-noise, long-context settings, confirming the framework's value in practical application domains, including grammar checking in educational or real-world document editing tools.

Implications and Future Directions

The CoCoGEC framework bridges counterfactual reasoning with structured-generation tasks beyond binary classification, providing a general blueprint for data-centric robustness in sequence transduction. By decoupling error representation from context artifacts, the approach encourages more reliable and interpretable model behavior, aligns closely with human error correction, and reduces domain and input shift fragility.

Potential avenues of extension include:

• Generalization to non-English and multilingual GEC, verified by positive cross-lingual transfer results.
• Porting the LLM-based generator to lighter-weight alternatives, minimizing dependency on large proprietary models.
• Integration of counterfactual selection into adversarial training or curriculum learning frameworks, leveraging MI for active augmentation.

Conclusion

CoCoGEC presents a robust, data-centric method for improving GEC systems under realistic, contextually variable settings. By leveraging targeted counterfactual generation and rigorous filtering via mutual information, it delivers state-of-the-art robustness on established perturbation benchmarks without sacrificing canonical performance. Its paradigm readily extends to broader sequence-to-sequence NLP tasks sensitive to contextual distributional shifts, marking a significant methodological advance in the systematic engineering of robust NLP systems.