CoCoGEC: Counterfactual GEC Framework

• CoCoGEC is a counterfactual generation framework that isolates grammatical error patterns by perturbing non-error spans without altering the core correction.
• It uses LLM-driven intra-sentence rewrites and adds coherent prefix/suffix to simulate discourse-level context shifts while preserving gold edit regions.
• Experimental results show marked robustness improvements on diverse benchmarks, especially in long-context settings, using a curated subset of generated examples.

CoCoGEC is a counterfactual generation framework for robust grammatical error correction (GEC) that constructs context-decoupled training examples in which error-irrelevant context is altered while the original grammatical error pattern is intended to remain valid (Wang et al., 13 Jun 2026). It is motivated by the observation that GEC systems often perform well on benchmark sentence pairs yet degrade sharply when nearby lexical material is changed or when the same erroneous sentence is embedded in longer discourse. In this formulation, robustness is treated as invariance of the correction pattern under controlled context perturbation rather than as generic tolerance to noise.

1. Problem formulation and robustness failure mode

The framework addresses robustness in GEC under subtle context perturbation. Standard GEC training and evaluation typically use sentence pairs $(s,t)$, where $s$ is an erroneous source sentence and $t$ is its correction. The paper argues that these benchmarks mostly test whether a model can map a familiar local sentence context to a correction, but do not adequately probe whether the model has isolated the true grammatical error signal from nuisance lexical or discourse context (Wang et al., 13 Jun 2026).

The key failure mode is label flipping at the edit level. In this setting, a small change to context can cause a model to stop predicting an edit that should still be applied. The paper formalizes this as the resulting edit set becoming a subset of the original edit set. This differs from the more common classification-style notion of counterfactuals, because the goal is not to change the gold correction itself, but to expose how a model’s edit decisions become unstable when error-irrelevant context changes.

Three limitations motivate the method. First, conventional training pairs provide too little controlled context variation around the same edit pattern. Second, standard evaluation benchmarks largely remain in-distribution and therefore obscure contextual brittleness. Third, existing augmentation methods may diversify data, but do not explicitly preserve the original error pattern while targeting context robustness. CoCoGEC is designed to make models focus on error patterns that should remain invariant across such changes.

2. Counterfactual definition for GEC

The paper begins from a generic counterfactual objective,

$$\arg\operatorname*{min}_{c} \quad \text{dist}(x, c) \qquad \text{s.t.} \qquad f(c) \neq f(x),$$

and adapts it to edit-level structured prediction in GEC (Wang et al., 13 Jun 2026).

Let $(s,t)$ be a source-target pair and let the original edit mapping $s \rightarrow t$ be $\mathcal{E}$. A GEC counterfactual is written as

$c = p \oplus s' \oplus q,$

where $s'$ is an intra-sentence variant of $s$, $s$0 is an added prefix, $s$1 is an added suffix, and $s$2 denotes concatenation. The paper defines the main objective as

$s$3

where $s$4 is the edit set for the counterfactual pair.

This formalization encodes three design principles. Minimal syntactic revision requires that only error-irrelevant spans inside $s$5 be changed, preserving syntax and the original error pattern. Semantic coherence requires that the added prefix and suffix fit naturally with the sentence. Flipped edit labels require that good counterfactuals be those for which a GEC model struggles to recover original edits, but no new edits should be introduced; hence $s$6. A plausible implication is that robustness is being defined at the level of edit invariance under context shift, rather than at the level of sentence similarity alone.

3. Generation pipeline

CoCoGEC consists of two major phases: counterfactual generation and counterfactual revision or selection. The first phase constructs intra-sentence and inter-sentence perturbations that alter context while attempting to preserve the original correction structure (Wang et al., 13 Jun 2026).

For intra-sentence counterfactuals, each source sentence $s$7 is segmented into candidate spans $s$8 using a Flair chunker. Any span overlapping the gold edit region $s$9 is discarded. Each remaining non-error span is replaced with a special token $t$0, and an LLM is prompted to fill the blank with an alternative word or phrase, allowing insertion, deletion, or replacement. This produces multiple controlled variants $t$1.

A crucial alignment rule preserves the gold correction structure. If a source span $t$2 is replaced by a generated fragment $t$3, the aligned span in the target $t$4 is replaced by the same $t$5. This yields a perturbed pair $t$6 whose correction should still express the original grammatical edits. The procedure changes word-level context while avoiding direct corruption of the error span itself.

For inter-sentence counterfactuals, an LLM generates a fluent, coherent prefix $t$7 and suffix $t$8 for each pair $t$9. These added contexts are constrained to be error-free. The same $$\arg\operatorname*{min}_{c} \quad \text{dist}(x, c) \qquad \text{s.t.} \qquad f(c) \neq f(x),$$0 and $$\arg\operatorname*{min}_{c} \quad \text{dist}(x, c) \qquad \text{s.t.} \qquad f(c) \neq f(x),$$1 are attached to the target, so the source becomes $$\arg\operatorname*{min}_{c} \quad \text{dist}(x, c) \qquad \text{s.t.} \qquad f(c) \neq f(x),$$2, while the target is conceptually expanded with the same surrounding context. This simulates paragraph-level context shifts without introducing new grammatical errors outside the original sentence.

In prose, the full pipeline is: start from gold GEC pairs $$\arg\operatorname*{min}_{c} \quad \text{dist}(x, c) \qquad \text{s.t.} \qquad f(c) \neq f(x),$$3; detect gold edit regions $$\arg\operatorname*{min}_{c} \quad \text{dist}(x, c) \qquad \text{s.t.} \qquad f(c) \neq f(x),$$4; chunk $$\arg\operatorname*{min}_{c} \quad \text{dist}(x, c) \qquad \text{s.t.} \qquad f(c) \neq f(x),$$5 and choose only spans outside $$\arg\operatorname*{min}_{c} \quad \text{dist}(x, c) \qquad \text{s.t.} \qquad f(c) \neq f(x),$$6; rewrite these non-error spans to obtain $$\arg\operatorname*{min}_{c} \quad \text{dist}(x, c) \qquad \text{s.t.} \qquad f(c) \neq f(x),$$7; update target spans correspondingly to form $$\arg\operatorname*{min}_{c} \quad \text{dist}(x, c) \qquad \text{s.t.} \qquad f(c) \neq f(x),$$8; add coherent error-free prefix and suffix to obtain $$\arg\operatorname*{min}_{c} \quad \text{dist}(x, c) \qquad \text{s.t.} \qquad f(c) \neq f(x),$$9; compute edit sets for original and generated pairs; retain only valid candidates; rank them; and augment the training set with selected counterfactuals.

4. Revision, validity control, and mutual-information selection

Not every generated candidate is valid. Some rewrites may alter the correction pattern or introduce new edits. CoCoGEC therefore applies an edit-subset filter using ERRANT. For the original pair $(s,t)$0, it computes edit set $(s,t)$1; for the candidate pair $(s,t)$2, it computes edit set $(s,t)$3; and it keeps only candidates satisfying

$(s,t)$4

This is the core validity criterion of the method (Wang et al., 13 Jun 2026).

After filtering, the remaining candidates are ranked by a GEC Mutual Information (MI) coefficient. The aim is to prefer hard, near-miss counterfactuals that remain strongly associated with the original target: $(s,t)$5 and for GEC,

$(s,t)$6

The conditional term is approximated differently for Seq2Seq and Seq2Edit GEC models. For a Seq2Seq model,

$(s,t)$7

and for a Seq2Edit model,

$(s,t)$8

The target prior is estimated with a LLM,

$(s,t)$9

Here GPT-2-medium is used for $s \rightarrow t$0, and candidates are ranked by MI before selecting the top $s \rightarrow t$1. In the main ablation, $s \rightarrow t$2 performs better than keeping all examples. This suggests that selection quality, rather than raw augmentation volume, is central to the method’s effectiveness.

5. Experimental setting and empirical results

The core experiments are conducted on RobustGEC, built from BEA-19, CoNLL-14, and TEM-8, with train, dev, and test split by case in a 7:1:2 ratio so that all variants of a case remain in the same split (Wang et al., 13 Jun 2026). For long-context evaluation, GPT-4 is used to prepend and append shared context, creating BEA-19*, CoNLL-14*, and TEM-8*. Appendix statistics report 22,848 training sentences, 2,538 development sentences, 3,018 test sentences for BEA-19, 1,578 for CoNLL-14, and 1,590 for TEM-8.

The main backbones are GECToR-large as Seq2Edit, T5-large as Seq2Seq, and Qwen3-8B as an LLM-based GEC model. Additional zero-shot LLM baselines include Qwen3-4B, Qwen3-14B, Qwen3-235B, GPT-4o, and LLaMA3-8B. CoCoGEC is compared against CPR, DISCO, and TypeDA. Candidate spans are chunked with Flair, gold edit constraints are extracted with ERRANT, and LLM-based fine-tuning uses LLaMA-Factory and LoRA. For Qwen3-8B, LoRA uses rank $s \rightarrow t$3, learning rate $s \rightarrow t$4, 2 epochs, per-device batch size 2, gradient accumulation 8, warmup ratio 0.1, maximum sequence length 1024, cosine scheduler, and bf16 precision. Inter-sentence expansion generates roughly 3–5 prefix/suffix sentences.

Evaluation uses ERRANT-based precision, recall, and $s \rightarrow t$5, together with robustness measures: $s \rightarrow t$6 For attacked sets, the paper also reports SR and TR.

The main result is that CoCoGEC improves robustness across all three backbone families and all perturbed datasets. For Qwen3-8B, the paper reports absolute $s \rightarrow t$7 gains of +9.9 on BEA-19*, +11.3 on CoNLL-14*, and +20.8 on TEM-8*, corresponding to 28.29 $s \rightarrow t$8 38.22, 22.50 $s \rightarrow t$9 33.83, and 33.32 $\mathcal{E}$0 54.09, respectively. For GECToR-large, the gains are 26.36 $\mathcal{E}$1 30.07 on BEA-19*, 22.08 $\mathcal{E}$2 25.29 on CoNLL-14*, and 38.35 $\mathcal{E}$3 46.77 on TEM-8*. For T5-large, the corresponding values are 28.92 $\mathcal{E}$4 32.64, 24.04 $\mathcal{E}$5 26.40, and 25.18 $\mathcal{E}$6 33.51. Across every backbone-dataset pair, CoCoGEC outperforms CPR, DISCO, and TypeDA while using only 25K generated instances versus 30K–42K for some baselines.

An important pattern is that the strongest improvements occur on TEM-8*, the longest-context setting. The paper interprets this as evidence that the method is particularly effective for long-context robustness. It also reports that standard benchmark performance is not degraded: for GECToR, BEA-19 dev improves from 59.6 to 63.2, CoNLL-14 test from 64.5 to 68.2, and BEA-19 test from 73.1 to 74.1.

6. Ablations, interpretation, limitations, and significance

The ablation on TEM-8 with GECToR separates the effects of generation and selection (Wang et al., 13 Jun 2026). Starting from baseline $\mathcal{E}$7, adding a GPT-based global $\mathcal{E}$8 yields 39.00, adding intra-sentence $\mathcal{E}$9 yields 40.83, and adding inter-sentence $c = p \oplus s' \oplus q,$0 yields 42.33. Using all generated counterfactuals with $c = p \oplus s' \oplus q,$1 gives 41.31, whereas MI selection with $c = p \oplus s' \oplus q,$2 gives 46.77. This shows that both intra-sentence and inter-sentence perturbations help, but that MI-based selection is crucial; retaining all generated data dilutes usefulness.

The same ablation reports robustness improvements from baseline CRS/P-CRS of 6.41 / 64.30 to 16.98 / 72.52 for the full method. On TEM-8 with GECToR, perturbation-type breakdown further indicates that sentence-level context is especially challenging: source $c = p \oplus s' \oplus q,$3 is 55.6 for the baseline versus 60.6 with CoCoGEC; word-level perturbation is 53.0 versus 59.9; sentence-level perturbation is 37.3 versus 53.4; and combined perturbation is 38.3 versus 46.7. The sentence-level robustness drop $c = p \oplus s' \oplus q,$4 decreases from 18.3 to 7.2. This suggests that discourse context shifts are more damaging than local lexical substitutions, and that CoCoGEC is especially effective against that failure mode.

With Qwen3-8B on TEM-8, as added context length increases, the vanilla model degrades steadily, whereas the CoCoGEC-trained model remains more stable. Under attack budgets ATK1 to ATK5, CoCoGEC also yields higher $c = p \oplus s' \oplus q,$5, TR, and SR; for example, ATK1 $c = p \oplus s' \oplus q,$6 improves from 7.98 to 9.48, and ATK5 from 21.29 to 22.16. These gains are smaller than the long-context gains, but they are directionally consistent with the same robustness claim.

The paper attributes the framework’s effectiveness to three interacting factors: perturbations are controlled because only non-error spans are changed; gold correction fidelity is enforced through $c = p \oplus s' \oplus q,$7; and hard examples are selected rather than indiscriminately added. The method appears especially suited to small lexical substitutions inside a sentence, insertion, deletion, or replacement of non-error spans, and long-range context shifts produced by coherent surrounding sentences.

The main limitations are practical and methodological. The framework depends on external LLMs for generation, and performance may depend on prompting quality and generator choice. Some generated candidates require additional filtering for grammar, coherence, semantic drift, and leakage. The appendix mentions further quality-control checks such as automatic grammaticality checking, external GEC verification of $c = p \oplus s' \oplus q,$8, perplexity filtering, semantic-similarity filtering, leakage prevention, and manual verification. A plausible implication is that CoCoGEC is best understood as a high-curation data-centric robustness pipeline rather than a lightweight augmentation heuristic.

More broadly, CoCoGEC reframes robust GEC as a counterfactual invariance problem. Its central claim is that robustness improves when augmentation is structurally aligned with the task, explicitly constrained to preserve gold edits, targeted at context variation rather than arbitrary corruption, and selectively filtered for challenging validity. In that sense, the framework contributes both a practical training procedure and a more specific robustness criterion for GEC: the same underlying grammatical error should be corrected the same way even when irrelevant context changes.