SoftCorrect: Selective ASR Error Correction

• SoftCorrect is an ASR error correction method that uses a soft per-token detection signal to selectively edit only error tokens while preserving correct ones.
• It fuses language model detector scores with acoustic ASR scores and employs a constrained CTC decoder that duplicates only flagged tokens for focused correction.
• The method achieves significant CER reductions (26.1% on AISHELL-1 and 9.4% on Aidatatang) while maintaining fast parallel inference performance.

SoftCorrect is an automatic speech recognition (ASR) error-correction method that treats post-ASR correction as a selective editing problem: because recent ASR systems usually have low word error rate, only a small subset of tokens should be changed, and originally correct tokens should be preserved (Leng et al., 2022). Its defining contribution is a soft error detection mechanism that supplies an explicit per-token correctness signal to a non-autoregressive corrector, combining a dedicated language-model detector with a constrained CTC decoder so that only detected error tokens are duplicated and edited. The method operates on text-only ASR outputs, supports parallel generation, and was reported to achieve 26.1% and 9.4% CER reduction on AISHELL-1 and Aidatatang, respectively, while maintaining fast inference (Leng et al., 2022).

1. Problem formulation and design rationale

ASR error correction takes the text hypotheses produced by a first-pass ASR system and edits only those tokens that are incorrect, preserving correct tokens. In typical modern ASR, WER or CER is low, often below 10%, so a corrector that edits indiscriminately risks degrading already-correct output. This makes error detection central to the correction problem.

Prior approaches in the paper are divided into implicit and explicit detection. Encoder-decoder autoregressive correctors with attention implicitly discover which source positions to edit through target-source attention, but they do not provide a clear training or inference signal about which tokens are wrong. Vanilla CTC-based correction that duplicates all tokens is flexible and parallel, but it also lacks an explicit focus signal and tends to propose edits everywhere, which risks changing correct tokens. Explicit detection methods, including duration-based detection such as FastCorrect, predict insertion, deletion, or substitution types, but misclassification of the error type can introduce new errors and reduce robustness.

SoftCorrect is designed to combine the advantages of both families while avoiding their failure modes. It does not attempt to classify a token into deletion, substitution, or insertion categories. Instead, it predicts whether a token is likely correct or not through a soft probability produced by a dedicated LLM, then leaves the fine-grained resolution of edit operations to CTC. This gives the decoder an explicit focus signal without constraining it to a possibly incorrect discrete error taxonomy.

A plausible implication is that SoftCorrect reframes post-ASR correction as constrained editing around suspected error spans rather than full-sequence rewriting. That is the central structural difference from both implicit attention-based methods and explicit duration-based methods.

2. Architecture and soft error detection

SoftCorrect takes text-only inputs rather than acoustic features. The input is an aligned N-best list from the ASR beam. Multiple candidates are first aligned to the same length with a multi-candidate alignment algorithm using token and phoneme similarity, inserting blanks $\phi$ where needed. This alignment step produces a position-wise representation over competing beam hypotheses.

The detector is a 12-layer Transformer encoder with hidden size 512. At each aligned position it outputs a probability distribution over the vocabulary $V$ plus one special token $\mathrm{GT}$. This detector is not a pretrained external LM; it is trained end-to-end for SoftCorrect with an anti-copy language-model objective. Using the detector’s per-position distributions, SoftCorrect constructs a selected candidate by choosing the token with the highest probability at each position. The paper states that this selected candidate typically has fewer errors than any single beam candidate and improves downstream detection and correction.

For a selected token $y_i$, the detector score is $p_i(y_i)$. SoftCorrect then fuses this with an ASR score reflecting acoustic alignment and pronunciation similarity, producing a soft correctness score

$$S_i = \alpha \cdot p_i(y_i) + (1-\alpha)\cdot p_i^{\mathrm{ASR}}(y_i).$$

A token is detected as incorrect if $S_i < \tau$, where $\tau$ is chosen on development data; otherwise it is treated as correct and should pass through unchanged (Leng et al., 2022).

The role of the special token $\mathrm{GT}$ is to create a decision boundary between correct and incorrect tokens. Let $H_i \in \mathbb{R}^d$ be the encoder hidden state at position $V$0, and let $V$1 be the softmax weights. The detector defines logits

$V$2

and probabilities

$V$3

The anti-copy objective is

$V$4

with $V$5 in experiments, where

$V$6

and

$V$7

The intended effect is that $V$8 gives correct tokens high probability, while $V$9 removes $\mathrm{GT}$0 from the candidate set and trains the model to prefer $\mathrm{GT}$1 over all incorrect alternatives, discouraging a degenerate always-copy behavior.

The paper’s ablations indicate that this bidirectional anti-copy detector is structurally important. Replacing it with standard cross-entropy, BERT-style N-pass masking, GPT-style left-to-right LM, or binary classification degrades correction quality (Leng et al., 2022).

3. Constrained CTC correction

The corrector is a 6-layer Transformer decoder trained with a constrained CTC loss. Its input is the selected candidate sequence, but only tokens detected as incorrect are duplicated. The standard duplication factor is $\mathrm{GT}$2. Tokens detected as correct remain as single fixed anchors.

If the selected candidate is $\mathrm{GT}$3 and the detection mask is $\mathrm{GT}$4, where $\mathrm{GT}$5 when $\mathrm{GT}$6, SoftCorrect constructs the decoder input by repeating only the positions with $\mathrm{GT}$7. Non-duplicated positions are intended to pass through unchanged. The constrained CTC objective then restricts dynamic alignment to the duplicated spans.

Standard CTC is written as

$\mathrm{GT}$8

where $\mathrm{GT}$9 is the set of paths collapsing to target $y_i$0. SoftCorrect instead uses

$y_i$1

where $y_i$2 is restricted by the mask $y_i$3 (Leng et al., 2022).

For anchor positions with $y_i$4, the path must emit $y_i$5 exactly once and cannot substitute it with a different symbol; transitions at anchor positions are restricted to $y_i$6, with exactly one $y_i$7 emission surviving collapse. For duplicated error spans with $y_i$8, the path is unconstrained within the repeated segment, allowing substitution, insertion, and deletion resolution through ordinary CTC alignment.

This structure is the mechanism by which SoftCorrect converts soft detection into focused correction. Compared with duplicating every token, constrained CTC reduces the alignment search space, reduces the chance of modifying correct tokens, and preserves the parallel decoding properties of CTC-style generation. In inference, the decoder predicts all positions in parallel; the output is obtained by greedy selection, blank removal, and collapse of repeats. If all tokens are detected as correct, the decoder can be skipped to reduce latency.

4. Training regime and inference pipeline

The full inference procedure is a six-stage pipeline. First, beam search produces N-best ASR candidates. Second, these candidates are aligned to equal length with token- and phoneme-aware alignment. Third, the encoder produces per-position distributions over $y_i$9. Fourth, position-wise token selection forms a selected candidate and computes the fused score $p_i(y_i)$0 for each position. Fifth, detected error tokens are duplicated, typically three times. Sixth, the constrained CTC decoder produces the final corrected output in parallel.

The paper also describes a robustness augmentation during decoder training: 5% of correct tokens are randomly marked as pseudo-errors and forced to decode back to themselves. This is meant to teach the decoder to leave falsely flagged tokens unchanged. A plausible implication is that the method tries to make detection errors less damaging by training the corrector under mild detector noise.

SoftCorrect is pretrained on 400M pseudo text pairs synthesized via a BERT generator with homophone-aware masking, then fine-tuned on paired ASR outputs and ground-truth text for each dataset. Optimization follows Fairseq Transformer defaults, with training on 16 V100 GPUs and batch size around 6000 tokens. The detector uses $p_i(y_i)$1, the duplication factor is $p_i(y_i)$2, and the fusion weight $p_i(y_i)$3 and threshold $p_i(y_i)$4 are tuned on development sets (Leng et al., 2022).

The method’s efficiency claims derive from two distinct design choices. The anti-copy detector produces per-position scores in a single bidirectional pass, unlike BERT-style masking approaches that scale with sequence length. The constrained CTC decoder duplicates only a small fraction of positions when error rates are low, shortening the effective sequence length and reducing memory overhead relative to methods that duplicate every token.

5. Empirical performance and ablation evidence

SoftCorrect is evaluated on AISHELL-1, Aidatatang-200zh, and additionally on WenetSpeech for scalability. AISHELL-1 contains 150h train, 10h dev, and 5h test; Aidatatang-200zh contains 140h train, 20h dev, and 40h test. The main metric is CER, together with CERR, defined as CER reduction relative to the ASR output (Leng et al., 2022).

Setting	No correction CER	SoftCorrect CER
AISHELL-1 test	4.83	3.57
AISHELL-1 dev	4.46	3.40
Aidatatang test	4.47	4.05
Aidatatang dev	3.82	3.44

These results correspond to 26.09% CERR on AISHELL-1 test and 9.40% CERR on Aidatatang test. The paper reports that SoftCorrect outperforms previous works by a large margin while preserving fast parallel generation. On AISHELL-1 test, latency is 17.0 ms/sentence on GPU and 97.4 ms/sentence on CPU. This is reported as approximately $p_i(y_i)$5 faster than FC+Rescore and comparable to FastCorrect, while achieving higher CERR. MAPSSWE tests show $p_i(y_i)$6 versus baselines in most cases; SoftCorrect and FC+Rescore are reported as comparable on AISHELL-1, but SoftCorrect is approximately $p_i(y_i)$7 faster (Leng et al., 2022).

The ablations are especially diagnostic. Replacing the anti-copy LM with standard cross-entropy gives AISHELL-1 test CER 3.77; BERT-style N-pass masking gives 3.93; GPT-style left-to-right LM gives 4.76; binary classification gives 3.98; and removing constrained CTC by duplicating all tokens gives 3.95. The GPT-style result is particularly poor, which the paper interprets as evidence that bidirectional context is crucial.

The detector and corrector analysis further clarifies the operating regime. On AISHELL-1, SoftCorrect reports detection precision 84.06, recall 49.71, and F1 62.47, with correction precision 71.30. On Aidatatang, which is described as harder, the detector metrics drop to precision 80.52, recall 25.29, and F1 38.49, with correction precision 61.25. Even in that harder setting, SoftCorrect still achieves 9.40% CERR, whereas some baselines degrade CER.

The qualitative example in the paper illustrates the intended mechanism: if the selected candidate contains only a few likely errors, only those spans are duplicated, and constrained CTC dynamically repairs them while leaving surrounding anchors unchanged. This suggests that the model’s empirical advantage is tied less to aggressive correction capacity than to selective editing under explicit uncertainty estimates.

6. Limitations, failure modes, and relation to later work

The paper identifies several limitations. Threshold $p_i(y_i)$8 and fusion weight $p_i(y_i)$9 must be calibrated, and domain shift can reduce detector reliability. The method also relies on N-best hypotheses, so very low-beam settings may reduce its gains. A more specific failure mode concerns hard deletions: if the deleted token never appears in any beam candidate and neither neighboring token is wrong, SoftCorrect may fail to detect the error. The paper notes that such errors are rare and suggests that adding a dedicated binary deletion detector is a future direction (Leng et al., 2022).

The method’s design also expresses a deliberate trade-off. It prioritizes avoiding harmful edits over aggressively repairing every possible error. This is visible in the constrained CTC anchors, the option to skip decoding when all tokens are detected correct, and the robustness augmentation that teaches the decoder to reconstruct pseudo-errors back to themselves. A plausible implication is that SoftCorrect is optimized for low-to-moderate CER regimes where most tokens should survive intact.

A later contrastive point comes from UCorrect, which proposes an unsupervised Detector–Generator–Selector framework with no dependency on pseudo paired data or original paired data (Guo et al., 2024). UCorrect also begins with explicit error detection, but its detector is based on masked-language-model scoring, its generator proposes phonetic-similar candidates, and its selector chooses the sentence with the lowest average masked-LM score as defined in that paper. On AISHELL-1 with a Conformer ASR input, UCorrect reports 6.83% WERR without fine-tuning and 14.29% after fine-tuning, together with 35.12 ms/sentence latency and reduced FAR relative to several baselines (Guo et al., 2024). This suggests that subsequent work explored a different axis of the same problem: reducing dependence on paired correction data while retaining explicit modular detection.

SoftCorrect’s place in ASR post-correction is therefore defined by a specific synthesis: explicit but soft token-level detection, N-best candidate aggregation, and constrained non-autoregressive correction. Its contribution is not merely a new decoder objective, but a complete architecture for focusing correction capacity on likely error positions while preserving the fast inference profile of parallel CTC-style generation (Leng et al., 2022).