NLE: Non-autoregressive LLM-based ASR by Transcript Editing

Abstract: While autoregressive (AR) LLM-based ASR systems achieve strong accuracy, their sequential decoding limits parallelism and incurs high latency. We propose NLE, a non-autoregressive (NAR) approach that formulates speech recognition as conditional transcript editing, enabling fully parallel prediction. NLE extracts acoustic embeddings and an initial hypothesis from a pretrained speech encoder, then refines the hypothesis using a bidirectional LLM editor trained with a latent alignment objective. An interleaved padding strategy exploits the identity mapping bias of Transformers, allowing the model to focus on corrections rather than full reconstruction. On the Open ASR leaderboard, NLE++ achieves 5.67% average WER with an RTFx (inverse real-time factor) of 1630. In single-utterance scenarios, NLE achieves 27x speedup over the AR baseline, making it suitable for real-time applications.

• The paper demonstrates a novel non-autoregressive ASR pipeline that converts CTC outputs into editable transcripts using a bidirectional LLM editor.
• It employs an interleaved padding scheme and LoRA adaptation to enable parallel editing and robust error correction under noisy conditions.
• Empirical results reveal up to 27x speedup over autoregressive models with competitive word error rates on standard ASR benchmarks.

Non-Autoregressive LLM-Based ASR by Transcript Editing: A Technical Overview of NLE

Motivation and Design Principles

Autoregressive (AR) large language-model-based ASR systems, while delivering strong transcript accuracy, are inevitably bottlenecked by sequential token decoding that impedes parallelism and increases inference latency. For real-time conversational and low-latency ASR scenarios, this deficiency becomes a critical constraint. Despite the availability of efficient CTC-based architectures, their language modeling capacity and correction ability are limited, especially under noisy acoustic conditions.

The NLE system proposes a non-autoregressive (NAR) paradigm, casting ASR as conditional transcript editing. Rather than rely solely on AR LLM decoders, NLE utilizes a frozen pretrained CTC encoder to generate both acoustic embeddings and an initial (greedy) CTC transcript hypothesis, and then refines this draft via a parallel, bidirectional LLM-based editing mechanism. The core architectural innovation is an interleaved padding strategy with explicit insertion slots, allowing for insertion and deletion operations without sequence-wide shifts, fully exploiting the Transformer's identity mapping bias.

Figure 1: Architecture of NLE, detailing the flow from the frozen CTC encoder to the interleaved sequence editor and final output.

Methodology

CTC Hypothesis Extraction and Interleaved Representation

A pretrained Conformer-based CTC encoder, frozen during NLE training, processes input audio to produce frame-wise acoustic embeddings and a greedy CTC output hypothesis. This hypothesis is retokenized into LLM subwords, then interleaved with blank (insertion) slots: for a token sequence $x_1,\ldots,x_N$, the input is $$\epsilon, x_1, \epsilon, x_2, \ldots, x_N, \epsilon$$. This setup enables local insertions (including multi-token) without affecting the remainder of the sequence.

LLM-Based Editing Model

The NLE editor is based on a pretrained causal LLM adapted for bidirectional attention via LoRA (Low-Rank Adaptation), facilitating full sequence context for parallel editing. Acoustic embeddings, projected via a Q-Former, are concatenated with token embeddings and passed through the LLM; output logits are mapped directly to vocabulary predictions for each sequence position. Training is driven by a CTC-style loss, marginalizing over possible alignments, with an auxiliary copying regularization (CR) loss to strengthen the identity bias and facilitate preservation of correct tokens.

Inference and Iterative Refinement

Inference is performed in a single pass, achieving full sequence parallelism; optionally, the editing process may be iterated by refeeding output transcripts as new hypotheses for further refinement, though gains plateau after the first additional step due to distribution mismatch.

Empirical Results

Accuracy–Speed Tradeoff

On the Open ASR leaderboard comprising 39 models, NLE and NLE++ (an enlarged version with increased training budget and LoRA rank) establish a new Pareto frontier in the WER–RTFx space: NLE++ yields 5.67% average WER and RTFx of 1630, competitive with the best LLM-based models while significantly outperforming them in throughput. NLE achieves up to 27x speedup over AR baselines in single-utterance mode, confirming suitability for low-latency settings.

Figure 2: NLE and NLE++ set the Pareto frontier for WER versus RTFx against the top-6 ASR models, outperforming most models in inference speed at comparable or superior accuracy.

Component and Ablation Analyses

Key design choices were substantiated via ablation: removing CR, bidirectionality, interleaved padding, or acoustic conditioning significantly degrades validation loss and WER. The bidirectional LLM and interleaved insertion architecture are critical—end-of-sequence padding and causal attention both introduce inefficiencies and inferior editability.

Figure 3: Validation loss trajectories during ablation underline the necessity of each component in NLE’s design.

Error Profile and Qualitative Examples

NLE displays higher deletion rates but lower insertion rates relative to AR models, whose higher insertions indicate more hallucinations, especially in acoustically ambiguous regions. Substitution rates are dataset-dependent, with the model's corrections typically improving local errors in CTC outputs. Qualitative inspection shows robust correction of diverse error types and the capacity to excise metadata versus actual speech content.

Figure 4: Decomposition of WER into insertions, deletions, and substitutions for CTC, NLE, and AR systems, revealing NLE’s conservative error profile.

Computation Cost Breakdown

The encoder dominates inference cost at 66%, with the LLM editor at ~30%. The non-autoregressive architecture allows near-optimal hardware utilization, with all other stages negligible in compute share.

Figure 5: Profiling of inference stages for NLE; encoder operations are the dominant cost.

Practical and Theoretical Implications

NLE demonstrates that pretrained LLMs can be adapted, using lightweight modifications, for non-autoregressive conditional editing tasks in ASR. The approach combines high-accuracy transcript correction with extreme inference parallelism, making LLM-based ASR practical for real-time applications and demonstrating a template for efficient NAR editing in other sequence transduction tasks.

Crucially, the system design enables weights to be shared between ASR and general LLM applications, via LoRA and attention mask reconfiguration.

There remain tradeoffs: NLE’s conservative, hypothesis-anchored editing biases against large-scale rehypothesization, limiting performance if the initial CTC draft is poor or if large semantic transformations are required. Cross-tokenizer retokenization overhead is minor but suboptimal.

Future Directions

Several optimizations are outlined:

• Text augmentation: To close the gap between training and iterative inference distributions, increasing the robustness of multi-step editing.
• Hybrid editing-masking: Combining editing with mask-predict strategies may further enhance correction ability, especially for uncertain regions.
• Architectural modifications: Decoupling LLM processing for audio and text to switch from quadratic to linear complexity in audio length, enabling efficient long-form ASR.
• Encoder-editor joint optimization: End-to-end training could further elevate correction capacity, provided robust schemes are designed to stabilize training.
• Employ LLM as CTC LLM: Incorporating the LLM directly into CTC beam search may bring additional priors to initial draft generation.

Conclusion

NLE advances LLM-based ASR by recasting transcript generation as non-autoregressive editing, marrying CTC efficiency with LLM linguistic priors. With performance that rivals AR paradigms and massively outpaces them in inference speed—especially for real-time use—NLE substantiates the viability of bidirectionally-adapted LLM editors in ASR. This paradigm is likely extensible to other sequence transduction domains, representing a significant stride toward practical, low-latency, LLM-powered language technologies.