Transformer-Based Non-Autoregressive Approach

• The transformer-based non-autoregressive approach is a model paradigm that predicts output tokens in parallel, removing sequential dependencies and accelerating inference.
• It employs techniques like latent variable modeling, knowledge distillation, and iterative refinement to mitigate performance gaps compared to autoregressive models.
• Applications span machine translation, speech recognition, and summarization, offering significant speedups for latency-critical scenarios.

A transformer-based non-autoregressive approach refers to a class of sequence generation models that lift the left-to-right, token-wise sequential dependency characteristic of canonical autoregressive transformers. In these models, all output tokens are predicted in parallel, which unlocks significant inference speedup and is particularly advantageous for latency-critical applications like machine translation (MT), speech recognition, and summarization. The field, originally defined by the seminal fertility-based non-autoregressive neural machine translation (NAT) architecture (Gu et al., 2017), has grown to encompass a diverse set of designs, training strategies, and application areas, with ongoing research focused on bridging the accuracy gap to autoregressive (AR) baselines.

1. The Principle of Parallel Sequence Generation

Traditional AR transformers factorize the conditional output probability as

$$p_\text{AR}(Y|X) = \prod_{t=1}^T p(y_t | y_{<t}, X)$$

which requires sequential generation and is inherently non-parallelizable at inference. In contrast, non-autoregressive transformers posit conditional independence across output positions:

$p_\text{NAR}(Y|X) \approx \prod_{t=1}^T p(y_t|X)$

In practice, this enables a single forward pass to generate all $T$ outputs, dramatically reducing inference latency (e.g., by $10\times$ or more (Gu et al., 2017)).

The non-autoregressive formulation is achieved by modifying the Transformer decoder to remove causal masking, eliminate position-wise shifting of target embeddings, and allow full-attention across all slots in the output sequence. Decoder initialization typically uses source representations, repeated or rearranged to match target length, and may involve additional architectural modules such as fertility predictors, CTC or CIF token extractors, or frequency-mixing layers.

2. Latent Variable Modeling and Output Length Prediction

Naïve NAR models suffer severe performance drops, most notably due to "multimodality"—the target conditional distribution is sharply peaked in correlated output configurations, violating the independence assumption and leading to over-/under-translation, repetition, or misalignment.

Early NAT models address this by introducing discrete latent variables. The fertility-based NAT (Gu et al., 2017) predicts, for each source token $x_i$, a discrete fertility $f_i$—the number of times a source token should be mapped to the target. The joint distribution is marginalized over all fertility sequences $\mathbf{f}$ compatible with the target length:

$$p_\text{NAR}(Y|X; \theta) = \sum_{\mathbf{f} \in \mathcal{A}} \left[ \prod_{i} p_F(f_i|X;\theta) \prod_{t} p(y_t|x^{f_1}_1, ..., x^{f_{T'}}_{T'}; \theta) \right]$$

where $x^f_i$ indicates $f$ repeated copies of token $x_i$ and $\mathcal{A}$ is the set of valid fertility assignments. This constrains the combinatorial space and increases compatibility between source and target, partially restoring inter-output dependencies.

Subsequent work has generalized latent variable mechanisms: position modeling as a permutation latent variable (Bao et al., 2019), CTC alignments as continuous or discrete latent variables for token extraction (Fan et al., 2023, An et al., 26 Sep 2024), and DAG-structured hidden representations (Huang et al., 2022).

Table: Latent Variable Mechanisms in NAR Transformers

Approach	Latent Variable	Effect
Fertility-based NAT	Fertility per src	Output length/planning
PNAT (PosNAT)	Positional perm. z	Target order/reordering
CTC-NAT / Paraformer-v2	CTC alignment	Robust token embeddings
DA-Transformer	Path in DAG	Parallel multimodality

3. Training Strategies: Knowledge Distillation, Policy Gradient, and Proxy Objectives

NAR models are notorious for learning instability and subpar maximum likelihood (MLE) training results compared to AR baselines. Several key techniques have been established to compensate:

• Sequence-Level Knowledge Distillation (KD): A high-capacity AR "teacher" model generates synthetic outputs, which are more deterministic and less multimodal than human-generated translations. The NAR "student" is then trained on these teacher outputs, significantly reducing the performance gap (Gu et al., 2017). This is essential to reduce the dependency gap (Huang et al., 2022).
• Policy Gradient Fine-Tuning: Due to non-differentiable sampling (e.g., in fertility or alignment sampling), REINFORCE-based or other policy-gradient methods are employed to optimize non-differentiable sequence-level losses such as KL divergence to the teacher, BLEU, or reverse-KL. Fine-tuning the loss as

$$L_\mathrm{FT} = \lambda [ \mathbb{E}_{f \sim p_F} (L_\mathrm{RKL}(f) - L_\mathrm{RKL}(\bar{f})) ] + (1-\lambda)L_\mathrm{KD}$$

helps recover additional BLEU points (Gu et al., 2017).

• Proxy Distribution Training (Maximum Proxy-Likelihood Estimation, MPLE): Recent theoretical analysis (Huang et al., 2022) proves that MLE on the original multimodal data is fundamentally limited by the target's conditional total correlation $\Delta$, resulting in irreducible information loss. Instead, the effective practice is to maximize the likelihood on a simplified (proxy) target distribution $Q(T|Z,X)$—knowledge-distilled, aligned, or otherwise—whose dependency gap is smaller, thus enabling better realizable NAR performance.

4. Decoding and Inference: Heuristic and Structured Strategies

Because an NAR model's conditional independence assumption is imprecise, naive decoding (greedy or argmax) can yield low-quality predictions. Various strategies are used to close this gap:

• Noisy Parallel Decoding (NPD): Sample multiple latent variable configurations (e.g., fertility, alignment, or mask patterns), generate translations for each in parallel, and re-score them with the AR teacher to select the best. NPD with 100 samples effectively recovers many of the missing BLEU points (Gu et al., 2017), yet maintains significant speedup compared to beam search AR decoding.
• Iterative Refinement (Mask-Predict, Parallel Easy-First): Iteratively mask and re-predict uncertain tokens using confidence-based selection (e.g., "easy first" decoding (Kasai et al., 2020), mask-predict approaches). These iterative methods allow partial incorporation of sequential dependencies while keeping most predictions parallel.
• CTC-Based Decoding: For speech recognition and some language generation models, use CTC alignment to robustly extract token boundaries and condition transformations on the most likely frame alignments, with error-based sampling to mitigate alignment mismatch between training and inference (Fan et al., 2023, An et al., 26 Sep 2024).

5. Architectural Innovations and Extensions

Beyond the canonical NAT, diverse architectures have been introduced to address global or local dependency deficits:

• FS-Decoder: Combines non-autoregressive bottom layers with an autoregressive top layer, fusing bottom-up parallel context with sequential modeling to bridge the accuracy-speed trade-off (Shao et al., 2019).
• Disentangled Context (DisCo) Transformer: Enables every token to be predicted conditioned on arbitrary subsets of other tokens, using masked attention and parallel easy-first inference (Kasai et al., 2020).
• Position Learning (PNAT): Models explicit position permutations as latent variables, improving reorderings in tasks with strong monotonicity violations (Bao et al., 2019).
• DAG-structured Decoders (DA-Transformer, CoDAT): Represents all possible translation paths in a DAG, marginalizing over them to capture multiple modes and reduce multimodality (Huang et al., 2022, 2305.13667).
• FourierNAT: Incorporates discrete Fourier transforms in the decoder, enabling spectral global context mixing in a parallel, non-autoregressive fashion (Kiruluta et al., 4 Mar 2025).

In the speech domain, the usage of CTC or CIF modules for token boundary detection (Gao et al., 2022, An et al., 26 Sep 2024), glancing LLM samplers (Paraformer), and contrastive lexical-aware training (LA-NAT) (Lin et al., 2023) further expands the NAR transformer paradigm to noisy, low-resource, and language-universal ASR problems.

6. Empirical Performance and Application Domains

NAR transformers have demonstrated rapid progress in performance metrics and application breadth:

• Translation: The original NAT achieved 25–30 BLEU on IWSLT/WMT, ~2–5 BLEU below AR models—a gap that shrinks to below 2 BLEU points with advanced decoding/training (Gu et al., 2017, Huang et al., 2022, 2305.13667).
• Speech Recognition: Paraformer, CASS-NAT, Paraformer-v2, and NAR-BERT-ASR deliver ASR error rates on par with AR models, while delivering up to $24\times$ inference speedup and improved noise robustness (Gao et al., 2022, Fan et al., 2023, An et al., 26 Sep 2024, Yu et al., 2021).
• Conditional Generation: Tracformer demonstrates that non-autoregressive, sparse attention-based encoders can outperform both diffusion-based and AR models in generalization for conditional tasks, such as text infilling or summarization, especially under varying masking/query patterns (Liu et al., 11 Feb 2025).
• Large-scale Unsupervised Pretraining: UT5 shows that unrolled denoising enables fast pretraining of giant decoder-only models with SoTA downstream quality, using multi-iteration, parallelized denoising objectives (Salem et al., 2023).

NAR's speedup enables deployment in real-time translation, large-batch ASR, edge device inference, and other low-latency environments, with growing applicability in multi-modal and generative AI tasks.

7. Limitations, Trade-offs, and Future Directions

The principal limitation of the transformer-based NAR approach remains the dependency gap: loss of target-side conditional dependencies inherent to parallel decoding, as formally measured by conditional total correlation (Huang et al., 2022). While proxy distribution training, iterative refinement, or explicit structural modeling partially alleviate these issues, a residual quality deficit remains for complex, open-domain sequence generation. Additional points of concern are:

• NAR models often require significantly larger compute budgets and more complex training schedules (KD, RL fine-tuning, heuristic search, multi-stage training).
• Parallelization gains are sometimes offset by the higher computational cost per sample or issues with output length prediction/alignment.
• Certain domains with extreme output multimodality or requirement for fine-grained sequencing (e.g., rich dialog, program synthesis) remain challenging for current NAR designs.
• Emerging hybrid, sparse, and frequency-domain architectures (e.g., Tracformer, FourierNAT) signal ongoing innovation in balancing locality, global structure, and parallelism.

Research trends are converging on further reducing the dependency gap via improved proxy objectives, principled latent variable modeling, compositional architecture, and informed iterative decoding, with future directions likely to see NAR approaches scale into large, general-purpose sequence models for both text and speech (Liu et al., 11 Feb 2025, Kiruluta et al., 4 Mar 2025, Salem et al., 2023).