Synthetic Audio Generation Framework for Air Traffic Control Speech Recognition

Abstract: Automatic Speech Recognition (ASR) systems, despite achieving remarkable accuracy in general-purpose domains with native speech (L1), struggle in domains like Air Traffic Control (ATC) due to strong channel noise, a presence of non-native (L2) English accents, and data scarcity. We propose a synthetic data generation pipeline with acoustical properties simulations specifically designed to address this lack of real data to improve recognition accuracy in the ATC domain. Our approach leverages a combination of neural generation techniques, including Text-to-Speech, Voice Conversion, L2-to-L1 accent conversion, and a novel controllable L1-to-L2 accent conversion framework built to simulate accented speech. Our experiments with the Whisper model on the ATCO2 corpus demonstrate that fine-tuning with either synthetic data alone, or a mix of real and synthetic data, significantly improves the word error rate over out-of-the-box and real data only baselines respectively.

• The paper presents a novel synthetic audio generation pipeline that combines TTS, VC, and a controllable L1-to-L2 accent conversion to simulate diverse ATC speech.
• It demonstrates significant improvements in ASR by reducing WER from 63.32% to as low as 31.40% with synthetic data, outperforming real-only baselines.
• The framework underscores the pivotal role of accent diversity in robust ASR, offering scalable augmentation for domain-specific acoustic challenges.

Synthetic Data Augmentation for Robust Air Traffic Control Speech Recognition

Motivation and Context

Automatic Speech Recognition (ASR) in Air Traffic Control (ATC) applications presents significant challenges due to channel noise, pronounced presence of L2 English accents, rapid speech rates, domain-specific phraseology, and severe annotated data scarcity. The performance of state-of-the-art ASR models, such as Whisper and wav2vec 2.0, degrades notably in this domain. Traditional data augmentation techniques—speed perturbation, pitch shifting, noise injection, and SpecAugment—introduce acoustic diversity but lack the capacity to simulate realistic and varied accent and speaker characteristics that are critical for ATC communications. The paper introduces a novel synthetic data generation pipeline explicitly designed for the ATC domain, combining Text-to-Speech (TTS), Voice Conversion (VC), and Accent Conversion (AC), including a new controllable L1-to-L2 accent conversion framework.

Synthetic Audio Generation Framework

The proposed framework is architected to maximize diversity in synthetic ATC training data via modular, composable generation strategies. Starting from real ATC data, it applies speech/noise separation, super-resolution to match higher sampling rates, TTS for native accent, VC for speaker diversity, conventional L2-to-L1 AC for accent normalization, and a novel controllable L1-to-L2 AC module to simulate varied non-native speech variants. An ATC acoustic simulation (AAS) module resamples, band-pass filters, and injects realistic background noise extracted from original ATC recordings. The synthetic generation pipeline leverages the F5-TTS flow-matching diffusion Transformer, kNN-VC, and TokAN-based AC modules.

Figure 1: Synthetic audio generation framework for ATC speech recognition.

Novel L1-to-L2 Accent Conversion

Accent conversion has typically focused on L2-to-L1 normalization to facilitate recognition by mapping non-native input to standard native output. However, in the ATC context, increasing accent diversity is vital. The paper repurposes the TokAN architecture for controllable L1-to-L2 accent generation, fine-tuning both its token conversion module and token-to-mel synthesizer to map native (L1) tokens to target L2 tokens based on reference accent embeddings. This enables the synthesis of non-native accented ATC speech from native utterances, promoting robust ASR generalization across accent spectra.

Figure 2: Repurposed TokAN architecture for controllable L1-to-L2 accent conversion.

Experimental Results

The evaluation leveraged the ATCO2 corpus (4 h transcribed ATC speech) and synthetic data generated by each strategy independently and in combination. Fine-tuning Whisper-small exclusively on synthetic data outperformed the out-of-the-box baseline (WER reduced from 63.32% to 24.18% with VC). Applying ATC acoustic simulation to synthetic TTS speech reduced WER from 53.88% to 33.77%. Accent conversion provided further improvements, with the best synthetic-only results (31.40% WER) achieved by combined L1-to-L2 AC and VC.

When fine-tuning on a mix of real and synthetic data, the controllable L1-to-L2 AC module yielded the lowest WER (21.64%), surpassing real-only data (22.69%) by a margin that was statistically significant. Conversely, L2-to-L1 accent normalization degraded recognition performance when mixed with real data. These results emphasize that accent diversity, rather than normalization toward native speech, is critical for robust ATC ASR. The application of VC was less impactful in mixed settings, indicating that accent diversity trumps speaker diversity for this domain.

Theoretical and Practical Implications

The strong empirical results substantiate generative augmentation beyond traditional acoustic transformations. The finding that synthetic L1-to-L2 accent-converted speech yields superior recognition performance, contrary to the convention of accent normalization, demonstrates that simulating the accentual variability present in ATC environments is more conducive to robust ASR. The practical value is clear: synthetically generated, domain-matched speech can be used to effectively fine-tune ASR models without requiring extensive real labeled data.

On the theoretical side, the work validates modular generative pipelines and controllable accent conversion as scalable augmentation strategies for ASR in linguistically heterogeneous, low-resource domains. The modular architecture allows for comprehensive ablation and multi-factor experimentation, supporting rigorous evaluation of augmentation effects. The approach can be generalized to other domains where channel effects, accent heterogeneity, and resource scarcity are similarly pronounced.

Future Prospects

Potential future directions highlighted include generating L2 speech with controlled fluency, modelling domain-specific environmental noise, leveraging LLMs for expansion of synthetic corpora, and scaling to larger ASR models. Detailed perceptual evaluation, per-accent analysis, and integration with active learning or online adaptation schemes are warranted to optimize synthetic data quality and ASR robustness.

Conclusion

The paper presents a robust, modular synthetic data generation framework for ATC ASR, with a strong emphasis on controllable accent diversity via a novel L1-to-L2 accent conversion method. Quantitative results demonstrate significant improvements in ASR performance on ATC speech, with synthetic data alone or mixed with a limited amount of real data outperforming all baselines. The implication is that generative augmentation targeting accent and channel diversity, rather than normalization, is pivotal for domain-adaptive ASR in safety-critical, low-resource settings. The methodology and insights are directly applicable to other specialized communication domains with similar acoustic and linguistic complexities.