FSC-Net: Integrating Fast Fourier Convolutions and Progressive Learning for Speech Bandwidth Extension

Abstract: Speech bandwidth extension (BWE) aims to reconstruct high-fidelity wideband audio from narrowband inputs. While recent approaches have made significant progress, they often struggle to reconstruct realistic high-frequency phase and harmonic structures, leading to perceptual artifacts. In this paper, we propose FSC-Net (Full-Spectrum Context Network), a parameter-efficient architecture designed to explicitly model cross-band harmonic dependencies. By integrating Fast Fourier Convolutions (FFCs) into a complex spectral mapping framework, FSC-Net expands its receptive field to the entire spectrum, capturing long-range frequency interactions effectively. To address the ill-posed nature of high-frequency generation, our novel frequency-progressive learning curriculum guides the network to reconstruct spectral details from coarse to fine. Experimental results on the VCTK and unseen EARS datasets demonstrate that FSC-Net delivers consistently strong reconstruction quality and generalization, particularly in the challenging VCTK 4 kHz-to-48 kHz task. Compared to scaled-up baselines, our model attains leading LSD and PESQ scores while maintaining a highly compact parameter footprint (1.54 M).

• The paper presents FSC-Net, which leverages Fast Fourier Convolutions and a frequency-progressive curriculum to extend speech bandwidth with reduced artifacts.
• It employs a hybrid spectral-spatial architecture with channel-wise subband processing to capture global spectral dependencies and refine harmonic details.
• Experimental evaluations on VCTK and EARS datasets demonstrate superior performance, efficiency, and generalization over state-of-the-art bandwidth extension models.

FSC-Net: Fast Fourier Convolutions and Progressive Learning for Speech Bandwidth Extension

Introduction and Motivation

Speech bandwidth extension (BWE) remains a central problem in audio restoration, aiming to reconstruct high-fidelity wideband audio from narrowband signals. The challenge is particularly acute when the bandwidth truncation is extreme, such as mapping from 4 kHz to 48 kHz, where high-frequency content and phase information are severely lost or distorted. While deep learning approaches, including waveform-based and spectrum-based models, have made advances in performance, they frequently generate perceptual artifacts, notably metallic timbres or spectral discontinuities, due to limitations in phase reconstruction and insufficient capture of global spectral dependencies.

FSC-Net ("FSC-Net: Integrating Fast Fourier Convolutions and Progressive Learning for Speech Bandwidth Extension" (2606.06962)) addresses these limitations by integrating Fast Fourier Convolutions (FFC) into a TF-GridNet backbone and introducing a frequency-progressive learning strategy. The architecture is designed to enlarge the receptive field for better spectral context aggregation while reducing parameter count. In addition, the learning curriculum guides the model from coarse global envelopes to fine spectral details, providing a structured pathway for high-frequency reconstruction.

Figure 1: The general architecture of FSC-Net and the framework for progressive learning, leveraging $N$ TF-FFC Blocks and frequency-progressive supervision.

FSC-Net Architecture

FSC-Net is built upon two principal innovations:

Channel-wise Subband (CWS) Processing:

The frequency axis is divided into $B$ subbands and rearranged as separate channels. This formulation enhances the model's capacity to capture inter-subband (i.e., cross-frequency) dependencies and facilitates efficient computation. In the implementation, $B=3$ suffices to balance computational load and contextual modeling.

Integration of Fast Fourier Convolutions (FFC):

FFC expands the receptive field by leveraging both local convolutions and global operations in the frequency domain. The input feature map is split: one branch processes local structure with spatial convolution, and the other encodes global interactions via real FFT, spatial convolution in Fourier space, and inverse FFT. This hybrid approach allows the model to effectively bridge "spectral gaps" that are pervasive in BWE, modeling harmonic dependencies and phase information more globally than spatial convolutions alone.

Frequency-Progressive Learning

BWE is inherently ill-posed; direct high-frequency hallucination from a deficient input spectrum often induces artifacts. FSC-Net proposes a novel frequency-progressive curriculum: each intermediate block in the network is supervised to reconstruct targets of increasing spectral resolution. Specifically, a sliding-window averaging scheme is applied to the residual between ground-truth and input spectra, with progressively shrinking window sizes across network depth ($W_i$ from large to small), forcing the earlier blocks to capture global spectral envelope and the later blocks to focus on local harmonic detail refinement. This approach effectively guides the generator towards physically plausible solutions and mitigates the risk of generating spurious or over-smoothed spectral content.

Training Objectives

The model is trained using a combination of:

• Multi-Resolution STFT Loss:

Aggregates spectral convergence and log-magnitude errors across multiple FFT analysis windows, directly penalizing both global and local spectral mismatches.

• Log-Spectral Distance (LSD):

A primary metric in audio super-resolution, measuring the dB-scale mismatch between the reconstructed and reference magnitude spectra.

• Adversarial and Feature-Matching Losses:

Least Squares GANs (LS-GAN) are used with multiple discriminators at each progressive stage output. Feature-matching is also employed to stabilize adversarial training and further encourage perceptual quality.

Experimental Evaluation

Quantitative Results

FSC-Net is evaluated on VCTK (4 kHz → 48 kHz and 16 kHz → 48 kHz) and generalization is assessed on the unseen EARS dataset. Metrics include LSD, NISQA, and PESQ, benchmarking against state-of-the-art models including AP-BWE, BAE-Net (and its upscaled version), AERO, and SFNet.

FSC-Net demonstrates leading numerical results in both challenging (extreme bandwidth) and typical settings. For example, on VCTK 4 kHz → 48 kHz:

• LSD: FSC-Net achieves 0.8771, outperforming all baselines.
• NISQA: 4.3134, indicating higher perceptual quality.
• PESQ: 2.8092, notably exceeding scaled-up BAE-Net (2.5519) and much higher than AP-BWE and AERO.

The model maintains a compact footprint (1.54M parameters), substantially smaller than AP-BWE's 29.76M and competitive with BAE-Net variants.

Qualitative Analysis

Spectrogram visualizations confirm the suppression of characteristic artifacts in prior models. FSC-Net reconstructs continuous, high-frequency harmonics closely resembling ground truth, while the baselines display excessive high-frequency noise (AP-BWE), spectral energy discontinuities (BAE-Net), and striated over-smoothed textures (AERO).

Figure 2: Spectrogram comparison on VCTK 4 kHz→48 kHz; FSC-Net restores clear harmonics, whereas baselines exhibit typical spectral artifacts.

Generalization

Zero-shot testing on the EARS dataset reveals strong robustness; FSC-Net retains superior LSD, NISQA, and PESQ scores compared to prior models, evidencing effective out-of-domain generalization.

Ablation Studies

Ablations show that:

• FFC alone yields notable gains in perceptual metrics without inflating parameter count.
• The additional frequency-progressive learning curricula further raises both NISQA and PESQ, with minor incremental improvements in LSD, confirming the primary impact is perceptual.

Implications and Future Directions

FSC-Net offers a highly parameter-efficient, context-aware approach for BWE, appropriate for practical deployment where model size and computation are constrained (e.g., embedded or real-time systems). The integration of FFC modules points to the utility of hybrid spectral-spatial architectures in tasks where long-range contextual modeling is critical, including but not limited to speech enhancement, separation, and reconstruction.

The frequency-progressive curriculum, as a guided solution to coarse-to-fine spectral detail synthesis, is potentially applicable to other generative audio tasks where high fidelity and perceptual quality are sensitive to global-meso-local spectral structures.

Future research may extend this approach via adaptive or learned curriculum schedules, broader spectrum-to-spectrum transformations (including music and non-speech audio), or hybridization with neural source-filter or diffusion-based models.

Conclusion

FSC-Net demonstrates that explicit modeling of global spectral dependencies—via Fast Fourier Convolutions—and a structured coarse-to-fine learning curriculum significantly improve speech bandwidth extension, surpassing both lightweight and computationally heavy baselines in both fidelity and perceptual quality. The demonstrated generalization capacity and parameter efficiency make it a compelling candidate for future advances in speech restoration and audio super-resolution.