LatentFlowSR: High-Fidelity Audio Super-Resolution via Noise-Robust Latent Flow Matching

Abstract: Audio super-resolution aims to recover missing high-frequency details from bandwidth-limited low-resolution audio, thereby improving the naturalness and perceptual quality of the reconstructed signal. However, most existing methods directly operate in the waveform or time-frequency domain, which not only involves high-dimensional generation spaces but is also largely limited to speech tasks, leaving substantial room for improvement on more complex audio types such as sound effects and music. To mitigate these limitations, we introduce LatentFlowSR, a new audio super-resolution approach that leverages conditional flow matching (CFM) within a latent representation space. Specifically, we first train a noise-robust autoencoder, which encodes low-resolution audio into a continuous latent space. Conditioned on the low-resolution latent representation, a CFM mechanism progressively generates the corresponding high-resolution latent representation from a Gaussian prior with a one-step ordinary differential equation (ODE) solver. The resulting high-resolution latent representation is then decoded by the pretrained autoencoder to reconstruct the high-resolution audio. Experimental results demonstrate that LatentFlowSR consistently outperforms baseline methods across various audio types and super-resolution settings. These results indicate that the proposed method possesses strong high-frequency reconstruction capability and robust generalization performance, providing compelling evidence for the effectiveness of latent-space modeling in audio super-resolution. All relevant code will be made publicly available upon completion of the paper review process.

• The paper introduces a noise-robust autoencoder with latent-space conditional flow matching that efficiently enhances high-fidelity audio reconstruction.
• It demonstrates superior objective performance, outperforming waveform and diffusion-based models using metrics like ViSQOL and LSD.
• The method generalizes effectively across varied audio domains while reducing computational complexity and enabling one-step inference.

LatentFlowSR: High-Fidelity Audio Super-Resolution via Noise-Robust Latent Flow Matching

Introduction

LatentFlowSR implements audio super-resolution by leveraging conditional flow matching (CFM) within a continuous latent representation space. The approach focuses on overcoming the inefficiencies and limitations of waveform- and spectrogram-based generative models, particularly their restricted coverage across general audio domains and high computational cost. Through a pipeline comprising a noise-robust autoencoder and a latent-space CFM mechanism driven by a hierarchical velocity field estimator, LatentFlowSR achieves high perceptual reconstruction quality and strong generalization. Theoretical advancements are substantiated by state-of-the-art empirical results spanning speech, sound effects, and music under rigorous in-domain and out-of-domain conditions.

Figure 1: The overall LatentFlowSR framework comprises a noise-robust autoencoder, a CFM-based super-resolution module, and the integration of low- and high-resolution latent representations.

Noise-Robust Latent Representation Learning

LatentFlowSR’s front-end is a noise-robust autoencoder, constituting an encoder-decoder structure with residual and periodic activation mechanisms. The encoder progressively downsamples input audio via stacked convolutional, residual, and "Snake" activations, yielding compact latent representations with a preserved hierarchy of acoustic semantics.

During training, additive Gaussian perturbations are injected into the latent vectors before decoding. This denoising autoencoding regularization, operationalized with adversarial and L1 waveform losses, calibrates the decoder to robustly invert imperfect latent codes—a critical adaptation for subsequent generative modeling.

Figure 2: Schematic of the noise-robust autoencoder, illustrating the injection of noise into the latent space during training to improve decoder robustness.

Empirical results demonstrate that the noise-robust variant outperforms both standard autoencoders and alternative analysis–reconstruction architectures (e.g., vector-quantized VAEs, vocoders), providing higher ViSQOL scores and maintaining LSD under varying levels of latent noise. This mitigates train–test drift and stabilizes the generative process in latent space.

Conditional Flow Matching in Latent Space

The core generative process is built on the conditional flow matching paradigm, where a velocity field $v_\theta$ parameterized by a U-Net backbone transforms Gaussian noise toward the target latent distribution conditioned on upsampled low-resolution latents. The path from noise to latent target is formalized as a continuous-time ODE.

Figure 3: Architecture of the velocity field estimation network, which integrates time embeddings and conditional latent guidance through hierarchical feature extraction and fusion.

The velocity field network concatenates condition and state, incorporates explicit time embeddings, and alternates between local (ResNet) and global (Transformer) context modeling at each hierarchical stage. The output block projects these features into a velocity field, which is then used—with an Euler ODE solver—to discretely traverse from Gaussian prior to the decoded high-resolution latent.

Training utilizes OT-CFM, where the optimal transport formulation employs linear interpolation between $l_0$ and $l_{HR}$, simplifying supervision to a mean-squared error over local velocities. At inference, a single-step Euler solver is utilized, dramatically reducing computation compared to iterative score-based diffusion methods.

Figure 4: Illustration of CFM-based training and inference. The model learns local velocity fields between Gaussian prior and target latent as guided by low-resolution audio conditions.

Empirical Evaluation

Objective and Subjective Results

LatentFlowSR is evaluated across VCTK (speech), ESC-50 (sound effects), and two music datasets (internal and MUSDB18-HQ), benchmarked against diffusion, adversarial, and flow-based baselines.

• Objective Metrics: Across all domains, LatentFlowSR yields the lowest LSD/LSD-HF and highest ViSQOL, especially in aggressive bandwidth-case settings (e.g., 8 kHz to 44.1 kHz). Notably, even against highly-optimized frequency-domain discriminative models (e.g., AP-BWE, mdctGAN), LatentFlowSR matches or outperforms them as spectral fidelity requirements increase.
• Subjective Quality: MOS experiments confirm the perceptual advantage, with LatentFlowSR leading strong speech and music baselines by a margin of 0.05–0.6 points, often approaching ground truth audio naturalness.

Generalization and Robustness

Out-of-domain generalization is validated on ESC-50 and MUSDB18-HQ, where LatentFlowSR preserves performance superiority, especially when other flow-based or diffusion-based models degrade substantially. The latent modeling and denoising strategies demonstrably contribute to this robustness.

Computational Efficiency

LatentFlowSR achieves a substantial reduction in both parameter count and FLOPs relative to diffusion-based models (e.g., AudioSR, FlashSR), primarily because latent-space modeling allows a lighter generative model (as little as 4–8% the complexity) and single-step ODE integration. This efficiency does not come at the expense of reconstruction quality, as ablations with mel-spectrogram and discrete latent spaces show consistent performance drops.

Ablation and Design Analysis

Ablation studies identify two key drivers of performance:

• Noise-robust autoencoder: When replaced by a plain autoencoder, ViSQOL and LSD decline uniformly, emphasizing the necessity of robust representation learning.
• Latent space selection: Both mel-spectrogram and discrete latent alternatives exhibit inferior high-frequency recovery and spectral detail, demonstrating the superiority of the continuous latent space constructed by the proposed autoencoder.

Theoretical and Practical Implications

LatentFlowSR provides a compelling argument for latent-space CFM as the target configuration for high-fidelity audio super-resolution. The demonstrated one-step inference efficacy, capacity to generalize across disparate audio domains, and strong performance with lightweight architectures challenge the prevailing reliance on waveform or frequency-domain approaches for general audio restoration and modeling.

Potential future developments include:

• Extension to broader, unlabeled audio corpora for open-domain unsupervised generalization.
• Hybridization with self-supervised or contrastive representation learning for cross-modal alignment.
• Integration into real-time or edge-deployment scenarios, exploiting the low computational footprint for interactive audio tools.

Conclusion

LatentFlowSR systematically advances the state of the art in audio super-resolution by combining noise-robust latent representation learning with conditional flow matching in latent space. The approach enables efficient, robust, and high-quality recovery of high-frequency audio detail across diverse domains, backed by strong theoretical foundations and empirical superiority. This work substantiates continuous latent-space CFM as a preferred backbone for future developments in neural generative audio processing.