MR-CQTdiff: High-Fidelity Diffusion Audio

• The paper presents an invertible, octave-adjustable CQT framework that refines time-frequency trade-offs for advanced diffusion-based audio synthesis.
• MR-CQTdiff employs a dual-path U-Net architecture to balance fine harmonic detail in higher frequencies with improved temporal resolution in lower octaves.
• Empirical evaluations using Fréchet Audio Distance indicate that MR-CQTdiff outperforms competing models in generating high-quality musical and vocal audio.

MR-CQTdiff is a neural network architecture designed for diffusion-based audio generation, utilizing a multi-resolution Constant-Q Transform (CQT) to address limitations in time-frequency representations for audio synthesis. Its primary innovation lies in an invertible, octave-wise adjustable CQT framework that enables flexible temporal and spectral resolution, facilitating high-fidelity generation of complex audio, including music and vocal signals (Costa et al., 20 Sep 2025).

1. Architectural Principles

MR-CQTdiff is constructed on a U-Net-based diffusion framework operating exclusively in the time–frequency domain. Audio waveforms are processed through one or more CQT filter banks, resulting in a multi-resolution, logarithmically spaced time-frequency representation. The filter banks are designed such that:

• Higher octaves (upper frequencies) utilize a greater number of bins per octave, providing fine-grained frequency resolution.
• Lower octaves (lower frequencies) employ fewer bins per octave, reducing filter length and improving temporal resolution.

The encoder-decoder structure of the U-Net facilitates feature extraction and reconstruction, with skip connections ("concatenative paths") that preserve details at multiple resolutions. The neural processing module $U_{(\theta)}$ operates on the CQT features within the U-Net. The entire generative process is a composition:

$F_{(\theta)} = ICQT \circ U_{(\theta)} \circ CQT$

where $CQT$ denotes the Constant-Q transform, $U_{(\theta)}$ the neural module, and $ICQT$ the differentiable inverse CQT for reconstruction.

At the architectural level, frequency-domain resolution transitions—rather than standard time-domain downsampling—are used to change feature dimensions across octaves, enabling rich modeling of both harmonic content and transient phenomena.

2. Constant-Q Transform Framework

The CQT employed in MR-CQTdiff is an invertible, FFT-based formulation. Frequency bins are spaced logarithmically; for the $k$th bin,

$$f_k = f_{min} \cdot 2^{\frac{(k-1)}{b}}, \quad k = 1, 2, \ldots, K$$

where $b$ is bins-per-octave and $f_{min}$ is the minimum center frequency. Each filter maintains a constant ratio between frequency resolution and center frequency—this establishes an octave-specific time-frequency trade-off:

• Lower frequencies rely on longer filters for higher frequency resolution.
• Higher frequencies use shorter filters for enhanced time resolution.

This octave-adjustable structure is tailored to the inherent properties of musical audio, which presents fine harmonic structures at high frequencies and salient transients or rhythmically dense content at low frequencies.

3. Multi-Resolution Temporal Handling

Standard CQT architectures suffer from poor temporal resolution at low frequencies due to elongated filter impulse responses, leading to smearing of percussive or transient details. MR-CQTdiff introduces multi-resolution CQT processing:

• Parallel CQT transforms with different bins-per-octave settings are computed across octaves.
• Bin count is reduced in lower octaves, resulting in shorter filters and improved time resolution.
• Architectural transitions align feature dimensions across resolutions, including optional $2\times$ time-domain downsampling, allowing concatenation and skip connections to preserve both locality and global structure.

This results in improved representation of rapidly changing sound events (e.g., onsets, pitch modulations), with minimal loss of frequency discrimination, particularly in lower spectral regions.

4. Training Objective and Diffusion Process

The diffusion generative process is structured around denoising score matching, formalized as:

$$L(\theta) = \mathbb{E}_{x_0, \epsilon \sim \mathcal{N}(0, I)} [\lambda(\tau) \cdot \| s_{(\theta)}(x_0 + \sigma(\tau) \epsilon, \tau) - (x_0 - x_t) / \sigma^2(\tau) \|^2_2 ]$$

where $s_{(\theta)}(\cdot, \tau)$ is the estimated score function, $\sigma(\tau)$ represents the noise schedule, and $\lambda(\tau)$ is a temporal weighting function. Conditioning on the diffusion noise level is achieved via noise-level embeddings using Random Fourier Features and a multilayer perceptron (MLP), enabling the model to adaptively process features according to the current noise scale.

5. Empirical Performance and Evaluation

MR-CQTdiff was rigorously evaluated using the Fréchet Audio Distance (FAD) metric, which quantifies perceptual similarity between generated and real audio using CLAP-based embeddings. The following empirical observations were reported (Costa et al., 20 Sep 2025):

Model	Dataset	FAD Median Score	Notable Behavior
MR-CQTdiff	FMA	Lowest	State-of-the-art fidelity
MR-CQTdiff	OpenSinger	Lower than baselines	Stable by 200k iterations
CQTdiff+	FMA	Higher	Less accurate transients
1D U-Net	FMA	Higher	Lacks harmonics/transients
2D STFT U-Net	OpenSinger	Higher	Less vocal detail

MR-CQTdiff outperformed competing architectures (1D U-Net, 2D U-Net with STFT), as well as previous CQTdiff+ and latent diffusion methods, on both FMA and OpenSinger datasets. It exhibited rapid convergence on OpenSinger and consistently superior FAD scores on heterogeneous musical material.

6. Innovations and State-of-the-Art Contributions

Several technical choices underpin MR-CQTdiff’s high fidelity:

• Multi-resolution CQT filter banks circumvent the coarse time resolution at low frequencies endemic to standard CQTs.
• Dual-path U-Net structure with both inner and outer skip connections boosts feature reuse and stability.
• Explicit noise-level conditioning allows adaptation across varying diffusion trajectories.
• Full differentiability and invertibility of the CQT/ICQT chain avoid autoencoder reconstruction penalties and enable strict end-to-end training.

Collectively, these features ensure that MR-CQTdiff maintains both transient and harmonic integrity in generated audio, achieving demonstrably superior perceptual quality.

7. Broader Context and Significance

MR-CQTdiff’s design aligns with contemporary trends in neural audio generation that emphasize spectral domain modeling and robust temporal handling. Its octave-based multiresolution approach constitutes a refinement of CQT-based synthesis techniques, offering a tunable mechanism for managing the time-frequency trade-offs crucial in realistic audio generation. The invertible architecture and empirical superiority across diverse datasets position MR-CQTdiff as a reference point for future developments in trainable, end-to-end diffusion models within the audio domain (Costa et al., 20 Sep 2025).

By effectively enabling expressive synthesis of complex audio through principled spectral analysis and generative modeling, MR-CQTdiff advances both technical methodology and practical capabilities in high-fidelity neural audio generation.