Complex TF-Domain Super-Resolution

• Complex TF-domain super-resolution is a neural approach that restores missing high-frequency spectral content by directly modeling both magnitude and phase.
• CTFT-Net uses a deep U-Net architecture with complex convolutions, global attention, and skip connections to preserve fine temporal and spectral structures.
• Empirical evaluations demonstrate significant improvements in metrics such as LSD, PESQ, and SI-SDR, especially at extreme upsampling ratios.

Complex time–frequency (TF) domain super-resolution refers to a class of neural techniques for inferring missing high-frequency spectral content from low-resolution audio, reconstructing both magnitude and phase information directly in the complex spectrogram domain. Distinguished from prior approaches that focus on magnitude (or use band-concatenation heuristics), state-of-the-art models such as CTFT-Net perform end-to-end learning on complex-valued spectrograms and jointly optimize time-domain and frequency-domain objectives. This paradigm achieves high-fidelity recovery of speech and music, demonstrating improvements over both real-valued and magnitude-only methods, especially for extreme upsampling ratios and in scenarios where phase coherence critically impacts perceptual quality (Tamiti et al., 30 Jun 2025, Mandel et al., 2022).

1. Mathematical Foundations of Complex TF-Domain Representation

The process begins by converting a time-domain waveform $x[n]$ into its Short-Time Fourier Transform (STFT), yielding a complex-valued spectrogram $S(t, f) = X_r(t,f) + j X_i(t,f)$, where $t$ is frame index, $f$ denotes frequency bin, and $N$ is the FFT length: $$X(t,f) = \sum_{m=0}^{N-1} x[tH+m] w[m] e^{-j2\pi f m/N}$$ Inverse STFT reconstructs $\hat x[n]$ via overlap–add from estimated spectrograms. In complex TF-domain super-resolution, the neural network operates directly on (or predicts) $S(t,f)$, explicitly handling both $X_r$ and $X_i$. After every complex operation, normalization (CBN) and nonlinearity through CReLU are applied: $$\text{CReLU}(X_r + j X_i) = \mathrm{ReLU}(X_r) + j\,\mathrm{ReLU}(X_i)$$

2. Architectural Innovations for Complex Spectrogram Modeling

CTFT-Net exemplifies the most elaborate architecture for this task. It is structured as a deep U-Net operating in the complex TF domain:

• Encoders and Decoders: Eight level encoder-decoder stacks, using complex 2D convolutions with progressive downsampling/restoration in frequency and time.
• Complex Conformer Bottleneck: A complex-valued Conformer module processes spectral “bottleneck” representations, modeling both long-range and local spectro-temporal dependencies using multi-head self-attention (MHSA) and convolutional blocks.
• Complex Global Attention Block (CGAB): Placed after selected encoders, the CGAB aggregates features independently along time and frequency axes, calculating global non-local dependencies between phonetic and harmonic components.
• Skip Connections and Complex Skip-Blocks: Each encoder output is projected via a skip-block (complex conv + norm + CReLU) before summing into the matching decoder layer, preserving fine temporal and spectral structure.

The architecture is designed to directly predict the full frequency range of the complex spectrogram, eschewing explicit band-splitting or band-concatenation tactics used in previous super-resolution models (Tamiti et al., 30 Jun 2025, Mandel et al., 2022).

3. Loss Functions Integrating Time and Frequency Domains

Success in complex TF-domain super-resolution hinges on loss functions that enforce both temporal and spectral consistency:

• Scale-Invariant SDR Loss ($\mathcal L_{\mathrm{SI\text{-}SDR}}$): Evaluates time-domain reconstruction quality via the scale-invariant signal-to-distortion ratio.
• Multi-Resolution STFT Loss ($L_{\mathrm{MR\text{-}STFT}^r}$): Imposes constraints at multiple spectral granularities by measuring spectral convergence and log-magnitude errors across several STFT settings.

$$\mathcal L_{\rm total} = \mathcal L_{\rm SI\text{-}SDR} + L_{\mathrm{MR\text{-}STFT}^r}$$

This combination compels the network to produce reconstructions that are perceptually coherent in waveform and spectrogram metrics without need for post-hoc vocoders or phase heuristics.

4. Empirical Results and Performance Evaluation

CTFT-Net has been evaluated on extensive speech SSR experiments (VCTK corpus, 48 kHz target), with low-rate signals prepared by low-pass filtering and downsampling to $\{2,4,8,12\}$ kHz, then upsampled to 48 kHz for STFT computation. Metrics such as Log Spectral Distance (LSD ↓), Short-Time Objective Intelligibility (STOI ↑), Perceptual Evaluation of Speech Quality (PESQ ↑), and SI-SDR ↑ are used.

Method / Input→Output	2 kHz → 48 kHz	4 kHz → 48 kHz	8 kHz → 48 kHz	12 kHz → 48 kHz
Unprocessed	3.06	2.85	2.44	1.34
NU-Wave	1.85	1.48	1.45	1.27
WSRGlow	1.45	1.18	1.02	0.91
NVSR	1.10	0.99	0.93	0.87
AERO	1.15	1.09	1.01	0.93
CTFT-Net	1.06	0.96	0.81	0.62

LSD reductions reach 66% for 2 kHz upsampling. PESQ shows 28–47% improvement; STOI remains stable, indicating no loss of intelligibility. SI-SDR improves slightly, suggesting the network does not introduce additional artifacts (Tamiti et al., 30 Jun 2025).

5. Core Challenges and Model Capabilities

Major SSR challenges include reconstructing fine high-frequency details without musical noise, handling phase directly (since real-valued methods typically ignore phase), and avoiding seam artifacts between observed and synthesized bands. Complex TF-domain approaches offer:

• Joint modeling of magnitude and phase, minimizing reliance on separate vocoders.
• CGABs for non-local time–frequency dependencies within the spectrogram.
• Robustness to extreme low-pass inputs, e.g., 2 kHz→48 kHz upsampling, without artifacts.

The use of cross-domain objectives, complex batch normalization, and activations enhances convergence and generalization, promoting robustness to temporal and spectral distortions.

6. Future Research Directions

Potential avenues include adaptive or trainable filterbanks for variable upsampling ratios, hybrid models that combine diffusion or GAN priors in the complex TF domain, and complex spatio-temporal self-attention across spectrogram sequences. These expansions are anticipated to further enhance fidelity and flexibility in both speech and general audio super-resolution (Tamiti et al., 30 Jun 2025, Mandel et al., 2022).