SonicMaster: Unified Music Restoration

• SonicMaster is a unified generative framework for music restoration and mastering that uses text instructions and audio cues to correct multiple degradation artifacts.
• It employs a VAE-based latent encoding and a flow-matching training paradigm to integrate text-guided and automated enhancement with consistent quality.
• The system efficiently restores equalization, dynamics, reverberation, amplitude, and stereo artifacts, validated by objective metrics and subjective listening tests.

SonicMaster is a unified generative framework for music restoration and mastering, designed to correct a broad range of audio degradations in music recordings through a single, text-controlled or automatic interface. Distinguishing itself from conventional pipelines reliant on cascaded, task-specific modules, SonicMaster applies a generative flow-matching paradigm to jointly address equalization, dynamic range, reverberation, amplitude, and stereoization artifacts. It leverages large paired datasets of degraded and high-quality tracks, and enables both targeted, prompt-driven correction and fully automated enhancement. This system provides a new paradigm for music production by enabling direct, interpretable control over restoration and mastering via natural language, simplifying workflows for both professionals and non-experts.

1. Unified Generative Model Architecture

SonicMaster employs a single generative model encompassing restoration and mastering functionalities that were traditionally addressed by a series of specialized tools. The model ingests stereo audio waveforms (sampled at 44.1 kHz), which are encoded into a compact spectro-temporal latent space via a variational autoencoder (VAE) codec. This facilitates the modeling of long-range temporal dependencies with reduced computational cost.

The network adopts a dual conditioning mechanism:

• Text-Based Control: Natural language instructions, embedded using a frozen FLAN-T5 encoder, allow users to specify desired enhancements (e.g., “remove distortion,” “increase treble,” “reduce reverb tail”).
• Audio Cue Conditioning: An optional pooled audio branch injects a clean reference segment (typically 5–15 sec) into every layer, maintaining intra-track consistency, particularly during chunked inference of long recordings.

The architecture is constructed from multimodal DiT (MM-DiT) blocks, which integrate textual and latent audio representations. The model’s output is a “velocity” vector in the latent domain, directing the degraded embedding toward the target mastered embedding. For inference, velocities are integrated iteratively (e.g., using forward Euler) to synthesize the restored output.

Modes of operation include:

• Text-guided restoration and mastering, where corrective actions are dictated by user-supplied prompts.
• Automatic (auto-correction) mode, which invokes learned perceptual heuristics for end-to-end mastering without explicit prompt input.

2. Dataset Construction and Degradation Simulation

Training relies on the SonicMaster dataset, which comprises high-fidelity/degraded music pairs with associated text prompts. Source data are curated from approximately 580,000 Jamendo recordings, covering 10 musical genres, with about 25,000 selected as high-quality 30 sec canonical segments.

Degradation simulation applies one to three out of nineteen distinct degradation functions, grouped as follows:

Enhancement Group	Example Degradations
Equalization	Brightness, muddiness, microphone coloration
Dynamics	Compression, “punch” (transient shaping)
Reverb	RT60 extension, real/virtual room IRs
Amplitude	Clipping, volume attenuation
Stereo	Collapse to mono, spatial narrowing

Both algorithmically parametrized transformations (e.g., EQ curves, dynamic range compression) and empirical impulse responses (from Pyroomacoustics or real environments) are used. Each degraded sample is paired with a natural language description of the artifact and requested fix, enabling supervised learning for both restoration and controllable enhancement.

3. Flow-Matching Training Paradigm

SonicMaster is trained to perform an audio transformation from degraded latent representations ($x_1$) to clean/mastered ones ($x_0$) in a text-conditional manner using flow-matching. For each sample, an interpolated latent $x_t$ is computed as:

$x_t = t \cdot x_1 + (1-t) \cdot x_0$

where $t \in [0,1]$ is drawn to emphasize more challenging (more degraded) inputs with $t$ closer to $1$. The model is trained to predict the “velocity” in latent space:

$v_t = x_0 - x_1$

with the prediction $f_\theta(x_t, t, c_\text{text}) \approx v_t$. The loss is:

$$L(\theta) = \mathbb{E}_{t,x_1,x_0}\left[ \| f_\theta(x_t, t, c_\text{text}) - v_t \|_2^2 \right]$$

Classifier-free guidance is implemented by randomly omitting or replacing text prompts during training, thus increasing robustness for both conditioned and unconditioned inference.

During inference, an integration procedure (e.g., forward Euler) is used to “walk” from the degraded to clean latent, applying the predicted velocity vector in successive steps:

$x_{t+h} = x_t + h \cdot \hat{v}_t$

This process reconstructs the mastered waveform from the latent representation.

4. Objective and Subjective Evaluation

SonicMaster’s enhancements are validated with both objective signal quality measures and subjective listening tests.

• Objective metrics:
  • Fréchet Audio Distance (FAD) using CLAP embeddings for global perceptual similarity.
  • Kullback–Leibler (KL) divergence and SSIM on 128-bin Mel-spectrograms.
  • Production Quality (PQ) score.
  • Degradation-specific metrics: cosine distance for X-band EQ, energy ratios for EQ artifacts, onset envelope statistics for dynamics (“punch”), and modulation spectra for reverb effects.
• Subjective listening tests:
  • Eight participants (five music experts) rated paired degraded and restored samples for text relevance, pre- and post-restoration quality, consistency, and overall preference.
  • SonicMaster outputs were consistently rated better than original degraded audio, with largest gains reported for reverb attenuation and amplitude correction.

The system addresses the complex interdependence of quality degradations. For instance, the unified model can simultaneously suppress excessive reverb, mitigate clipping, and rebalance stereo image—tasks that in traditional workflows often require sequential or iterative processing.

5. Technical Implementation Details

Key technical aspects of SonicMaster include:

• Latent Space Operations: Use of a VAE codec to encode 44.1 kHz stereo into manageable spectro-temporal latents, balancing high modeling capacity with tractable memory and compute loads.
• Chunked and Overlapping Inference: For long tracks, chunked inference (e.g., overlapping 30 sec segments) with continuity cues prevents boundary artifacts and maintains cross-chunk consistency.
• Multimodal Conditioning: Concatenation and joint processing of text and audio cues in MM-DiT blocks facilitate precise and interpretable control over enhancement operations.
• Classifier-Free Guidance: Random prompt dropout increases the robustness and versatility for both prompted and auto-correction modes.
• Integration Procedure: Explicit velocity integration for denoising and enhancement ensures that corrections are continuous in latent space, which is critical for avoiding discontinuities or “musical” artifacts.

6. Applications and Broader Implications

SonicMaster is applicable to a range of professional and consumer scenarios, including:

• Audio Restoration and Remastering: Restoration of archival or live audio affected by compounded degradations; mastering of home or venue recordings lacking professional studio resources.
• User-Guided Enhancement: Empowerment of non-experts to “master” recordings by describing desired outcomes in natural language, effectively abstracting low-level technicalities (e.g., specific EQ parameters, dynamics settings).
• Workflow Simplification: Replacement of cascaded, task-specific audio tools and iterative manual adjustments with a singular, interpretable framework.
• Joint Artifact Correction: Demonstrates feasibility of learning transformation across multiple interconnected degradations jointly, which can lead to more coherent and less artifact-prone results than task-isolated processing.
• Research Directions: Paves the way for further investigation into latent generative models for audio, integration with less lossy latent representations (to address occasional “robotic” vocal artifacts), and new paradigms for high-level controllability in audio post-production.

A plausible implication is that text-conditional generative restoration frameworks may standardize interfaces for audio enhancement tasks across fields (music, speech, broadcast) and improve accessibility of high-quality mastering irrespective of resources or expertise.

7. Limitations and Prospects

While SonicMaster demonstrates significant advances, certain limitations are acknowledged:

• Occasional Artifacts: The use of a VAE latent space can introduce artifacts such as “robotic” vocal coloration in some failure modes. This suggests further work into less lossy or alternative latent audio representations is warranted.
• Dependency on Prompt Quality: The precision of text-guided enhancement currently depends on the accuracy and specificity of the natural language input.
• Future Extensions: Research directions include extending the range of correctable artifacts, improving audio fidelity at the latent-reconstruction interface, and advancing the integration with interactive, prompt-driven interfaces for both music professionals and general users.

SonicMaster exemplifies how unifying multiple restoration and mastering tasks within a generative, prompt-controllable paradigm offers marked benefits in flexibility, efficiency, and interpretability, and sets a precedent for future systems in automatic and user-driven music enhancement (Melechovsky et al., 5 Aug 2025).