DAFMSVC: One-Shot Singing Voice Conversion

• DAFMSVC is an advanced one-shot singing voice conversion system that transforms a source voice to a target timbre while preserving melody and lyrics.
• It employs a frame-level SSL feature replacement using a matching pool strategy to minimize timbre leakage and maintain linguistic integrity.
• The system integrates dual cross-attention fusion and a conditional flow matching module to optimize high-fidelity audio synthesis with superior objective and subjective performance.

DAFMSVC is an advanced one-shot singing voice conversion (SVC) system designed for “any-to-any” transfer of timbre, enabling the transformation of a source singer’s voice to match that of a target singer while preserving melody and linguistic content. DAFMSVC integrates feature replacement via self-supervised learning (SSL), a dual cross-attention mechanism for adaptive fusion of speaker and musical attributes, and a conditional flow matching (CFM) generative model for high-fidelity audio synthesis. The architecture addresses fundamental challenges in SVC, especially timbre leakage and audio quality degradation, and achieves state-of-the-art subjective and objective performance in song similarity, naturalness, pitch correlation, and loudness consistency (Chen et al., 8 Aug 2025).

1. Problem Formulation and System Overview

In singing voice conversion, the central task is mapping a source vocal recording $A_s$ onto a synthesized output $A_{t\gets s}$ that preserves the original melody and lyrics but adopts the timbre of a target singer. DAFMSVC approaches this as a one-shot “any-to-any” SVC problem: the system must generalize to unseen speaker timbres with minimal reference data, typically a single clip from the target singer. The model’s core innovation is replacing the SSL features of the source audio—normally prone to carry timbre information—with those most similar (by cosine distance) from the target’s reference pool, thus ideally preventing timbre leakage. The feature replacement is followed by a dual cross-attention fusion of content, pitch, and speaker identity, conditioning audio generation on these attributes through a CFM module designed to optimize the synthesis trajectory in latent space.

2. SSL Feature Replacement and Matching Pool Strategy

DAFMSVC uses SSL features extracted from a pre-trained WavLM-large encoder. These features are highly informative for both linguistic content and timbre. To avoid timbre leakage, the approach implements a matching pool: given the source SSL feature sequence $\{f_s\}$, the system searches for the closest-matching SSL features $\{f_t^*\}$ from the target pool using a $k$-nearest neighbor strategy and cosine similarity, averaging representations from the last layers of WavLM to ensure timbre is captured robustly. The replacement is performed at the frame level:

• For each source frame, identify the target frame whose SSL feature is maximally similar under the selected metric.
• Replace the source SSL features with those of the best-matched target, yielding a hybrid representation that retains source linguistic/melodic content but imposes the target’s timbre.

This mechanism is critical for enforcing strict timbre transfer and suppressing residual source vocal characteristics, a common failure mode in prior architectures.

3. Dual Cross-Attention Fusion Mechanism

The fusion module comprises a dual cross-attention network responsible for selectively injecting speaker, content, and melody information. Inputs are:

• Content representation $C$
• Speaker embedding $S$ (from a verification model)
• Melody features $P$ (pitch, loudness)

Cross-attention is performed over the content queries, attending both to speaker (timbre) and melody axes: $$O = \text{softmax}\left( \frac{C_Q P_K^T}{\sqrt{d}} \right) P_V + \tanh(\alpha) \cdot \text{softmax}\left( \frac{C_Q S_K^T}{\sqrt{d}} \right) S_V$$ where $d$ is the query dimensionality and $\alpha$ is a learnable scalar gating parameter (initialized at zero for stable training). $C_Q$ denotes content query, $P_K$, $P_V$ are melody keys/values, $S_K$, $S_V$ are speaker keys/values. This structure allows adaptive and stable fusion—$\tanh(\alpha)$ acts as a gate controlling the degree of speaker embedding injection, mitigating instability or overshooting during training. The final fused output encapsulates timbre, melody, and linguistic content suitable for conditional generative modeling.

4. Conditional Flow Matching Generative Module

DAFMSVC employs a CFM module for waveform reconstruction, leveraging an ODE-based flow matching approach. The dynamical system describes the trajectory between two distributions $\pi_0$ (prior, typically noise or latent vector) and $\pi_1$ (target): $\frac{dZ_t}{dt} = v(Z_t, t)$ where $Z_0 \sim \pi_0$, $Z_1 \sim \pi_1$, and $v$ is a neural velocity field conditioned on $C$. Training minimizes the flow matching loss: $$L_{rf} = \mathbb{E}_{X_0 \sim \pi_0, (X_1, C) \sim D}\left[ \int_0^1 \left\| \frac{X_1 - X_0}{\sigma} - \frac{\nu(X_t, t | C)}{\sigma} \right\|^2 dt \right]$$ with $\sigma = \sqrt{\mathrm{Var}_1(X_1 - X_0)}$ and $X_t$ as linear interpolation. Auxiliary losses are employed:

• Overlap loss $L_{overlap}$ (reducing prediction discrepancies across consecutive frames)
• STFT loss $L_{stft}$ (mitigating background noise and enhancing spectral fidelity)

The overall objective is: $L = L_{rf} + \lambda ( L_{overlap} + L_{stft} )$ with $\lambda = 0.01$. This module enables direct synthesis of natural, high-fidelity audio from fused representations.

5. Training and Evaluation Protocols

The system is trained and evaluated on the OpenSinger dataset (50 hours, Chinese singing, male and female singers). Melody features are extracted at $24$ kHz. Speaker embeddings are provided by CAM++. DAFMSVC is compared against NeuCoSVC, DDSP-SVC, and So-VITS-SVC, all using identical preprocessing and model selection protocols. Objective measures include:

• Singer similarity (SSIM)
• Pitch correlation (F0CORR)
• Loudness RMSE
• Mel Cepstral Distortion (MCD)

Subjective listening tests (MOS-Similarity, MOS-Naturalness) consistently favor DAFMSVC over baselines, indicating notable improvements in both target timbre similarity and overall perceptual quality. Ablation studies confirm the essentiality of dual cross-attention and speaker embeddings, as their removal leads to significant drops in performance.

6. Technical Significance and Future Directions

DAFMSVC addresses core limitations of previous any-to-any SVC systems:

• The matching pool strategy solves timbre leakage without requiring multi-shot adaptation or extensive speaker data.
• Dual cross-attention fusion integrates speaker and musical content with stability and adaptivity.
• CFM provides principled waveform generation aligned with fused representation statistics.

A plausible implication is that robust one-shot SVC frameworks are becoming increasingly reliant on large-scale SSL features and attention-based fusion. Future research may pursue model efficiency, reduced computational cost, and greater robustness to noisy or mismatched recording conditions. Improvements in SSL extraction or matching strategies could further enhance timbre fidelity. Extension to complex acoustic scenarios is also a logical direction.

In summary, DAFMSVC establishes a high-performance standard for one-shot singing voice conversion by fusing state-of-the-art SSL, attention, and flow matching methodologies. The modular design allows further refinement and scaling for diverse application environments (Chen et al., 8 Aug 2025).