PerformSinger: Multimodal Singing Synthesis

• PerformSinger is a multimodal singing synthesis framework that integrates text, pitch, visual lip cues, and timbre reference to internally predict phoneme durations without external inputs.
• It employs parallel encoders, a visual complementary fusion module, and diffusion-based decoders to accurately model pitch, style, and mel-spectrogram output.
• The framework is validated on the VisualSinger dataset using metrics like MCD, FFE, and MOS, demonstrating advancements in singer conversion and audiovisual synchronization.

PerformSinger most directly denotes a multimodal singing voice synthesis framework that uses synchronized lip cues from singing performance videos to enable high-quality “duration-free” synthesis, meaning that no phoneme-level duration sequence is provided as an input at inference and durations are predicted internally from fused multimodal features. In the broader singer-centered literature represented here, the same label is also used in implementation-oriented descriptions of singer conversion, singer identification, singer separation, speech-to-singing transfer, and streaming singing synthesis. This suggests a broader systems-level usage in which PerformSinger refers to architectures that model singer identity, timing, timbre, accompaniment confounds, and, increasingly, audiovisual synchronization (Gu et al., 24 Sep 2025, Daher et al., 2019, Zhang et al., 2022, Hsieh et al., 2020, Xue et al., 2020, Lin et al., 2021, Eliav et al., 2024, Cui et al., 2024, Li et al., 2024).

1. Problem space and conceptual scope

The named PerformSinger framework addresses a specific bottleneck in contemporary SVS: dependence on externally supplied phoneme-level durations. In conventional FastSpeech-style systems, durations drive a Length Regulator that expands phoneme-level features into frame-level features; in singing, these durations are difficult to obtain because timing, embellishments, and expressive rubato are less regular than in speech. PerformSinger replaces that requirement at inference with a multimodal duration prediction network conditioned on lyrics, pitch, visual lip video, and a timbre reference. Visual lip cues are treated as complementary timing evidence because lip movements encode visemes and syllabic timing tightly coupled with textual content and phonetic realization (Gu et al., 24 Sep 2025).

Across related work, the surrounding research space divides into several recurring formulations. Song-to-song singer conversion replaces the original singer’s timbre in an existing performance while preserving the instrumental accompaniment, timing, phrasing, and overall mix. Singer identification estimates the real singer behind an audio performance and must discount accompaniment and genre confounds. Singer separation isolates one or two lead singers rather than returning a single undifferentiated vocal stem. Speech-to-singing transfer and speech-only SVS ask whether singing can be synthesized from speech corpora, either with a teacher singer, with zero-shot style embeddings, or with end-to-end streaming latent generation. These formulations share a common decomposition into content, singer identity, pitch or timing control, and nuisance factors such as accompaniment, reverberation, or video alignment (Daher et al., 2019, Zhang et al., 2022, Hsieh et al., 2020, Xue et al., 2020, Lin et al., 2021, Eliav et al., 2024, Cui et al., 2024, Li et al., 2024).

2. Multimodal duration-free SVS architecture

PerformSinger comprises parallel multi-branch multimodal encoders for text and pitch, visual lip video, and reference audio timbre; a Visual Complementary Fusion Module; a duration prediction network; a diffusion-based pitch prediction network; a style extraction network using residual vector quantization; a diffusion-based mel-spectrogram decoder; and a HiFi-GAN vocoder. The input streams are a phoneme sequence $T$, a pitch sequence $P$, a lip image sequence $V$, and a short reference audio clip $A_{ref}$. The model computes

$$TF = Enc_T(T), \quad PF = Enc_P(P), \quad CF = TF + PF, \quad SF = Enc_S(A_{ref}), \quad VF = Enc_V(V).$$

Here, the phoneme encoder is FastSpeech2-style with a four-layer Transformer encoder and hidden dimension $256$; the pitch encoder is a learnable embedding layer; the speaker encoder is a d-vector encoder; and the visual encoder is a pretrained lip-reading encoder with a 3D convolution front-end and a ResNet-18 trunk (Gu et al., 24 Sep 2025).

The Visual Complementary Fusion Module progressively fuses $VF$ into the content features $CF$ in an aligned semantic space. First, an adapter projects visual features into the content feature distribution:

$VF' = L_1(\mathrm{GELU}(L_2(VF))),$

with $L_2: \mathbb{R}^{H} \to \mathbb{R}^{2H}$ and $P$0, where $P$1. Then $P$2 residual fusion blocks apply self-attention to both modalities and cross-attention from content queries to visual keys and values:

$P$3

The fused features $P$4 feed a FastSpeech2-style duration predictor, whose outputs are expanded by a Length Regulator. This is the operational meaning of “duration-free”: durations are supervised during training through annotated phoneme boundaries, but no phoneme-level duration sequence is required from the user at inference (Gu et al., 24 Sep 2025).

Pitch and style are modeled separately. PerformSinger adopts a diffusion-based pitch prediction network following RMSSinger to predict frame-wise $P$5 and unvoiced/voiced labels. Style extraction follows StyleSinger and uses residual vector quantization as a bottleneck so that style can be aligned to frame-level content without leaking content from the reference audio. Mel generation follows DiffSinger with a 20-layer residual diffusion decoder. The full objective is

$P$6

where $P$7 is mel-spectrogram reconstruction loss, $P$8 duration loss, $P$9 pitch loss, and $V$0 quantization loss. The implementation uses 80-bin mel-spectrograms, 48 kHz sampling rate, FFT size 1024, hop size 256, Adam with $V$1 and $V$2, two NVIDIA A40 GPUs, and a two-stage hierarchical training strategy in which the visual encoder is frozen in the second stage (Gu et al., 24 Sep 2025).

3. Dataset, evaluation protocol, and empirical profile

PerformSinger is trained and evaluated on VisualSinger, a new SVS dataset with synchronized audio-video singing streams and precise phoneme-level annotations. The corpus is built primarily from the Chinese subset of URSing plus additional Chinese singing videos collected from the web. Construction includes vocal isolation with MDX-KimVocal2 and VR 5HP Karaoke, segmentation with silero-VAD at silence boundaries, manual refinement in Audacity, synchronous extraction of audio and video clips via FFmpeg, Chinese lyric conversion to phonemes via pypinyin, initial alignment with Montreal Forced Aligner, and manual correction in Praat TextGrid by five musically trained annotators. Manual TextGrid annotation is reported as requiring 40–45 human-hours per hour of audio. The resulting dataset contains 9 singers, 69 songs, 1,248 utterances, and total duration $V$3 s ($V$4 hours). Excluding two speakers for out-of-domain testing, the remaining seven speakers are split 8:1:1 into 967 train, 121 validation, and 122 test utterances (Gu et al., 24 Sep 2025).

Objective evaluation uses Mel Cepstral Distortion, F0 Frame Error, Singer Cosine Similarity computed from WavLM speaker embeddings fine-tuned for speaker verification, and Lip-Sync Error via SyncNet, reported as confidence (LSE-C) and distance (LSE-D). Subjective evaluation uses MOS-Q for audio quality, MOS-N for pitch naturalness, MOS-S for timbre similarity, and MOS-M for audio-video synchronization. In Setting 1, where the timbre reference comes from the same clip, PerformSinger reports MCD 3.1125, FFE 0.3921, COS 0.9206, LSE-C 1.4270, LSE-D 10.2782, MOS-Q $V$5, MOS-N $V$6, MOS-S $V$7, and MOS-M $V$8. Under the same setting, StyleSinger reports MCD 3.3580, FFE 0.4494, COS 0.9141, LSE-C 1.3057, LSE-D 10.4129, MOS-Q $V$9, MOS-N $A_{ref}$0, MOS-S $A_{ref}$1, and MOS-M $A_{ref}$2; HPMDubbing and HPMDubbing-P are substantially worse on all reported metrics (Gu et al., 24 Sep 2025).

The ablation study identifies the fusion strategy as central. Stepwise monotonic multihead attention degrades performance relative to the duration-free StyleSinger baseline. VCFM-S, which uses single-stage training, improves the baseline to MCD 3.1307, FFE 0.4022, COS 0.9195, LSE-C 1.3888, and LSE-D 10.3146. VCFM-T, the two-stage training scheme, yields the best results, confirming the benefit of introducing visual fusion after stabilizing the acoustic stack. A common misunderstanding is that “duration-free” eliminates duration supervision entirely; in this framework, the absence of duration inputs applies to inference, not to training, where phoneme boundaries remain essential supervision (Gu et al., 24 Sep 2025).

4. Conversion and synthesis lineages

One major antecedent is SCM-GAN, an end-to-end non-parallel song conversion system that separates vocals and instrumentals with a U-Net, converts the vocal segments to the target singer using CycleGAN-VC, and merges the converted vocals with the background music. The system is initialized from a state-of-the-art voice-to-voice conversion model and trained on 228 non-parallel vocal segments of approximately 3 seconds each. Evaluation uses Global Variance and Modulation Spectra over 24 Mel-cepstral coefficients. With transfer learning, GV RMSE to target is $A_{ref}$3 versus $A_{ref}$4 from scratch, and MS RMSE to target is $A_{ref}$5 versus $A_{ref}$6. In subjective tests, Split-Convert-Merge reaches 3.46/5 similarity to the target and 2.68/5 naturalness, compared with 2.75/5 and 1.55/5 when training directly on full songs. The reported average improvements are approximately 35% for GV and 13% for MS, with similarity to target increasing by 26% and naturalness by 73% when separation is used (Daher et al., 2019).

A second lineage, Learn2Sing, removes the need for target-speaker singing data. It trains on a teacher’s singing corpus and speech from multiple target speakers, using a frame-level autoregressive acoustic model in which singing and speaking share a speaker embedding and a style tag embedding. Because target speakers have no musical-score annotations, Learn2Sing uses log-scale fundamental frequency as an auxiliary feature and applies domain adversarial training to disentangle style from acoustic features. It also trains a duration model and an LF0 prediction model so that no singing reference audio is required at inference. Reported objective results include Duration Accuracy 85.43%, LF0 prediction PCC 0.8835, and RMSE 25.97. The paper states that DAT markedly improves singing naturalness, expressiveness, and timbre similarity for target speakers (Xue et al., 2020).

A third line, SingIt!, emphasizes zero-shot, many-to-many style transfer from ordinary speech. Its pipeline uses Spleeter for vocal separation, Resemblyzer for a 256-dimensional style embedding, a modified AutoVC-like encoder-decoder operating on $A_{ref}$7 log-STFT spectrograms, and Griffin-Lim for waveform reconstruction. The training objective is

$A_{ref}$8

with $A_{ref}$9, where $$TF = Enc_T(T), \quad PF = Enc_P(P), \quad CF = TF + PF, \quad SF = Enc_S(A_{ref}), \quad VF = Enc_V(V).$$0 and $$TF = Enc_T(T), \quad PF = Enc_P(P), \quad CF = TF + PF, \quad SF = Enc_S(A_{ref}), \quad VF = Enc_V(V).$$1 are reconstruction MSE terms and $$TF = Enc_T(T), \quad PF = Enc_P(P), \quad CF = TF + PF, \quad SF = Enc_S(A_{ref}), \quad VF = Enc_V(V).$$2 is a latent consistency $$TF = Enc_T(T), \quad PF = Enc_P(P), \quad CF = TF + PF, \quad SF = Enc_S(A_{ref}), \quad VF = Enc_V(V).$$3 loss. In a subjective listening test with 25 non-expert listeners, word intelligibility is $$TF = Enc_T(T), \quad PF = Enc_P(P), \quad CF = TF + PF, \quad SF = Enc_S(A_{ref}), \quad VF = Enc_V(V).$$4, melody similarity to the source is $$TF = Enc_T(T), \quad PF = Enc_P(P), \quad CF = TF + PF, \quad SF = Enc_S(A_{ref}), \quad VF = Enc_V(V).$$5, resemblance to the target style is $$TF = Enc_T(T), \quad PF = Enc_P(P), \quad CF = TF + PF, \quad SF = Enc_S(A_{ref}), \quad VF = Enc_V(V).$$6, and human-like naturalness is $$TF = Enc_T(T), \quad PF = Enc_P(P), \quad CF = TF + PF, \quad SF = Enc_S(A_{ref}), \quad VF = Enc_V(V).$$7. This positions SingIt! as a practical zero-shot transfer system, but also shows the persistent difficulty of naturalness (Eliav et al., 2024).

5. Singer identification and singer separation usages

In singer identification, PerformSinger is used in the material as a robust SID pipeline for metaverse settings. The underlying model combines frame-level mel-spectrograms, a 512-dimensional X-vector timbral feature, and middle-level perceptual embeddings from an Inception V3 network. Fusion is implemented with a CRNN that first combines mel-spectrogram and timbral features and then concatenates middle-level features before two GRU layers and a final classifier over the 20 artists of Artist20. Under album-split evaluation, the best model, CRNN + X-vector + L4 middle-level features, reaches F1/best 0.86 and F1/avg 0.81. The ablation study reports CRNN + X-vector at 0.71 average F1, L3 at 0.80, L4 at 0.81, and L5 at 0.78, while combining multiple middle-level layers can reduce performance, for example L3+L4 at 0.70 and all layers at 0.77. The paper attributes this degradation to possible overfitting or multicollinearity (Zhang et al., 2022).

A complementary identification line focuses on accompaniment confounds. This work uses open-unmix to separate vocal and instrumental tracks and compares training on vocal-only stems, on shuffle-and-remix mixtures, and on a combined Data aug setting that unions Origin, Vocal-only, and Remix. It also adds melodic features from CREPE-based vocal melody contours through a CRNNM architecture. On Artist20 with album split, CRNNM + Data aug reaches song-level F1 0.75, improving over the CRNN + Origin baseline at 0.67. Segment-level F1 remains lower, and a reported correlation coefficient of 0.39 links correct-singer likelihood to higher “vocalness,” indicating that non-vocal segments remain a significant error source (Hsieh et al., 2020).

Singer separation extends the problem from singer recognition to source decomposition. For karaoke content generation, a two-stage system first applies Wave-U-Net+ for vocal-accompaniment separation and then uses DPRNN or DPTNet models specialized for English duets, Chinese duets, or English self-harmonic singing. The MIR-SingerSeparation dataset contains 976 songs total, 476 English and 500 Chinese, all converted to 8 kHz and segmented into 10-second excerpts. On English duet data, a direct DPRNN 3-channel baseline yields SI-SNRi 3.24 dB and SDRi 4.04 dB, whereas the two-stage SSSYS with DPRNN reaches SI-SNRi 8.27 dB and SDRi 8.78 dB, and SSSYS with DPTNet reaches SI-SNRi 9.37 dB and SDRi 8.89 dB. An automatic model-selection heuristic based on CREPE pitch-trend similarity achieves 71.43% accuracy on 14 real karaoke APP songs and an average SI-SNRi 8.95 dB across all selections (Lin et al., 2021).

6. Streaming synthesis and audiovisual generation

Streaming SVS introduces a different interpretation of performer-centered synthesis. CSSinger is a fully end-to-end chunkwise streaming singing voice synthesis system based on a conditional variational autoencoder. It replaces the conventional acoustic-model-plus-vocoder cascade with latent waveform synthesis using a ChunkStream Decoder, a causal posterior encoder, a causal HiFi-GAN generator, a Natural Padding strategy that uses real latent values from prior context instead of constant one-sided padding, and a Causal Smooth Layer to reduce chunk-boundary artifacts. The reported configuration uses chunk size 20 frames, left context 10, and right context 4. On a V100 GPU, CSSinger-FS reaches latency 0.051 s, process time 0.476 s, and RTF 0.099; on a 10-core Xeon CPU, latency is 0.483 s and RTF 0.347. On Opencpop, CSSinger-FS reports MOS $$TF = Enc_T(T), \quad PF = Enc_P(P), \quad CF = TF + PF, \quad SF = Enc_S(A_{ref}), \quad VF = Enc_V(V).$$8, F0 RMSE 28.601, F0 Corr 0.919, U/UV Err 0.107, MSE 1.093, and MCD 6.715 (Cui et al., 2024).

Audiovisual generation extends the singer-centered problem from sound to visible performance. SINGER is an audio-driven singing video generation model that augments a latent diffusion backbone with a Multi-scale Spectral Module and a Self-adaptive Filter Module. MSM decomposes audio embeddings with Haar wavelets into $$TF = Enc_T(T), \quad PF = Enc_P(P), \quad CF = TF + PF, \quad SF = Enc_S(A_{ref}), \quad VF = Enc_V(V).$$9, $256$0, $256$1, and $256$2 components and reweights them according to the current video latent, while SFM applies a wavelet-domain gating mechanism to visual feature maps inside the denoising UNet. The SHV dataset assembled for this task contains more than 200 videos, approximately 20 hours total duration, and more than 200 subjects. On SHV, SINGER reports SSIM 0.6364, PSNR 30.686, CPBD 0.5430, FVD 503.78, LMD 53.373, LSE-D 9.1269, LSE-C 1.6209, Diversity 14.445, and BAS 0.2405, outperforming the compared methods on FVD and BAS. This line of work is directly relevant to PerformSinger because the named multimodal SVS framework already treats synchronized lip cues as a first-class signal, and SINGER shows a parallel trend in which singing-specific audiovisual synchronization is no longer modeled as a simple extension of talking-face generation (Li et al., 2024).

7. Limitations, misconceptions, and ethical issues

Several limitations recur across the literature. The named PerformSinger framework depends on a relatively small Mandarin-only corpus of approximately 3.07 hours and requires labor-intensive manual phoneme-boundary annotation. Robustness to occlusions, off-axis views, and noisy lip tracks is not explicitly evaluated. This implies that “duration-free” at inference does not remove dependence on highly curated supervision during training (Gu et al., 24 Sep 2025).

A second recurring limitation is artifact formation. In SCM-GAN, using mixture phase in U-Net reconstruction can introduce musical artifacts, and WORLD can produce buzzy artifacts under extreme AP/F0 manipulations. In SingIt!, explicit F0, tempo, and vibrato modeling are absent, and the mean naturalness rating of $256$3 indicates that style transfer alone does not guarantee convincing singing. In CSSinger, expressiveness is captured implicitly in latent $256$4, but the reported datasets are single-singer, so multi-singer generalization remains outside the demonstrated scope (Daher et al., 2019, Eliav et al., 2024, Cui et al., 2024).

Ethical concerns are explicit in the singer-conversion literature. Singer identity cloning raises consent and licensing issues, and explicit rights and consent from target singers or public-domain data are recommended. The same material also recommends compliance with platform policies and regional laws on synthetic voice usage and disclosure of synthesized content to avoid impersonation harms. In the audiovisual setting, the use of performance videos also raises privacy and consent questions. A plausible implication is that future PerformSinger systems will need both stronger multimodal modeling and clearer governance over identity, attribution, and disclosure (Daher et al., 2019, Gu et al., 24 Sep 2025).