LAPS-Diff: Diffusion SVS for Hindi Singing

• LAPS-Diff is a diffusion-based singing voice synthesis framework that targets low-resource Bollywood Hindi singing by using language-aware prosody and conditional priors.
• It fuses IndicBERT and XPhoneBERT embeddings with auxiliary style and pitch supervision to address alignment, expressiveness, and data scarcity challenges.
• Empirical results show improved metrics and listener ratings compared to DiffSinger, underscoring its enhanced acoustic refinement and prosody modeling.

LAPS-Diff is a diffusion-based Singing Voice Synthesis (SVS) framework designed for low-resource, Bollywood-style Hindi singing. It extends DiffSinger’s two-stage SVS design by combining language-aware word and phone embeddings, prosody-style guided supervision in the auxiliary decoder, and audio-derived conditional priors in the diffusion denoiser. The framework targets three coupled difficulties: singer-specific style and expressiveness, language-dependent prosody and alignment in Hindi singing, and data scarcity arising from a corpus of only about an hour of labeled singing (Dhar et al., 7 Jul 2025).

1. Problem setting and research context

LAPS-Diff is formulated for SVS rather than speech TTS or generic audio generation. Its stated target is Bollywood Hindi singing, where long vowels, stylistic stress patterns, slides, melismas, and language-specific schwa behavior complicate lyrics alignment, phone duration modeling, and pitch contour generation. The framework is explicitly motivated by the observation that existing SVS systems such as DiffSinger, Learn2Sing 2.0, MakeSinger, and VISinger2 tend to sound less expressive when trained on small datasets, and that Hindi singing introduces phonotactic and prosodic phenomena that differ from speech and from the better-resourced SVS languages emphasized in prior work (Dhar et al., 7 Jul 2025).

The paper situates the problem in a low-resource regime with no large, clean, labeled Hindi singing corpus and with singing acoustics that differ substantially from speech in timing, pitch, and articulation. LAPS-Diff addresses this by enriching text and score conditioning with IndicBERT and XPhoneBERT embeddings, supervising style and pitch through auxiliary losses, and injecting MERT and IndicWav2Vec embeddings as conditional priors for diffusion-based acoustic refinement.

A plausible implication is that the method is not primarily an architectural departure from diffusion SVS in the abstract, but a targeted reconfiguration of conditioning and supervision to make shallow diffusion viable under extreme data constraints.

2. System architecture and diffusion formulation

LAPS-Diff inherits DiffSinger’s two-stage structure. The first stage is an auxiliary decoder that transforms content into a baseline mel-spectrogram. The second stage is a denoiser, implemented as a diffusion model, that refines this mel-spectrogram through a shallow diffusion process. A HiFi-GAN vocoder converts the final mel-spectrogram to waveform (Dhar et al., 7 Jul 2025).

The model operates on 80-bin mel-spectrograms with frame size 512, hop size 128, and normalization to $[-1,1]$. Inputs include lyrics in Hindi text, phoneme sequence in the IIT-M Hindi phoneset, MIDI note, note duration, phone duration, and slur information. During inference, no reference audio is required.

The diffusion component follows standard DDPM-style forward and reverse processes, but in the shallow-diffusion form used by DiffSinger. During inference, the auxiliary decoder first predicts a baseline mel $\hat{M}$. Noise is then added to $\hat{M}$ to obtain $\hat{M}_q$ for a fixed $q=60$ out of $T=100$ steps, and denoising proceeds from $M_q$ to $M_0$. The denoiser is conditioned not only on content and timestep embeddings but also on musical and contextual priors:

$$\hat{\epsilon}_\theta = D(M_t, \mathbf{e}_c, \mathbf{e}_t, \mathbf{e}_{mert}, \mathbf{e}_{vec}).$$

The denoising loss is

$$\mathcal{L}_{\text{denoise}} = \mathrm{MAE}(\epsilon, \hat{\epsilon}_\theta).$$

Architecturally, the denoiser is a non-causal WaveNet-style convolutional network, as in DiffSinger. The paper conceptually describes MERT and IndicWav2Vec features as extra conditioning, with inner wiring following the DiffSinger codebase.

This decomposition suggests a division of labor: the auxiliary decoder handles coarse acoustic structure and alignment-sensitive prediction, while shallow diffusion performs conditional refinement.

3. Language-aware content representation

A central component of LAPS-Diff is its enriched content embedding. The framework uses IndicBERT as a text encoder and XPhoneBERT as a phone encoder. IndicBERT receives Devanagari lyrics and outputs contextual word-level embeddings $\hat{M}$0. XPhoneBERT receives the phoneme sequence and outputs contextual phone-level embeddings $\hat{M}$1. Both are projected to 256 dimensions and summed with the score encoder output $\hat{M}$2:

$\hat{M}$3

This fused representation is then passed to the FastSpeech2-like auxiliary decoder (Dhar et al., 7 Jul 2025).

The paper attributes distinct functions to the two linguistic encoders. Word-level embeddings capture lexical context, word boundaries, and syntactic or semantic cues that influence phrasing and stress placement. Phone-level embeddings capture detailed phonemic context, including coarticulation and vowel-consonant structure relevant to syllable timing and rhythm. Because the auxiliary decoder is trained with phone-level duration, pitch, and energy predictors in the FastSpeech2 style, the system is intended to learn Hindi-specific timing and prosody, including longer vowels in singing, optional schwa insertions at word-final consonants, and note-to-syllable mismatches with slurs.

The paper does not describe an explicit attention mechanism for this alignment problem. Instead, alignment is learned end-to-end through duration predictors for words, phones, and slurs. This is an important design choice: LAPS-Diff treats alignment as a structured prediction problem mediated by duration supervision rather than an unconstrained cross-modal attention problem.

4. Prosody-style guided learning and conditional priors

LAPS-Diff augments the auxiliary decoder with explicit style and pitch supervision. The style encoder $\hat{M}$4 is a 2D CNN with residual blocks inspired by StyleTTS2. It has four residual 2D convolution blocks and outputs a fixed-dimensional style embedding $\hat{M}$5. Style loss compares embeddings extracted from ground-truth and predicted mel-spectrograms:

$\hat{M}$6

with $\hat{M}$7 (Dhar et al., 7 Jul 2025).

Pitch supervision is provided by JDCNet, described as a Conv-BiLSTM pitch extractor pre-trained on large singing data and adopted from StyleTTS2. The pitch loss is not a plain pointwise loss. Instead, LAPS-Diff introduces a concordance correlation coefficient (CCC)-weighted MSE:

$\hat{M}$8

The CCC term measures both correlation and agreement in mean and variance, and the paper states that a lower bound of 0.01 is applied to avoid the factor vanishing as CCC approaches 1. The stated purpose is to capture melodic shape, including bends, slides, and vibrato, rather than only pointwise $\hat{M}$9 accuracy.

The diffusion denoiser is additionally conditioned on two audio-derived priors. MERT provides musical embeddings $\hat{M}$0, intended to encode timbre, pitch, harmony, and tonal structure. IndicWav2Vec provides contextual embeddings $\hat{M}$1, intended to preserve Hindi content and pronunciation. These embeddings are extracted from ground-truth audio during training and from vocoder output during inference.

The auxiliary loss is summarized as

$\hat{M}$2

and the paper conceptually gives the total objective as

$\hat{M}$3

No extra style or pitch loss is applied directly to the denoiser; those terms are used solely for auxiliary decoder training.

5. Dataset, preprocessing, and training configuration

The dataset consists of 37–38 songs by Arijit Singh from 2012–2020, yielding approximately 65 minutes of singing from a single male voice. It contains 397 phrase-level segments of 5–15 seconds. The reported split is 31 training songs with 344 segments and 57.14 minutes, 3 validation songs with 25 segments and 3.76 minutes, and 4 test songs with 28 segments and 3.76 minutes (Dhar et al., 7 Jul 2025).

Preprocessing includes vocal isolation from mixed tracks using a commercial separation tool, segmentation based on silences greater than 500 ms, manual lyric association, phone-level forced alignment using a Kaldi hybrid ASR model trained on 180 hours of Hindi speech from 400 speakers, lexicon adaptation for singing pronunciations, syllable derivation by merging phones, and pitch extraction at 10 ms via autocorrelation in a Praat-style procedure. Each syllable receives a MIDI note defined as the mode of $\hat{M}$4 over the syllable. Syllables longer than 200 ms are split into non-overlapping 200 ms sub-segments; if MIDI changes between sub-segments, a slur flag is added.

Training uses AdamW with learning rate $\hat{M}$5, batch size 48, and $\hat{M}$6 iterations, with validation every 2000 iterations. The diffusion schedule uses $\hat{M}$7 steps and linearly increases $\hat{M}$8 from $\hat{M}$9 to $\hat{M}_q$0, with shallow diffusion parameter $\hat{M}_q$1. Implementation is reported in Python 3.8.20, PyTorch 2.4.1, and Librosa 0.8.0 on an NVIDIA A100 80GB. Training time is approximately 2 GPU days, with the extra cost predominantly attributed to running JDCNet for pitch at each iteration.

These details matter because the method’s claims are explicitly tied to a constrained, single-singer, genre-specific setting rather than to large-scale multilingual SVS.

6. Empirical results, ablations, and terminological clarifications

The main baseline is DiffSinger retrained from scratch on the same Hindi dataset using the official implementation, including the score encoder, FastSpeech2-like auxiliary decoder, WaveNet denoiser, and duration, $\hat{M}_q$2, and energy predictors. Relative to this baseline, LAPS-Diff reports the following objective scores: cosine similarity 0.987 versus 0.982, MAE 0.165 versus 0.197, V/UV accuracy 0.907 versus 0.890, log-$\hat{M}_q$3 RMSE 0.141 versus 0.155, MCD 7.897 versus 8.200, Content Enjoyment 4.770 versus 4.004, and Production Quality 6.552 versus 6.340. In MOS evaluation with 15 listeners, the reference audio scores 4.59 ± 0.26, LAPS-Diff 3.36 ± 0.34, and the DiffSinger baseline 2.85 ± 0.44 (Dhar et al., 7 Jul 2025).

The ablation study isolates four components: linguistic fusion only, priors only, pitch loss only, and style loss only. The reported conclusions are differentiated rather than uniform. Linguistic embeddings alone slightly improve MAE and MCD but leave prosody and style weak, with MOS essentially unchanged from baseline. MERT and IndicWav2Vec priors alone improve MAE, MCD, and log-$\hat{M}_q$4 RMSE, but cosine similarity and V/UV ratio degrade. The JDCNet plus CCC pitch loss yields the best log-$\hat{M}_q$5 RMSE among all models at 0.118, but the paper notes reduced intelligibility in some cases. Style loss alone raises cosine similarity to 0.986, close to the full model, but does not fully address prosody and technical fidelity. The full system achieves the best or near-best scores across most metrics and the highest MOS.

The paper also reports qualitative findings: PCA analysis of IndicWav2Vec embeddings places LAPS-Diff outputs closer to ground truth than DiffSinger, and visual analyses of $\hat{M}_q$6 and mel-spectrograms show better modeling of high-pitch peaks, long vowels, and unvoiced segments, with reduced harmonic distortion and fewer artifacts.

The stated limitations are narrow training coverage, computational overhead, and lack of evaluation on new singers or other genres. The model is trained on roughly 1 hour of Arijit Singh songs, and the authors identify larger datasets, multilingual SVS, better stylistic transfer, and computational optimization as future directions.

A frequent naming confusion deserves clarification. “LAPS-Diff” in (Dhar et al., 7 Jul 2025) denotes a singing voice synthesis framework with language-aware prosody-style guided learning. By contrast, “Local MAP Sampling for Diffusion Models” introduces LMAPS for inverse problems and explicitly notes that its method is “very close in spirit” to what one might have in mind as a local approximate posterior or MAP diffusion method, but it does not use the name “LAPS-Diff” and addresses a different problem class (Zhang et al., 7 Oct 2025). The two works share diffusion-based conditioning language only at a very abstract level; their domains, objectives, and mathematical formulations are distinct.