SingingSDS: Singing-Enabled Dialogue System

• SingingSDS is a singing-capable spoken dialogue system that integrates ASR, LLM, and SVS components to transform spoken input into in-character sung responses.
• Its modular, cascaded pipeline supports configurable backends and melody control, balancing latency and quality for dynamic interactive entertainment.
• The system bridges traditional spoken dialogue systems and singing voice synthesis by enabling roleplay interactions with consistent character persona and musical expression.

SingingSDS is a singing-capable spoken dialogue system that takes speech as input and returns a sung response in character. It is implemented as a modular, cascaded ASR–LLM–SVS pipeline with melody control, intended for character-based roleplay and interactive entertainment rather than conventional spoken-assistant use. The system supports configurable ASR and LLM backends, multiple melody sources, multi-speaker singing voice synthesis, and open-source deployment through a web demo and codebase (Han et al., 26 Nov 2025).

1. Concept and design objectives

Conventional spoken dialogue systems are described as combining ASR, LLMs, and TTS, but as outputting only spoken responses. SingingSDS replaces speech output with singing, motivated in the source description by memorability and working memory, pleasure and reward, and affective richness. The target application domain is character-based voiced interactive experiences such as virtual characters, games, theme park attractions, and live streaming, with explicit design goals of immediate, responsive singing replies, strong persona embodiment through the characters Limei and Yaoyin, and the perception of the system as a singing character rather than a generic assistant (Han et al., 26 Nov 2025).

The system is positioned between two previously separate research lines. On one side are SDS platforms that focus on speech output; on the other are singing voice synthesis systems that are typically offline and assume pre-written lyrics and scores. SingingSDS is therefore framed as a bridge between SDS and SVS: it performs speech-in, singing-out interaction by combining recognition, dialogue generation, melody control, and neural singing synthesis in one configurable pipeline (Han et al., 26 Nov 2025).

A common misconception is to treat SingingSDS as an end-to-end speech-to-song generator. The implementation is explicitly cascaded rather than end-to-end: ASR errors can propagate into the LLM and SVS stages, melody is provided or retrieved rather than jointly generated from dialogue semantics, and lyric–melody matching is managed through prompting and alignment rather than by a single monolithic model (Han et al., 26 Nov 2025).

2. System pipeline and modular architecture

The high-level pipeline consists of six stages. First, ASR transcribes user speech $s$ into text $s_t$ with an explicit language tag $\ell$: $s_t = \mathrm{ASR}(s, \ell)$ Second, the LLM generates an in-character lyrical reply $l$ from the transcript, character persona $c$, and optional structural constraints $\mathcal{C}$ derived from melody: $$l = \mathrm{LLM}(SystemPrompt(c, \mathcal{C}),\ UserPrompt(s_t))$$ Third, a melody controller provides or retrieves a note sequence

$$\mathcal{N} = \{(p_i, \tau_i^s, \tau_i^e)\}_{i=1}^n$$

with pitch $p_i$ and note times $s_t$0. Fourth, grapheme-to-phoneme conversion and alignment map the generated lyrics to the note sequence. Fifth, the SVS backend synthesizes a 44.1 kHz sung waveform $s_t$1 from the aligned phoneme and score representation together with speaker or voice information $s_t$2: $s_t$3 Finally, the interface renders audio, lyrics, character portrait, and configuration controls (Han et al., 26 Nov 2025).

Implementation is registry-based. The package structure separates ASR, LLM, SVS, Melody, and character-behavior modules behind a central interface, with configuration handled through YAML. The web interface is Gradio-based, and local use is supported through app.py for the web application and cli.py for batch synthesis, benchmarking, and non-interactive experimentation. The paper states that the configuration space spans about 350 possible system configurations across ASR, LLM, SVS, melody, and voice choices (Han et al., 26 Nov 2025).

This modularization is technically significant because it makes latency–quality trade-offs explicit rather than implicit. The paper notes, for example, that Paraformer plus Llama 3 yields lower latency, whereas Whisper plus Gemini gives better subjective quality at the cost of higher LLM latency. The architecture is therefore not merely compositional; it is an experimental framework for systematic backend substitution (Han et al., 26 Nov 2025).

3. Language processing, melody control, and lyric–melody alignment

The ASR module supports Whisper backends—small, medium, large-v3, and large-v3-turbo—and Paraformer from Alibaba FunASR. Paraformer-zh is described as a non-autoregressive transformer ASR optimized for speed and used for low-latency Chinese recognition. The language is supplied explicitly rather than inferred automatically, which the source text says avoids misclassification in multilingual settings and stabilizes recognition within a dialogue (Han et al., 26 Nov 2025).

The LLM module supports Gemini 2.5 Flash, Gemma 2 2B, Llama 3.1 8B Instruct, Llama 3.2 3B Instruct, Qwen3 8B, Qwen3 30B A3B, and MiniMax-Text-01. Prompting is divided into a system prompt and a user prompt. The system prompt encodes both character persona and structural constraints derived from melody, including number of syllables or characters per line and line counts, while also enforcing short, conversational, in-character replies without stage directions. For Japanese, input is converted to kana so that syllable counts can be expressed in kana units. The source explicitly states that persona prompts plus structural constraints are sufficient to produce “singable”, phrase-aware lyrics without additional training (Han et al., 26 Nov 2025).

Melody control is central rather than auxiliary. Melody is represented as a sequence of note tuples $s_t$4, optionally with phrase boundaries. Supported sources are random melodies, KiSing dataset melodies, the Touhou MIDI collection, and a synthesized Yue-based melody dataset containing 499 songs. Two alignment strategies are supported for dataset melodies. In pitch-based alignment, each syllable maps to one note and many-to-one mappings are collapsed into one-to-one form. In lyric-aware alignment, a single syllable may be sustained over multiple notes, represented by dashes indicating extension. Phrase annotations from KiSing and the synthesized corpus are used to compute per-phrase syllable budgets, which are then inserted into LLM prompts as soft constraints (Han et al., 26 Nov 2025).

The paper also defines a melody-dynamics analysis metric, the large jump ratio: $s_t$5 where $s_t$6 is the MIDI pitch of the $s_t$7-th note and $s_t$8 is the number of notes. This metric is not used as an optimization target, but as an analysis of how “jumpy” a melody is (Han et al., 26 Nov 2025).

4. Singing synthesis backend and voice realization

The SVS backend is VISinger2, used in two multi-speaker configurations: a Chinese multi-speaker VISinger2 trained on ACE-Opencpop with discrete speaker ID conditioning, and a bilingual Mandarin–Japanese multi-speaker VISinger2 trained on a mixture of OpenCpop, KiSing, ACE-KiSing, M4Singer, Kiritan, Onikuru Kurumi, PJS, and Namine Ritsu, with 192-dimensional learned speaker embeddings and 3-way language IDs $s_t$9. Common architecture settings reported for the SingingSDS deployment are $\ell$0, a 6-layer text encoder, an 8-layer posterior encoder, $\ell$1, FFN expansion to 768 dimensions, encoder dropout 0.1, 44.1 kHz sampling rate, AdamW with learning rate $\ell$2, exponential scheduler with $\ell$3, and loss weights of 45.0 for mel loss, 10.0 for pitch loss, 0.1 for duration loss, and 1.0 for KL loss; adversarial training uses MSE GAN, with up to 500 epochs and batch size 8 (Han et al., 26 Nov 2025).

Within the broader literature, VISinger2 is an end-to-end SVS model enhanced by a DSP synthesizer comprising harmonic and noise synthesizers, introduced to address the text-to-phase problem, glitches in voiced segments, and low sampling rate in the original VISinger architecture. It generates 44.1 kHz singing audio and, on OpenCpop, is reported to outperform VISinger, CpopSing, and RefineSinger in both subjective and objective metrics (Zhang et al., 2022). In the context of SingingSDS, this suggests that the choice of VISinger2 is not incidental: the backend already embodies a high-fidelity, multi-speaker, full-band SVS design compatible with low-latency interactive synthesis.

Voice realization in SingingSDS is determined by the SVS model choice, speaker ID or embedding, and language ID. Character persona and voice are linked only through configuration: Limei and Yaoyin are textual personas with default voices mapped to specific speaker IDs, while additional voices can be realized by selecting different speaker IDs or adding new models or embeddings. The paper does not describe an explicit learned persona-to-voice model; persona and timbre are coupled operationally through configuration rather than through a dedicated joint representation (Han et al., 26 Nov 2025).

5. Persona modeling and interaction design

Two original characters are provided: Limei and Yaoyin. Both are placed in the world of “Changge Plains,” where memories are passed on through song. Limei is a 19-year-old princess and guardian of the music city Lianming, associated with the “Millennial Songbook”; Yaoyin is a wandering singer and bard from Cloudsong Village. Their persona definitions include background, relationships, special abilities, personality traits, and speaking style. Limei is described as calm, resolute, reverent toward music, deeply empathetic, and self-disciplined; Yaoyin as free-spirited, warm, curious, and straightforward (Han et al., 26 Nov 2025).

Prompt constraints are used to maintain persona consistency. The system instructs the LLM always to remain in character, avoid breaking the fourth wall, use plain spoken language, keep replies short, avoid descriptions of actions or environment, and exclude annotations or stage directions. Additional constraints are character-specific: for Yaoyin, poetic or metaphorical language is discouraged unless storytelling or singing is explicitly requested. The paper states that these instructions stabilize character behavior across turns and keep outputs consistent with the fictional personality (Han et al., 26 Nov 2025).

This persona layer is important because SingingSDS is not merely a singing renderer attached to dialogue. Its target is roleplay interaction, and the text generation stage is explicitly optimized for short, singable, in-character replies rather than open-ended dialogue. The architecture therefore distributes character embodiment across prompt engineering, melody-aware structure, and voice assignment rather than localizing it in any single model component (Han et al., 26 Nov 2025).

6. Evaluation, latency, and reported performance

Evaluation is conducted on a 20-prompt roleplay test set targeting Yaoyin and on 450 questions from KdConv, with the user side synthesized using a VITS TTS system to control ASR difficulty. The reported metrics cover singing naturalness through SingMOS, content quality and entertainment through six-rater 5-point Likert judgments of Novelty and Fun, Character Consistency, and Lyric Quality, intelligibility through PER, latency through wall-clock component times, and melodic dynamics through large jump ratio (Han et al., 26 Nov 2025).

Among the reported roleplay configurations, Whisper + Gemini + KiSing achieves SingMOS 4.59, PER 0.48%, Novelty and Fun 4.21, Character Consistency 4.19, and Lyric Quality 3.86, with ASR latency 0.55 s, LLM latency 5.79 s, and SVS latency 0.18 s. Paraformer + Llama 3 + KiSing is the fastest configuration reported in the same table, with SingMOS 4.47, PER 0.12, Novelty and Fun 4.08, Character Consistency 4.13, Lyric Quality 3.41, and latencies of 0.44 s for ASR, 1.79 s for the LLM, and 0.16 s for SVS (Han et al., 26 Nov 2025).

Configuration	Reported quality metrics	Component latencies
Whisper + Gemini + KiSing	SingMOS 4.59; PER 0.48%; N&F 4.21; Char. Cons. 4.19; Lyric Qual. 3.86	ASR 0.55 s; LLM 5.79 s; SVS 0.18 s
Paraformer + Llama 3 + KiSing	SingMOS 4.47; PER 0.12; N&F 4.08; Char. Cons. 4.13; Lyric Qual. 3.41	ASR 0.44 s; LLM 1.79 s; SVS 0.16 s

The melody comparison under Whisper + Llama shows KiSing with SingMOS 4.53 and Jump Ratio 0.11, and Touhou with SingMOS 4.52 and Jump Ratio 0.28. The paper interprets Touhou melodies as more jumpy and dynamic while still maintaining high quality. On the KdConv experiments, three SVS-only settings are reported: SVS-1 (Random) with SingMOS 4.53, PER 25, latency 0.02 s, and Jump Ratio 35; SVS-2 (KiSing) with SingMOS 4.27, PER 36, latency 0.02 s, and Jump Ratio 4; and SVS-3 (Touhou) with SingMOS 4.43, PER 29, latency 0.02 s, and Jump Ratio 12 (Han et al., 26 Nov 2025).

Two practical conclusions are explicit in the source. First, SVS latency is very low, around 0.16–0.19 s in full pipeline settings and 0.02 s in SVS-only KdConv experiments. Second, the main latency bottleneck is the LLM, not the singing synthesizer. This explains the design choice to avoid direct text-to-song generation models such as Yue, which the appendix reports as requiring more than 40 s to generate 5 s of audio on a T4 GPU (Han et al., 26 Nov 2025).

7. Limitations, failure modes, and research context

The paper lists several limitations. Technically, the system is cascaded rather than end-to-end, so ASR errors propagate downstream. LLM-based syllable and phrase control is soft rather than exact, and can produce rhythmic mismatch. Melody is usually pre-existing—retrieved or random—rather than semantically tailored to content. Linguistically, the implementation currently focuses on Mandarin and Japanese, and extension to other languages requires additional SVS and G2P resources. Interactionally, some LLM configurations introduce substantial latency, and no explicit mechanism for multi-turn memory is described beyond whatever the chosen LLM backend supports (Han et al., 26 Nov 2025).

Typical failure modes are also specified: ASR errors can yield inconsistent lyrics; LLM failures can produce too many or too few syllables per phrase or drift out of character; SVS artifacts can arise when lyric length mismatches melody; and musical mismatch can occur when serious content is paired with playful melodies or when pitch jumps are very large. The paper further notes that automatic metrics such as Meta AudioBox Aesthetics did not correlate well with human enjoyment, sometimes favoring inharmonic random melodies (Han et al., 26 Nov 2025).

Relative to prior work, SingingSDS is contrasted with conventional SDS systems such as SALMONN-omni, Qwen2.5-Omni, Baichuan-audio, and ESPnet-SDS, which focus on speech output, and with SVS systems such as DiffSinger, VISinger2, TokSing, HiFiSinger, and Sinsy, which require pre-written lyrics and scores. The paper characterizes SingingSDS as the first open-source speech-in, singing-out SDS with configurable components and a demo (Han et al., 26 Nov 2025).

A broader research context is provided by parallel speech–singing corpora such as NHSS, which contains 100 songs sung and spoken by 10 singers, totaling about 7 hours of audio, and was released to support speech-to-singing conversion, speech–singing alignment, and comparative analysis of acoustic attributes (Sharma et al., 2020). A plausible implication is that resources of this kind could support future SingingSDS variants that move closer to tighter speech-to-singing coupling, especially where alignment, prosodic transfer, or end-to-end conditioning becomes central.

The future directions proposed for SingingSDS are tighter integration between LLM and SVS, improved melody generation that is controllable and semantic-aware, enhanced controllability over emotion and style, multilingual extension, and evaluation methods that better capture entertainment value and roleplay quality (Han et al., 26 Nov 2025).