NNSVS: A Neural Network-Based Singing Voice Synthesis Toolkit (2210.15987v2)

Abstract: This paper describes the design of NNSVS, an open-source software for neural network-based singing voice synthesis research. NNSVS is inspired by Sinsy, an open-source pioneer in singing voice synthesis research, and provides many additional features such as multi-stream models, autoregressive fundamental frequency models, and neural vocoders. Furthermore, NNSVS provides extensive documentation and numerous scripts to build complete singing voice synthesis systems. Experimental results demonstrate that our best system significantly outperforms our reproduction of Sinsy and other baseline systems. The toolkit is available at https://github.com/nnsvs/nnsvs.

• The paper introduces a modular SVS toolkit that splits synthesis into time-lag, duration, acoustic, and vocoder modules.
• It employs multi-stream and autoregressive models to capture dynamic pitch features such as vibrato more naturally.
• Experimental results, including an MOS of 3.86, demonstrate that neural vocoders significantly enhance synthesized voice quality.

Essay on "NNSVS: A Neural Network-Based Singing Voice Synthesis Toolkit"

The paper "NNSVS: A Neural Network-Based Singing Voice Synthesis Toolkit" details the creation and evaluation of NNSVS, an open-source toolkit designed to advance singing voice synthesis (SVS) research. This toolkit emerges as a significant contribution in the SVS domain, building upon existing frameworks and addressing several limitations noted in prior systems like Sinsy and Muskits.

Core Contributions

Modular and Extensible Architecture: NNSVS introduces a modular design, drawing inspiration from Sinsy while extending its capabilities. It decomposes the synthesis process into distinct modules: time-lag, duration, acoustic model, and vocoder. This approach facilitates customization, allowing researchers to modify or add components without destabilizing the entire system.

Multi-Stream and Autoregressive Models: The toolkit integrates multi-stream models to address the imbalance in feature weighting issues, previously noted in DNN-based models. Autoregressive models are employed specifically for $F_0$ modeling, capturing dynamic features like vibrato more naturally compared to static methodologies. The modular design combined with autoregressive capabilities represents a salient advancement over existing paradigms.

Neural Vocoders: NNSVS incorporates advanced neural vocoders, particularly the unified source-filter generative adversarial networks (uSFGAN), to enhance the pitch robustness and quality of synthesized voices. This is a strategic improvement over traditional vocoding techniques.

Experimental Evaluation

The paper presents a comprehensive evaluation of the NNSVS toolkit against baseline systems such as Sinsy, Muskits, and DiffSinger. Utilizing objective metrics like mean opinion scores (MOS), NNSVS demonstrates superior performance, particularly with configurations employing multi-stream and autoregressive models. The best-performing variant, NNSVS-WORLD~v4, achieved an MOS of 3.86, which is notably higher than other examined systems.

In terms of acoustic features, systems utilizing WORLD features, enhanced by multi-stream and autoregressive processing, outperformed those relying solely on mel-spectrogram data, underscoring the effectiveness of feature disentanglement.

Implications and Future Directions

Practically, NNSVS provides a robust and flexible platform for SVS, enabling broader experimentation and potential integration within commercial applications. The significant improvements in quality, driven by advanced model architectures and neural vocoders, pave the way for more expressive and natural synthetic singing voices.

Theoretically, the paper underlines the importance of modular and extensible design in SVS toolkits, promoting ease of feature integration and model adaptability. The success of autoregressive models in improving expressive capabilities marks a promising direction for future research.

Looking ahead, the authors suggest potential enhancements such as integrating variational auto-encoders, diffusion models, and fully end-to-end systems. Such developments could further refine the synthesis process, offering even tighter integration of learning components, and possibly reducing the dependence on feature engineering.

Conclusion

"NNSVS: A Neural Network-Based Singing Voice Synthesis Toolkit" is a well-constructed toolkit that addresses several limitations of previous SVS systems. Its strong experimental results, modular architecture, and open-source availability make it a valuable resource for researchers in the field. Moving forward, the application of more advanced AI models presents exciting avenues to augment both the toolkit’s capabilities and the auditory realism of synthesized singing voices.