Controllable Singing Style Conversion with Boundary-Aware Information Bottleneck

Abstract: This paper presents the submission of the S4 team to the Singing Voice Conversion Challenge 2025 (SVCC2025)-a novel singing style conversion system that advances fine-grained style conversion and control within in-domain settings. To address the critical challenges of style leakage, dynamic rendering, and high-fidelity generation with limited data, we introduce three key innovations: a boundary-aware Whisper bottleneck that pools phoneme-span representations to suppress residual source style while preserving linguistic content; an explicit frame-level technique matrix, enhanced by targeted F0 processing during inference, for stable and distinct dynamic style rendering; and a perceptually motivated high-frequency band completion strategy that leverages an auxiliary standard 48kHz SVC model to augment the high-frequency spectrum, thereby overcoming data scarcity without overfitting. In the official SVCC2025 subjective evaluation, our system achieves the best naturalness performance among all submissions while maintaining competitive results in speaker similarity and technique control, despite using significantly less extra singing data than other top-performing systems. Audio samples are available online.

• The paper introduces a boundary-aware information bottleneck that minimizes intra-phoneme style leakage, enabling precise singing style conversion.
• It employs an explicit frame-level technique matrix and deterministic pitch modulation to control expressive techniques like vibrato and glissando.
• Experimental results in SVCC2025 show that the approach enhances naturalness and data efficiency, achieving high MOS scores despite limited training data.

Controllable Singing Style Conversion with Boundary-Aware Information Bottleneck

Introduction and Problem Definition

The paper "Controllable Singing Style Conversion with Boundary-Aware Information Bottleneck" (2604.05526) addresses the problem of fine-grained singing style conversion (SSC), differentiating itself from classic Singing Voice Conversion (SVC) by focusing explicitly on the transformation of vocal techniques—such as breathiness, register, resonance, vibrato, and glissando—while strictly preserving lyrical and melodic content and singer identity. Prior SSC methods predominantly relied on coarse, global context embeddings or utterance-level representations. Such approaches are inherently limited by persistent style leakage and insufficient granularity, restricting both controllability and the fidelity of dynamic technique rendering. The increasing demand for expressive and data-efficient systems emphasizes the necessity for architectures that can disentangle content and style at a fine temporal resolution and operate robustly under data constraints.

System S4: Model Architecture and Methodology

System S4 introduces a targeted architecture for controllable SSC comprising three primary technical innovations:

1. Boundary-Aware Whisper Bottleneck: The system performs temporal pooling of Whisper encoder frame-level representations within phoneme spans, suppressing intra-phoneme style variability and minimizing source-style leakage in downstream synthesis. This action constrains the semantic condition to encode only stable linguistic information and not technique cues.
2. Explicit Frame-Level Technique Matrix and Pitch Processing: The approach introduces a binary frame-level technique matrix $\mathbf{T}$, aligned with acoustic frames and phoneme boundaries, which directly conditions the waveform generator. For pitch-driven expressive techniques (e.g., vibrato, glissando), system S4 applies deterministic transformations to refine the $F_0$ contour during inference, governed by the matrix.
3. High-Frequency Band Completion: High-frequency spectral information ($>10$ kHz) is completed post-hoc via an external, standard 48-kHz SVC model. The protocol involves extracting and splicing high-frequency bands from the auxiliary model into the main S4 output, enhancing perceived brightness without introducing style leakage or overfitting risks.

Figure 1: The overall architecture of the proposed System S4, integrating boundary-aware semantic bottlenecking and high-frequency augmentation.

The architecture follows a recognition–synthesis paradigm. Feature extraction begins with phoneme, MIDI, F0, UV, and semantic features (from Whisper). These are pooled by phoneme boundaries, and scaled, then concatenated with technique and singer identity conditions—creating a highly disentangled representation set for the VITS-based decoder. The high-frequency completion module augments the output to full-bandwidth quality only after core synthesis is complete.

Disentanglement and Control Mechanisms

A central contribution is a robust disentanglement mechanism ensuring technique features are isolated from semantic and content cues. The phoneme-aware pooling within the semantic bottleneck explicitly removes rapid intra-phoneme variations, which often carry residual style. The scaled semantic stream (with $\lambda=0.1$) further restricts any potential dominance of this branch.

Simultaneously, controllability is realized by the explicit frame-level injection of the technique matrix, enabling compositional and temporally localized manipulation of expressive attributes. Pitch-driven modulations leverage deterministic formulas: e.g., vibrato frames are algorithmically perturbed with sinusoidal log-frequency modulations whose amplitude, rate, and phase are set by the matrix, while glissando frames undergo continuous frequency transitions.

Figure 2: The Boundary-Aware Semantic Bottleneck suppresses source style, while the technique matrix $\mathbf{T}$ explicitly controls temporal style specification.

This approach ensures that dynamic techniques reflect only target instructions and are decoupled from the source input's performative artifacts. The use of a binary multi-hot matrix allows for concurrent multi-technique rendering, further improving flexibility and expressivity.

Experimental Results and Analysis

Comprehensive experiments in the official Singing Voice Conversion Challenge 2025 (SVCC2025) validate the S4 system's efficacy. Evaluations utilize both subjective Mean Opinion Scores (MOS) for naturalness and binary accuracy scores for singing style and singer identity similarity.

Figure 3: Scatter plot of system positions on naturalness and style similarity, demonstrating the superior naturalness of S4B.

Figure 4: Official SVCC2025 results. (a) Naturalness MOS distribution and rank. (b) Singing style similarity. (c) Singer identity similarity; binary accuracy is overlaid.

Key empirical findings:

• The S4B system (full model with boundary-aware bottleneck) achieves the highest naturalness (median MOS = 4.0), ranked first among all challenge submissions excluding ground truth.
• S4B achieves competitive, though not top, results in style and identity similarity—surpassing or matching other systems that utilize significantly more external data for pretraining.
• An ablation (S4A) that removes semantic pooling and scaling exhibits severe increases in style leakage and reduced technique transfer success, empirically confirming the bottleneck's necessity.
• In controlled objective experiments, the use of Whisper features with the semantic bottleneck outperforms ContentVec and WeNet-based encoders, aligning with low leakage and high conversion rates.

Contradictory claim: The results indicate that highly data-efficient, disentangled models can outperform larger pretrained systems on naturalness, challenging the prevailing assumption that more data and model size directly imply higher quality in SSC when explicit disentanglement is enforced.

Practical and Theoretical Implications

The architecture of S4 demonstrates that explicit bottlenecking of content representations, coupled with explicit dynamic control signals, can decisively address style leakage in highly expressive synthesis settings. This design suggests:

• In expressive TTS/SSC, frame-level explicit condition matrices and aggressive semantic bottlenecking are preferable to weak global or utterance-level conditioning, especially in data-limited regimes.
• Deterministic, rule-based pitch modulation (driven by technique labels) offers precise and interpretable dynamic control and can be more effective than amortized neural predictors, which are prone to over-smoothing.
• The separation of main-band and high-frequency band generation allows robust high-fidelity synthesis without requiring high-bandwidth model training, mitigating overfitting and improving perceptual appeal under data scarcity.

Theoretically, the findings align with the principle that information bottlenecks and supervised conditioning are critical for controlled generation and style disentanglement. These lessons are directly transferable to related domains such as expressive TTS, voice cloning, and multimodal generation where disentanglement and fine-grained control are paramount.

Future Directions

Building on S4, anticipated research trajectories include:

• Extension of the boundary-aware bottleneck and explicit control framework to multi-lingual or out-of-domain SSC and zero-shot singing style transfer.
• Investigation of more granular or hierarchical technique control, possibly integrating continuous-valued technique embeddings or adversarial disentanglement objectives.
• Exploration of joint or unified learning of content, style, and high-frequency synthesis modules, including end-to-end differentiable architectures where high-frequency augmentation is trainable, subject to data augmentation or self-supervised learning constraints.
• Application of similar architectures to other paralinguistic domains (emotion, accent, prosody) or music-related tasks (instrument technique conversion).

Conclusion

System S4 advances controllable singing style conversion by integrating a boundary-aware semantic bottleneck, explicit frame-level technique conditioning, and high-frequency augmentation modules. Empirical evaluations in SVCC2025 substantiate its superiority in naturalness and controllability, particularly under strict data constraints. The methodology confirms the necessity of explicit disentanglement and deterministic control for high-fidelity and data-efficient expressive synthesis, providing a blueprint for further research in expressive neural voice and audio generation.