Repurposing a Speech Classifier for Guided Diffusion-Based Speech Generation

Abstract: Classifier guidance is a way to control diffusion generation by using a noise-conditioned classifier to steer the sampling process toward a target class. One drawback of classifier guidance is that it requires two separately trained models: a classifier and a diffusion model. We therefore study a more compact alternative in which a conventionally trained speech classifier is repurposed as the backbone for diffusion generation. Starting from a frozen noise-conditioned classifier in log-Mel space, we attach a lightweight subnetwork that reuses intermediate classifier representations and train only this subnetwork under a Denoising Score Matching objective. Our work shows that a pretrained classifier can be repurposed for conditional generation, providing an appealing bridge between discriminative modeling and conditional speech synthesis resulting in high speech quality within a single-backbone model, with reduced memory footprint and computational cost.

• The paper introduces a method that repurposes frozen speech classifiers and trains a lightweight Score Subnet for diffusion-based speech generation.
• It leverages JEM-style gradient taps and cross-attention fusion to achieve competitive speech quality with reduced parameters and compute.
• Empirical results on SC09 demonstrate comparable MOS and FAD metrics to U-Net, highlighting efficiency especially in low-data settings.

Repurposing Pretrained Speech Classifiers for Diffusion-Based Speech Generation

Overview of Classifier-Based Diffusion Guidance

Classifier guidance in diffusion models leverages a noise-conditioned classifier to steer the generative sampling process toward a desired class, usually by exploiting gradients of the classifier’s output. Conventionally, this requires two separately trained models: a generative diffusion model and a classifier. While effective, such pipelines are computationally intensive and resource-demanding due to the necessity of maintaining and evaluating both models during generation.

The paper proposes a parameter-efficient alternative by repurposing a conventionally trained speech classifier as the backbone for diffusion generation. Rather than learning a full generative model from scratch, a lightweight Score Subnet is trained atop the frozen classifier, reusing intermediate representations and JEM-style activation gradients for score prediction. This design unifies discriminative and generative modeling in speech synthesis, aligning classifier-guidance with conditional diffusion in a single-backbone architecture.

Methodological Details

The generative process operates in log-Mel filterbank feature space, employing diffusion defined by stochastic differential equations on spectrogram distributions. The Score Subnet is trained under the Denoising Score Matching (DSM) objective, where model predictions are matched to perturbation scores that can be computed in closed form. For conditional generation, classifier guidance is invoked by computing class-dependent gradients from the frozen classifier.

The architectural core is a modified U-Net, with the upsampling path replaced by a classification head. For score-based generation, a decoder-style Score Subnet is attached to the frozen classifier. Intermediate feature maps (forward taps) and backpropagated gradients from JEM-based marginal log-density (gradient taps) at multiple resolution levels are fused via cross-attention mechanisms. Only the Score Subnet parameters are optimized, allowing efficient reuse of classifier feature extraction without retraining the backbone.

For class-conditional sampling, the Score Subnet’s score predictions are augmented with classifier-guidance gradients, parameterized by a guidance strength hyperparameter $\gamma$. This approach is compatible with both unconditional and conditional generation settings.

Empirical Evaluation

Experiments are conducted on the SC09 spoken digit benchmark. The Score Subnet achieves speech quality and fidelity comparable to or exceeding traditional diffusion U-Net and open-source baselines, with significant reductions in both trainable parameters (4.4M vs. 16.6M for U-Net) and computational cost (12.07 GMACs vs. 14.56 GMACs per reverse diffusion step).

Strong numerical results include:

• ScoreQ MOS: 3.10 (Score Subnet), 3.06 (U-Net)
• FAD: 0.84 (Score Subnet), 0.74 (U-Net)
• ResNeXt-based FID: 0.17 (both Score Subnet and U-Net)

In conditional generation with classifier guidance ($\gamma=3.0$), Score Subnet matches U-Net + Classifier on perceptual quality, yet achieves a ~50% reduction in compute and trainable parameters (4.4M vs. 16.6M trainable, 16.44 vs. 22.74 GMACs). The method’s advantage becomes pronounced in low-data and zero-shot regimes: Score Subnet consistently outperforms the classifier-guided U-Net pipeline as guidance strength increases, demonstrating robustness in constrained training settings.

Ablation reveals that gradient taps, extracted via backpropagation from the classifier, are critical for performance, with the variant lacking these taps exhibiting degraded metrics.

Implications and Future Directions

This approach bridges discriminative and generative paradigms in speech modeling, offering a compact alternative to conventional classifier-guided diffusion with performance parity or superiority at a fraction of the cost. It provides an operational pathway for leveraging pretrained noise-conditioned classifiers in generative tasks, including speech synthesis, with implications for practical deployment on resource-limited platforms.

Theoretically, the methodology aligns with joint energy-based model perspectives, where classifier logits constitute unnormalized joint models over input-label pairs. By sidestepping full JEM training—which is fraught with instabilities and normalization intractabilities—the Score Subnet approach extracts score-relevant signals from classifier representations, enhancing generative modeling with minimal adaptation.

Anticipated future developments may include extending this framework to other modalities (e.g., text, vision) and exploiting more sophisticated adaptation strategies. Exploration of other classifier architectures and transfer settings (multilingual, cross-domain speech) is plausible, given the generality of the approach.

Conclusion

Repurposing frozen speech classifiers for score-based diffusion generation via lightweight subnet adaptation yields substantial efficiency gains while maintaining strong speech quality and fidelity. The method elegantly unifies conditional guidance and generative modeling within a single-backbone architecture, with robust empirical performance across baseline comparisons and low-data settings. This paradigm offers practical implications for scalable, low-resource speech generation and motivates further research into classifier-centric generative synthesis mechanisms.