Harmonic-Plus-Noise Model (HPN/HNM)

• The Harmonic-Plus-Noise Model (HPN/HNM) is a signal decomposition framework that separates audio signals into deterministic harmonic and stochastic noise components.
• It employs both classical analytic methods and neural architectures, including adaptive MVF estimation and differentiable DSP, to optimize audio synthesis across speech and music domains.
• Empirical studies show that integrating adversarial training and latent modeling techniques within HPN/HNM enhances spectral fidelity and perceptual naturalness.

The Harmonic-Plus-Noise Model (HPN/HNM) represents a class of signal decomposition frameworks designed to separate deterministic harmonic (periodic) and stochastic noise (aperiodic) components of audio signals. Initially formulated for speech and singing synthesis and later extended to musical signals and neural vocoding, HPN/HNM delivers a flexible and interpretable structure for modeling quasi-periodicity and turbulence within a unified mathematical and signal-processing formalism.

1. Mathematical Foundations of Harmonic-Plus-Noise Decomposition

The central premise of HPN/HNM involves representing an audio frame $s(t)$ as the sum of time-varying harmonic and noise components: $s(t) = h(t) + n(t)$ where $h(t)$ is the sum of sinusoids (harmonics) up to a dynamically determined maximum voiced frequency (MVF), and $n(t)$ models the residual, typically via filtered noise or a stochastic process (Wang et al., 2015, Yoneyama et al., 2022, 1908.10256, Subramani et al., 2020, Grumiaux et al., 2023).

The harmonic part is rendered as: $h(t) = \sum_{k=1}^{K(t)} A_k(t)\,\cos[\phi_k(t)]$ where $A_k(t)$ and $\phi_k(t)$ denote the amplitude and phase trajectories of the $k$th harmonic, respectively. The noise part, $n(t)$, is typically band-limited above MVF and amplitude-modulated to match the high-frequency energy profile.

Musical and neural approaches further generalize this to permit smooth parameter flows, learned amplitude and frequency envelopes, and differentiable architectures, while retaining the key bandwise separation governed by MVF or algorithmically inferred transitions between periodic and aperiodic structure (Subramani et al., 2020, Grumiaux et al., 2023, Li et al., 2023).

2. Frequency Band Partitioning and MVF Estimation

A distinctive property of HPN/HNM is the separation between harmonic and noise components along the frequency axis, with the "Maximum Voiced Frequency" MVF (called $f_m$ or $s(t) = h(t) + n(t)$0) acting as the band boundary. Classical approaches estimate MVF by energy criteria or harmonic regularity in the spectrum (Wang et al., 2015). Neural approaches employ auxiliary networks to predict a time-varying, differentiable MVF from acoustic features or signal context, often parameterizing analytic low/pass and high/pass FIR filters (e.g., windowed-sinc kernels) with the MVF as cutoff (1908.10256).

A representative MVF prediction network upsamples frame-level F0 and voiced/unvoiced labels to waveform sampling rate, combines these with spectral features via Bi-LSTM and convolutional layers, and fuses this with a learned residual to give a smooth cutoff trajectory (1908.10256). This adaptive separation enables robust synthesis and transformation across voiced/unvoiced transitions and complex temporal structure.

3. Neural and Differentiable HPN Architectures

Modern HPN designs leverage deep neural networks to parameterize both the harmonic and noise synthesis branches, ensuring differentiability, data-driven envelope acquisition, and joint optimization. Three broad architectural patterns emerge:

• Explicit Signal Branching: Two separate branches, typically with pitch-conditioned, deep convolutions for the harmonic stream and standard convolutions for the noise stream (Yoneyama et al., 2022). Features from both branches are fused via periodicity masks or analytic filterbanks, ensuring dynamic weighting between periodic and aperiodic energy based on input characteristics or learned masks.
• Differentiable DSP (DDSP): The entire HPN process is implemented as a differentiable computation graph, allowing all synthesis steps (sinusoidal integration, noise filtering, mixing) to participate in end-to-end gradient optimization (Grumiaux et al., 2023). This enables backpropagation through the sinusoidal and noise generation, filter design, and amplitude/frequency envelope inference.
• Parametric Latent Modeling: Harmonic and noise spectral envelopes are encoded as low-dimensional latent representations, commonly via VAEs or conditional VAEs, facilitating generative manipulation of timbral and stochastic aspects (Subramani et al., 2020). Joint or independent latent modeling strategies are used to capture cross-dependencies between harmonic structure and residual noise.

A generic HPN neural pipeline involves (i) excitation generation (sinusoidal plus noise streams), (ii) adaptive or learned filtering to produce hybrid excitation, and (iii) filtering or waveform-level network post-processing to yield the final signal (Yoneyama et al., 2022, Grumiaux et al., 2023, Li et al., 2023).

4. Adversarial and Spectral Training Objectives

HPN-parameterized networks are trained using adversarial and non-adversarial losses:

• GAN-based Adversarial Training: Integration of discriminators (Multi-Period and Multi-Scale Discriminators) operating on waveform outputs enables perceptually motivated supervision. Generator losses blend adversarial terms (LSGAN, relativistic, or HiFiGAN-style) with feature-matching and auxiliary spectral losses (Yoneyama et al., 2022, Li et al., 2023).
• Mel-Spectral and Multi-Scale Spectral Losses: To enforce spectral fidelity across harmonic and noise bands, auxiliary losses compute L1 or L2 distance between generated and ground-truth spectrograms at various time-frequency resolutions (Yoneyama et al., 2022, Grumiaux et al., 2023).
• Residual-Spectra Regularization: Learning to match the statistics (e.g., distribution and dynamic range) of the true residual spectrum (post envelope subtraction) significantly enhances naturalness and timbral plausibility (Yoneyama et al., 2022).
• Closed-form Differentiable Filtering: Losses are backpropagated through FIR windowed-sinc or IIR parametric filters that implement the frequency band split, ensuring the MVF and downstream synthesis adapt directly to optimization signals (1908.10256).

No auxiliary time-domain reconstruction is needed in most state-of-the-art systems, and empirical results indicate superior perceptual robustness and fidelity to F0/phase mismatches when using spectral-domain targets (Yoneyama et al., 2022, Grumiaux et al., 2023).

5. Empirical Evaluations and Impact Across Modalities

HPN-based systems have been quantitatively and perceptually validated across speech, singing synthesis, and musical signal restoration tasks:

• Speech and Voice Synthesis: Neural HPN models demonstrate significant reductions in mel-cepstral distortion (MCD), decreased voiced/unvoiced error rates, and improved MOS for naturalness—HN-uSFGAN improves MCD from 3.09 to 2.82 dB and MOS from 3.66 to 3.79 (Yoneyama et al., 2022). Subjective tests confirm enhanced clarity and stability under pitch transformation (Yoneyama et al., 2022, 1908.10256).
• Bandwidth Extension and Musical Signal Modeling: Differentiable HPN pipelines outperform higher-parameter deep networks for high-frequency reconstruction, achieving lower log-spectral distances and higher MUSHRA scores across a wide variety of music signals (Grumiaux et al., 2023). Empirical evidence supports both the efficiency (e.g., 9% real-time on monophonic music with ~4M parameters) and the timbral superiority of hybrid models.
• Expressive Synthesis and Latent Modeling: Parametric and VAE-based HPN/HNM frameworks allow for explicit control and sampling of both harmonic and noise envelopes, supporting lifelike performance in expressive contexts such as singing voice and bowed-string instruments (Subramani et al., 2020, Wang et al., 2015).

Ablation studies consistently show that removing the HPN split, residual regularization, or GAN-based discrimination substantially degrades both objective and subjective output quality (Yoneyama et al., 2022, Grumiaux et al., 2023).

6. Design Variants, Implementation Choices, and Domain Extensions

HPN/HNM admits several design flexibilities and extensions:

• Choice of Filtering and Mixing: Explicit, parameterized FIR/IIR filters (windowed-sinc, AR models) afford interpretability and dynamic control, while mask-based and learned-determined interpolation allow more flexible, context-dependent time–frequency mixing (1908.10256, Yoneyama et al., 2022).
• Adaptive MVF vs. Static Boundary: Time-varying, data-driven MVF prediction improves transition handling and signal adaptation compared to static or hand-tuned boundaries, especially for challenging voiced–unvoiced segments and rapid prosodic change (1908.10256, Drugman et al., 2019).
• Latent and Joint Factor Modeling: Modeling both harmonic and noise spectral envelopes in low-dimensional latent spaces (CVAE, JNet) allows for explicit decoupling or soft-coupling of periodic and stochastic phenomena, yielding improved reconstruction and expressive control, notably in musical instrument synthesis (Subramani et al., 2020).
• Computational and Storage Implications: HPN-based models are markedly efficient relative to full waveform neural vocoders, owing to reduced parameter count, non-autoregressive factorization, and the ability to cache or predict envelope and filter parameters (1908.10256, Drugman et al., 2019).

7. Broader Significance and Continuing Research

The HPN/HNM framework represents a key evolutionary step bridging classical parametric and modern neural approaches to audio synthesis, restoration, and transformation. By architecting explicit hybridization of deterministic and stochastic signal models, HPN systems enable high-fidelity, editable, and interpretable generative pipelines. Recent research demonstrates their competitiveness with or superiority over large-scale neural vocoders in both quality and efficiency, and establishes differentiable HPN elements as robust inductive priors in music, speech, and cross-domain audio processing (Yoneyama et al., 2022, Grumiaux et al., 2023, Li et al., 2023).

Ongoing investigations include the role of learned vs. analytic frequency boundaries, joint hierarchical factor modeling, spectral domain GAN objectives, and applications beyond speech and music to environmental and biomedical signal analysis. The model’s flexibility for explicit control (e.g., pitch/vibrato, noise coloring, timbral morphing) suggests wide applicability in creative, scientific, and real-time contexts.

---

Key references:

• "Unified Source-Filter GAN with Harmonic-plus-Noise Source Excitation Generation" (Yoneyama et al., 2022)
• "Neural Harmonic-plus-Noise Waveform Model with Trainable Maximum Voice Frequency for Text-to-Speech Synthesis" (1908.10256)
• "Efficient bandwidth extension of musical signals using a differentiable harmonic plus noise model" (Grumiaux et al., 2023)
• "Mandarin Singing Voice Synthesis Based on Harmonic Plus Noise Model and Singing Expression Analysis" (Wang et al., 2015)
• "A Deterministic plus Stochastic Model of the Residual Signal for Improved Parametric Speech Synthesis" (Drugman et al., 2019)
• "HpRNet : Incorporating Residual Noise Modeling for Violin in a Variational Parametric Synthesizer" (Subramani et al., 2020)
• "HiFTNet: A Fast High-Quality Neural Vocoder with Harmonic-plus-Noise Filter and Inverse Short Time Fourier Transform" (Li et al., 2023)