Neural Harmonic Textures

• Neural Harmonic Textures are neural representations that explicitly decompose structured harmonic content into periodic and aperiodic components for enhanced interpretability.
• They leverage methods such as source–filter models, time–frequency convolution, and Gram-based texture transfer to achieve controllable synthesis in audio, music, and graphics.
• These approaches balance interpretability and performance by optimizing sound quality, harmonic density, and computational efficiency across diverse applications.

Searching arXiv for relevant papers on neural harmonic textures and closely related work. Neural harmonic textures are neural representations or generative mechanisms that explicitly organize signal structure into harmonic, noise-like, or otherwise interpretable components, often with controllable temporal evolution, local interpolation structure, or symbolic harmonic organization. Across audio synthesis, music generation, system identification, and neural rendering, the term denotes a family of methods that use neural networks to represent structured frequency content, cross-frequency coupling, harmonic-plus-noise decomposition, or harmonic basis expansions in ways that are more explicit and controllable than fully implicit black-box models. In audio, this includes source–filter waveform models that separate periodic and aperiodic components with a time-varying crossover (1908.10256), interpretable time–frequency adaptive filters that model cross-frequency interactions (Helwani et al., 2023), spectrogram-based neural texture transfer systems that implicitly preserve harmonic structure (Peng et al., 2018), and symbolic or neurosymbolic systems that generate harmonized or polyphonic textures from melody (Wu et al., 2021, Blanchard et al., 22 Jun 2025). In computer graphics, the term is used directly for primitive-bound neural representations that interpolate local features, apply periodic activations, and decode a weighted sum of harmonic components in deferred rendering (Condor et al., 1 Apr 2026).

1. Harmonic texture as a technical concept

In the source–filter and texture-analysis literature, harmonic texture refers to structured, temporally evolving energy aligned with periodicity, harmonic series, or stable cross-frequency organization, in contrast to broadband noise or impulsive structure. The harmonic-plus-noise neural source–filter model treats waveform generation as the combination of a harmonic component aligned with $F_0$ and its harmonics, a noise component capturing broadband aperiodic energy, and a time-varying spectral split controlled by a trainable Maximum Voice Frequency (MVF) (1908.10256). This establishes a concrete neural formulation in which texture is not merely an emergent by-product of a deep network, but an explicitly parameterized decomposition.

A related perceptual and signal-analysis formulation separates sound into tones, pulses, and broadband noises, corresponding respectively to horizontal, vertical, and stochastic structures in the time–frequency plane (Elburg et al., 2017). In that framework, tonal or harmonic texture is associated with extended horizontal ridges in a cochleagram, and global descriptors such as tonality, pulsality, and noisiness correlate with perceptual organization. This suggests that neural harmonic textures can be understood not only as an overview strategy but also as a representational axis grounded in production physics and auditory perception.

In symbolic music and harmony generation, harmonic texture is expressed through the density, timing, and diversity of chordal events. AutoHarmonizer generates frame-by-frame chord symbols at sixteenth-note resolution and introduces a scalar control $\Gamma$ that modulates harmonic density, thereby changing how often chords repeat or change and shaping the resulting harmonic rhythm (Wu et al., 2021). AI Harmonizer, by contrast, expands a monophonic vocal line into a four-part SATB texture, where texture is the combined effect of vertical chordal structure, note-synchronous voice alignment, and timbral blending in rendered audio (Blanchard et al., 22 Jun 2025).

A broader implication is that “neural harmonic textures” is not a single architecture class. It encompasses models in which harmonic structure is represented as periodic waveform components, interpretable time–frequency dependencies, spectrogram statistics, symbolic chordal organization, or local Fourier-like primitive features. What unifies these systems is explicit structure in the harmonic domain and some degree of controllability or interpretability.

2. Harmonic-plus-noise waveform models

The paper “Neural Harmonic-plus-Noise Waveform Model with Trainable Maximum Voice Frequency for Text-to-Speech Synthesis” defines one of the clearest neural harmonic texture engines in audio (1908.10256). It uses a neural source–filter architecture with three functional parts: a condition module, two source modules, and two filter modules. Given frame-rate acoustic features $\mathbf{c}_{1:B}$ with each frame $\mathbf{c}_b = [f_b, \mathbf{s}_b^\top]^\top$, where $f_b$ is $F_0$ and $\mathbf{s}_b$ is an 80-dim Mel spectrogram, the model produces a waveform $\hat{\mathbf{o}}_{1:T}$ (1908.10256).

The harmonic source produces a sum of sinusoids at $F_0$ and its first $I-1$ harmonics, with $\Gamma$0, additive Gaussian noise, and pure noise in unvoiced regions. The excitation for harmonic index $\Gamma$1 is defined piecewise for voiced and unvoiced conditions, with $\Gamma$2, $\Gamma$3, and random initial phase $\Gamma$4 (1908.10256). The harmonic excitations are then linearly mixed through

$\Gamma$5

This gives the model a source-level learned harmonic envelope before neural filtering.

The noise branch starts from Gaussian noise and is processed by a separate neural filter to generate an aperiodic waveform component. The harmonic branch uses five stacked residual dilated-convolution filter blocks, while the noise branch uses one. Each filter block contains 10 dilated convolution layers with 64 channels, kernel size 3, and dilation $\Gamma$6 cycled across layers, with conditional feature addition at each layer (1908.10256). The asymmetry in depth reflects the structural complexity assigned to the periodic component.

The defining operation is the time-varying merge of harmonic and noise outputs using FIR low-pass and high-pass filters whose cutoff is the predicted MVF. The final signal is

$\Gamma$7

The fixed-cutoff baseline h-NSF uses voiced-region cutoffs of approximately 5 kHz for the low-pass and 7 kHz for the high-pass filters, and unvoiced-region cutoffs of approximately 1 kHz and 3 kHz. The proposed sinc-h-NSF replaces these with differentiable windowed-sinc filters whose normalized cutoff $\Gamma$8 varies over time and is predicted from acoustic features (1908.10256).

Three MVF variants are compared. In sinc1-h-NSF,

$\Gamma$9

where $\mathbf{c}_{1:B}$0 for voiced and $\mathbf{c}_{1:B}$1 for unvoiced regions, and $\mathbf{c}_{1:B}$2. In sinc2-h-NSF, the cutoff ignores the voiced/unvoiced prior and depends only on $\mathbf{c}_{1:B}$3. In sinc3-h-NSF, a trainable sigmoid parameterization collapses to $\mathbf{c}_{1:B}$4, effectively eliminating the noise band (1908.10256). The experiments reported that sinc1-h-NSF predicts a good MVF trajectory and that sinc1-h-NSF, base-h-NSF, and WaveNet are comparable in MOS, while sinc2 and sinc3 are significantly worse. This indicates that a single time-varying scalar controlling harmonic versus noise allocation can be a meaningful neural texture parameter.

Training uses a sum of three spectral amplitude distances,

$\mathbf{c}_{1:B}$5

with STFT configurations of 20 ms / 5 ms / 512 DFT, 5 ms / 2.5 ms / 128 DFT, and 120 ms / 40 ms / 2048 DFT, all using Hann windows (1908.10256). There is no explicit branch-separation loss; the decomposition emerges from the structured excitation sources and the filtering architecture. This makes the model a canonical example of neural harmonic textures as an explicit decomposition of periodic and aperiodic signal structure.

3. Time–frequency coupling and interpretable harmonic structure

Where h-NSF expresses harmonic texture as a harmonic-plus-noise waveform decomposition, Neural Harmonium expresses it as structured cross-frequency interaction in an analytic time–frequency domain (Helwani et al., 2023). The model is designed for nonlinear dynamic system identification and represents signals using an analytic STFT-like filterbank: $\mathbf{c}_{1:B}$6 Synthesis is defined analogously, with an analytic condition ensuring causality in the time–frequency domain (Helwani et al., 2023).

The core dynamical block is a multiple-input single-output time–frequency convolution: $\mathbf{c}_{1:B}$7 Here, $\mathbf{c}_{1:B}$8 is the activation set of input bins that influence output bin $\mathbf{c}_{1:B}$9, and $\mathbf{c}_b = [f_b, \mathbf{s}_b^\top]^\top$0 is a time-lagged mixing kernel (Helwani et al., 2023). This formulation makes harmonic texture visible as a dependency map over frequency pairs and time lags. Diagonal structure corresponds to approximately linear per-band behavior; off-diagonal structure corresponds to nonlinear frequency coupling such as modulation, distortion, or hysteresis.

The system is organized as an order-recursive, multi-stage lattice architecture: $\mathbf{c}_b = [f_b, \mathbf{s}_b^\top]^\top$1 with a cost functional combining final-output loss and per-stage regularizers (Helwani et al., 2023). Forward and backward prediction errors satisfy a multichannel lattice recursion, and the corresponding Schur recursion ensures causality, stability, and uniqueness under analytic initial conditions. Because stage inputs are decorrelated, the Hessian becomes block-diagonal across stages, enabling exact second-order optimization through Kalman filtering rather than explicit full-Hessian computation.

The neural component is not a generic end-to-end waveform synthesizer but a CNN that predicts frequency interdependencies. Its input tensor is

$\mathbf{c}_b = [f_b, \mathbf{s}_b^\top]^\top$2

with channels $\mathbf{c}_b = [f_b, \mathbf{s}_b^\top]^\top$3, $\mathbf{c}_b = [f_b, \mathbf{s}_b^\top]^\top$4, $\mathbf{c}_b = [f_b, \mathbf{s}_b^\top]^\top$5, $\mathbf{c}_b = [f_b, \mathbf{s}_b^\top]^\top$6, and $\mathbf{c}_b = [f_b, \mathbf{s}_b^\top]^\top$7, and convolutions are performed only along the time axis (Helwani et al., 2023). The network outputs a per-bin dependency probability vector, thresholded to define the active input set $\mathbf{c}_b = [f_b, \mathbf{s}_b^\top]^\top$8. In this sense, the model learns a graph of harmonic couplings.

The framework is validated on nonlinear system identification and acoustic echo cancellation. In the Bouc–Wen hysteresis example, it reaches a modeling error $\mathbf{c}_b = [f_b, \mathbf{s}_b^\top]^\top$9 dB versus roughly $f_b$0 to $f_b$1 dB for Wiener–Hammerstein, Gaussian Process regression, and ARX models (Helwani et al., 2023). In the ICASSP 2022 AEC Challenge setup with window size $f_b$2, hop $f_b$3, and $f_b$4 stages, Neural Harmonium obtains ERLE $f_b$5 dB, PESQ $f_b$6, and MOS $f_b$7 in the speech double-talk case, compared with WebRTC’s ERLE $f_b$8 dB, PESQ $f_b$9, MOS $F_0$0, and an interference suppression NN’s ERLE $F_0$1 dB, PESQ $F_0$2, MOS $F_0$3 (Helwani et al., 2023). For music near-end, its MOS is $F_0$4, above WebRTC’s $F_0$5 and the interference suppression NN’s $F_0$6. These results indicate that explicit time–frequency harmonic structure can be both interpretable and competitive in real audio tasks.

4. Spectrogram statistics and neural texture transfer

A different branch of the literature models texture through stationary statistics of spectrogram features. “A Lightweight Music Texture Transfer System” defines texture as “the collective temporal homogeneity of acoustic events” and formalizes texture transfer as generating an output $F_0$7 that preserves the content relation to $F_0$8 while sharing recognizable texture with $F_0$9 (Peng et al., 2018). The system operates on STFT magnitude: $\mathbf{s}_b$0 using a Gaussian window, then takes log scaling

$\mathbf{s}_b$1

and applies a hard threshold mask with $\mathbf{s}_b$2 to suppress low-energy bins (Peng et al., 2018).

The resulting “texture spectrum” is mapped to a 3-channel pseudo-image $\mathbf{s}_b$3 for compatibility with 2D CNNs and a VGG-19 loss network. The feed-forward transfer network follows the Johnson et al. architecture: 3 convolutional layers with instance normalization and ReLU, 5 residual blocks, 3 transpose convolution layers, and a final tanh (Peng et al., 2018). The content loss is

$\mathbf{s}_b$4

the texture loss is

$\mathbf{s}_b$5

with Gram matrices

$\mathbf{s}_b$6

and the total objective is

$\mathbf{s}_b$7

with $\mathbf{s}_b$8, $\mathbf{s}_b$9, and $\hat{\mathbf{o}}_{1:T}$0 (Peng et al., 2018).

In this formulation, harmonic texture is implicit rather than explicitly factorized. Harmonic series, partial stacks, and chordal spectra appear as vertical structures in the spectrogram image, and their correlations contribute to the Gram statistics. The model preserves these to the extent that the content loss preserves large-scale time–frequency organization. It does not represent pitch, chords, or harmonic functions separately, and therefore does not target harmonic structure in the symbolic or psychoacoustic sense. This suggests that Gram-based spectrogram textures are well suited to timbral or surface-level harmonic color, but not to explicit harmonic organization.

The more general audio texture synthesis framework of “Synthesizing Diverse, High-Quality Audio Textures” develops this idea further by defining audio textures as stationary random processes with local sufficient statistics and synthesizing them by optimizing a spectrogram to match neural feature correlations (Antognini et al., 2018). Audio is represented as a 16 kHz log-magnitude spectrogram with STFT window 512, hop 64, and $\hat{\mathbf{o}}_{1:T}$1, treating frequency bins as channels in a 1D temporal CNN. The model uses six separate one-layer CNNs with kernel sizes $\hat{\mathbf{o}}_{1:T}$2 frames and 512 filters each (Antognini et al., 2018).

The base Gram loss is

$\hat{\mathbf{o}}_{1:T}$3

augmented by an autocorrelation loss over lags corresponding to 200 ms to 2 s,

$\hat{\mathbf{o}}_{1:T}$4

and a diversity term based on shift-invariant feature differences. The total loss is

$\hat{\mathbf{o}}_{1:T}$5

This framework can capture pitched textures such as wind chimes and church bells, but the paper emphasizes a trade-off between quality and diversity, and it states that speech and structured music are not true textures in this sense (Antognini et al., 2018). A plausible implication is that neural harmonic textures represented purely as stationary statistics are appropriate for local harmonic color and periodicity, but not for long-range harmonic progression.

5. Symbolic and neurosymbolic harmonic texture generation

In symbolic harmony generation, harmonic texture is represented through discrete events rather than continuous spectra. AutoHarmonizer models melody harmonization at sixteenth-note resolution using four aligned streams: melody, beat, key, and chord sequence (Wu et al., 2021). Melody frames are 128-dimensional one-hot vectors, beat frames are 4-way categories, key frames encode 15 key signatures, and chord frames are one-hot vectors over a vocabulary of 1,462 chord types (Wu et al., 2021). The model uses two Bi-LSTM encoders—one for melody and one for beat-plus-key—and a three-layer autoregressive decoder that predicts a chord at every frame.

The key control mechanism is gamma sampling. At generation time, the attribute-related token set is chosen as the previous chord $\hat{\mathbf{o}}_{1:T}$6, and a scalar $\hat{\mathbf{o}}_{1:T}$7 modifies the output distribution so that $\hat{\mathbf{o}}_{1:T}$8 reduces the probability of chord repetition and increases the probability of change, while $\hat{\mathbf{o}}_{1:T}$9 does the opposite (Wu et al., 2021). In effect, $F_0$0 is a global harmonic-density control. This directly shapes harmonic rhythm coverage, chord histogram entropy, chord onset distribution over beat strengths, and functional variety.

The evaluation includes metrics for chord diversity and harmonic rhythm such as Chord Coverage, Chord Histogram Entropy, Chord Tonal Distance, Harmonic Rhythm Coverage, Harmonic Rhythm Histogram Entropy, and Chord Beat Strength (Wu et al., 2021). AutoHarmonizer with higher $F_0$1 values produces richer and more flexible harmonic textures than bar-level or half-bar baselines, and listener studies indicate that these outputs are more often judged human-like than systems with fixed harmonic rhythm.

AI Harmonizer moves from symbolic chord generation to end-to-end neurosymbolic audio harmonization (Blanchard et al., 22 Jun 2025). The pipeline maps a raw sung melody to four-part SATB audio using four stages: Basic Pitch for voice-to-MIDI transcription, a fine-tuned Anticipatory Music Transformer for symbolic harmony generation, RMVPE for continuous $F_0$2 extraction, and RVC for voice synthesis (Blanchard et al., 22 Jun 2025). Harmony generation is constrained so that Alto, Tenor, and Bass each align note-by-note with the Soprano melody; time and duration tokens are fixed to match the melody, and only pitch tokens are sampled under voice-specific logit masking.

For each harmony voice, RMVPE extracts a fine-grained contour $F_0$3, and note-wise semitone shifts $F_0$4 from the symbolic output define

$F_0$5

RVC then renders these pitch-shifted contours in the singer’s timbre using HuBERT embeddings and a VITS-derived voice-conversion stack (Blanchard et al., 22 Jun 2025). The resulting texture is a homorhythmic, four-part choral blend, where symbolic harmony and audio rendering are tightly coupled. Unlike AutoHarmonizer, control is mainly architectural rather than scalar, but both systems show that harmonic texture can be explicitly generated as note-synchronous polyphonic structure.

6. Perception, multi-pitch structure, and emergent harmonic lines

A distinct perspective on neural harmonic textures emerges from “Insights on Harmonic Tones from a Generative Music Experiment” (Deruty et al., 8 Jun 2025). BassNet is described as “a Variational Gated Autoencoder for conditional generation of bass guitar tracks with learned interactive control,” producing per-frame outputs consisting of an $F_0$6 trajectory and a Constant-Q Transform log-magnitude spectrogram (Deruty et al., 8 Jun 2025). Audio is reconstructed by additive sinusoidal synthesis with frequencies

$F_0$7

and waveform

$F_0$8

The network therefore controls a parametric harmonic synthesizer rather than directly synthesizing a waveform.

The paper’s central observation is that producers used BassNet outputs to create the perception of multiple simultaneous melodic lines from single harmonic complex tones. In the track Melatonin, most BassNet outputs contain two or three simultaneous melodic lines even though each frame is generated as a single harmonic series (Deruty et al., 8 Jun 2025). The authors attribute this to three conditions: harmonics loud enough to be individually audible, configurations with only odd harmonics and a weak fundamental, and slight inharmonicity in one example.

This phenomenon is analyzed with STFT spectrograms weighted by ISO 226-2003 equal-loudness curves, and the paper reports listening-test results outside the BassNet context. In an excerpt of Alt-J’s “Hunger of the Pine,” responses to the question of how many simultaneous pitches were heard yielded a mean of 2.22 perceived pitches for a single harmonic complex tone. Other tones used in Hyper Music’s commercial work gave mean perceived pitch counts between 1.7 and 2.45 (Deruty et al., 8 Jun 2025). These data indicate that multi-pitch structured timbres are perceptually robust enough to be treated as a design target.

The BassNet study is significant because it reframes harmonic texture as potentially polyphonic even when the model is trained on monophonic material. A plausible implication is that neural harmonic textures can encode multiple independent perceptual trajectories within the spectral envelope of a single harmonic series, provided that upper-partial amplitudes evolve coherently over time.

The perceptual framework of tonality, pulsality, and noisiness provides a complementary analytical basis for such claims. In that work, energy-based tonality $F_0$9, pulsality $I-1$0, and noisiness $I-1$1 are derived from tract features over a gammachirp cochleagram, and energy-based tonality and pulsality correlate strongly with the first perceptual dimension of human similarity judgments, while energy-based noisiness correlates moderately with the second (Elburg et al., 2017). This suggests that harmonic texture is not only an overview primitive but also a measurable perceptual category.

7. Harmonic basis representations beyond audio

The term “Neural Harmonic Textures” appears directly in computer graphics in “Neural Harmonic Textures for High-Quality Primitive Based Neural Reconstruction” (Condor et al., 1 Apr 2026). In that work, each primitive—such as a 3D Gaussian or triangle—is surrounded by a virtual scaffold carrying local latent features. For 3D Gaussians, the scaffold is a tetrahedron in the primitive’s canonical space, with four vertex features. At the ray–primitive intersection point, features are barycentrically interpolated,

$I-1$2

then passed through periodic activations,

$I-1$3

and alpha-blended along the ray as

$I-1$4

A small deferred MLP then maps the accumulated harmonic vector and a low-order spherical-harmonic view encoding to RGB (Condor et al., 1 Apr 2026).

This method treats alpha blending as a weighted sum of harmonic components and achieves strong results in real-time novel-view synthesis. On MipNeRF360 it reports PSNR $I-1$5, SSIM $I-1$6, and LPIPS $I-1$7; on Tanks & Temples, PSNR $I-1$8, SSIM $I-1$9, and LPIPS $\Gamma$00; and on Deep Blending, PSNR $\Gamma$01, SSIM $\Gamma$02, and LPIPS $\Gamma$03 (Condor et al., 1 Apr 2026). In a controlled comparison with equal appearance-parameter counts, 3DGUT+NHT achieves PSNR $\Gamma$04, outperforming SH-based variants while maintaining real-time performance of roughly 140–240 FPS depending on dataset (Condor et al., 1 Apr 2026). The representation is also applied to semantic reconstruction and 2D image fitting.

Although this domain is distinct from audio, the structural analogy is direct. Local interpolated features act as latent coefficients, periodic activations impose a Fourier-like basis, and the final signal is a weighted harmonic sum. This suggests that “harmonic texture” can denote a broader neural design pattern: local features, basis expansion through periodic nonlinearities, and efficient deferred decoding.

A related but distinct use of neural textures appears in “Dynamic Neural Textures: Generating Talking-Face Videos with Continuously Controllable Expressions” (Ye et al., 2022). There, expression-dependent facial appearance is encoded by dynamic neural textures in UV space, generated from a Continuous Intensity Expression Coding vector $\Gamma$05, effectively as a linear combination of learned neural texture bases. The paper states that dynamic neural textures are obtained by “blending a set of neural textures bases,” using a fully connected transcoding network to infer weights from expression codes (Ye et al., 2022). While the paper does not use the term harmonic in the Fourier sense, it presents a basis-mixing view of neural texture in which semantic control variables modulate latent texture components.

8. Scope, limitations, and synthesis

The surveyed literature supports several distinct interpretations of neural harmonic textures. In waveform generation, they are explicit periodic and aperiodic branches with a learnable crossover frequency (1908.10256). In interpretable system models, they are time–frequency dependency maps and cross-frequency kernels (Helwani et al., 2023). In spectrogram texture models, they are stationary feature correlations and rhythmic autocorrelations (Peng et al., 2018, Antognini et al., 2018). In symbolic and neurosymbolic music systems, they are controllable chord-change patterns, SATB voice structures, and rendered choral blends (Wu et al., 2021, Blanchard et al., 22 Jun 2025). In perceptual and creative practice, they include multi-pitch structured timbres formed from controlled partial trajectories (Deruty et al., 8 Jun 2025). In graphics, they become primitive-local harmonic feature fields decoded by periodic activations and deferred neural shading (Condor et al., 1 Apr 2026).

Several limitations recur across these formulations. Gram-based audio texture models assume stationarity and therefore do not handle speech or structured music well (Antognini et al., 2018). Spectrogram style-transfer systems ignore phase and do not explicitly model pitch or harmony (Peng et al., 2018). AutoHarmonizer controls density with a single global scalar $\Gamma$06, not a time-varying profile, and remains symbolic rather than voicing-level (Wu et al., 2021). AI Harmonizer is restricted by its JSB Chorales fine-tuning and strict note-synchronous voice alignment (Blanchard et al., 22 Jun 2025). BassNet’s multi-pitch behavior emerges in practice but is not directly supervised as polyphony (Deruty et al., 8 Jun 2025). Neural Harmonium requires analytic time–frequency representations and incurs nontrivial complexity (Helwani et al., 2023). Primitive-based NHT gains expressivity but can overfit under sparse supervision and runs slightly slower than pure SH-based Gaussian splatting (Condor et al., 1 Apr 2026).

Taken together, these results indicate that neural harmonic textures are best understood as a design family rather than a single method. Their defining property is explicit structure in how harmonic information is represented, controlled, or decoded. In audio, this usually means periodic excitation, cross-frequency organization, or symbolic harmony; in rendering, it means periodic basis functions over local latent coordinates. This suggests a unifying characterization: neural harmonic textures are neural representations in which structured harmonic components are made first-class objects of modeling, rather than being left entirely implicit in generic deep features.