Higher Harmony Music Generator

• Higher Harmony Music Generator is a computational system that generates rich, multi-voice harmonies by jointly modeling vertical and horizontal musical structures.
• It employs techniques such as chord tokenization, hierarchical Transformers, and rule-based optimization to regulate harmonic rhythm and voice leading.
• Recent advancements include end-to-end audio augmentations and quantum-inspired optimization that enhance perceptual coherence and musical plausibility.

Searching arXiv for the cited harmonization and higher-harmony generation papers to ground the article in current literature. Higher harmony music generation denotes a class of computational music systems that generate harmony at a level above isolated note continuation: chord progressions for a given melody, higher-voice lines, multi-track accompaniment, chorale-style textures, and orchestrated realizations conditioned by harmonic scaffolds. Taken together, the literature suggests that the defining objective is not merely polyphony, but explicit control over harmonic function, harmonic rhythm, voice leading, textural coherence, tonal structure, or perceptual tension. Early deep-learning work already framed the problem as generating music with both harmony and melody rather than only a single monophonic line (Huang et al., 2016). More recent systems make harmony a first-class control variable through chord tokens, phrase tokens, melody-derived chord summaries, rule-based optimization, perceptual-feature trajectories, or beat-quantized harmony skeletons (Zhang et al., 2021, Wu et al., 2021, Blanchard et al., 22 Jun 2025, He et al., 28 Apr 2026).

1. Conceptual scope and problem formulation

A higher harmony generator differs from a monophonic melody model by treating vertical and horizontal structure jointly. In polyphonic settings, harmony is the vertical dependency among simultaneous notes, while melody is the horizontal dependency across time; several systems explicitly state that both must be modeled together for musically plausible output (Zhou et al., 2023). In accompaniment settings, harmony is not only chord identity but also chord progression, harmonic rhythm, texture, and the dependency of one track on another, as emphasized by pop accompaniment and harmony-aware pop generation models (Ren et al., 2020, Zhang et al., 2021).

Melody harmonization remains a central formulation. Here, the input is a melody, and the output is a chord progression, a higher-voice line, a chorale texture, or a full accompaniment. AutoHarmonizer formulates the task as frame-level chord generation with flexible harmonic rhythm and controllable harmonic density (Wu et al., 2021). The 2026 hybrid systems formulate it as structured decision-making over a constrained combinatorial space in which multiple chord candidates are explored and then refined by tonal rules (Pavlicek, 7 Jul 2026, Pavlíček et al., 6 Jul 2026). AI Harmonizer extends the formulation from symbolic harmonization to vocal augmentation: a solo sung melody in audio becomes a four-part chorale-style arrangement without requiring a key, scale, or tonal center from the performer (Blanchard et al., 22 Jun 2025).

A plausible implication is that “higher harmony” names a design orientation rather than a single architecture. In the surveyed literature it includes symbolic melody-to-chord systems, multi-track accompaniment models, harmony-aware form models, orchestral generators with harmony skeletons, and end-to-end vocal or audio systems.

2. Representational foundations for harmonic structure

A recurring design choice is to encode harmony explicitly rather than infer it only from raw note streams. Harmony-Aware Hierarchical Music Transformer uses an event-based tokenization with nine event types—Type, Bar, Beat, Tempo, Phrase, Chord, Track, Pitch, and Duration—so that Phrase and Chord are explicit structure-bearing events (Zhang et al., 2021). Choir Transformer similarly introduces a representation with note events and chord events in a chord-first manner, combining chord information with the note sequence to model melody and harmony together (Zhou et al., 2023).

Several systems compress melody into a higher-level harmonic summary before generating accompaniment. A-Muze-Net generates the right hand first, then conditions the left-hand LSTM on a chord embedding derived from the right-hand bar; each bar is mapped to the closest chord from a set of 253 chord types, including major and minor chords, 7th chords, 6th chords, 9th chords, 13th chords, diminished and augmented chords (Goren et al., 2021). This bar-level conditioning is a compact harmonic abstraction rather than raw note-to-note copying.

Key-relative and function-relative representations are also prominent. The emotion-driven harmonization model replaces raw pitch/chord encoding with a functional representation centered on key, Roman numerals, and scale degrees; it begins sequences with Emotion_Positive or Emotion_Negative, uses 24 key types, represents chords relative to the key, and decomposes melody pitch into Octave_* and Degree_* (Huang et al., 2024). AutoHarmonizer aligns melody, beat, key, and chord sequences at sixteenth-note resolution and uses a 1,462-way chord output space, including a rest symbol (Wu et al., 2021). CPFG-Net moves one level higher still: the target representation is not chords directly but sequences of tension, distance, strain, and a 24-class tonality variable (Huang et al., 18 Nov 2025).

At the orchestral scale, SymphonyGen uses a beat-quantized multi-voice harmony skeleton as a “short-score” condition that specifies what pitches are allowed to be used in the music as harmonic tones while preserving textural diversity (He et al., 28 Apr 2026). Taken together, these systems suggest that higher harmony generation depends heavily on choosing representations that expose tonal function, phrase structure, or harmonic admissibility before detailed note realization.

3. Architectural families and generative mechanisms

One major family uses sequential neural generators whose conditioning signal is itself harmonic. A-Muze-Net employs two LSTM-based networks: the right-hand LSTM generates melody, and the left-hand LSTM generates harmony conditioned on the melody-derived chord representation (Goren et al., 2021). PopMAG uses MuMIDI, a recurrent Transformer, and extra long-context as memory to model multiple note attributes together and preserve long-range accompaniment consistency across tracks and bars (Ren et al., 2020). Choir Transformer replaces RNN-based polyphonic modeling with a Transformer using relative positional attention, which is intended to capture repeated motifs, voice-leading patterns, harmonic resolutions, and long-range chord progressions (Zhou et al., 2023).

A second family uses hierarchical structure models. HAT places a Bottom Song Transformer below a Hierarchical Structure-Enhanced Module and a Top Song Transformer above it. Inside the HSE, a Texture Transformer models chord progressions within phrases, and a Form Transformer models the evolution of phrase-level texture summaries across phrases (Zhang et al., 2021). SymphonyGen extends this logic to a 3D Bar–Track–Event hierarchy, separating macro-time planning, orchestral layering, and event-level realization (He et al., 28 Apr 2026).

A third family treats harmonization as search or optimization rather than next-token prediction. The multi-objective genetic algorithm system uses GA1 to generate candidate polyphonic pieces, trains two Bi-LSTM regressors on expert and ordinary listener scores, and then uses GA2 to optimize grammar accuracy together with learned expert and ordinary listener satisfaction (Majidi et al., 2021). The 2026 quantum-inspired systems maintain weighted distributions over candidate chords rather than selecting one immediately, evaluate full harmonizations with global objectives, and then apply classical optimization for cadence stabilization, inversion selection, harmonic smoothing, and voice-leading corrections (Pavlicek, 7 Jul 2026, Pavlíček et al., 6 Jul 2026).

A fourth family adds melody guidance to broader generative tasks. MG2 first aligns text, waveform, and melody using Contrastive Language-Music Pretraining, retrieves a relevant melody via HNSW, fuses the retrieved melody with the text query, and conditions a retrieval-augmented diffusion module on both (Wei et al., 2024). Although its task is text-to-music generation rather than melody harmonization alone, it explicitly frames melody as the mechanism that maintains intrinsic harmony across segments.

4. Control, theory, and harmonic decision-making

Higher harmony generation is frequently defined by controllability. AutoHarmonizer uses gamma sampling to bias whether the model repeats the previous chord token or moves away from it, thereby controlling harmonic density without retraining; because chords are generated frame-by-frame at sixteenth-note resolution, harmonic rhythm is flexible rather than fixed at one chord per bar or half-bar (Wu et al., 2021). The paper introduces Harmonic Rhythm Coverage, Harmonic Rhythm Histogram Entropy, and Chord Beat Strength to quantify this behavior (Wu et al., 2021).

Other systems encode explicit music theory directly into the generator. Harmonizer uses Schoenberg’s Chart of Regions as a 24×24 2D array with the melody’s key at the center; horizontal movement follows the circle of fifths, and vertical movement navigates relative minor, major, and parallel minor regions (Fernandes, 5 Jan 2025). The same prototype offers simple harmonizations, Schoenberg-region traversal, Giant Steps-style motion, and church-mode harmonizations, with explicit use of secondary dominants and ii–V–I turnarounds (Fernandes, 5 Jan 2025). The Java algorithm and app utility take an even more enumerative approach, generating all possible four-chord and eight-chord progressions under directed major and minor transition rules and reporting 3,297 total 4-Chord Progressions and 405,216 Total 8-Chord Progressions (Lakshminarasimhan, 2024).

Perceptual control constitutes a different strand of theory-informed generation. CPFG-Net predicts tension, distance, strain, and tonality from melody with a CVAE, then recovers chords by minimizing deviation from target perceptual trajectories in Spiral Array space (Huang et al., 18 Nov 2025). The emotion-driven Transformer likewise makes key a first-class control variable, factorizing the task as

$$p(k, C \mid e, M) = p(k \mid e)\, p(C \mid e, k, M),$$

and reports that rule-based key choice performs better for valence control than model-based sampling (Huang et al., 2024).

The hybrid 2026 harmony systems articulate control as a separation between exploration and correction. One module explores a huge search space—reported as exceeding $10^{50}$ alternative harmonic configurations in the maintainability-oriented system—while a rule-based optimizer reduces excessive chord changes, smooths bass motion, strengthens cadences, and adjusts harmonic function and chord complexity (Pavlicek, 7 Jul 2026). This suggests a general trend toward transparent systems in which multiple valid harmonic realizations are preserved, but stylistic plausibility is still enforced explicitly.

5. From symbolic harmony to audio realization

A significant recent development is the move from symbolic harmonization to end-to-end audio systems. AI Harmonizer is an offline, end-to-end vocal augmentation system with four stages: Basic Pitch transcribes the input vocal recording to MIDI; a custom-trained Anticipatory Music Transformer generates three additional harmony voices; RMVPE extracts the detailed continuous pitch contour $f_0$; and Retrieval-based Voice Conversion synthesizes the three harmony voices so that they sound like natural extensions of the singer (Blanchard et al., 22 Jun 2025). The bridge from symbolic harmony to audio is the explicit pitch-shifting rule

$f_0^{out} = f_0^{in}(t)\cdot 2^{h_i/12}$

within each note region, where $h_i$ is the semitone interval assigned by the harmony model (Blanchard et al., 22 Jun 2025). The system is trained on JSB Chorales and constrained so that, for each input melody note, each harmony voice generates at most one corresponding note and only the pitch token is sampled (Blanchard et al., 22 Jun 2025).

The agent-based Higher Harmony Music Generator generalizes this end-to-end idea to symbolic scores. It defines the pipeline

$\mathcal{G}: S_{XML} \rightarrow W_{audio},$

with a Music-Ingestion Agent, a Chord-Knowledge Agent powered by Chord-Former, a Harmony-Generation Agent using Harmony-GPT and Rhythm-Net, and an Audio-Production Agent using a GAN-based Symbolic-to-Audio Synthesizer (Ganapathy et al., 29 Sep 2025). The paper explicitly describes the higher harmony line as a new higher-voice symbolic line above the input melody (Ganapathy et al., 29 Sep 2025).

SymphonyGen remains symbolic at output time, but its refinement pipeline is audio-aware. After pretraining, it uses Group Relative Policy Optimization with a cross-modal audio-perceptual reward based on CLaMP 3 and MuseScore-rendered audio, and it adds a dissonance-averse sampling algorithm that subtracts dissonance contributions from pitch logits relative to the harmony skeleton (He et al., 28 Apr 2026). This places the harmonic control signal and the audio perceptual objective in the same training loop.

A more unconventional case is BassNet, reported in the Sony CSL Paris and Hyper Music studio-lab experiment. BassNet predicts $f_0$ trajectories and Constant-Q Transform log-magnitude spectrograms and renders them by additive synthesis. Although it is not designed as a harmony generator, the outputs often contained two or three simultaneous melodic lines perceptually embedded in a single harmonic complex tone sequence (Deruty et al., 8 Jun 2025). A plausible implication is that audio-domain harmonic structure can emerge not only from explicit chord generation, but also from the time-varying salience of upper partials.

6. Evaluation regimes, applications, and open issues

Evaluation remains heterogeneous across the literature. HAT introduces structure-oriented objective metrics: Accompaniment Groove Stability for texture and Chord Progression Realism for form, and reports AGS values of 0.193 for CP-Transformer, 0.256 for Music Transformer, and 0.474 for HAT (Zhang et al., 2021). MG2 evaluates with FAD, KL divergence, and IS, reporting on MusicCaps $FAD = 1.91$, $KL = 1.21$, $IS = 2.11$, and on MusicBench $10^{50}$0, $10^{50}$1, $10^{50}$2, together with human evaluations from 163 subjects across recognizability, text relevance, satisfaction, quality, and market potential (Wei et al., 2024). AutoHarmonizer uses ACC, CC, CHE, CTD, CTnCTR, PCS, MCTD, HRC, HRHE, and CBS, and complements them with a listener discrimination study involving 83 participants (Wu et al., 2021).

Other systems foreground internal structural metrics rather than reference imitation. The maintainable hybrid system compares Raw Generator and Optimized Generator on chord density, average chord duration, average bass jump, segment length standard deviation, exact match, functional agreement, harmonic similarity, and final function match; optimization changes chord density from 1.5924 to 2.2673 and average bass jump from 3.5615 to 1.2165 while leaving Final Function Match at 90.91% (Pavlicek, 7 Jul 2026). The related quantum-inspired decision model reports that optimization reduces chord density from 4.00 to 2.71 on Autumn Leaves and from 4.00 to 2.53 on It’s a Long Way to Tipperary, increases average chord duration, and shifts function balance toward stronger dominant presence (Pavlíček et al., 6 Jul 2026).

Applications are equally broad. AI Harmonizer is positioned for composition, rehearsal, performance enhancement, and experimental live vocal layering, even though the current implementation is offline (Blanchard et al., 22 Jun 2025). MG2 explicitly studies short-video creation as an application domain and reports that 75.72% of short-video bloggers said the music is suitable for short-video editing (Wei et al., 2024). Harmonizer is presented as a composer’s tool rather than only auto-accompaniment (Fernandes, 5 Jan 2025). SymphonyGen targets contemporary cinematic orchestration (He et al., 28 Apr 2026).

Several limitations recur. AI Harmonizer does not report a formal quantitative music-theoretic benchmark or a listening-test study in the manuscript and remains too slow for true real-time operation, with AMT generation and $10^{50}$3 extraction identified as the slowest components (Blanchard et al., 22 Jun 2025). The quantum-inspired decision model notes that increased harmonic complexity is not always perceived as more natural, especially outside jazz-like contexts (Pavlíček et al., 6 Jul 2026). The MO-GA system depends on a relatively small dataset of 235 polyphonic music pieces and human scoring from 15 expert and 15 ordinary listeners (Majidi et al., 2021). Style restriction is also explicit in systems trained on JSB Chorales or Bach-oriented corpora, where coherence is gained at the cost of stylistic diversity (Blanchard et al., 22 Jun 2025, Goren et al., 2021).

Taken together, the field has moved from proof-of-concept polyphonic LSTMs toward controllable, theory-aware, and increasingly end-to-end systems. The dominant research direction is not the elimination of constraint, but the reorganization of constraint: chord and phrase tokens, harmonic density controls, perceptual trajectories, harmony skeletons, weighted candidate distributions, and audio-aware refinement all serve to make harmony generation more autonomous while keeping harmonic decisions legible, steerable, and musically structured.