Libretto: Giving LLM Agents a Sense of Musical Structure

Abstract: Generative music systems can now produce impressive audio from text prompts, but audio outputs are difficult to inspect, edit, and diagnose as musical structure. We introduce Libretto, an agent-facing framework for symbolic music generation and revision. Libretto uses an LLM-native grammar with explicit onset slots, voices, and bar-level organization, then evaluates each piece in a corpus-calibrated statistical space over rhythm, harmony, melody, texture, form, and variation. The same structural axes support retrieval, diagnosis, copy-risk control, and iterative self-revision. Across gap filling, reference-guided full-piece generation, gradual morphing, and educational music generation, Libretto turns symbolic music from a raw token sequence into a measurable and editable object for language-model agents.

• The paper introduces a novel LLM-native grammar that explicitly encodes musical structure with onset slots, voice assignments, and bar segmentation.
• It leverages a corpus-calibrated 29-dimensional fingerprint to measure rhythmic, harmonic, melodic, texture, and formal elements for robust symbolic music evaluation.
• Empirical results show that iterative retrieval and revision significantly improve generation pass rates for tasks like gap filling and full-piece composition.

Libretto: Agentic Symbolic Music Creation via Corpus-Calibrated Structural Measurement

Motivation and Context

Libretto addresses symbolic music generation in the context of LLM-driven agentic workflows. While audio-domain generative models such as MusicGen [Copet et al., 2023] and MusicLM [Agostinelli et al., 2023] yield high-quality audio, they lack transparent symbolic structure for precise editing, diagnosis, and musical reasoning. Symbolic music—MIDI-like or ABC notations—supports editable representations but often poses challenges in timing clarity, structure inspection, and agentic local editing, especially due to implicit event timing or cumbersome grammar. Libretto proposes a novel framework designed explicitly for LLM agents, leveraging a grammar with explicit onset slots, voice assignments, and bar-level segmentation. Structural evaluation is grounded not in heuristics or subjective ratings, but in corpus-calibrated statistical measurement over interpretable axes.

Agent-Facing Grammar and Structural Axes

Libretto introduces a plain-text, LLM-native grammar optimized for symbolic music representation. Each piece contains a global header for metadata (key, meter, tempo, grid, bar count), explicit voice declarations, and per-bar blocks with chord labels and voice-specific note events. Timing is encoded via integer onset slots, eliminating the ambiguity and computational overhead of duration-based recovery typical in ABC notation.

From these tokens, Libretto computes a 29-dimensional structural fingerprint spanning rhythm (syncopation, onset density, triplet share), harmony (chromaticism, distinct pitch classes, chord vocabulary density), melody (range, step ratio, ascending ratio), texture (voice count, chord width, simultaneity), form (self-similarity, novelty, section density), and within-song variation. Corpus percentile ranking is used for each axis, making structural extremes and idiomatic bands directly measurable, independent of subjective preferences.

Strong separation of idiomatic similarity and direct copying is maintained. Copy risk is quantified at the note-level via bar-aligned onset–pitch pairs, calibrated to corpus statistics. Novelty gates enforce pedagogical non-redundancy in education tasks.

Retrieval, Feedback, and Revision Loop

Libretto frames generation as an iterative generate–measure–revise process. The agent:

1. Retrieves genre-grounded concepts and real musical excerpts for reference.
2. Generates a candidate piece in grammar form.
3. Measures it via structural fingerprint and copy risk, evaluates gates for degeneracy and genre fit.
4. Receives musician-readable feedback highlighting structural axes needing adjustment, not raw metrics.
5. Iteratively revises the candidate, retaining improvements according to corpus-calibrated diagnostics.

These mechanisms enable targeted musical diagnosis (e.g., texture sparsity, harmonic instability) without imposing hard numerical targets, thus supporting nuanced agentic self-improvement.

Application-Level Evaluation

Libretto supports multiple symbolic-music generation tasks with a unified workflow:

• Gap Filling: The agent infers missing regions in the context, passing only if structural degeneracy and copying are avoided.
• Full-Piece Generation: The agent generates complete multi-voice, long-form pieces (≈100 bars), leveraging retrieval for genre grounding and structural integrity.
• Morphing: Gradual stylistic transition between source and target, monitored via fingerprint progress curves.
• Education Drills: Generation of short pedagogical examples matching explicit theory concepts and constraints, enforcing novelty relative to shown material.

Empirical results demonstrate the impact of the retrieval+revision loop: For gap-filling, pass rates improve from 12% (single-shot) to 39% (looped); in full-piece generation, from 62% to 94%. Retrieval triples pass rate (to 75%) in full-piece generation by reducing structural degeneracy rather than increasing corpus overlap.

Genre-specific fingerprints are interpretable and discriminate stylistic tendencies (e.g., jazz harmonic complexity, electronic self-similarity) with corpus percentile gaps up to 50 points. Gates are calibrated on real music, avoiding excessive rejection of human pieces and targeting structural collapse and copy failures.

Implications and Future Prospects

Practically, Libretto’s grammar and structural axes afford explicit, agent-driven music generation, retrieval-based grounding, and nuanced iterative revision, transforming symbolic music from opaque token sequence to a measurable, editable object for LLM agents. The corpus-calibrated fingerprint enables structural diagnosis and genre fit monitoring without recourse to subjective listen tests.

Theoretically, Libretto provides a formal substrate for agentic composition, paving the path for reinforcement learning or hierarchical control in symbolic music. Structural axes could be expanded or pruned, with decorrelation and meaningful variation as guiding principles. The framework motivates symbolic agents capable of reasoned revision, controlled idiomatic divergence, and pedagogically grounded creation.

Speculatively, future AI developments could leverage Libretto-like representation-evaluation loops for interactive, adaptive music learning platforms, compositional assistants, and automated music pedagogy, extending to broader creative domains where structure and editability are paramount. Furthermore, reinforcement on structural axes, copy risk minimization, and human preference integration could drive agentic symbolic music toward both generative robustness and creative diversity.

Conclusion

Libretto presents an agent-facing framework for symbolic music generation and revision leveraging an explicit grammar and corpus-calibrated structural axes. The system supports multiple compositional tasks through retrieval-grounded iterative revision, enabling LLM agents to not only emit notes but to reason over and refine musical structure. Its methodology provides robust, interpretable diagnostics, augments generation with real music retrieval, and raises task pass rates via explicit structural measurement, suggesting promising directions for future agentic symbolic-music applications and reinforcement-driven workflows (2606.22708).