CoComposer: Multi-Agent Music Composition
- CoComposer is a multi-agent symbolic music composition framework that decomposes music creation into planning, melodic crafting, accompaniment arrangement, revision, and review.
- The system employs five specialized agents coordinated through AutoGen, using ABC notation and a MIDI backend to enhance controllability, interpretability, and iterative editing.
- Evaluated with AudioBox-Aesthetics, CoComposer shows improved production complexity and transparent score editing, addressing limitations in prompt expressiveness and musical nuance.
CoComposer is a multi-agent symbolic music composition system that uses generic LLMs to transform natural-language music requests into multi-track music output. It is organized around five collaborating agents, uses ABC notation as its intermediate representation, and renders output through a MIDI backend. The system is positioned as a response to several limitations in existing AI music tools—short generation duration, limited prompt expressiveness, musical quality concerns, poor controllability, and weak editability and transparency—and it is evaluated with AudioBox-Aesthetics on four compositional criteria (Xing et al., 29 Aug 2025).
1. Definition and objectives
CoComposer, expanded in the paper as “Collaborative Composer,” is presented as an LLM multi-agent collaborative music composition framework whose internal workflow follows a traditional composition logic: “main melody - arrangement and orchestration - revision.” Its core design premise is that music composition is not treated as a single monolithic generation act, but as a decomposable process involving planning, melodic writing, accompaniment construction, notation repair, and critical review. This decomposition is intended to improve controllability, interpretability, and editability relative to end-to-end text-to-music generation systems (Xing et al., 29 Aug 2025).
The problem formulation is explicitly practical. The paper identifies short generation duration in some systems, gives MusicFX as an example of producing only 30-second outputs, describes platforms such as Suno as not supporting long prompt inputs well, and argues that generated music often has a “toy-like” character rather than the nuance expected by professional users. It also emphasizes that many existing systems make detailed edits difficult, and that end-to-end generation often sacrifices inspectability of structure. Within LLM-based music systems, CoComposer is additionally framed as a redesign of ComposerX, whose weaknesses are described as including lack of delicacy in expression, a gap between natural-language intentions and actual score realization, instrument-range violations, difficulty aligning multiple voices in polyphonic notation, and weak cadence resolution (Xing et al., 29 Aug 2025).
A central implication is that CoComposer is not primarily optimized as a raw audio generator. It instead occupies the symbolic-compositional end of the design space: natural language is mapped into score-like structure, that structure is iteratively revised, and the resulting notation remains human-readable. This suggests a system aimed less at black-box synthesis and more at “low-threshold, high-controllability” composition.
2. Multi-agent architecture
The system uses five role-playing language agents coordinated through AutoGen. Communication occurs through group chats, and intermediate outputs are posted into a shared dialogue pool that acts as the common context for subsequent rounds. The architecture is summarized by the paper as “task decomposition - creative execution - corrective feedback” (Xing et al., 29 Aug 2025).
| Agent | Function | Main output |
|---|---|---|
| Leader Agent | Parses user request, extracts musical elements, assigns tasks, coordinates revisions | Task instructions |
| Melody Agent | Writes the single-line main melody and selects a lead instrument | ABC melody with %%MIDI program |
| Accompaniment Agent | Creates harmonic and textural support aligned with the melody | Multi-voice ABC accompaniment |
| Revision Agent | Fixes timing and formatting errors under a minimum intervention principle | Corrected ABC notation |
| Review Agent | Evaluates the score from a music-theoretic perspective and proposes improvements | Structured critique |
The Leader Agent functions as planner and controller. It extracts title, genre or style, key or tonality, chord progression, instruments, tempo, and rhythm from the user request, then decomposes the task for the Melody Agent and Accompaniment Agent. In later rounds it reads reviewer feedback and decides what should be changed. The Melody Agent composes the single-line main melody, chooses an instrument, annotates it using %%MIDI program, and outputs only ABC notation. The Accompaniment Agent constructs harmonic progression, accompaniment texture, and supporting instrumental layers, while ensuring alignment with melody in rhythm, key, and structure. The Revision Agent is intentionally narrow: it corrects only confirmed timing and format errors and is instructed not to alter creative content. The Review Agent acts as a critic, evaluating the result across five internal dimensions: Melodic Structure, Harmony and Counterpoint, Rhythmic Complexity, Instrumentation and Timbre, and Form and Structure (Xing et al., 29 Aug 2025).
Architecturally, CoComposer is also a simplification of ComposerX. ComposerX is described as having six agents—Group Leader, Melody Agent, Harmony Agent, Instrument Agent, Reviewer Agent, and Arrangement Agent—whereas CoComposer compresses this to five by folding instrument selection into the Melody and Accompaniment agents and replacing the final arrangement role with a more focused Revision Agent. The paper attributes part of CoComposer’s improvement to this streamlined structure and to “creation-orchestration synchronization” (Xing et al., 29 Aug 2025).
3. Symbolic representation and iterative workflow
CoComposer’s central representation is ABC notation. The full pipeline is natural-language request to ABC notation to MIDI backend to rendered music. This choice is central to the paper’s claims about interpretability and editability: melody, accompaniment, instrumentation metadata, and score structure remain inspectable and directly revisable at the symbolic level, rather than being embedded in an opaque audio latent (Xing et al., 29 Aug 2025).
The workflow has two phases. In the Initialization Creation Phase, the Leader Agent analyzes the prompt and assigns subtasks, the Melody Agent produces the main melody, the Accompaniment Agent builds accompaniment conditioned on that melody, the Revision Agent repairs timing and format issues, and the Review Agent critiques the resulting score. In the Iterative Creation Phase, the Leader Agent reads the review output from the shared dialogue pool, identifies key issues, and instructs the Melody and Accompaniment agents to revise their parts. The Revision Agent then checks the new notation again, after which the process may continue (Xing et al., 29 Aug 2025).
Prompt design is strongly role-constrained. The Melody Agent and Accompaniment Agent are instructed to output only sheet music in ABC format, include MIDI instrument information, wrap the result in code fencing, and avoid explanatory text. The Revision Agent is given a “minimum intervention principle,” meaning that only confirmed timing and formatting errors should be repaired. This is not a secondary creative agent; it is a syntax-and-meter validator. The Review Agent, by contrast, is explicitly evaluative rather than generative.
Several formal details are notably absent. The paper does not provide equations, optimization objectives, pseudocode, a fixed stopping criterion, a formal convergence rule, or a memory module beyond the shared dialogue history. This suggests that CoComposer is better described as a prompt-engineered orchestration system than as a learned planning architecture. The paper itself identifies richer feedback analysis and explicit memory as future directions (Xing et al., 29 Aug 2025).
4. Evaluation methodology and reported performance
The evaluation uses Meta’s AudioBox-Aesthetics model on 20 prompts drawn from the ComposerX prompt set, with 10 prompts abridged to leave more freedom by removing some explicit descriptions such as major keys, specific notes, and musical instruments. The four reported criteria are Production Quality (PQ), Production Complexity (PC), Content Enjoyment (CE), and Content Usefulness (CU). All reported LLM symbolic systems achieved 100% generation success (Xing et al., 29 Aug 2025).
Against ComposerX with GPT-4o, CoComposer with GPT-4o improves on all four reported metrics: CE rises from 6.52 to 6.75, CU from 7.61 to 7.76, PC from 3.72 to 4.13, and PQ from 7.76 to 7.86. Against a single-agent GPT-4o baseline, the main improvement is in Production Complexity: CoComposer scores 4.13 versus 3.92, while CE is 6.75 versus 6.72, CU is 7.76 versus 7.77, and PQ is 7.86 versus 7.88. The paper therefore emphasizes a multi-agent advantage most clearly in production complexity rather than in uniform dominance across all dimensions (Xing et al., 29 Aug 2025).
The paper also evaluates CoComposer with three LLM backbones. CoComposer with GPT-4o reports CE 6.75, CU 7.76, PC 4.13, and PQ 7.86. The DeepSeek-V3-0324 version is close, with CE 6.77, CU 7.70, PC 3.98, and PQ 7.85. Gemini-2.5-Flash is weaker, with CE 6.37, CU 7.57, PC 3.92, and PQ 7.73. The paper’s summary is that GPT-4o performs best overall, although DeepSeek-V3-0324 is numerically slightly higher on CE (Xing et al., 29 Aug 2025).
The comparison to dedicated music generation is more mixed. The reported table compares against MusicFX, which scores CE 7.37, CU 7.93, PC 4.96, and PQ 7.84. On those results, MusicFX exceeds CoComposer in enjoyment, usefulness, and complexity, while production quality is roughly comparable. The paper interprets this as a tradeoff: specialized music models remain stronger in pure musical output, whereas CoComposer is stronger in interpretability and editability because it is symbolic and open. An objective complication is that the abstract refers to MusicLM, whereas the experimental section and table report MusicFX. This suggests a naming inconsistency in the paper text rather than a stable experimental distinction (Xing et al., 29 Aug 2025).
5. Position within related research
CoComposer belongs to a broader line of symbolic and agentic composition systems, but its specific configuration is distinctive. ByteComposer, for example, is also an LLM-centered symbolic music agent, yet it uses a four-stage workflow—“Conception Analysis - Draft Composition - Self-Evaluation and Modification - Aesthetic Selection”—with an Expert, Generator, Voter, and Memory architecture. It is explicitly designed as a human-like melody-composition system and is evaluated as reaching the level of a novice melody composer (Liang et al., 2024). CoComposer is narrower in architectural scope but more explicit about five specialized collaborative agents, ABC notation repair, and comparison to another multi-agent baseline.
JEN-1 Composer occupies a different point in the design space. It is not a symbolic score system but a latent diffusion framework for multi-track audio generation that models marginal, conditional, and joint distributions over aligned tracks within a single model. Its co-composition interface is iterative track selection and regeneration over audio stems rather than ABC-level score editing (Yao et al., 2023). Relative to JEN-1 Composer, CoComposer trades off audio-native fidelity for symbolic transparency.
An earlier precursor is MusECI, introduced in “Composition by Conversation,” which addresses score-level composition and revision through natural language by mapping parsed utterances to symbolic queries and operations over notes, rests, and sequential or parallel structures (Quick et al., 2017). CoComposer can be read as extending this broad conversational-editing tradition into an LLM multi-agent setting, but without MusECI’s explicit query-then-operation formalism.
CoComposer should also not be conflated with similarly named systems outside music. “Composer: Creative and Controllable Image Synthesis with Composable Conditions” is an image-generation framework based on diffusion and composable visual conditions, not a music-composition agent (Huang et al., 2023). The similarity of names reflects a broader use of “composer” for controllable generative systems, but the underlying modalities, representations, and control mechanisms are unrelated.
6. Strengths, limitations, and outlook
The paper presents CoComposer’s main strengths as controllability, interpretability, and editability. Because it uses ABC notation, users can inspect melody, accompaniment, and instrumentation metadata directly. Because it decomposes the workflow into roles, it exposes intermediate reasoning stages that are hidden in end-to-end generators. Because input remains natural language, it also aims to lower the barrier to composition for non-expert users. The authors further emphasize that the framework uses generic LLMs and open-source orchestration tools rather than requiring additional music-specific pretraining (Xing et al., 29 Aug 2025).
The limitations are equally explicit. CoComposer depends on ABC notation and a MIDI backend, and the paper notes that MIDI sound libraries are oriented toward traditional acoustic instruments and are limited for modern synthetic timbres. The system does not use music-specific fine-tuning, which constrains its handling of complex musical structures, polyphony, and long-form organization. The authors also argue that music contains personal experience, cultural background, and spontaneous inspiration that remain difficult for the system to capture. Most importantly, dedicated music models still produce stronger music according to the reported AudioBox metrics (Xing et al., 29 Aug 2025).
Two additional limitations concern system organization. First, symbolic multi-voice generation by general LLMs remains fragile, which is why CoComposer needs a dedicated Revision Agent. Second, the architecture has no explicit long-term memory mechanism beyond the shared dialogue pool. The paper proposes a feedback analysis agent and a richer memory mechanism as future work. This suggests that the current system is best understood as an interpretable collaborative scaffold over generic LLMs rather than a fully stabilized music-composition platform (Xing et al., 29 Aug 2025).
Taken together, CoComposer marks a specific research direction within AI music: not end-to-end text-to-audio generation, and not fully formal symbolic reasoning, but prompt-engineered, role-specialized collaborative composition over editable notation. Its reported gains over ComposerX and over a single-agent baseline in production complexity indicate that multi-agent decomposition is operationally useful. Its weaker performance relative to MusicFX indicates that symbolic openness and direct editability remain distinct from, and in tension with, raw musical output quality.