ACE-Step 1.5: Pushing the Boundaries of Open-Source Music Generation

Abstract: We present ACE-Step v1.5, a highly efficient open-source music foundation model that brings commercial-grade generation to consumer hardware. On commonly used evaluation metrics, ACE-Step v1.5 achieves quality beyond most commercial music models while remaining extremely fast -- under 2 seconds per full song on an A100 and under 10 seconds on an RTX 3090. The model runs locally with less than 4GB of VRAM, and supports lightweight personalization: users can train a LoRA from just a few songs to capture their own style. At its core lies a novel hybrid architecture where the LLM (LM) functions as an omni-capable planner: it transforms simple user queries into comprehensive song blueprints -- scaling from short loops to 10-minute compositions -- while synthesizing metadata, lyrics, and captions via Chain-of-Thought to guide the Diffusion Transformer (DiT). Uniquely, this alignment is achieved through intrinsic reinforcement learning relying solely on the model's internal mechanisms, thereby eliminating the biases inherent in external reward models or human preferences. Beyond standard synthesis, ACE-Step v1.5 unifies precise stylistic control with versatile editing capabilities -- such as cover generation, repainting, and vocal-to-BGM conversion -- while maintaining strict adherence to prompts across 50+ languages. This paves the way for powerful tools that seamlessly integrate into the creative workflows of music artists, producers, and content creators. The code, the model weights and the demo are available at: https://ace-step.github.io/ace-step-v1.5.github.io/

• The paper presents a hybrid model that integrates a language model for planning with a diffusion transformer for precise acoustic rendering.
• It employs an RL-driven data pipeline and advanced annotation techniques to optimize semantic-to-acoustic alignment.
• Evaluation demonstrates significant improvements in audio quality, efficiency, and multi-modal integration compared to proprietary systems.

ACE-Step 1.5: Advancements in Open-Source Music Generation Architecture

Overview

ACE-Step v1.5 introduces a highly optimized, open-source music generation framework that bridges the gap between experimental diffusion-based models and production-grade music synthesis, combining high fidelity, versatile controllability, and operational efficiency on consumer-grade hardware. The architecture leverages a hybrid reasoning-diffusion paradigm, employing a LLM (LM) as a high-level planner and a Diffusion Transformer (DiT) as the acoustic renderer. Enhanced by a rigorous, RL-driven pipeline and a sophisticated data annotation scheme, ACE-Step 1.5 establishes a unified platform for synthesis, editing, and multi-modal music manipulation, achieving performance metrics previously limited to proprietary systems.

RL-Driven Data Infrastructure and Annotation

A fundamental source of improvement in ACE-Step 1.5 is the realignment of its data pipeline, shifting from maximizing dataset size to optimizing for semantic alignment and annotation density. The initial annotation employed Gemini 2.5 Pro to transcribe and structure a Golden Set of 5 million audio samples, which seeded subsequent domain-specialist models (ACE-Captioner and ACE-Transcriber). Advanced reward models, trained using contrastive negative pairs and enhanced with RL (GRPO), provided a robust filter and refinement mechanism—effectively eliminating hallucinations and optimizing semantic-to-acoustic alignment.

Figure 1: The RL-driven annotation pipeline maximizes semantic-audio correlation through multi-stage filtering and self-evolving supervision.

The infrastructure leverages extensive augmentation strategies, such as Qwen3-30B-A3B-based prompt rewrites, fostering robustness over a wide input distribution (from terse to verbose prompts). For global coverage, a stochastic Romanization procedure during training exposes the model to phonological variations, improving multilingual lyric generation and recognition.

Hybrid Model Architecture

Central to ACE-Step 1.5 is its bifurcated architecture:

• High-Fidelity 1D VAE: Surpassing the limitations of mel-spectrogram conditioning, a pure waveform-domain 1D VAE achieves $1920\times$ compression with 48kHz stereo audio, preserving transient structure crucial for professional sound quality.
• Diffusion Transformer Backbone: A 2B-parameter DiT, equipped with hybrid attention (sliding window + global group query), facilitates long-range musical consistency and efficient conditioning via FSQ tokenization. This enables scalable, multi-modal operations across source/mask configurations, drastically reducing inference steps (from 50 to 4–8) through dynamic-shift adversarial distillation.

  Figure 2: The ACE-Step 1.5 model framework separates structural planning (LM) from acoustic rendering (DiT), supporting flexible, modular conditioning.

This architecture unifies six core tasks (standard synthesis, cover generation, repainting, track extraction, layering, and completion) through a masked generative framework, leveraging shared latent spaces for high task efficiency and fidelity.

Figure 3: The flexible masked generative framework enables ACE-Step 1.5 to generalize across diverse music generation and editing tasks.

LLM as Composer Agent

The Qwen-based LM is trained in a Chain-of-Thought (CoT) paradigm, outputting structured metadata (BPM, key, arrangement) to explicitly ground generation. The LM supports planner, listener, co-pilot, and refiner interaction modes, enabling conversion from vague prompts to detailed compositions, semantic reverse-mapping, creative structure expansion, and prompt normalization.

The explicit YAML-based CoT reasoning not only increases structural and stylistic coherence but also facilitates interoperability with external music production and management systems.

Training Pipeline and Distillation

ACE-Step 1.5 adopts a progressive, curriculum-based training regimen:

• Foundation Pre-training: 20M Source = Silence generative samples for broad acoustic and linguistic priors.
• Omni-Task Fine-Tuning: 17M samples emphasizing editing, layering, and manipulation, leveraging separated stems and mask-conditioned targets.
• High-Quality SFT: Final 2M high-alignment samples curated for strong instruction adherence.

Distillation utilizes a dynamic-shift DMD2 framework, augmented with adversarial objectives and ConvNeXt-latent discriminators operating under Flow Matching criteria. Notably, stochastically varying the time-step shift restricts overfitting, further accelerating inference and, in ablation, enables the distilled student model to surpass its teacher in both fidelity and efficiency.

Figure 4: The training pipeline systematically progresses from general generative modeling to precision multi-task fine-tuning and fast adversarial distillation.

Intrinsic Reinforcement Learning for Alignment

Intrinsic RL is employed both at the DiT and LM levels:

• DiT Alignment: The Attention Alignment Score (AAS), derived from dynamic time warping of cross-attention heatmaps, directly regularizes lyric-to-audio and structure-to-output adherence without exogenous reward signals—improving synchronization and reducing output hallucinations.
• LM Alignment: PMI-based reward modeling penalizes genericity and rewards precise semantic match between generated audio codes and descriptive captions, dynamically weighted for style, lyric, and metadata fidelity.

This policy shaping yields superior prompt adherence and multi-language robustness while reducing dependence on potentially biased external evaluations.

Evaluation and Benchmarking

ACE-Step 1.5 demonstrates strong performance across commercial and open-source benchmarks. It achieves best-in-class AudioBox CU (8.09) and PQ (8.35) scores, high temporal consistency, and bridges the style/lyric alignment gap compared to leading proprietary systems. Subjective evaluations via Music Arena locate its perceptual quality between Suno-v4.5 and Suno-v5, indicating commercial-competitive generation at a fraction of the computational footprint.

The authors foreground a rigorous "Usability" checklist, encompassing deployment, speed, robustness, editing capability, creative support, and professional knowledge—arguing persuasively for a shift from narrow score-based evaluation toward ecosystem readiness and workflow integration.

Implications and Future Outlook

ACE-Step 1.5 evidences that a modular, decoupled planning-rendering framework, supported by sophisticated data engineering and intrinsic RL, can deliver high-fidelity, multi-modal music generation within consumer hardware constraints. The implications for open-source creative tools are substantial: democratization of advanced generative models, transparent extensibility (e.g., lightweight LoRA fine-tuning for user personalization), and a unified substrate for further symbolic-audio alignment research.

Practically, ACE-Step 1.5 paves the way for DAW integration, agentic music editing workflows, and real-time, style-consistent generation in production settings. Theoretically, its approach to intrinsic reward modeling and attention regularization may generalize to other multi-modal conditional synthesis domains.

Future efforts should focus on scaling model capacity to further enhance acoustic texture and global style coverage, integrating richer world knowledge for culturally nuanced generation, and enhancing agentic control (e.g., fine-grained lyric synchronization, real-time editing). The framework provides a robust baseline for cross-disciplinary advances at the intersection of generative audio, symbolic music theory, and AI-driven creative assistance.

Conclusion

ACE-Step 1.5 constitutes a comprehensive, production-ready open-source solution for music generation that systematically addresses previous efficiency, fidelity, and control limitations. Through architectural modularity, advanced data infrastructure, and RL-driven alignment, it delivers commercial-grade performance while retaining accessibility. The paradigm and engineering principles established by ACE-Step 1.5 are poised to accelerate further developments in open, democratized, multi-modal generative AI for music and beyond.