AutoMashup: Automated Music Mashup System

• AutoMashup is a system for automatic mashup creation that recombines song stems using source separation, musical analysis, and compatibility estimation.
• The method employs a three-stage pipeline—source separation with Demucs, musical alignment using Allin1 and Pyrubberband, and ranking with COCOLA for compatibility.
• Empirical results show that compatibility is asymmetric and that zero-shot embeddings like CLAP and MERT do not capture the perceptual coherence of COCOLA.

AutoMashup denotes a line of systems for automatic music mashup creation in which audio elements from two or more pre-existing songs are recombined into a new work through source separation, musical analysis, alignment, and compatibility estimation. In its 2025 formulation, AutoMashup is organized around three stages—source separation, musical analysis and alignment, and compatibility-based song selection and mashup generation—and uses COCOLA as a reference estimator of stem compatibility while testing whether CLAP and MERT embeddings can support zero-shot ranking of candidate vocal–accompaniment pairs. Its central findings are that mashup compatibility is asymmetric with respect to stem roles and that current general-purpose pretrained audio embeddings do not reproduce the perceptual coherence encoded by COCOLA (Delabaere et al., 30 Jul 2025).

1. Conceptual scope and problem formulation

A music mashup combines audio elements from two or more songs to create a new work (Huang et al., 2021). AutoMashup addresses the automation of that process under the assumption that source separation can isolate musically meaningful components, especially vocals and accompaniment, from full mixes. This shifts the problem from direct mixing of unaltered excerpts toward stem-level recombination, where harmonic fit, tempo consistency, structural alignment, and perceptual coherence must all be managed jointly.

In the 2025 system, the core operational unit is a candidate pair consisting of a vocal stem and an accompaniment stem. Compatibility estimation is then used to rank such pairs before alignment and rendering (Delabaere et al., 30 Jul 2025). This framing makes song selection a first-class problem rather than a by-product of rule-based key and tempo matching.

Earlier work on automatic mashup generation had already treated key and tempo adjustment as prerequisites for high-quality mashups and had trained neural models to predict a binary “mashability” score from separated stems (Huang et al., 2021). The later AutoMashup system retains the prerequisite role of alignment, but focuses more sharply on the compatibility function itself and on the feasibility of scaling compatibility search across larger libraries (Delabaere et al., 30 Jul 2025).

2. End-to-end pipeline and system architecture

AutoMashup is organized into three major stages tied together by a simple API implemented with Streamlit / Barfi (Delabaere et al., 30 Jul 2025). The first stage is source separation. The input consists of full mixes of two or more pre-existing songs, and Demucs, described as the latest hybrid U-Net + transformer model, is used to extract separate stems, specifically vocals versus accompaniment.

The second stage is musical analysis and alignment. AutoMashup uses Allin1 to predict key, tempo (BPM), beat and downbeat positions, and musical segments such as verses and choruses (Delabaere et al., 30 Jul 2025). BPM and beat alignment are performed by matching the accompaniment’s beats to the vocal stem’s beats, and vice versa, using Pyrubberband for time-stretching. Key alignment is handled by shifting one stem by integer semitones so that its estimated key coincides with the other’s key, thereby avoiding harmonic clashes. Segment alignment aligns or loops matching song sections, for example chorus-to-chorus, so that structure remains coherent.

The third stage is compatibility-driven selection and mashup generation. An offline or interactive module computes a compatibility score between a candidate vocal stem and a candidate accompaniment stem and uses that score to rank possible pairs (Delabaere et al., 30 Jul 2025). High-compatibility pairs are then passed through the alignment pipeline to produce the final mashup.

This workflow closely parallels an earlier stem-based mashup pipeline that first performed music source separation, then constructed matched and unmatched combinations under explicit key and tempo constraints, and finally ranked matched-condition candidates using a neural compatibility model (Huang et al., 2021). A plausible implication is that the 2025 AutoMashup system should be understood as an evolution from pipeline-centric mashup assembly toward compatibility-centric retrieval.

3. Compatibility estimation with COCOLA

AutoMashup employs COCOLA as its reference compatibility estimator (Delabaere et al., 30 Jul 2025). Let $v \in \mathbb{R}^d$ be the embedding of a vocal stem and $i \in \mathbb{R}^d$ the embedding of an instrumental stem, produced by neural encoders $f_v(\cdot)$ and $f_i(\cdot)$. COCOLA defines compatibility through cosine similarity,

$$s_{\text{COCOLA}}(v,i) = \frac{\langle v, i \rangle}{\|v\| \, \|i\|}.$$

Training uses positive pairs $(v_k, i_k)$ from the same song and negative pairs $(v_k, i_j)$ with $j \neq k$, optimized with an InfoNCE-style objective,

$$L = - \frac{1}{N} \sum_{k=1}^N \log \frac{\exp\bigl(s_{\text{COCOLA}}(v_k,i_k)/\tau\bigr)} {\sum_{j=1}^N \exp\bigl(s_{\text{COCOLA}}(v_k,i_j)/\tau\bigr)}.$$

After convergence, $s_{\text{COCOLA}}(v,i)$ serves as a perceptual-compatibility score, where higher values indicate better harmonic and rhythmic coherence (Delabaere et al., 30 Jul 2025).

Within AutoMashup, COCOLA functions as the reference against which alternative similarity mechanisms are assessed. The paper explicitly motivates this by the computational burden of exhaustive pairwise scoring: for large libraries, COCOLA incurs an $i \in \mathbb{R}^d$0 cost when all candidate pairs must be evaluated (Delabaere et al., 30 Jul 2025). This computational characteristic is central to the later exploration of zero-shot retrieval proxies.

4. Zero-shot similarity with CLAP and MERT, and the asymmetry result

To avoid the $i \in \mathbb{R}^d$1 cost of pairwise COCOLA scoring, AutoMashup explores a zero-shot alternative based on pretrained audio encoders: CLAP, trained on audio/text pairs, and MERT, trained on large music collections (Delabaere et al., 30 Jul 2025). For each separated stem, a fixed-length embedding $i \in \mathbb{R}^d$2 is computed. For any candidate pair $i \in \mathbb{R}^d$3, the system calculates

$i \in \mathbb{R}^d$4

and ranks candidate mashups by this cosine similarity. The hypothesis is that $i \in \mathbb{R}^d$5 should correlate with $i \in \mathbb{R}^d$6 (Delabaere et al., 30 Jul 2025).

A key empirical finding is that compatibility is not symmetric in practice. Specifically, the system observes that

$i \in \mathbb{R}^d$7

even when the same two full-song sources $i \in \mathbb{R}^d$8 and $i \in \mathbb{R}^d$9 are involved (Delabaere et al., 30 Jul 2025). The paper states that auditory inspection confirms that $f_v(\cdot)$0 and $f_v(\cdot)$1 can have very different perceptual quality. The stated implication is that models and ranking strategies must treat vocal-to-instrumental and instrumental-to-vocal directions separately rather than assuming a single compatibility score per unordered song pair.

This asymmetry result is important because many retrieval settings implicitly collapse pairwise relations into unordered similarity. Here, the empirical evidence indicates that stem role assignment is part of the compatibility relation itself. This suggests that mashup retrieval should be formulated as a directional matching problem rather than a symmetric nearest-neighbor search.

5. Experimental study and empirical findings

The evaluation uses a subset of FMA (“Free Music Archive”) filtered to 21 popular-music tracks in major keys between B♭ and D and with durations 184–194 s (Delabaere et al., 30 Jul 2025). All 21 vocals and 21 instrumentals were separated, producing $f_v(\cdot)$2 candidate mashups. Ground-truth compatibility was defined by COCOLA scores computed for each of the 421 valid pairings.

The embedding structure was examined with t-SNE and hierarchical clustering over the 42 stem embeddings. CLAP produced three clusters, described as mostly vocals, mostly instrumentals, and synthetic voices, whereas MERT produced two clusters mixing vocals and instrumentals (Delabaere et al., 30 Jul 2025). The Adjusted Rand Index between the CLAP and MERT clusterings was $f_v(\cdot)$3, indicating no agreement beyond chance.

Correlation analysis compared $f_v(\cdot)$4 with $f_v(\cdot)$5 for both CLAP and MERT across all pairs (Delabaere et al., 30 Jul 2025).

Correlation metric	CLAP	MERT
Pearson	0.051	-0.018
Spearman	0.079	-0.017
Kendall’s $f_v(\cdot)$6	0.053	-0.010

The paper characterizes all of these correlations as essentially zero, concluding that neither CLAP nor MERT zero-shot similarities capture the musical coherence encoded by COCOLA (Delabaere et al., 30 Jul 2025). In that sense, the failure is not merely quantitative but representational: music-oriented or general-purpose pretrained embeddings do not appear to preserve the compatibility structure required for mashup creation.

The principal insights and limitations are stated directly. Compatibility must be modeled asymmetrically; general-purpose embeddings, including the music-specific MERT, fail to align with perceptually validated compatibility metrics; and fully automatic, large-scale song selection for mashups still requires either an efficient proxy for COCOLA or fine-tuning/distillation of large audio models on mashup compatibility data (Delabaere et al., 30 Jul 2025).

6. Relation to earlier automatic mashup systems and adjacent methods

A closely related predecessor is the stem-based mashup system presented in “Modeling the Compatibility of Stem Tracks to Generate Music Mashups” (Huang et al., 2021). That system generated positives from stems of the same clip, negatives from unmatched key and/or tempo combinations, and unlabeled “average mashups” from matched key and tempo combinations whose true compatibility was unknown. It trained two architectures, PreMixNet and PostMixNet, with Label Smoothing Regularization for Outliers (LSRO), on 51,507 generated mashups derived from 33,192 preview clips. On the Harmonix test halves, reported objective results included 99.8% accuracy and 1.0000 average retrieval rank for PreMixNet, and subjective listening tests found that all neural models significantly outperformed the rule-based AutoMashUpper, with PreMixNet + LSRO achieving the highest Mean Opinion Score across most criteria (Huang et al., 2021).

The contrast between the 2021 and 2025 systems is methodologically significant. The earlier work learned compatibility from constructed positive, negative, and unlabeled mashups after key and tempo had already been adjusted (Huang et al., 2021). The later AutoMashup paper instead treats COCOLA as the reference perceptual metric and asks whether zero-shot pretrained embeddings can stand in for that metric at retrieval time (Delabaere et al., 30 Jul 2025). This suggests a shift from supervised mashability classification over processed candidates toward scalable compatibility estimation over stem embeddings.

Another adjacent line of work concerns cue-point detection for DJ mixing rather than full mashup construction. “Automatic Detection of Cue Points for DJ Mixing” formalizes switch-point selection through novelty, section-boundary, and salience rules, reporting approximately 86% precision and 49% recall on 134 EDM tracks, with a net usable rate of approximately 96% after subjective review of false positives (Zehren et al., 2020). That work is explicitly limited to cue-point detection and does not prescribe a full mashup pipeline, but it identifies structural transition points that are relevant to later beat-aligned or phrase-aligned recombination.

Across these strands, the recurrent technical issue is that alignment and compatibility are separable only up to a point. AutoMashup in its 2025 form resolves alignment with Demucs, Allin1, and Pyrubberband, but its main contribution lies in showing that compatibility estimation remains the unresolved bottleneck: it is role-dependent, perceptually grounded, and not recoverable by zero-shot cosine similarity in current pretrained audio embedding spaces (Delabaere et al., 30 Jul 2025).