Artificial Audio Multitracks

• Artificial Audio Multitracks (AAM) are structured audio representations defined by native stem generation, latent editing, and aligned symbolic tracks for coherent music production.
• AAM encompasses methods such as latent diffusion, transformer-based parameter estimation, and 3D convolution to jointly model synchronized instrument stems.
• Key applications include generating synchronized instrument tracks, enabling mixing style transfer, and facilitating source-specific manipulation in production workflows.

Artificial Audio Multitracks (AAM) denotes audio representations and systems that treat a recording as a structured collection of track-like sources rather than only as a final mixed waveform. In the strictest sense, AAM refers to native generation of synchronized, separate audio stems that sum to a coherent mixture. In broader research usage, it also includes artificially generated multitrack datasets with aligned annotations, mixture-domain source-aware editing that learns latent track-like channels inside a mixture, AI systems that predict coordinated processing over raw multitracks, and real-audio corpora paired with aligned symbolic multitracks rather than true isolated stems (Karchkhadze et al., 2024, Majchrzak et al., 7 Aug 2025, Choi et al., 2021, Vanka et al., 2024, Pruyne et al., 9 Jul 2026). Across these variants, the defining object is not a single audio file alone but a source-structured representation tied to a shared musical or acoustic context.

1. Conceptual scope and taxonomy

AAM is not a single technical task. One branch is native multistem audio generation, in which the output is a set of synchronized stem waveforms intended to be individual instrument tracks and to sum into a coherent musical mixture. Another branch is AI-based multitrack manipulation and production assistance, where the system does not invent new source audio but predicts coordinated processing parameters across multiple tracks. A third branch is mixture-domain multitrack-aware editing, where track-like structure is inferred latently from a mixture and edited without explicit stems at inference time. A fourth branch is symbolic multitrack alignment, where the multitrack object is a time-aligned MIDI representation paired with real commercial audio rather than a bank of isolated studio stems (Karchkhadze et al., 2024, Vanka et al., 2024, Choi et al., 2021, Pruyne et al., 9 Jul 2026).

A persistent misconception is that every multitrack AI system is a generator of new stems. That is not the case. Diff-MST$^C$ is explicitly a parameter estimation system for mixing style transfer rather than a source generator; MulTTiPop provides real commercial audio + aligned symbolic multitracks rather than true stems; and AMSS-Net operates on a mixture while learning an internal latent multitrack-like decomposition rather than outputting a full multitrack session (Vanka et al., 2024, Pruyne et al., 9 Jul 2026, Choi et al., 2021). Conversely, not every separation-oriented or symbolic system qualifies as AAM in the strict sense. The strictest formulation is the one in which separate, synchronized audio stems are generated natively as the primary output.

2. Native multistem audio generation and arrangement

The clearest literal AAM formulation is Multi-Track MusicLDM. A musical example is defined by a mixture $x_{\text{mix}}$ and $S$ aligned stems $x_s$, with

$x_{\text{mix}} = \sum_{s=1}^S x_s$

and the stack of stems represented as $x$ with dimensions $S \times T_{\text{mix}}$. The stems are time-synchronous, have shared duration, and belong to a shared musical context. In the reported experiments, the stem inventory is fixed to four classes: Bass, Drums, Guitar, Piano. The processing chain is waveform $\rightarrow$ Mel-spectrogram $\rightarrow$ pretrained MusicLDM VAE encoder $\rightarrow$ latent diffusion over the whole stem stack $x_{\text{mix}}$0 VAE decoder $x_{\text{mix}}$1 pretrained HiFi-GAN vocoder (Karchkhadze et al., 2024).

The central architectural choice is to model all stems jointly rather than independently. The spectrogram stack $x_{\text{mix}}$2 has dimensions $x_{\text{mix}}$3, and the latent representation is

$x_{\text{mix}}$4

For $x_{\text{mix}}$5, $x_{\text{mix}}$6, $x_{\text{mix}}$7, and $x_{\text{mix}}$8, the latent tensor is

$x_{\text{mix}}$9

To exploit the stem dimension explicitly, the model converts the MusicLDM UNet into a 3D LDM using 3D convolutions. The latent at diffusion step $S$0, $S$1, has shape $S$2, and the 3D convolution jointly models inter-stem and time-frequency dependencies. This is the mechanism by which the model learns a joint distribution over synchronized stems rather than independent marginals.

The diffusion formulation follows latent DDPM/LDM notation. The forward process is

$S$3

with $S$4, and the reverse process is

$S$5

$S$6

Training uses the standard denoising objective

$S$7

There are no auxiliary multitrack-specific losses, no explicit inter-stem consistency loss, no adversarial loss, and no jointly optimized reconstruction loss for stems during MT-MusicLDM training. The VAE and HiFi-GAN are imported from pretrained MusicLDM and kept frozen.

Conditioning is global rather than per-stem. The condition $S$8 can come from CLAP embeddings of text prompts or reference audio, and classifier-free guidance is trained by dropping conditioning with probability $S$9. At inference,

$x_s$0

with guidance scale $x_s$1 for audio- and text-conditioned generation. This means text or reference audio steers the character of the whole arrangement rather than specifying exact notes for each instrument.

Arrangement generation is a particularly important AAM capability. For a subset of observed tracks $x_s$2, the model seeks missing tracks $x_s$3 conditioned on $x_s$4. In latent space it defines a binary mask $x_s$5 over stems and enforces the observed region during reverse diffusion by

$x_s$6

This is latent inpainting over the stem dimension and supports arbitrary subset conditioning within the fixed four-stem set: any one, two, or three missing stems can be generated from the remainder.

The training clips are 10.24 seconds at 16 kHz, with Mel-spectrograms of size $x_s$7 per stem. The model uses a 3D UNet with 2 encoder blocks, 1 middle transformer block, and 2 decoder blocks, Adam with learning rate $x_s$8, diffusion training with $x_s$9 steps, and DDIM inference with $x_{\text{mix}} = \sum_{s=1}^S x_s$0 steps. Objective evaluation is based on Fréchet Audio Distance (FAD). For total generation, FAD is 6.55 for MSDM, 1.36 for MT-MusicLDM (unconditional), and 1.13 for MT-MusicLDM (audio-conditioned). Under direct stem FAD for arrangement generation, examples include Bass 6.88 vs 0.76, Drums 5.48 vs 1.07, Guitar 4.25 vs 2.76, Piano 6.45 vs 1.80, and pair or triple combinations such as BD 4.47 vs 0.89 and BDP 4.06 vs 1.01.

The limitations are substantial. The system is confined to a fixed set of four instruments; the pretrained VAE and HiFi-GAN were trained on music mixtures rather than isolated stems; the temporal horizon is limited to 10.24-second clips; and there is no explicit stem-disentanglement objective or direct evaluation of bleed, independence, or crosstalk. The paper also states that source separation is not well handled by the latent approach because mixture and stem latents do not maintain linear relationships.

3. Mixing, style transfer, and production assistance

AAM also includes systems that operate on multitracks as editable structured objects without generating new source performances. Diff-MST$x_{\text{mix}} = \sum_{s=1}^S x_s$1 is a prototype integrated into Cubase for mixing style transfer. It processes up to 20 raw tracks together with a reference song and predicts mixing console parameters that can be used to create an initial mix. The task is explicitly placed in the Parameter Estimation (PE) category rather than direct audio transformation. Its outputs are per-track gain, panorama, EQ, and compressor settings, plus master-bus EQ, compressor, and fader. The inference pipeline is: select a segment of the reference song and of the multitrack session, encode both, contextualize the embeddings with a transformer encoder using self-attention, predict console parameters with a controller of linear layers, and apply those parameters through a differentiable mixing console. The prototype is implemented in PyTorch, optimized for Cubase using TorchScript, and integrated as a Steinberg Kernel Interface (SKI) plugin extending VST3. Its significance for AAM is indirect but strong: tracks remain separate, controls remain editable, and the user can continue the production process after AI assistance (Vanka et al., 2024).

A second production-oriented formulation appears in the automatic mixing speech enhancement system for multitrack speech audio. Here the inputs are isolated speech tracks rather than mixtures, and the objective is to reduce auditory masking while preserving the audibility of multiple simultaneous speakers. The system applies level balance, equalization, dynamic range compression, and spatialization and uses the psychoacoustic front-end of the ITU-R BS.1387 PEAQ model to estimate masking thresholds in critical bands. The optimization objective combines total masking

$x_{\text{mix}} = \sum_{s=1}^S x_s$2

with a balance term

$x_{\text{mix}} = \sum_{s=1}^S x_s$3

and the intended goal is to minimize $x_{\text{mix}} = \sum_{s=1}^S x_s$4 through Harmony Search. The evaluated scenarios use 3 tracks for teleconferencing, 4 tracks for gaming, and 6 tracks for live streaming. In a subjective listening test with 16 participants, the proposed system was competitive with mixes by professional sound engineers and outperformed an existing automatic mixing system in the more complex gaming and live-streaming conditions (Marrinan et al., 2024).

These systems clarify a second major AAM misconception: multitrack AI is not only about synthesizing new stems. It also includes track-aware optimization, parameter transfer, and assistive production workflows in which the multitrack object is preserved as a manipulable session.

4. Generative multitrack effect rendering

MEGAMI extends AAM from deterministic mixing assistance to one-to-many multitrack effect generation. It formulates automatic mixing as conditional generative modeling over processed tracks $x_{\text{mix}} = \sum_{s=1}^S x_s$5 given dry multitracks $x_{\text{mix}} = \sum_{s=1}^S x_s$6, with

$x_{\text{mix}} = \sum_{s=1}^S x_s$7

Rather than generating audio directly, MEGAMI generates track-level effect embeddings $x_{\text{mix}} = \sum_{s=1}^S x_s$8 conditioned on CLAP-based content embeddings $x_{\text{mix}} = \sum_{s=1}^S x_s$9. “Ground-truth” effect embeddings are extracted from professionally mixed wet stems using FxEncoder++, then augmented with deterministic gain and dynamics features such as log-RMS, crest factor, dynamic spread, stereo width, and stereo imbalance. Each dry track is rendered by a deterministic track-agnostic processor

$x$0

This factorization concentrates stochasticity in the latent effect space rather than in direct audio synthesis (Moliner et al., 11 Nov 2025).

The generator uses EDM diffusion over time-invariant, track-level embeddings:

$x$1

The score model is a permutation-equivariant transformer with self-attention across tracks and cross-attention to the conditioning set. This is designed for arbitrary numbers of unlabeled tracks, with random permutation during training and padding up to $x$2. The renderer $x$3 is a TCN conditioned by the concatenated content and effect embeddings through FiLM-like modulation. The system models static per-track mixing decisions rather than time-varying automation.

A notable practical contribution is domain adaptation for wet-only training data. A two-layer MLP $x$4 is trained so that wet-track content embeddings are mapped toward the dry domain:

$x$5

using an $x$6 loss on paired dry/wet single tracks. This permits diffusion training on large wet-only multitrack collections. The internal datasets are Internal-Small with about 400 professionally mixed songs with paired dry and wet isolated stems, Internal-Large with about 20k professionally mixed songs with wet stems only, and Public with 248 openly licensed multitracks. Audio is stereo at 44.1 kHz, trained on 11.9 s excerpts.

Because mixing is one-to-many, evaluation uses Kernel Audio Distance (KAD) rather than simple pairwise regression metrics. The strongest non-oracle system is generally MEGAMI I-L, with KAD values 5.21 on AFxRep, 1.72 on FxEncoder, 3.90 on FxEncoder++, and 0.84 on CLAP. In a listening test with 12 participants, 7 songs, and a multi-stimulus design, MEGAMI I-L approached human-level quality and outperformed baselines in the majority of cases. Its AAM significance lies in the generation of multiple plausible professionally mixed multitrack variants from the same dry stems rather than in musical composition from scratch.

5. Mixture-domain editing, decomposition, and symbolic multitracks

AAM also includes systems that infer track-like structure from mixtures. AMSS-Net defines Audio Manipulation on Specific Sources (AMSS): given a mixture $x$7 and a textual description $x$8, produce a manipulated track $x$9 that edits only the addressed sources while preserving irrelevant content. The mixture is modeled as

$S \times T_{\text{mix}}$0

and the desired manipulated mixture is

$S \times T_{\text{mix}}$1

where $S \times T_{\text{mix}}$2 is the set of target sources named in the query and $S \times T_{\text{mix}}$3 is the manipulation implied by the text. The language interface is a controlled Audio Manipulation Language (AML) rather than open-ended natural language, supporting tasks such as separate, mute, increase volume, decrease volume, pan left, pan right, lowpass, highpass, and dereverb over sources such as vocals, drums, and bass. The model uses a Description Encoder and a Spectrogram Encoder-Decoder Network with latent source channels $S \times T_{\text{mix}}$4 of size $S \times T_{\text{mix}}$5, with $S \times T_{\text{mix}}$6 in experiments. Selective editing is applied by SMPoCM:

$S \times T_{\text{mix}}$7

with gates

$S \times T_{\text{mix}}$8

Training uses only an $S \times T_{\text{mix}}$9 loss on spectrogram reconstruction. On Musdb18, AMSS-Net outperforms its ablations across most tasks and supports source-aware editing from mixture audio alone, but its latent channels are not guaranteed to align with clean semantic stems (Choi et al., 2021).

A second line couples decomposition and symbolic recovery. The source-separation and automatic-transcription prototype begins from a single mixed music waveform, converts it to STFT, predicts a spectrogram mask with an LSTM separator, reconstructs separated audio, then applies a piano AMT stack based on log-CQT, Conv2D, BiLSTM, and a sigmoid piano-roll output before MIDI and MuseScore conversion. The separator is only vocals vs accompaniment, while the AMT model is piano-only, so the pipeline is structurally mismatched. It sketches an AAM workflow—mixture $\rightarrow$0 source-specific audio $\rightarrow$1 symbolic transcription $\rightarrow$2 sheet music—but does not constitute a robust internally consistent multitrack system (Derby et al., 2024).

MulTTiPop occupies a different position in the AAM landscape. It is a dataset of 572 segments totaling 3.5 hours of commercial pop audio paired with aligned multitrack MIDI recordings, drawn from 374 unique songs and 263 unique artists. The data are built by metadata-based matching between TheoryTab segments and the LMD-matched subset of Lakh MIDI, then by manual anchor-beat selection and beat-synchronous warping. The alignment is defined by an anchor beat $\rightarrow$3 linking the first audio beat $\rightarrow$4 to MIDI beat $\rightarrow$5, with control points

$\rightarrow$6

Linear interpolation then retimes MIDI note events to the audio. MulTTiPop is therefore real commercial mixture audio + aligned symbolic multitrack labels, not true isolated stems. For AAM, it supports proxy stem rendering by synthesizing individual MIDI tracks, but those renderings remain synthetic approximations rather than authentic studio stems (Pruyne et al., 9 Jul 2026).

6. Data resources, benchmark regimes, and adjacent evaluation frameworks

In one explicit MIR usage, AAM denotes the dataset introduced by Ostermann et al.: 3000 artificial music tracks generated by algorithmic composition, rendered with real instrument samples, and released with MIDI, mp3 audio, separate instrument stems, mixes, and aligned annotations including onsets, pitches, instruments, keys, tempos, chords, beats, and segment boundaries. In chord-recognition experiments, the labels are reduced to a MajMin vocabulary of 25 classes. The central conclusion is qualified rather than absolute: AAM can enrich a smaller human-composed training set and can even be used as a standalone training set for pop-oriented chord recognition if no other data are available, but it does not remove the domain gap to human music, especially for Winterreise-like classical material (Majchrzak et al., 7 Aug 2025).

Benchmarking has also expanded toward multi-audio reasoning, which is directly relevant when stems or channels are provided as separate inputs. MAE is the first multi-audio evaluation benchmark for audio LLMs and contains 20 datasets from 11 multi-audio tasks, each with two audio inputs and one instruction. Tasks include speech comparison, dialogue response generation, hotword detection, speech identification, keyword comparison, intent identification, story generation, sound comparison, caption retrieval, event retrieval, and event detection. MALLM, built on Qwen-Audio with Whisper-large-v2 and Qwen-7B, is trained on synthetic speech and sound pairs without human annotations and raises average MAE-Speech performance from 39.6 for Qwen-Audio to 73.8, and average MAE-Sound from 57.0 to about 74. The benchmark is not a multitrack music-production benchmark, but it supplies a concrete evaluation model for reasoning over multiple separate audio streams (Chen et al., 2024).

A further adjacent framework concerns provenance rather than generation. ArtifactNet is a forensic detector of AI-generated music based on codec residual physics. It uses ArtifactUNet with 3.6M parameters to extract bounded-mask residuals from magnitude spectrograms, then forms a 7-channel forensic representation after HPSS and classifies 4-second segments with a compact CNN of 0.4M parameters. On ArtifactBench, comprising 6,183 tracks with 4,383 AI tracks from 22 generators and 1,800 real tracks from 6 sources, ArtifactNet reaches F1 = 0.9829 and FPR = 1.49% on an unseen test partition of 2,263 tracks. Codec-aware training over WAV, MP3, AAC, and Opus reduces cross-codec probability drift by 83%. The method is not an AAM generator, but it is relevant to AAM as a provenance and authenticity framework for synthetic audio (Oh, 17 Apr 2026).

7. Limitations, controversies, and likely directions

Current AAM systems remain fragmented. Native stem generation is demonstrated, but usually with a fixed and small instrument inventory, short clip duration, synthetic training data, and stem-domain decoders inherited from models trained on mixtures rather than isolated sources. Generative mixing systems can sample multiple plausible processed stem sets, but they typically model static track-level decisions rather than automation and often omit master-bus effects. Mixture-domain editors recover useful latent source channels, yet those channels are not guaranteed to correspond to clean semantic stems. Symbolic multitrack datasets provide strong alignment but not authentic isolated waveforms. Synthetic datasets can reduce copyright dependence, but transfer to human commercial audio remains limited by domain gap (Karchkhadze et al., 2024, Moliner et al., 11 Nov 2025, Choi et al., 2021, Pruyne et al., 9 Jul 2026, Majchrzak et al., 7 Aug 2025).

Several controversies follow directly from these constraints. One concerns what should count as a “multitrack” object: native stems, latent source channels, differentiable-console parameter sets, or aligned symbolic tracks. Another concerns evaluation. FAD on summed mixtures measures overall realism but not stem independence; KAD better matches one-to-many mixing but does not expose automation or interpretability; and symbolic transcription benchmarks can be strong on aligned MIDI while remaining silent on stem fidelity. A third concerns realism of training resources: Slakh2100 is synthesized from MIDI using premium virtual instruments, the AAM dataset is algorithmically composed and rendered from samples, and MulTTiPop uses commercial audio only through aligned symbolic labels rather than released stems (Karchkhadze et al., 2024, Moliner et al., 11 Nov 2025, Majchrzak et al., 7 Aug 2025, Pruyne et al., 9 Jul 2026).

The research trajectory nevertheless points toward a clearer synthesis. This suggests that future AAM systems will need larger and more diverse stem inventories, longer-form temporal modeling, stem-specialized reconstruction back ends, time-varying or automation-aware control, stronger guarantees of stem purity and crosstalk suppression, benchmark designs that distinguish native generation from parameter estimation and proxy symbolic alignment, and provenance tools that can verify whether a multitrack session is wholly synthetic or only partially synthetic. The available literature already delineates the major technical components of that agenda: joint stem generation, arbitrary subset arrangement, permutation-equivariant multitrack rendering, latent source editing from mixtures, aligned symbolic multitrack corpora, multi-audio reasoning benchmarks, and codec-aware forensic analysis (Karchkhadze et al., 2024, Moliner et al., 11 Nov 2025, Choi et al., 2021, Pruyne et al., 9 Jul 2026, Chen et al., 2024, Oh, 17 Apr 2026).