Music-Token-Conditioned Image Synthesis

• The paper introduces a novel framework that redefines dance pose generation as an image synthesis task by conditioning on music tokens.
• It employs a VAE-based compression of multi-channel pose data and a DiT-style diffusion backbone to achieve beat-aligned, temporally coherent outputs.
• Empirical results show significant improvements in FID, BAS, and diversity metrics, confirming state-of-the-art performance for music-driven pose synthesis.

Music-token-conditioned multi-channel image synthesis refers to a class of generative approaches that leverage music-derived latent representations (music tokens) to control the synthesis of high-dimensional, temporally structured image outputs, typically with multi-channel encodings. The framework generalizes recent advances in audio-conditioned and text-conditioned diffusion models to the domain of structured time-series data, such as 2D human pose sequences for dance, by re-encoding these sequences as multi-channel images and conditioning generation on music tokens extracted from pretrained music encoders. This approach enables rhythm-aligned, temporally coherent synthesis that is scalable to high-variance, in-the-wild distributions, and is empirically validated as yielding state-of-the-art results for music-driven pose generation (Zhang et al., 12 Dec 2025).

1. Conceptual Foundation and Problem Formulation

Music-token-conditioned multi-channel image synthesis reconceptualizes sequence generation—originally treated as sequential autoregressive prediction or direct pose regression—as an image generation task in a high-dimensional latent space. The primary inputs are (a) a raw music clip, (b) a reference pose specifying canonical body proportions and on-screen scale, and (c) optionally, previous pose frames for segmentwise continuity. The target output is a temporally indexed pose sequence of length $T$ at frame rate $r$, encoded as an image-like tensor $X \in \mathbb{R}^{C\times W\times T}$, where $C=2K$ (channels for keypoint coordinates) and $W$ (spatial resolution, typically 512) (Zhang et al., 12 Dec 2025).

Music is passed through a frozen music encoder $\psi$ (e.g., Jukebox), yielding a sequence of $T'$ music tokens:

$$A = \psi(\text{music}) \in \mathbb{R}^{T' \times d_a}$$

The synthesis goal is thus to generate $X$ conditioned on $A$ and $x_{\mathrm{ref}}$ (the reference pose), enabling the adoption of transformer- and diffusion-based architectures developed for text-to-image modeling (Zhang et al., 12 Dec 2025).

2. Pose Sequence Representation and Compression

2D poses at each time step $u$ are encoded as tuples $(x_{k,u}, y_{k,u}, s_{k,u})$ for $K$ keypoints, with normalized image coordinates and detector confidence scores. Each coordinate is transformed into a one-hot vector of length $W$ using integer indices ($i_x = \lfloor W x_{k,u} \rfloor$), where the confidence $s_{k,u}$ is placed at the corresponding position (Zhang et al., 12 Dec 2025):

$X[:,:,u] \in \mathbb{R}^{2K \times W}$

Over all $T$ frames, this yields a pose "image" $X \in \mathbb{R}^{C \times W \times T}$.

To facilitate scalable, high-fidelity synthesis and enable latent diffusion methods, $X$ is compressed using a frozen pretrained image VAE (e.g., SD-VAE). Channels are grouped into triplets, each processed as a pseudo-RGB image—$\tilde{X}_g \in \mathbb{R}^{3 \times W \times T}$—and encoded into a $4$-channel latent grid ($\tilde{Z}_g \in \mathbb{R}^{4 \times W' \times T'}$ with $W'=W/8, T'=T/8$). Concatenation yields the full latent tensor $Z \in \mathbb{R}^{(4G) \times W' \times T'}$, leveraging pretrained VAE priors for reconstruction and capacity (Zhang et al., 12 Dec 2025).

3. Backbone Diffusion Model and Conditioning Mechanisms

The core generative engine is a DiT-style (Diffusion Transformer) backbone operating in the VAE latent space, extending the image tokens with music-derived conditioning and reference-shape information. Diffusion noise is added via a schedule:

$$Z_\tau = \alpha_\tau Z + \sigma_\tau \epsilon, \quad \epsilon \sim \mathcal{N}(0,I)$$

Pose latents are flattened into $N_p$ pose tokens, appended with $N_m$ music tokens from $A$, optionally prefixed by $N_r$ reference tokens and a binary mask $M$ that guides the reliance on reference frames. Each transformer layer applies:

• Multi-head self-attention over (pose $\parallel$ music $\parallel$ ref $\parallel$ mask).
• Cross-attention from pose and ref tokens to music tokens $A$.
• Feed-forward networks.

Rotary Positional Embedding (RoPE) is injected using shared 3D indices, synchronizing temporal alignment between pose and music tokens (Zhang et al., 12 Dec 2025).

4. Temporal Indexing and Reference Conditioning Strategies

Temporal alignment between music and pose is realized via a time-shared indexing scheme: pose tokens at $(w, t)$ are assigned $(0, w, t)$, music tokens at time $t$ as $(0, 0, t)$. This structure, with temporal coordinate $t$ shared, enhances beat-synchronous cross-attention—effectively coupling rhythmic patterns between modalities (Zhang et al., 12 Dec 2025). Empirical ablation shows this raises beat-align score (BAS) by 14.6% and nearly doubles diversity (DIV).

Reference-pose conditioning secures body proportions and scale. A "shape-only" reference tensor is built by repeating the one-hot representation of $x_{\mathrm{ref}}$ across frames. For continuity, the initial frames may interpolate with ground-truth positions. The corresponding reference latent $Z_{\mathrm{ref}}$ is concatenated, and $M$ marks pose-aware regions. Inference for long music clips uses segmentwise generation with segment-and-stitch, each segment seeded by prior outputs to ensure seamless long-form choreography (Zhang et al., 12 Dec 2025).

5. Training Objectives, Losses, and Optimization

Only the DiT core is trainable; both the VAE encoder/decoder and the music encoder remain frozen. The primary objective is $\epsilon$-prediction:

$$\mathcal{L}_{\mathrm{diff}} = \mathbb{E}_{Z, \tau, \epsilon} \left\| \epsilon - \hat{\epsilon}_\theta([Z_\tau \Vert Z_{\mathrm{ref}} \Vert M], A) \right\|_2^2$$

No auxiliary alignment or adversarial losses are used. VAE was pretrained with conventional reconstruction and KL terms. During inference, diffusion steps are unrolled, latent pose images decoded, and continuous coordinates recovered by one-hot inversion (Zhang et al., 12 Dec 2025).

Key hyperparameters include DiT architecture size, VAE channel grouping, and music encoder hop size ($L=T'$ for tight alignment). The model is trained on extensive in-the-wild dance datasets (600 hours, 240K segments), with additional benchmarking on AIST++2D (Zhang et al., 12 Dec 2025).

6. Quantitative and Qualitative Results

Music-token-conditioned multi-channel image synthesis achieves state-of-the-art results in music-to-dance pose generation, validated on both in-the-wild and calibrated benchmarks. Metrics include:

• Pose-space FID$_p$, DIV, Beat-Align Score (BAS)
• Video-space FVD, FID$_{\text{vid}}$
• Human Win Rate (WR) via side-by-side evaluations

On in-the-wild test sets, FID$_p$ drops from 80.4 to 45.2, BAS increases from 0.2270 to 0.2524, FVD decreases from 986.5 to 682.9, and WR exceeds 95% against all baselines. Ablations confirm one-hot encoding reduces jitter and improves FID$_p$ by ~12 points. On AIST++2D, FID$_p$ improves to 29.3, DIV reaches 8.4, and BAS rises to 0.2715.

Qualitatively, the system demonstrates sharp responsiveness to tempo/energy shifts, stochastic diversity under fixed music, reliable scaling and camera mirroring, and seamless long-form generation with segment-and-stitch plus reference conditioning (Zhang et al., 12 Dec 2025).

7. Extensions and Relation to Audio-Conditioned Image Synthesis

The pipeline generalizes prior approaches from audio-token-conditioned image generation, as studied in AudioToken (Yariv et al., 2023), wherein audio recordings are encoded into continuous tokens via a frozen audio backbone, projected and pooled via an "Embedder," and injected as conditioning into a frozen latent diffusion model. Application to music-token and multi-channel settings (e.g., spectrograms, high-dimensional images) is straightforward: substitute $\phi(\cdot)$ for music-specific encoders, adjust pooling and token length for the target temporal scale, and retrain the lightweight conditioning modules with paired data (Yariv et al., 2023). Both frameworks share the principle of leveraging pretrained encoders and latent-diffusion backbones; in the dance synthesis context, additional temporal and shape conditioning yield superior rhythmic and structural fidelity.

A plausible implication is that music-token-conditioned models could extend to broader multimodal settings where temporally indexed semantic signals guide high-dimensional image synthesis, offering a unified approach for rhythm-aligned generation, synchronization tasks, and structured sequence modeling.