Expotion: Multimodal Music Generation

• Expotion is a multimodal music-generation system that integrates facial expressions, upper-body motion, and text prompts to control a pretrained MusicGen.
• It uses a lightweight adaptor with gated prefix injection and temporal smoothing to align 5 fps visual signals with 50 Hz audio token sequences.
• Empirical evaluations show improvements in expressivity and synchronization, with metrics such as FAD 1.93 and CLAP 0.54 achieved using the motion Syncformer variant.

Searching arXiv for the Expotion paper and its immediate backbone/context papers to ground citations. Expotion is a multimodal music-generation system introduced as “Facial Expression and Motion Control for Multimodal Music Generation,” designed to condition a pretrained text-to-music model on human facial expressions, upper-body motion, and text prompts in order to generate expressive and temporally aligned music (Izzati et al., 7 Jul 2025). The model addresses a specific limitation of text-only generation: high utterance-level audio quality without fine-grained, time-varying control. Its central design combines compact visual embeddings, a parameter-efficient condition adaptor inserted into the last layers of a frozen MusicGen decoder, and a temporal smoothing strategy that aligns low-frame-rate visual signals with 50 Hz audio token sequences. The system is trained on a 7-hour synchronized dataset of video, instrumental music, and captions, and is evaluated with perceptual, rhythmic, and cross-modal metrics (Izzati et al., 7 Jul 2025).

1. Conceptual scope and problem formulation

Expotion is situated at the intersection of text-to-music generation, video-to-music generation, and multimodal control. Its stated objective is not merely to map video to accompaniment, but to use facial expression and upper-body motion as continuous control signals for musical expressivity and temporal synchronization. In this formulation, facial expression is treated as a proxy for affective and expressive state, while upper-body motion carries stronger rhythmic and kinetic information. Text prompts remain part of the conditioning stack, preserving semantic controllability alongside visual guidance (Izzati et al., 7 Jul 2025).

The system is explicitly built around the claim that text-only models such as MusicGen lack mechanisms to reflect subtle, time-varying expressive cues. Expotion therefore augments, rather than replaces, a pretrained backbone. This design choice is important: the method assumes that high-quality audio priors are already present in the frozen text-to-music model, and that the research problem is one of conditioning and alignment rather than de novo sequence modeling.

A common misconception is to treat Expotion as a full multimodal end-to-end retraining of a music generator. It is instead a parameter-efficient fine-tuning scheme with a frozen backbone. Another misconception is to interpret it as a purely video-driven model. The architecture retains text prompts throughout generation, and some of its empirical trade-offs are specifically prompt-dependent: generic prompts improve tempo error in some settings, whereas generated captions improve text–audio similarity (Izzati et al., 7 Jul 2025).

2. Architecture and conditioning pathway

The backbone is MusicGen, with a pretrained T5 text encoder, a pretrained EnCodec RVQ tokenizer operating on 32 kHz audio at 50 Hz, and a frozen acoustic transformer decoder with 48 causal self-attention layers (Izzati et al., 7 Jul 2025). Expotion modifies this stack by introducing a lightweight multimodal adaptor rather than altering the backbone weights.

Facial features are extracted with MARLIN. Videos are resampled from 30 fps to 80 fps, and MARLIN processes 16-frame windows, yielding a 5 fps feature stream with per-frame output $z_f \in \mathbb{R}^{768}$. Motion is encoded by one of two alternatives. The first is Synchformer, which operates on 16-frame clips with stride 5 and is normalized to an effective 5 fps stream. The second is RAFT optical flow, followed by a Flow Embedding CNN that maps dense flow fields to $z_{\text{flow}}^{(t)} \in \mathbb{R}^{256}$ (Izzati et al., 7 Jul 2025).

Both facial and motion features are compressed through low-rank projections with $d_1 = 12$ and $d_2 = 12$. The compact vectors are concatenated, combined with learnable positional encodings, and fused to the MusicGen hidden dimension through $W_e$. The resulting joint sequence $z = \{z_1,\dots,z_{T_v}\}$ acts as the multimodal control signal.

Conditioning occurs through a time-varying “condition prefix” injected into the last $L$ decoder layers. Prefix states $H_l^p \in \mathbb{R}^{T_v \times d}$ are initialized by learnable $H_0^p$, updated through self-attention, and mixed into the main decoder path through a gated attention term. The scalar gates $g_l$ are learnable and initialized to $z_{\text{flow}}^{(t)} \in \mathbb{R}^{256}$0, so the model begins training from the frozen MusicGen behavior and gradually increases multimodal influence (Izzati et al., 7 Jul 2025).

This architecture has three notable consequences. First, it localizes trainable capacity in the adaptor rather than in the backbone. Second, it preserves MusicGen’s existing text-conditioning pathway. Third, it treats visual control as a time-indexed memory that modulates only the final decoder layers, which suggests that Expotion prioritizes fine-grained conditioning over global representational restructuring.

3. Temporal alignment and multimodal synchronization

A defining technical problem for Expotion is the mismatch between visual and audio temporal resolutions. Visual streams are represented at 5 fps, whereas EnCodec tokens are generated at 50 Hz. Expotion resolves this mismatch with a temporal smoothing strategy based on linear interpolation rather than with an explicit synchronization loss (Izzati et al., 7 Jul 2025).

For a fractional time index $z_{\text{flow}}^{(t)} \in \mathbb{R}^{256}$1, facial features are interpolated as

$z_{\text{flow}}^{(t)} \in \mathbb{R}^{256}$2

with $z_{\text{flow}}^{(t)} \in \mathbb{R}^{256}$3 and $z_{\text{flow}}^{(t)} \in \mathbb{R}^{256}$4. Motion features are interpolated analogously. Audio time steps are mapped to visual time through $z_{\text{flow}}^{(t)} \in \mathbb{R}^{256}$5, so each audio-frame conditioning vector is derived from a smoothly varying visual representation (Izzati et al., 7 Jul 2025).

The paper emphasizes that no dedicated synchronization loss, contrastive objective, or explicit dynamic time warping module is used. Alignment is therefore architectural and algorithmic rather than learned through an auxiliary optimization target. This is a substantive design choice. It simplifies training and avoids introducing additional objectives, but it also constrains alignment capacity to first-order temporal smoothing and time-indexed prefix injection.

This suggests a precise interpretation of Expotion’s synchronization behavior. The model is not solving a general audiovisual correspondence problem. It is performing conditioned autoregressive decoding with interpolated visual controls, relying on the pretrained decoder prior and the adaptor’s gates to translate those controls into musically meaningful timing. The reported empirical gains indicate that this is already effective, but the absence of explicit synchronization supervision is also one of the system’s clearest limitations (Izzati et al., 7 Jul 2025).

4. Training protocol, dataset, and inference workflow

Expotion is trained on a dataset of 7 hours of synchronized video–audio recordings. Volunteers recorded facial expressions and upper-body movements while listening to 30-second instrumental clips licensed from Epidemic Sound. The audio contains no vocals. Before recording, subjects listened once in order to plan their expressions and movements. The data are segmented into 10-second clips for training, with 30 minutes held out for validation and test (Izzati et al., 7 Jul 2025).

The captions used as text prompts are generated by SALMONN with the instruction “Please describe the music.” Audio is resampled to 32 kHz and tokenized by EnCodec at 50 Hz. Visual preprocessing includes 30-to-80 fps resampling for MARLIN and 5 fps extraction for motion representations. There are no manual labels; supervision is entirely based on paired audiovisual data and generated captions (Izzati et al., 7 Jul 2025).

Training optimizes only token-level reconstruction cross-entropy over EnCodec code indices:

$z_{\text{flow}}^{(t)} \in \mathbb{R}^{256}$6

No auxiliary synchronization or text-adherence losses are introduced. The initial learning rate is $z_{\text{flow}}^{(t)} \in \mathbb{R}^{256}$7, batch size is 10, training runs for 40 epochs, and compute resources are four A1000 GPUs (Izzati et al., 7 Jul 2025).

At inference time, the pipeline proceeds in five stages: visual feature extraction, temporal interpolation, low-rank projection and fusion, prefix propagation through the adaptor, and autoregressive decoding of EnCodec tokens followed by waveform reconstruction. The system is designed for offline generation rather than real-time deployment. That distinction matters because feature extraction with MARLIN, Synchformer, or RAFT, combined with transformer decoding, introduces nontrivial latency.

5. Empirical performance and ablation structure

Evaluation covers general music quality, rhythm alignment, and cross-modal adherence. The reported metrics are FAD-VGG, average KL divergence over audio-label distributions, Inter-Sample Score, tempo error $z_{\text{flow}}^{(t)} \in \mathbb{R}^{256}$8, Beat F1, CLAP text–audio similarity, and LanguageBind video–audio similarity (Izzati et al., 7 Jul 2025).

The strongest quality/diversity result is obtained by motion Syncformer with generated prompts, with FAD $z_{\text{flow}}^{(t)} \in \mathbb{R}^{256}$9, KL $d_1 = 12$0, and IS $d_1 = 12$1 (Izzati et al., 7 Jul 2025). The best reported text–audio similarity is achieved by face+motion Syncformer with generated prompts, with CLAP $d_1 = 12$2. The strongest video–audio similarity is obtained by motion RAFT with generic prompts, with LanguageBind $d_1 = 12$3. Expotion variants report Beat F1 scores in the range $d_1 = 12$4–$d_1 = 12$5, compared with $d_1 = 12$6 for the MusicGen baseline (Izzati et al., 7 Jul 2025).

Metric	Best reported Expotion variant	Value
FAD	motion Syncformer + generated prompts	1.93
KL	motion Syncformer + generated prompts	0.65
IS	motion Syncformer + generated prompts	1.57
CLAP	face+motion Syncformer + generated prompts	0.54
LanguageBind	motion RAFT + generic prompts	0.65
Tempo error	motion RAFT + generic prompts	28.07 BPM

The tempo results are more nuanced than the quality results. The lowest overall tempo error in the comparison set is achieved by the Video2Music baseline at $d_1 = 12$7 BPM, but that baseline also shows poor FAD ($d_1 = 12$8) and text adherence ($d_1 = 12$9 CLAP). Expotion does not dominate every scalar metric individually; its claim is instead one of better overall balance across musicality, creativity, beat-tempo consistency, temporal alignment, and text adherence (Izzati et al., 7 Jul 2025).

The ablations make the control structure clearer. Face-only conditioning with generated prompts yields FAD $d_2 = 12$0, CLAP $d_2 = 12$1, and LanguageBind $d_2 = 12$2. Motion-only conditioning consistently outperforms face-only on quality and diversity, and Syncformer generally outperforms RAFT on FAD and IS. Facial expression still contributes to text adherence when fused with motion, as seen in the best CLAP result for face+motion Syncformer. This supports the interpretation that motion carries the stronger rhythmic prior, while facial expression provides complementary expressive information (Izzati et al., 7 Jul 2025).

6. Limitations, interpretive issues, and future directions

Several limitations are explicit. The dataset is modest at 7 hours and is domain-specific: volunteers responding to instrumental tracks in a controlled setting. Generalization to unconstrained videos, broader genres, different camera conditions, or occlusion-heavy settings is not established (Izzati et al., 7 Jul 2025). The model also depends on generated captions from SALMONN for semantic grounding, which introduces an additional upstream model dependency.

Another important limitation is that synchronization is achieved without an explicit alignment objective. The paper itself notes that this may be suboptimal under complex temporal dynamics such as rapid visual changes. This suggests that the current alignment mechanism is sufficient for coarse and mesoscopic synchrony, but possibly brittle for fine onset-level correspondence or nonlinearly warped audiovisual timing.

The ablations also indicate a meaningful trade-off between semantics and rhythm. Generic prompts improve tempo error—for example, RAFT+generic reports $d_2 = 12$3 BPM versus $d_2 = 12$4 BPM for RAFT+generated—but this comes at the cost of lower CLAP because the prompt content is semantically sparse (Izzati et al., 7 Jul 2025). A plausible implication is that prompt specificity competes with some forms of temporal regularization in the frozen backbone.

The paper proposes several future directions: explicit synchronization objectives such as

$d_2 = 12$5

contrastive alignment via InfoNCE, adaptive temporal alignment such as dynamic time warping, richer adaptor designs including FiLM or LoRA on selected final-layer projections, real-time systems, and broader datasets (Izzati et al., 7 Jul 2025). These proposals mark out the main unresolved research frontier: moving from lightweight multimodal control toward more direct audiovisual co-structuring without losing the parameter efficiency that makes Expotion viable with limited paired data.

In that sense, Expotion is best understood as a PEFT-based conditioning framework for multimodal music generation rather than a complete solution to audiovisual music synthesis. Its main technical contribution is to show that compact facial and motion controls, injected through a gated condition-prefix pathway into a frozen MusicGen decoder, are sufficient to improve several dimensions of generation quality and alignment on a small synchronized dataset (Izzati et al., 7 Jul 2025).