LARA-Gen: Continuous Emotion Control

• LARA-Gen is a framework that enables continuous emotion control in text-to-music generation by aligning the generative model’s hidden states with an external music understanding model.
• It integrates a latent affective representation alignment module and continuous valence–arousal conditioning to overcome limitations of text-only prompts.
• Evaluations indicate superior arousal controllability and improved generation quality while highlighting challenges in fine-tuning valence control.

LARA-Gen is a framework for continuous emotion control in text-to-music generation that aligns the internal hidden states of a generative model with an external music understanding model through Latent Affective Representation Alignment (LARA), while conditioning generation on a continuous valence–arousal representation rather than relying solely on text prompts (Mei et al., 7 Oct 2025). Introduced for text-to-music models built on MusicGen, it couples a latent alignment objective with an emotion control module and an evaluation benchmark, with the stated aim of achieving fine-grained emotional controllability without the bottlenecks of text-based prompting (Mei et al., 7 Oct 2025).

1. Conceptual definition and scope

LARA-Gen addresses a specific limitation in text-to-music generation: coherent music generation from text prompts has advanced, but fine-grained emotional control remains unresolved (Mei et al., 7 Oct 2025). Its central premise is that emotional controllability can be improved by supervising the generative model’s hidden states with affective representations extracted from an external model and by representing target emotion in a continuous valence–arousal space (Mei et al., 7 Oct 2025).

The framework is defined by three coupled components. First, it builds on MusicGen as an autoregressive text-to-music backbone. Second, it introduces the LARA module, which aligns hidden states with “ground-truth” emotion features from a frozen MERT encoder. Third, it introduces an emotion control module that parameterizes emotion as $\mathbf{e}=(v,a)$ with $v,a\in[1,9]$, normalizes this target to $[-1,1]$, and injects the resulting embedding alongside text conditioning into every cross-attention layer (Mei et al., 7 Oct 2025).

A common misconception in controllable music generation is that emotional control can be reduced to prompt engineering alone. LARA-Gen explicitly treats text-based prompting as a bottleneck for continuous control, and its design therefore disentangles emotional attributes from textual content by introducing a dedicated continuous emotion pathway (Mei et al., 7 Oct 2025). This suggests a shift from lexical affect specification toward latent affect supervision.

2. Backbone model and latent affective representation alignment

The generative backbone is MusicGen, described here as “a 24-layer, 1024-dimensional Transformer with 16 attention heads,” operating autoregressively over a sequence of discrete acoustic tokens $c_{1},\dots,c_{T}$ and conditioning embeddings $\mathbf{E}$ (Mei et al., 7 Oct 2025). At each Transformer block, $\mathbf{E}$ is injected via cross-attention, and the final hidden-state sequence at layer $L$ is

$$\mathbf{H} = \bigl[\mathbf{h}_{1}, \dots, \mathbf{h}_{T-1}\bigr]\in\mathbb{R}^{B\times (T-1)\times D},$$

with batch size $B$ and $D=1024$ (Mei et al., 7 Oct 2025).

The LARA module supplies explicit affective supervision. “Ground-truth” emotion features are extracted from a frozen MERT encoder as

$v,a\in[1,9]$0

where $v,a\in[1,9]$1 (Mei et al., 7 Oct 2025). A lightweight Proxy Network $v,a\in[1,9]$2, implemented as a Transformer decoder with $v,a\in[1,9]$3 learnable queries $v,a\in[1,9]$4, attends over $v,a\in[1,9]$5 and predicts

$v,a\in[1,9]$6

The alignment objective minimizes the mean squared error between predicted and target affective features:

$v,a\in[1,9]$7

The paper’s interpretation is explicit: this loss “forces the high-dimensional hidden states of MusicGen to carry the same affective information as seen by the external MERT model, yielding an explicit emotion supervision signal” (Mei et al., 7 Oct 2025). In practical terms, LARA-Gen does not merely condition output on emotion labels; it constrains the latent space so that internal representations become affect-aware.

3. Continuous valence–arousal conditioning

Emotion targets are represented in a continuous two-dimensional valence–arousal space. Specifically, the target is $v,a\in[1,9]$8 with $v,a\in[1,9]$9, and this pair is normalized to $[-1,1]$0 before being processed by a small MLP, the Arousal-Valence Encoder, to obtain a $[-1,1]$1-dimensional embedding:

$[-1,1]$2

In parallel, the text prompt $[-1,1]$3 is encoded by a frozen T5 encoder into $[-1,1]$4 (Mei et al., 7 Oct 2025). The text and emotion embeddings are then concatenated, with the emotion embedding padded as a one-token sequence:

$[-1,1]$5

This combined conditioning is used in every cross-attention layer of the Transformer (Mei et al., 7 Oct 2025).

The architectural consequence is that emotional conditioning is not mediated indirectly through natural-language descriptions such as mood adjectives alone. Instead, LARA-Gen injects a dedicated continuous control signal throughout the model’s depth. The paper states that this “effectively gat[es] the generation to follow the precise continuous emotion target” (Mei et al., 7 Oct 2025). A plausible implication is that the model can separate semantic content specified by text from affective state specified by $[-1,1]$6 more cleanly than text-only approaches.

4. Objective function, training regime, and optimization

LARA-Gen is trained with a composite loss comprising standard token prediction and latent affective alignment. The acoustic-token objective is the standard cross-entropy loss

$[-1,1]$7

The total objective is

$[-1,1]$8

Thus, latent affective supervision is not auxiliary in a negligible sense; it is explicitly weighted in the training criterion (Mei et al., 7 Oct 2025).

The training data comprise “22 067 thirty-second instrumental clips (valence–arousal labels in $[-1,1]$9) from multiple open platforms” (Mei et al., 7 Oct 2025). MusicGen is fine-tuned for 20 000 steps, the Proxy Network is trained jointly, and the MERT encoder remains frozen. The learning rate schedule is stated to be identical to MusicGen’s original setup, while the batch size is noted as omitted in the paper but “typically 32” (Mei et al., 7 Oct 2025).

These design choices place LARA-Gen within a hybrid conditioning-and-representation-learning paradigm. It is neither a purely prompt-based control method nor solely a supervised regression head attached to a generator. Instead, it combines autoregressive music modeling, continuous conditioning, and hidden-state alignment. This suggests that controllability is treated as an internal representation problem as much as an input-format problem.

5. Benchmark, evaluation protocol, and empirical results

LARA-Gen is evaluated on a benchmark built around a curated test set and an Emotion Predictor (Mei et al., 7 Oct 2025). The test set is an out-of-domain subset of DEAM consisting of 986 thirty-second clips with no vocals and expert valence–arousal annotations (Mei et al., 7 Oct 2025).

The Emotion Predictor uses a frozen MERT-300M encoder with “75 Hz, 768-dim features” and a trainable 3-layer MLP with dimensions “512→256→128→2 outputs” (Mei et al., 7 Oct 2025). Audio is segmented into 5-second non-overlapping windows, features are extracted and pooled over time, and each window is regressed to $c_{1},\dots,c_{T}$0. The clip-level estimate is

$c_{1},\dots,c_{T}$1

The predictor is trained with Concordance Correlation Coefficient loss and validated on held-out in-domain data, reporting $c_{1},\dots,c_{T}$2, $c_{1},\dots,c_{T}$3, $c_{1},\dots,c_{T}$4, and $c_{1},\dots,c_{T}$5 (Mei et al., 7 Oct 2025).

Evaluation uses objective emotion adherence measured by Pearson’s $c_{1},\dots,c_{T}$6 and $c_{1},\dots,c_{T}$7 between ground-truth and predicted $c_{1},\dots,c_{T}$8, music quality measured by Fréchet Audio Distance (FAD), and a subjective user study with 8 non-musician raters assessing Overall Quality (OVL, 1–5) and perceived $c_{1},\dots,c_{T}$9 (Mei et al., 7 Oct 2025).

Method	Objective results on DEAM test set
Emotion Text Prompting (nearest ANEW word)	FAD = 4.81; $\mathbf{E}$0; $\mathbf{E}$1; $\mathbf{E}$2; $\mathbf{E}$3
Vanilla CE Fine-tuning	FAD = 2.34; $\mathbf{E}$4; $\mathbf{E}$5; $\mathbf{E}$6; $\mathbf{E}$7
LARA-Gen	FAD = 2.14; $\mathbf{E}$8; $\mathbf{E}$9; $\mathbf{E}$0; $\mathbf{E}$1

In the subjective evaluation, ground-truth received OVL $\mathbf{E}$2, $\mathbf{E}$3, $\mathbf{E}$4; Text Prompting received OVL $\mathbf{E}$5, $\mathbf{E}$6, $\mathbf{E}$7; and LARA-Gen received OVL $\mathbf{E}$8, $\mathbf{E}$9, $L$0 (Mei et al., 7 Oct 2025).

The paper summarizes these findings by stating that LARA-Gen achieves state-of-the-art continuous emotion control, “especially in arousal,” while maintaining or improving generation quality (Mei et al., 7 Oct 2025). More conservatively, the reported numbers indicate stronger objective arousal adherence than the listed baselines and a lower FAD than both text prompting and vanilla CE fine-tuning.

6. Qualitative behavior, interpretive significance, and limitations

The qualitative analysis associates different regions of the valence–arousal plane with distinct musical attributes. Low-valence/low-arousal settings such as $L$1 are described as exhibiting “slow tempo, minor-key motifs, sparse instrumentation,” whereas high-valence/high-arousal settings such as $L$2 are described as “upbeat, faster rhythmic patterns, major harmonies with bright timbres” (Mei et al., 7 Oct 2025). Mixed settings, for example $L$3, are described as “slow but consonant progressions with moderate dynamics, conveying ‘calm happiness’” (Mei et al., 7 Oct 2025).

The framework also supports emotional trajectories over time. Varying $L$4 dynamically, such as ramping arousal from $L$5, is reported to “smoothly modulate features such as note density, attack strength, and spectral centroid,” producing “a perceivable arousal climb” (Mei et al., 7 Oct 2025). Similarly, sweeping valence from $L$6 yields “a gradual shift from darker (minor-oriented) to brighter (major-oriented) chord voicings” (Mei et al., 7 Oct 2025). This suggests that the continuous conditioning mechanism is intended not merely for static affect labels but for temporally evolving expressive control.

Several boundaries should nonetheless be noted. The reported objective results show stronger gains in arousal than in valence, with LARA-Gen improving $L$7 to $L$8 while $L$9 is $$\mathbf{H} = \bigl[\mathbf{h}_{1}, \dots, \mathbf{h}_{T-1}\bigr]\in\mathbb{R}^{B\times (T-1)\times D},$$0 and $$\mathbf{H} = \bigl[\mathbf{h}_{1}, \dots, \mathbf{h}_{T-1}\bigr]\in\mathbb{R}^{B\times (T-1)\times D},$$1 remains negative at $$\mathbf{H} = \bigl[\mathbf{h}_{1}, \dots, \mathbf{h}_{T-1}\bigr]\in\mathbb{R}^{B\times (T-1)\times D},$$2 on the DEAM test set (Mei et al., 7 Oct 2025). Accordingly, any reading that treats the model as uniformly solving both dimensions of affect would overstate the evidence. The results support improved continuous emotion control, but they also indicate that valence prediction and control remain more difficult under this benchmark.

More broadly, LARA-Gen frames emotional controllability as a representational alignment problem: internal hidden states are trained to encode affective information seen by a frozen music-understanding model, while a separate continuous control pathway injects target emotion directly into the generator (Mei et al., 7 Oct 2025). Within that framing, its main significance lies in making affect a first-class latent variable in text-to-music generation rather than an indirect by-product of textual conditioning alone.