---
title: 'LARA-Gen: Continuous Emotion Control'
url: https://www.emergentmind.com/topics/lara-gen
type: topic
---

# LARA-Gen: Continuous Emotion Control

LARA-Gen is a framework for continuous emotion control in text-to-music generation that aligns the internal hidden states of a generative model with an external music understanding model through Latent Affective Representation Alignment (LARA), while conditioning generation on a continuous valence–arousal representation rather than relying solely on text prompts [2510.05875]. Introduced for text-to-music models built on MusicGen, it couples a latent alignment objective with an emotion control module and an evaluation benchmark, with the stated aim of achieving fine-grained emotional controllability without the bottlenecks of text-based prompting [2510.05875].

## 1. Conceptual definition and scope

LARA-Gen addresses a specific limitation in text-to-music generation: coherent music generation from text prompts has advanced, but fine-grained emotional control remains unresolved [2510.05875]. Its central premise is that emotional controllability can be improved by supervising the generative model’s hidden states with affective representations extracted from an external model and by representing target emotion in a continuous valence–arousal space [2510.05875].

The framework is defined by three coupled components. First, it builds on MusicGen as an autoregressive text-to-music backbone. Second, it introduces the LARA module, which aligns hidden states with “ground-truth” emotion features from a frozen MERT encoder. Third, it introduces an emotion control module that parameterizes emotion as $\mathbf{e}=(v,a)$ with $v,a\in[1,9]$, normalizes this target to $[-1,1]$, and injects the resulting embedding alongside text conditioning into every cross-attention layer [2510.05875].

A common misconception in controllable music generation is that emotional control can be reduced to prompt engineering alone. LARA-Gen explicitly treats text-based prompting as a bottleneck for continuous control, and its design therefore disentangles emotional attributes from textual content by introducing a dedicated continuous emotion pathway [2510.05875]. This suggests a shift from lexical affect specification toward latent affect supervision.

## 2. Backbone model and latent affective representation alignment

The generative backbone is MusicGen, described here as “a 24-layer, 1024-dimensional Transformer with 16 attention heads,” operating autoregressively over a sequence of discrete acoustic tokens $c_{1},\dots,c_{T}$ and conditioning embeddings $\mathbf{E}$ [2510.05875]. At each Transformer block, $\mathbf{E}$ is injected via cross-attention, and the final hidden-state sequence at layer $L$ is

$$
\mathbf{H} = \bigl[\mathbf{h}_{1}, \dots, \mathbf{h}_{T-1}\bigr]\in\mathbb{R}^{B\times (T-1)\times D},
$$

with batch size $B$ and $D=1024$ [2510.05875].

The LARA module supplies explicit affective supervision. “Ground-truth” emotion features are extracted from a frozen MERT encoder as

$$
\bar{\mathbf{M}} = [\bar{\mathbf{m}}_{1},\ldots,\bar{\mathbf{m}}_{N}]\in\mathbb{R}^{B\times N\times D_{\mathrm{MERT}}},
$$

where $N\ll T$ [2510.05875]. A lightweight Proxy Network $\mathcal{P}_{\theta}$, implemented as a Transformer decoder with $N$ learnable queries $\mathbf{Q}\in\mathbb{R}^{N\times D}$, attends over $\mathbf{H}$ and predicts

$$
\hat{\mathbf{M}}
= \mathrm{Linear}\bigl(\mathrm{TransformerDecoder}(\mathbf{Q},\,\mathbf{H})\bigr)
\in\mathbb{R}^{B\times N\times D_{\mathrm{MERT}}}.
$$

The alignment objective minimizes the mean squared error between predicted and target affective features:

$$
\mathcal{L}_{\mathrm{LARA}}
= \frac{1}{B\,N}\sum_{b=1}^{B}\sum_{i=1}^{N}
\bigl\lVert \hat{\mathbf{m}}_{i}^{(b)} - \bar{\mathbf{m}}_{i}^{(b)}\bigr\rVert_{2}^{2}.
$$

The paper’s interpretation is explicit: this loss “forces the high-dimensional hidden states of MusicGen to carry the same affective information as seen by the external MERT model, yielding an explicit emotion supervision signal” [2510.05875]. In practical terms, LARA-Gen does not merely condition output on emotion labels; it constrains the latent space so that internal representations become affect-aware.

## 3. Continuous valence–arousal conditioning

Emotion targets are represented in a continuous two-dimensional valence–arousal space. Specifically, the target is $\mathbf{e}=(v,a)$ with $v,a\in[1,9]$, and this pair is normalized to $[-1,1]$ before being processed by a small MLP, the Arousal-Valence Encoder, to obtain a $D$-dimensional embedding:

$$
\mathbf{E}_{\mathrm{emo}}
= \mathrm{MLP}_{\mathrm{AV}}\bigl((v,a)\bigr)
\in\mathbb{R}^{B\times D}.
$$

In parallel, the text prompt $\mathrm{p}_{\mathrm{text}}$ is encoded by a frozen T5 encoder into $\mathbf{E}_{\mathrm{text}}\in\mathbb{R}^{B\times T_{\mathrm{T5}}\times D}$ [2510.05875]. The text and emotion embeddings are then concatenated, with the emotion embedding padded as a one-token sequence:

$$
\mathbf{E} = \bigl[\mathbf{E}_{\mathrm{text}} \,\|\, \mathbf{E}_{\mathrm{emo}}\bigr]
\;\in\;\mathbb{R}^{B\times (T_{\mathrm{T5}}+1)\times D}.
$$

This combined conditioning is used in every cross-attention layer of the Transformer [2510.05875].

The architectural consequence is that emotional conditioning is not mediated indirectly through natural-language descriptions such as mood adjectives alone. Instead, LARA-Gen injects a dedicated continuous control signal throughout the model’s depth. The paper states that this “effectively gat[es] the generation to follow the precise continuous emotion target” [2510.05875]. A plausible implication is that the model can separate semantic content specified by text from affective state specified by $(v,a)$ more cleanly than text-only approaches.

## 4. Objective function, training regime, and optimization

LARA-Gen is trained with a composite loss comprising standard token prediction and latent affective alignment. The acoustic-token objective is the standard cross-entropy loss

$$
\mathcal{L}_{\mathrm{CE}}
= \mathbb{E}_{(\mathbf{C},\mathbf{E})\sim\mathcal{D}}
\Bigl[\sum_{t=1}^{T-1}
\mathrm{CrossEntropy}\bigl(\text{logit}_{t},\,c_{t+1}\bigr)\Bigr].
$$

The total objective is

$$
\mathcal{L}_{\mathrm{total}}
= \mathcal{L}_{\mathrm{CE}}
\;+\;\alpha\,\mathcal{L}_{\mathrm{LARA}}
\qquad \text{with}\;\alpha=100.
$$

Thus, latent affective supervision is not auxiliary in a negligible sense; it is explicitly weighted in the training criterion [2510.05875].

The training data comprise “22 067 thirty-second instrumental clips (valence–arousal labels in $[1,9]$) from multiple open platforms” [2510.05875]. MusicGen is fine-tuned for 20 000 steps, the Proxy Network is trained jointly, and the MERT encoder remains frozen. The learning rate schedule is stated to be identical to MusicGen’s original setup, while the batch size is noted as omitted in the paper but “typically 32” [2510.05875].

These design choices place LARA-Gen within a hybrid conditioning-and-representation-learning paradigm. It is neither a purely prompt-based control method nor solely a supervised regression head attached to a generator. Instead, it combines autoregressive music modeling, continuous conditioning, and hidden-state alignment. This suggests that controllability is treated as an internal representation problem as much as an input-format problem.

## 5. Benchmark, evaluation protocol, and empirical results

LARA-Gen is evaluated on a benchmark built around a curated test set and an Emotion Predictor [2510.05875]. The test set is an out-of-domain subset of DEAM consisting of 986 thirty-second clips with no vocals and expert valence–arousal annotations [2510.05875].

The Emotion Predictor uses a frozen MERT-300M encoder with “75 Hz, 768-dim features” and a trainable 3-layer MLP with dimensions “512→256→128→2 outputs” [2510.05875]. Audio is segmented into 5-second non-overlapping windows, features are extracted and pooled over time, and each window is regressed to $\hat{\mathbf{e}}_{i}=(\hat v_{i},\hat a_{i})$. The clip-level estimate is

$$
\hat{\mathbf{e}}=\tfrac1N\sum_{i}\hat{\mathbf{e}}_{i}.
$$

The predictor is trained with Concordance Correlation Coefficient loss and validated on held-out in-domain data, reporting $r_{A}=0.83$, $r_{V}=0.70$, $R^{2}_{A}=0.68$, and $R^{2}_{V}=0.47$ [2510.05875].

Evaluation uses objective emotion adherence measured by Pearson’s $r$ and $R^{2}$ between ground-truth and predicted $(v,a)$, music quality measured by Fréchet Audio Distance (FAD), and a subjective user study with 8 non-musician raters assessing Overall Quality (OVL, 1–5) and perceived $(v,a)$ [2510.05875].

| Method | Objective results on DEAM test set |
|---|---|
| Emotion Text Prompting (nearest ANEW word) | FAD = 4.81; $r_{A}=0.34$; $r_{V}=0.12$; $R^{2}_{A}=-1.45$; $R^{2}_{V}=-2.85$ |
| Vanilla CE Fine-tuning | FAD = 2.34; $r_{A}=0.49$; $r_{V}=0.31$; $R^{2}_{A}=-0.13$; $R^{2}_{V}=-0.91$ |
| LARA-Gen | FAD = 2.14; $r_{A}=0.69$; $r_{V}=0.27$; $R^{2}_{A}=0.16$; $R^{2}_{V}=-0.99$ |

In the subjective evaluation, ground-truth received OVL $= 3.94 \pm 0.98$, $r_{A}=0.55$, $r_{V}=0.59$; Text Prompting received OVL $= 3.30 \pm 1.14$, $r_{A}=0.17$, $r_{V}=0.09$; and LARA-Gen received OVL $= 3.48 \pm 1.08$, $r_{A}=0.48$, $r_{V}=0.17$ [2510.05875].

The paper summarizes these findings by stating that LARA-Gen achieves state-of-the-art continuous emotion control, “especially in arousal,” while maintaining or improving generation quality [2510.05875]. More conservatively, the reported numbers indicate stronger objective arousal adherence than the listed baselines and a lower FAD than both text prompting and vanilla CE fine-tuning.

## 6. Qualitative behavior, interpretive significance, and limitations

The qualitative analysis associates different regions of the valence–arousal plane with distinct musical attributes. Low-valence/low-arousal settings such as $v\approx2,a\approx2$ are described as exhibiting “slow tempo, minor-key motifs, sparse instrumentation,” whereas high-valence/high-arousal settings such as $v\approx8,a\approx8$ are described as “upbeat, faster rhythmic patterns, major harmonies with bright timbres” [2510.05875]. Mixed settings, for example $v=7,a=3$, are described as “slow but consonant progressions with moderate dynamics, conveying ‘calm happiness’” [2510.05875].

The framework also supports emotional trajectories over time. Varying $\mathbf{e}$ dynamically, such as ramping arousal from $3\to7$, is reported to “smoothly modulate features such as note density, attack strength, and spectral centroid,” producing “a perceivable arousal climb” [2510.05875]. Similarly, sweeping valence from $3\to8$ yields “a gradual shift from darker (minor-oriented) to brighter (major-oriented) chord voicings” [2510.05875]. This suggests that the continuous conditioning mechanism is intended not merely for static affect labels but for temporally evolving expressive control.

Several boundaries should nonetheless be noted. The reported objective results show stronger gains in arousal than in valence, with LARA-Gen improving $r_{A}$ to $0.69$ while $r_{V}$ is $0.27$ and $R^{2}_{V}$ remains negative at $-0.99$ on the DEAM test set [2510.05875]. Accordingly, any reading that treats the model as uniformly solving both dimensions of affect would overstate the evidence. The results support improved continuous emotion control, but they also indicate that valence prediction and control remain more difficult under this benchmark.

More broadly, LARA-Gen frames emotional controllability as a representational alignment problem: internal hidden states are trained to encode affective information seen by a frozen music-understanding model, while a separate continuous control pathway injects target emotion directly into the generator [2510.05875]. Within that framing, its main significance lies in making affect a first-class latent variable in text-to-music generation rather than an indirect by-product of textual conditioning alone.

Source: https://www.emergentmind.com/topics/lara-gen