MorphFader: Granular Audio Morphing

• MorphFader is a method for fine-grained audio morphing that interpolates cross-attention components from disparate text prompts in diffusion-based text-to-audio models.
• It integrates with latent diffusion architectures by intercepting and linearly combining Q, K, V matrices to control semantic content during the denoising process.
• Empirical evaluations indicate that MorphFader achieves perceptually smooth transitions and superior metrics compared to baseline techniques on standard audio datasets.

MorphFader is a method enabling granular, fine-grained morphing of audio generated from disparate text prompts using diffusion-based text-to-audio models with cross-attention mechanisms. By directly intercepting and interpolating attention components in the latent diffusion process, MorphFader introduces precise semantic control over the gradual transformation between sounds, producing perceptually smooth and semantically meaningful audio hybrids without retraining or model modification (Kamath et al., 2024).

1. Architectural Foundations and Integration

MorphFader operates atop a pre-trained text-to-audio latent diffusion model (LDM) architecture, incorporating models such as AudioLDM (“audioldm_16k_crossattn_t5”), TANGO, or Stable Audio. The core inference pipeline involves the following steps:

• Sample a noise latent $z_T \sim \mathcal{N}(0, I)$.
• Iteratively denoise $z_t$ to $z_{t-1}$ across $T$ steps via a U-Net, conditioned on a prompt embedding $P$.
• Decode the final latent $z_0$ into a spectrogram with a VAE decoder, then vocode to obtain the waveform.

MorphFader integrates within every cross-attention block of the U-Net. Instead of utilizing the query (Q), key (K), and value (V) matrices computed for a single prompt, the method intercepts and stores the Q, K, V triplets for both source and target prompts over the full diffusion trajectory. These are linearly interpolated during inference, yielding morphed latent representations for the subsequent denoising steps.

2. Cross-Attention Interception and Interpolation

Within each timestep $t$ and layer $\ell$ of the diffusion U-Net, the cross-attention operates as follows:

• The attention map $$A_t = \operatorname{Softmax}( Q_t K_t^\top / \sqrt{d} )$$ is computed.
• The cross-attention matrix $M_t = A_t V_t$ determines latent activation.
• $Q_t \in \mathbb{R}^{n_q \times d}$ is the projected latent query; $K_t, V_t \in \mathbb{R}^{n_k \times d}$ derive from the text prompt embedding; $d$ is the attention dimensionality.

For both the source prompt $P^{(s)}$ and target prompt $P^{(\tau)}$, $Q_t^{(s)}, K_t^{(s)}, V_t^{(s)}$ and $Q_t^{(\tau)}, K_t^{(\tau)}, V_t^{(\tau)}$ are logged at each $t, \ell$. MorphFader interpolates these using scalar $\alpha \in [0,1]$: $$Q_t^{(m)} = (1-\alpha) Q_t^{(s)} + \alpha Q_t^{(\tau)}$$

$$K_t^{(m)} = (1-\alpha) K_t^{(s)} + \alpha K_t^{(\tau)}$$

$$V_t^{(m)} = (1-\alpha) V_t^{(s)} + \alpha V_t^{(\tau)}$$

These interpolated matrices $(Q_t^{(m)}, K_t^{(m)}, V_t^{(m)})$ are injected in lieu of recomputed single-prompt matrices during inference, with an unconditional prompt $P^\phi$ used internally to bypass additional attention. To amplify or attenuate specific words $w$ in the prompt, word-level scaling is applied: $\bar{V}_t = wts \circ V_t$ where $wts$ is a vector of per-token weights.

3. Morphing Algorithm and Computational Procedure

The morphing workflow comprises two main phases: attention component precomputation and morphed audio synthesis. The outline is as follows:

Precompute and cache cross-attention components:

• For both source and target prompts,
  • Sample $z_T \sim \mathcal{N}(0, I)$.
  • For $t = T \dots 1$, at each layer $\ell$,
  • Compute $$Q_{t, \ell}^{(\text{run})}, K_{t, \ell}^{(\text{run})}, V_{t, \ell}^{(\text{run})}$$ from U-Net attention.
  • Denoise $$z_{t-1} \leftarrow \text{DM-step}(z_t, P^{(\text{run})}, t, \text{seed})$$.

Morphed inference for chosen $\alpha$ (or schedule $\alpha_k$):

• Use the same $z_T$ seed as precomputation.
• For $t = T \dots 1$, at each layer $\ell$,
  • Form interpolated $$Q_{t,\ell}^{(m)}, K_{t,\ell}^{(m)}, V_{t,\ell}^{(m)}$$.
  • Denoise $$z_{t-1} \leftarrow \text{DM-step}(z_t, P^\phi, t, \text{seed}; \text{override\_attention})$$.
• Decode $z_0$ to spectrogram, vocode to waveform.

Repeating the synthesis for $\alpha \in \{0, 0.1, \dots, 1.0\}$ yields a smooth morph sequence between source and target sounds.

4. Semantic Control, Theory, and Empirical Observations

Cross-attention matrices directly modulate the influence of prompt word tokens on latent locations throughout the diffusion process. By interpolating Q, K, V, MorphFader enables continuous, fine-grained control over semantic content in the output audio, yielding perceptually smooth transformations. This approach parallels “Prompt-to-Prompt” techniques in image editing but operates in the Q, K, V space for audio. Variable scaling of V’s rows allows differential emphasis of specified adjectives or verbs, refining morph dynamics without altering unrelated tokens.

Empirical findings indicate that manipulating V alone is computationally efficient but joint Q, K, V interpolation achieves optimal perceptual and objective performance. Ablation studies confirm that the full combination of Q, K, V is necessary for highest-quality morphs.

5. Evaluation Methodology and Findings

MorphFader’s performance was evaluated using the AudioPairBank dataset (≈1,100 adjective/verb–noun pairs) and the AudioLDM model with $T=20$ diffusion steps producing 10 s outputs. Metrics used encompass both objective and subjective criteria:

Metric/Setup	Description	Outcome/Observation
FAD–AudioSet, FD–AudioSet	Embedding distance to AudioSet (lower is better)	MorphFader superior to baselines
Inception Score (IS)	Via PANN (higher is better)	MorphFader outperforms engineered mixes
Smoothness (ρ)	Pearson correlation CLAP similarity vs. $\alpha$	Equal to raw mixing; novel hybrid timbres
Subjective MOS	N=18 listeners, 20 pairs, $\alpha=0.5$	$50.5 \pm 1.7$, above baselines

Ablation (100 pairs, $\alpha$ in $[0,1]$, steps 0.1) shows superior results when interpolating Q, K, V together. In word-type analyses, verb weighting yields higher smoothness ($\rho = 0.56$) compared to adjectives ($\rho = 0.23$); yet morphing smoothness is comparable for both, confirmed by listener judgements.

6. Limitations and Open Questions

MorphFader requires storing Q, K, V matrices for every timestep and layer, incurring memory and compute overhead, though this is tractable for $T=20$. The interpolation parameter $\alpha$ is fixed per synthesized audio clip; dynamic $\alpha(t)$ schedules have not been explored. The methodology has been validated primarily on AudioLDM, and behavior on alternative architectures or with different $T$ remains unknown. Outputs are currently stationary morphs; temporally dynamic, continuous transformation within one clip (dynamic morphing) is proposed for future investigation. Output quality is dependent on the expressivity and capability of the underlying text-to-audio (TTA) model, particularly for out-of-distribution prompts that may degrade attention map quality.

MorphFader constitutes a plug-and-play procedure: intercept cross-attention Q, K, V, interpolate by user-controlled $\alpha$, and reinject to generate smooth, fine-grained morphs between text-prompted sounds without additional training (Kamath et al., 2024).