---
title: 'SmoothCLAP: Soft-Target Enhanced CLAP'
url: https://www.emergentmind.com/topics/smoothclap
type: topic
---

# SmoothCLAP: Soft-Target Enhanced CLAP

SmoothCLAP is a soft-target enhanced variant of Contrastive Language–Audio Pretraining (CLAP) designed to address the inherent ambiguity and graded boundaries of human emotions in affective computing tasks. By diffusing supervisory signal among semantically related and paralinguistically similar samples, SmoothCLAP produces embeddings that better respect psychological emotion spaces and achieves improved zero-shot performance across diverse benchmarks [2601.12591].

## 1. Foundations: CLAP and Its Limitations

Contrastive Language–Audio Pretraining (CLAP) models aim to map paired audio and text samples into a shared latent space by jointly optimizing an audio encoder $f_a$ and a text encoder $f_t$. The embedded representations:
- Audio: $z_i^a = \mathrm{proj}_A\bigl(f_a(A_i)\bigr) \in \mathbb{R}^d$
- Text: $z_i^t = \mathrm{proj}_T\bigl(f_t(T_i)\bigr) \in \mathbb{R}^d$

Pairwise similarity is computed via scaled cosine similarity:
$$
s_{ij} = \frac{z_i^a \cdot z_j^t}{\tau}
$$
where $\tau$ is learnable.

Supervision relies on a symmetric InfoNCE objective over a batch of $B$ samples:
$$
\mathcal{L}_\mathrm{CLAP} = -\frac{1}{2B} \sum_{i=1}^B \left[\log\frac{\exp(s_{ii})}{\sum_{j=1}^B \exp(s_{ij})} + \log\frac{\exp(s_{ii})}{\sum_{j=1}^B \exp(s_{ji})}\right]
$$
This enforces rigid one-to-one alignment of audio–text pairs and treats all non-matching pairs as equally negative.

For affective-computing, such hard supervision is mismatched to the fuzzy category structure of emotions (e.g., "fear" and "disgust" overlap more than "fear" and "happiness"), as emotional states are not cleanly separable.

## 2. Motivations for Soft-Target Supervision

SmoothCLAP introduces soft targets to remedy CLAP’s disregard of intra-modal similarity and rich paralinguistic structure:
- **Intra-modal similarity:** Emotional neighboring samples often share acoustic or semantic features that should partially overlap in supervision.
- **Paralinguistic features:** Computationally derived vocal cues (pitch, intensity, jitter, etc.) encode nuanced affective information.

Instead of enforcing all non-matching samples as strictly negative, SmoothCLAP diffuses some probability mass to samples identified as semantically or paralinguistically close in “emotion space”.

## 3. Architecture and Training Objective

SmoothCLAP consists of parallel audio and text branches:
- **Audio branch:** wav2vec2.0-large encoder (finetuned on Speech Emotion Recognition) is used for both global and frame-level (“local”) feature extraction. Parameters are frozen during training.
- **Text branch:** BERT-base encoder, trainable, with a fully-connected projection.

### Soft Target Construction

Within each minibatch ($B$ samples), intra-modal similarity is computed:

**Audio-to-audio (a2a):** For mean-pooled local features $\bar\ell^a_i$:
$$
q^{a2a}_{ij} = \frac{\exp\bigl( \bar\ell^a_i \cdot \bar\ell^a_j / \tau_{a2a} \bigr)}{ \sum_{k=1}^B \exp\bigl( \bar\ell^a_i \cdot \bar\ell^a_k / \tau_{a2a} \bigr) }
$$

**Text-to-text (t2t):** With $\ell_2$-normalized embeddings $e^t_i$:
$$
q^{t2t}_{ij} = \frac{ \exp\bigl( e^t_i \cdot e^t_j / \tau_{t2t}\bigr) }{ \sum_{k=1}^B \exp \bigl( e^t_i \cdot e^t_k / \tau_{t2t}\bigr) }
$$

**Mixture:** $\gamma$ controls mixing:
$$
q_{ij} = (1-\gamma) q^{a2a}_{ij} + \gamma q^{t2t}_{ij}
$$

**Fusion with one-hot targets ($\beta$ fusion factor):**
$$
y_{ij} = (1-\beta) \delta_{ij} + \beta q_{ij}
$$

### Objective

Predicted cross-modal distributions $p^{a2t}_{ij}, p^{t2a}_{ij}$ are computed from the final projected and normalized embeddings. A symmetric KL divergence loss is used:
$$
\mathcal{L}_{\mathrm{soft}} = \frac{1}{2B} \sum_{i=1}^B \left\{ KL(y_i \| p^{a2t}_i) + KL(p^{a2t}_i \| y_i) + KL(y_i \| p^{t2a}_i) + KL(p^{t2a}_i \| y_i) \right\}
$$
Setting $\beta=0$ recovers the original InfoNCE.

**Hyperparameters:** Grid search found optimal $\gamma \approx 0.5$, $\beta \approx 0.1$. Text encoder LR: $1\times10^{-5}$; projection and temperature LR: $1\times10^{-3}$; batch size 32, 10 epochs, Adam optimizer.

## 4. Inference Pipeline

Inference with SmoothCLAP is identical to CLAP:
1. Audio input is embedded via $f_a$ and $\mathrm{proj}_A$.
2. Candidate text labels are embedded via $f_t$ and $\mathrm{proj}_T$.
3. Cosine similarity $z^a \cdot z^t_c$ is computed.
4. The label with highest score is selected, or retrieval is thresholded.

No computation of intra-modal similarity is performed at test time; the forward path is unchanged.

## 5. Experimental Evaluation

**Datasets:** Training uses MSP-Podcast v1.9 (45,619 utterances, 10 emotion categories). Zero-shot evaluation spans eight corpora in English and German, including IEMOCAP, RAVDESS, CREMA-D, TESS, FAU-Aibo (2/5 class), ALC (intoxication), and SLD (likability).

**Baselines:** CLAP, Pengi (audio-LDM), ParaCLAP.

**Main Results (Unweighted Average Recall, UAR):**

| Dataset                | CLAP | Pengi | ParaCLAP | SmoothCLAP |
|------------------------|------|-------|----------|------------|
| IEMOCAP (4cl/en)       | .353 | .345  | .600     | **.606**   |
| RAVDESS (8cl/en)       | .199 | .148  | .228     | .175       |
| CREMA-D (6cl/en)       | .230 | .245  | .177     | **.266**   |
| TESS (7cl/en)          | .232 | .177  | .170     | **.275**   |
| FAU-Aibo (2cl/de)      | .500 | .470  | .526     | **.555**   |
| FAU-Aibo (5cl/de)      | .211 | .185  | .197     | .204       |
| ALC (2cl/de)           | .511 | .473  | .537     | **.541**   |
| SLD (2cl/de)           | .472 | .485  | .507     | .496       |

SmoothCLAP achieves best UAR on 5/8 tasks and second-best on 2/8, especially improving on IEMOCAP, CREMA-D, TESS, and FAU-Aibo (2-class). Cross-lingual transfer from English to German remains strong.

## 6. Ablation Studies

**Local Feature Extractor:**
Performance varies across local feature encoders. No single extractor excels universally, suggesting dataset-specific paralinguistic sensitivity.

| LocFeat Encoder               | IEMOCAP | RAVDESS | CREMA-D | TESS  |
|-------------------------------|---------|---------|---------|-------|
| wav2vec2.0-Emo (finetuned)    | **.606**| .175    | .266    | .275  |
| wav2vec2.0-Large (self-sup.)  | .594    | .173    | .260    | **.368** |
| WavLM-Large                   | .595    | .212    | .201    | .267  |
| HuBERT-Large                  | .574    | **.260**| **.259**| .433  |

**Impact of $\gamma$ and $\beta$:**
Grid search shows no monotonic trend for mixing $\gamma$; fusion $\beta$ favors smaller values, indicating the need to retain strong hard-target signals.

## 7. Analysis and Future Directions

Soft targets facilitate improved learning by bridging fuzzy emotional categories—diffusing probability mass to semantically or paralinguistically neighboring samples enables a smoother embedding space, approximating human perceptual emotion similarity.

SmoothCLAP underperforms on 3/8 benchmarks, implying no approach is universally optimal for emotion recognition. Performance may depend on training data scale and diversity. Future improvements may involve adopting alternative similarity kernels (e.g., Gaussian over CP features), supporting multi-label or fuzzy annotation schemes, or end-to-end training of the local feature extractor.

SmoothCLAP demonstrates consistent zero-shot performance gains for affect-aware language–audio models by incorporating soft supervision rooted in intra-modal and paralinguistic similarities, producing more graded, psychologically plausible embeddings [2601.12591].

Source: https://www.emergentmind.com/topics/smoothclap