---
title: 'TRILLsson: Distilled Paralinguistic Speech Models'
url: https://www.emergentmind.com/topics/trillsson
type: topic
---

# TRILLsson: Distilled Paralinguistic Speech Models

TRILLsson is a family of distilled universal paralinguistic speech representation models introduced to make strong non-semantic speech embeddings publicly available in a compact, deployable form. It was proposed as a public, lightweight alternative to the much larger CAP12 teacher, while retaining most of the teacher’s performance on the Non-Semantic Speech (NOSS) benchmark and substantially improving over earlier public baselines such as TRILL on several paralinguistic tasks [2203.00236]. In subsequent work, TRILLsson has been treated not as an automatic speech recognition front end, but as a task-aligned representation for speech emotion recognition, depression detection, synthetic-speech source tracing, audio spoofing detection, and multimodal clinical assessment, typically in frozen-feature pipelines using 1024-dimensional embeddings [2402.01579].

## 1. Definition and lineage

TRILLsson was introduced as a collection of small, publicly released paralinguistic speech models obtained by distilling the 12th-layer, 1024-dimensional representation of CAP12, a 606M-parameter Conformer trained with a modified Wav2Vec 2.0 self-supervised training loss on about 900K hours of YouTube-derived speech (YT-U) [2203.00236]. The original motivation was twofold: strong paralinguistic embedding models were often too large for deployment, and many of the strongest systems were not publicly available. TRILLsson therefore emphasized public release, public-data-only distillation, and a practical size–performance frontier.

The model family sits in a lineage that includes TRILL, TRILL-Distilled, FRILL, and then TRILLsson as a stronger later-generation family. Later papers consistently describe TRILLsson as distilled from the “SOTA universal paralinguistic conformer (CAP12),” note that TRILLsson is open-sourced while CAP12 is not, and treat it as a representative paralinguistic or non-semantic speech pre-trained model rather than a linguistic or ASR-centric encoder [2509.16182].

| Variant | Parameters | Size |
|---|---:|---:|
| TRILLsson 5 (AST) | 88.6M | 314 MB |
| TRILLsson 4 (AST) | 63.4M | 224 MB |
| TRILLsson 3 (EffNetv2) | 21.5M | 99 MB |
| TRILLsson 2 (EffNetv2) | 8.1M | 42 MB |
| TRILLsson 1 (ResNet) | 5.0M | 22 MB |

The original paper’s headline result is that the largest distilled model is less than 15% the size of the 2.2 GB teacher and achieves over 96% of the accuracy on 6 of 7 NOSS tasks, while the smallest model is 1% in size and achieves over 90% of the accuracy on 6 of 7 tasks [2203.00236]. That framing established TRILLsson as a public, distilled approximation to a much larger paralinguistic teacher, not as a replacement for speech transcription models.

## 2. Distillation procedure and representational form

In the original formulation, TRILLsson students operate on fixed 2-second windows of log-magnitude mel-frequency spectrograms with 80 mel bins spanning 125 Hz to 7500 Hz, using a 25 ms window and 10 ms hop [2203.00236]. The distillation strategy uses local matching rather than global matching: for each 2-second student input window, the student is trained to match the teacher embedding computed on that same window. Three student architecture families were explored—AST, EfficientNetV2, and ResNetish—and the downstream evaluation protocol time-averages embeddings before linear probing on the benchmark tasks [2203.00236].

Later application papers use TRILLsson in several frozen-feature modes. In multilingual SER, TRILLsson1 from TensorFlow Hub is described as aggregating over time and returning a 1024-dimensional vector for each input audio [2402.01579]. In cross-corpus SER, the authors state, “For each frozen PTM, we extract the last hidden states through average pooling,” with TRILLsson producing 1024-dimensional utterance representations from audio resampled to 16 kHz [2509.16182]. In source tracing of synthetic speech generation systems, TRILLsson is again used as a frozen 1024-dimensional representation obtained from the last hidden state with mean pooling after resampling to 16 kHz [2506.01157]. In generalizable spoofing detection, the model is used in a chunk-level scheme: audio up to 6 seconds is divided into 50, 100, 200, or 300 ms chunks, each chunk is encoded by a frozen TRILLsson model, and the stacked chunk embeddings are sent to a learned temporal backend [2509.00186].

This variation in extraction protocol suggests that TRILLsson is treated in the literature as a flexible paralinguistic front end whose outputs can support utterance-level classification, chunk-level temporal modeling, or multimodal fusion, provided that the downstream pipeline preserves its non-semantic emphasis.

## 3. Benchmark performance on non-semantic speech tasks

The NOSS benchmark in the original paper covers seven tasks: VoxCeleb1, VoxForge, Speech Commands, ASVSpoof 2019, Euphonia, CREMA-D, and IEMOCAP [2203.00236]. Evaluation is deliberately simple: embeddings are extracted, time-averaged, and scored with linear models, while aggregate comparison across tasks uses equivalent d-prime. Within that protocol, CAP12 remains strongest overall, but TRILLsson preserves much of its performance at a fraction of the size.

On the two emotion tasks, the larger variants remain close to CAP12: TRILLsson 5 reports 86.1 on CREMA-D and 72.7 on IEMOCAP, compared with 88.2 and 75.5 for CAP12; even the smallest TRILLsson 1 still reaches 81.3 and 68.5 [2203.00236]. Against the best open-source Wav2Vec 2.0 layer reported in the same study, TRILLsson 5 and TRILLsson 4 outperform it on 6 of 7 tasks, losing only on Speech Commands, while the smallest TRILLsson 1 outperforms the open-source Wav2Vec 2.0 model on both emotion tasks despite being about 7% the size [2203.00236].

Later work on BRILLsson treats TRILLsson as a strong teacher model for non-semantic speech representation learning. There the teacher is a TRILLsson EfficientNet-V2 v3 model with 21.5M parameters, 1024-dimensional embeddings, and training ancestry traced through AudioSet, Libri-light, and the CAP12 teacher [2207.05784]. BRILLsson shows that much of TRILLsson’s utility can be distilled again into binary neural networks around 2 MB with latency less than 8 ms, although the student is clearly not a full substitute on harder tasks such as CREMA-D and Voxforge [2207.05784]. This establishes a second compression layer in the TRILLsson lineage: CAP12 to TRILLsson, then TRILLsson to BRILLsson.

## 4. Central role in speech emotion recognition

TRILLsson’s strongest sustained downstream evidence is in speech emotion recognition. In multilingual SER across English, Urdu, Arabic, German, and Greek, the model is presented as the paralinguistic PTM in a direct comparison with wav2vec2, XLS-R, Whisper, and x-vector, and it is reported as the best representation on all five datasets [2402.01579]. The reported scores are 83.28 / 81.66 on CREMA-D, 98.75 / 98.71 on URDU, 89.15 / 88.88 on BAVED, 96.26 / 96.20 on Emo-DB, and 94.21 / 93.83 on AESDD, all in Accuracy / F1 format [2402.01579]. A dimension-controlled comparison projects higher-dimensional features to 512 dimensions and still finds that TRILLsson remains the best, which the authors use to argue that its advantage is not merely a dimensionality artifact [2402.01579].

A later cross-corpus study makes TRILLsson the central missing baseline in cross-corpus SER benchmarking and argues that its earlier omission made benchmark conclusions incomplete because SER is “inherently a paralinguistic task” [2509.16182]. In that study, TRILLsson is benchmarked against monolingual PTMs (WavLM, UniSpeech-SAT, Wav2Vec2), multilingual PTMs (XLS-R, Whisper, MMS), and a speaker-recognition PTM (x-vector), using frozen representations, average pooling, and lightweight FCN or CNN heads. The paper states that “TRILLsson consistently outperforms monolingual, multilingual, and speaker recognition PTMs across CREMA-D, MESD, Emo-DB, AESDD, and RAVDESS,” and that “For cross-corpus experiments : TRILLsson outperforms other PTMs—including monolingual, multilingual, and speaker recognition PTMs across all datasets” [2509.16182].

The quantitative margins are often large. In same-corpus settings, TRILLsson reaches 79.70 | 79.63 on CREMA-D, 88.89 | 88.85 on MESD, 98.67 | 98.13 on Emo-DB, 93.39 | 93.35 on AESDD, and 96.36 | 96.35 on RAVDESS with CNN heads [2509.16182]. In cross-corpus transfer, representative CNN results include 51.67 | 47.55 for CREMA-D → MESD, 79.33 | 74.16 for CREMA-D → Emo-DB, 91.08 | 90.67 for AESDD → Emo-DB, 73.90 | 73.55 for Emo-DB → AESDD, and 62.23 | 61.59 for RAVDESS → CREMA-D [2509.16182]. The authors attribute this to better capture of “intonation, pitch, rhythm, and prosody,” and their t-SNE analysis reports that same-emotion samples from different datasets are mapped into a joint representational space “irrespective of linguistic difference” [2509.16182].

A persistent misconception challenged by these studies is that the strongest multilingual or ASR-oriented speech encoder should transfer best to SER. The combined evidence from multilingual and cross-corpus experiments instead supports the narrower claim that, within frozen-feature benchmarking, task-aligned paralinguistic pre-training can be more appropriate than generic multilingual or transcription-centric pre-training for emotion recognition [2402.01579].

## 5. Clinical and forensic applications

TRILLsson has also been used as a non-semantic feature extractor in clinical and forensic speech analysis. In depression detection, it is one of three core non-semantic features—together with x-vector and emoHuBERT—and is the best individual feature on both datasets evaluated in that paper [2409.14312]. On E-DAIC, the best TRILLsson-only result is achieved by the CNN model with RMSE 6.56 and MAE 5.03; on the ANDROID benchmark, TRILLsson CNN reaches 80.52 / 78.68 on the Interview task and 75.13 / 73.94 on the Reading task in Accuracy / F1 terms [2409.14312]. The same study shows that fusion systems containing TRILLsson outperform corresponding systems without it, culminating in x-vector + emoHuBERT + TRILLsson with FuSeR at RMSE 5.51 and MAE 4.48 on E-DAIC [2409.14312].

In synthetic-speech source tracing, TRILLsson is the best single speech pre-trained model among monolingual, multilingual, speaker-recognition, and paralinguistic alternatives [2506.01157]. Its CNN results are 97.16% accuracy with 1.69% EER on ASVspoof 2019 and 92.81% accuracy with 3.37% EER on CFAD, both the best single-representation scores in the table [2506.01157]. When fused with x-vector in TRIO, performance rises to 99.56% accuracy and 0.19% EER on ASV, and 99.04% accuracy and 0.95% EER on CFAD, establishing the paper’s reported new SOTA [2506.01157].

In audio spoofing detection under severe domain shift, TRILLsson is used as the non-semantic front end in a chunk-based detector trained only on LA19 [2509.00186]. The best in-domain score is 0.96% EER from TRILLsson3 with 200 ms direct chunking, but the paper’s central claim concerns out-of-domain behavior: TRILLsson3 gives 13.27% EER on DF21, and TRILLsson1 gives 20.08% EER on In-the-Wild, outperforming a controlled XLS-R + same backend ablation and several end-to-end baselines on those OOD settings [2509.00186]. The authors explicitly argue that non-semantic representations from TRILL and TRILLsson exhibit greater temporal stability than semantic embeddings and therefore generalize better under unseen attacks and channel conditions [2509.00186].

In Indic CodecFake detection, TRILLsson forms the paralinguistic branch of SATYAM, paired with Whisper as the semantic branch [2604.19949]. The paper reports that a TRILLsson-only model is slightly better than a Whisper-only model on both ICF and CodecFake, and the full SATYAM system—combining Whisper, TRILLsson, hyperbolic Bhattacharya-distance alignment, and a frozen Qwen2 decoder—reaches 98.32 ACC / 3.27 EER on ICF and 99.11 / 1.94 on CodecFake [2604.19949]. The authors interpret this as evidence for the “predominantly paralinguistic nature of speech deepfake detection” in the studied setting [2604.19949].

## 6. Multimodal use, compact descendants, and limitations

TRILLsson is frequently used as a frozen encoder inside larger multimodal systems. In DIVINE, it is described as the strongest audio branch and is paired with DeepSeek-VL2 for synchronized audio-video neurological disorder assessment [2601.07014]. The best pair, DeepSeek-VL2 + TRILLsson, reaches 98.26% accuracy and 97.51% F1 when both audio and video embeddings are provided, and remains comparatively strong under missing-modality evaluation [2601.07014]. The paper’s interpretation is that TRILLsson contributes prosodic variation and speaker-dependent cues, while the downstream architecture supplies hierarchical disentanglement, sparse gating, and symptom-token reasoning [2601.07014].

At the opposite end of the design space, BRILLsson shows that TRILLsson can itself be a teacher for ultra-light binary students [2207.05784]. The reported BRILLsson models are about 2.0–2.1 MB with 6.4–7.6 ms latency on a Snapdragon 855, and a tiny variant is 0.65 MB with 6.1 ms latency, all obtained by directly imitating TRILLsson embeddings rather than downstream labels [2207.05784]. A plausible implication is that TRILLsson has become a representational anchor for both high-capacity and low-resource non-semantic speech pipelines.

The literature also makes clear that TRILLsson’s strengths are conditional rather than universal. Many favorable comparisons are reported under frozen-feature evaluation with lightweight FCN, CNN, linear, or shallow multimodal heads rather than full end-to-end fine-tuning [2402.01579]. Cross-corpus SER evidence is strong but limited to five acted emotion corpora, a five-emotion intersection, and no reported significance tests [2509.16182]. In multilingual SER, TRILLsson is best among the five PTMs tested, but it does not surpass prior external SOTA on CREMA-D [2402.01579]. In spoofing detection, the best In-the-Wild result is still 20.08% EER, which shows that real-world generalization remains difficult even when TRILLsson outperforms alternative front ends [2509.00186]. SATYAM explicitly notes that alternative encoder choices may lead to minor performance variations and does not benchmark TRILLsson against a broad set of other paralinguistic encoders inside the same framework [2604.19949].

Across these caveats, one methodological conclusion recurs: benchmark reliability depends on evaluating task-aligned pre-training families. The cross-corpus SER study therefore argues that future benchmarks should include TRILLsson and, more broadly, paralinguistic speech processing PTMs as required baselines [2509.16182]. In that narrower but increasingly well-supported sense, TRILLsson is both a model family and a benchmark correction: a distilled public representation designed for non-semantic speech analysis, and a recurring empirical reminder that paralinguistic pre-training can dominate when the downstream task is governed more by prosody, expression, and source-specific acoustic behavior than by lexical content.

Source: https://www.emergentmind.com/topics/trillsson