InterpTRQE-SptME Benchmark Explained

• The paper introduces InterpTRQE-SptME to directly quantify residual speaker timbre in content embeddings using Gradient SHAP attribution.
• It concatenates pre-trained content and speaker embeddings to train a speaker classifier at near-perfect accuracy, isolating residual timbre contributions.
• The benchmark provides unified TRQ metrics and ranking for model disentanglement, guiding layer sensitivity studies and post hoc timbre reduction techniques.

InterpTRQE-SptME Benchmark, expanded as Interpretability-based Timbre Residual Recognition and Quantitative Evaluation Benchmark of Speech pre-training Models Encoding, is a benchmark for directly quantifying speaker timbre residual in content embeddings produced by speech self-supervised or pre-trained models. Its central purpose is to measure, for a given model and layer, how much speaker-related information remains encoded in embeddings intended to carry linguistic content, without modifying those embeddings or conflating the measurement with downstream task performance. The benchmark operationalizes this with a speaker classifier defined on concatenated content and explicit timbre embeddings, and then uses Gradient SHAP attribution to estimate the relative contribution of each partition to speaker classification decisions (Zhu et al., 19 Jul 2025).

1. Concept and motivation

InterpTRQE-SptME was introduced to address a specific deficiency in prior evaluation practice: speaker disentanglement in speech representations had typically been assessed only indirectly, through downstream measures such as ASR word error rate, speaker identification accuracy on embeddings, phone purity, or cluster purity. Those measures are task-dependent, depend on particular optimization pipelines and labels, and do not provide a direct, unified estimate of the speaker timbre that leaks into content embeddings. The benchmark therefore defines timbre residual itself as the target of measurement, rather than treating it as an unobserved factor inferred from task outcomes (Zhu et al., 19 Jul 2025).

The conceptual premise is diagnostic rather than interventionist. A speaker classifier is deliberately trained on inputs that include both a content embedding and an explicit speaker embedding. If the classifier can already recover speaker identity from the explicit timbre embedding, then any additional attribution assigned to the content embedding indicates residual timbre encoded there. Because the interpretability step is post hoc and additive, the benchmark does not alter the speech pre-trained model and does not destroy or rewrite its content representation. This design is intended to isolate measurement from representation learning.

The benchmark’s emphasis on interpretability is encoded in the acronym itself. “Interp” denotes the use of interpretability methods; “TRQE” denotes Timbre Residual Recognition and Quantitative Evaluation; and “SptME” denotes Speech pre-training Models Encoding. In this formulation, the benchmark is model-agnostic at the level of attribution analysis, while remaining layer-specific at the level of representation auditing.

2. Formal definition and metric construction

Let $M$ denote a speech pre-trained model and $w$ an utterance. From a designated content-relevant layer, the benchmark extracts a content embedding $c \in \mathbb{R}^{d_c}$. A separate pre-trained speaker model provides a timbre embedding $s \in \mathbb{R}^{d_s}$. These are concatenated into a single input

$x = [c; s] \in \mathbb{R}^{d_c + d_s}.$

A speaker classifier

$$f: \mathbb{R}^{d_c + d_s} \rightarrow \mathbb{R}^{N_{\text{spk}}}$$

is trained on this concatenated representation. InterpTRQE-SptME then interprets $f$ using Gradient SHAP. The abstract Shapley value for feature $j$ is given as

$$\phi_j = \sum_{S \subseteq \{1,\dots,p\}\setminus \{j\}} \frac{|S|!(p-|S|-1)!}{p!}\,[f(S \cup \{j\}) - f(S)].$$

Gradient SHAP approximates these values with expected gradients over a baseline distribution $D$. For input $w$0 and baseline $w$1, the attribution for feature $w$2 is

$w$3

In the benchmark, attribution values are computed for every dimension of $w$4 and then partitioned into content indices $w$5 and timbre indices $w$6. Timbre residual for model $w$7 is defined as the normalized ratio

$w$8

The benchmark reports $w$9 as a percentage and interprets it as content-attribution magnitude relative to speaker-attribution magnitude. Lower percentages indicate better speaker disentanglement, because they imply that speaker classification relies less on the content embedding. Alongside this dimension-normalized Mean Score (%), the benchmark also reports a Sum Score (%), defined by the same ratio without dividing by $c \in \mathbb{R}^{d_c}$0 or $c \in \mathbb{R}^{d_c}$1. An intuitive residual rate is additionally written as

$c \in \mathbb{R}^{d_c}$2

although the primary reported metric remains $c \in \mathbb{R}^{d_c}$3 (Zhu et al., 19 Jul 2025).

3. Benchmark pipeline and implementation protocol

The benchmark is instantiated on the VCTK corpus using 20 English speakers, identified as p225–p246 with p235 and p242 excluded, for a total of 7,758 utterances, approximately 8 hours of speech. Audio is standardized by resampling to 16 kHz, converting to 16-bit mono WAV, and applying energy normalization to a peak of $c \in \mathbb{R}^{d_c}$4 dB with SoX, with the stated purpose of reducing loudness variance impacts (Zhu et al., 19 Jul 2025).

Speaker embeddings are produced by ECAPA-TDNN using SpeechBrain spkrec-ecapa-voxceleb, pre-trained on VoxCeleb, yielding $c \in \mathbb{R}^{d_c}$5. Content embeddings are extracted from designated layers of target speech pre-training models through forward hooks. The benchmark adopts embedding vector sequence concatenation: content representations are average-pooled across frames to produce a single utterance-level vector $c \in \mathbb{R}^{d_c}$6, which is then concatenated with the utterance-level speaker vector $c \in \mathbb{R}^{d_c}$7. Depending on the probed model, $c \in \mathbb{R}^{d_c}$8 is either 768 or 1024, so the concatenated input has dimensionality 960 or 1216.

The speaker classifier is a 4-layer MLP with ReLU activations and a softmax output over $c \in \mathbb{R}^{d_c}$9, trained with cross-entropy loss. The protocol explicitly aims to overfit the classifier to near 1.0, and the paper emphasizes achieving 100% speaker classification accuracy so that SHAP has a clear target function and yields stable attributions. Reported explainability settings are a fixed baseline set of 256 concatenated embeddings, batch size 256 for SHAP processing, use of the entire dataset for explanation sampling, and local smoothing of 0.1. The benchmark’s TRQ computation uses raw absolute SHAP magnitudes; no per-feature z-scoring of embeddings is required for TRQ (Zhu et al., 19 Jul 2025).

The evaluation sequence is fixed. For each model and layer, the benchmark extracts $s \in \mathbb{R}^{d_s}$0 and $s \in \mathbb{R}^{d_s}$1, trains the speaker MLP to 1.0 accuracy, runs Gradient SHAP over all inputs with the fixed baseline distribution and smoothing, aggregates $s \in \mathbb{R}^{d_s}$2 separately over content and speaker partitions, and computes the corpus-level Mean Score and Sum Score. The paper also reports consistency checks through visualization of feature-importance distributions, heatmaps across speakers, and batch-stability plots.

4. Models probed and empirical ranking

InterpTRQE-SptME probes seven variations across five model families. The benchmark evaluates HuBERT BASE at layer 9, HuBERT LARGE at layer 21, HuBERT-CH at layer 9, DPHuBERT at layer 12, ContentVec at layer 12, WavLM Base+ at layer 12, and Whisper-ppg at encoder output. Speaker information is consistently isolated through the external ECAPA-TDNN embedding, while content information is taken from the layer-specific mean-pooled representation (Zhu et al., 19 Jul 2025).

Model / layer	Mean Score (%)	Sum Score (%)
ContentVec, layer 12	5.20	17.21
Whisper-ppg, encoder	7.46	28.47
DPHuBERT, layer 12	7.73	23.62
WavLM Base+, layer 12	9.02	26.51
HuBERT BASE, layer 9	13.72	35.44
HuBERT-CH, layer 9	13.93	35.78
HuBERT LARGE, layer 21	18.65	49.86

By the benchmark’s ranking, ContentVec exhibits the best disentanglement, with a Mean Score of 5.20%, while HuBERT LARGE at layer 21 is the worst among the probed settings, with 18.65%. The paper further states that ContentVec removes approximately 62% residual timbre relative to HuBERT BASE’s 13.72%. Whisper-ppg also shows strong disentanglement at 7.46% despite multilingual training, while DPHuBERT retains 7.73% despite pruning and distillation. Lower values are interpreted as lower reliance on content features for speaker identification and thus stronger speaker disentanglement (Zhu et al., 19 Jul 2025).

These results are significant within the benchmark’s own measurement framework because they convert an otherwise indirect question—how much timbre remains in “content” representations—into a directly comparable scalar ratio across heterogeneous architectures and layers. The comparison is therefore not framed as a downstream-task leaderboard, but as an attribution-based audit of residual speaker information.

5. Layer sensitivity and relation to SHAP-based filtering

The paper supplements benchmark results with a layer-sensitivity study on HuBERT LARGE, comparing layers 18 and 21 using ASR CTC loss as a content-preservation probe. The reported CTC loss on raw waveform input is 1.2190. For layer 18 embeddings, the CTC loss is 1.2171, described as approximately no degradation versus raw input, with Mean residual 10.58% and Sum residual 36.08%. For layer 21 embeddings, the CTC loss rises to 1.2612, with Mean residual 18.05% and Sum residual 49.04%. This suggests that layer 18 is more content-pure than layer 21 for ASR and has less timbre leakage (Zhu et al., 19 Jul 2025).

InterpTRQE-SptME is also used to evaluate the companion filtering method InterpTF-SptME, which applies SHAP-guided transformations to reduce timbre residual without retraining the speech model. On HuBERT LARGE layer 21, SHAP Noise with $s \in \mathbb{R}^{d_s}$3 raises CTC loss by 5.08%, from 1.2612 to 1.3252, while reducing Mean residual from 18.05% to approximately $s \in \mathbb{R}^{d_s}$4, effectively near 0%, for a Score↓rate of 18.09%. With $s \in \mathbb{R}^{d_s}$5, CTC loss increases by 0.93% to 1.2730 and Mean residual becomes 2.21%; with $s \in \mathbb{R}^{d_s}$6, CTC loss increases by 0.11% to 1.2626 and Mean residual becomes 7.23%. Under SHAP Cropping, $s \in \mathbb{R}^{d_s}$7 gives CTC loss 1.2969 with Mean residual 4.65%, and $s \in \mathbb{R}^{d_s}$8 gives CTC loss 1.2779 with Mean residual 10.07%.

The paper’s interpretation is that SHAP Noise removes timbre more aggressively and preserves content better than Cropping for comparable residual reduction, and that negative SHAP components help suppress timbre, consistent with timbre’s dependence on relative spectral components rather than sign. Within the article’s broader framework, this makes the benchmark not only an auditing tool but also a quantitative criterion for evaluating post hoc timbre-mitigation procedures (Zhu et al., 19 Jul 2025).

6. Comparative position, limitations, and practical use

InterpTRQE-SptME is positioned against two broad classes of prior practice. First, it contrasts with indirect downstream evaluations such as ASR WER, SID accuracy, phone purity, or cluster purity. Those measures remain task-relevant, but the benchmark argues that they are non-unified and sensitive to datasets, fine-tuning protocols, and optimization goals. Second, it contrasts with training-based disentanglement methods, including adversarial approaches based on GANs, VAEs, bottlenecks, instance normalization, and information bottleneck formulations. Such methods change representations through objectives and architectures and may harm content, whereas InterpTRQE-SptME performs post hoc quantification without altering the pre-trained model (Zhu et al., 19 Jul 2025).

The benchmark also has explicit limitations. Attribution assigned to the content partition may be inflated by genuine correlations between phonetic realization and speaker traits, such as accent-related covariation. Absolute values depend on the baseline distribution $s \in \mathbb{R}^{d_s}$9 and smoothing parameters, even though the benchmark fixes $x = [c; s] \in \mathbb{R}^{d_c + d_s}.$0 to 256 samples to improve comparability. Residual timbre is layer-dependent, making layer selection critical; the HuBERT LARGE 18-versus-21 comparison is used as a concrete example. Domain shift is another constraint: evaluation on English VCTK is appropriate for English, but may not generalize to other languages, including multilingual models such as Whisper. The SHAP procedure is computationally expensive, and some standard training hyperparameters for the classifier—optimizer, learning rate, epochs, random seeds, hardware, and runtime—are not specified (Zhu et al., 19 Jul 2025).

For practical application to a new model, the benchmark recommends selecting candidate content layers from prior phoneme-purity or ASR-utility evidence, extracting mean-pooled content embeddings and a strong speaker embedding such as ECAPA-TDNN, training a simple MLP to near-perfect speaker identification, and using Gradient SHAP with a representative baseline distribution of approximately 256 samples, batch size approximately 256, and local smoothing approximately 0.1. Interpreting the resulting scores is straightforward within the benchmark’s framework: lower $x = [c; s] \in \mathbb{R}^{d_c + d_s}.$1 indicates better disentanglement; near 0% indicates minimal timbre leakage; and values above roughly 15–20% suggest substantial leakage. In this sense, InterpTRQE-SptME functions as a model-agnostic auditing and selection benchmark for content-centric speech processing, with an additional role in identifying privacy leakage risk from residual timbre (Zhu et al., 19 Jul 2025).