InterpTF-SptME: Timbre Filtering for Speech Models

• InterpTF-SptME is an interpretability-based timbre filtering approach that decouples speaker timbre from content in speech model representations using SHAP-derived attributions.
• The method employs two operators, SHAP Noise and SHAP Cropping, to suppress timbre-bearing dimensions in intermediate embeddings, balancing timbre reduction with minimal ASR performance loss.
• Benchmark results demonstrate its effectiveness in lowering timbre residuals, offering practical improvements for ASR, TTS, and voice privacy without retraining the underlying model.

InterpTF-SptME is an interpretability-based timbre filtering method for speech pretrained model representations that operates on intermediate encodings rather than on waveform synthesis or end-to-end retraining. Introduced in conjunction with the InterpTRQE-SptME benchmark, it addresses the observation that ostensibly content-oriented embeddings from speech self-supervised models still retain residual speaker timbre information. The method uses SHAP-derived attributions to identify timbre-bearing dimensions or directions in a content embedding and then suppresses them through either SHAP Noise or SHAP Cropping, with the stated goal of reducing speaker leakage while preserving linguistic content for downstream content-related speech processing (Zhu et al., 19 Jul 2025).

1. Conceptual framing and target of the method

The method is motivated by a decomposition of speech representations into at least three components: content information, timbre information, and paralinguistic information. In the paper’s formulation, content refers to linguistic or phonetic semantics, timbre refers to speaker identity or voiceprint, and paralinguistic information includes style, prosody, and emotion. For tasks such as ASR, TTS, voice conversion, and speech translation, the desired intermediate representation is one that preserves content while suppressing speaker identity (Zhu et al., 19 Jul 2025).

For a speech sample $w_j$ and a speech pretrained model $M_i$, the content embedding is defined as

$\mathbf{c}_{ij} = f_i(w_j) \in \mathbb{R}^{d_c},$

and a separate pretrained speaker model produces a speaker embedding

$\mathbf{s}_j \in \mathbb{R}^{d_s},$

with $d_s = 192$. These are concatenated into

$$\mathbf{x}_{ij} = [\mathbf{c}_{ij}; \mathbf{s}_j] \in \mathbb{R}^{d_c + d_s}.$$

This construction makes the central diagnostic explicit. If the content embedding were fully speaker-independent, speaker classification should depend almost entirely on $\mathbf{s}_j$ and minimally on $\mathbf{c}_{ij}$. The paper treats any measurable contribution of $\mathbf{c}_{ij}$ to speaker prediction as timbre residual. InterpTF-SptME is the post-hoc mechanism designed to reduce that residual after it has been quantified.

2. Dependence on the InterpTRQE-SptME benchmark

InterpTF-SptME is not presented as an isolated filter. It is the second stage of a two-stage framework in which InterpTRQE-SptME first measures timbre residual in speech pretrained model encodings, and InterpTF-SptME then suppresses it (Zhu et al., 19 Jul 2025).

The benchmark proceeds by extracting content embeddings from a speech pretrained model, extracting speaker embeddings from a pretrained speaker verification model, concatenating them, training a speaker classifier on the fused features, and then using SHAP-based interpretation to measure how much of the speaker prediction comes from the content embedding. The classifier is intentionally made highly accurate, and the interpretability analysis is applied to that classifier to estimate the contribution of content dimensions versus speaker dimensions.

The timbre residual quantification score for model $M_i$ is defined as

$M_i$0

where $M_i$1 is the SHAP contribution of content dimension $M_i$2 for sample $M_i$3, $M_i$4 is the SHAP contribution of speaker dimension $M_i$5 for sample $M_i$6, and $M_i$7 is the number of samples. A smaller ratio indicates less timbre leakage in content representations.

This benchmark context is essential for interpreting InterpTF-SptME. The filtering method assumes that timbre-bearing structure has already been localized by attribution analysis, rather than learned through a specialized disentanglement objective.

3. Attribution basis: Gradient SHAP as a probe of speaker leakage

The paper uses Gradient SHAP Explainer to interpret the speaker classifier. The Shapley-value formulation is given as

$M_i$8

where $M_i$9 is the attribution for feature $\mathbf{c}_{ij} = f_i(w_j) \in \mathbb{R}^{d_c},$0, $\mathbf{c}_{ij} = f_i(w_j) \in \mathbb{R}^{d_c},$1 is a subset of the remaining features, $\mathbf{c}_{ij} = f_i(w_j) \in \mathbb{R}^{d_c},$2 is the number of features, and $\mathbf{c}_{ij} = f_i(w_j) \in \mathbb{R}^{d_c},$3 is the model’s value function (Zhu et al., 19 Jul 2025).

The paper emphasizes that SHAP is used not merely for visualization but as a quantitative probe of how speaker identity is encoded across content dimensions. In practice, the explanation is stabilized by overfitting the speaker classifier to 100% accuracy, using the whole dataset as explanation data, using a fixed baseline set of 256 samples, using batch size 256, and applying local smoothing $\mathbf{c}_{ij} = f_i(w_j) \in \mathbb{R}^{d_c},$4.

This makes InterpTF-SptME fundamentally interpretability-driven. The filter is guided by attribution mass rather than by an auxiliary adversarial objective, latent prior, or reconstruction-disentanglement loss. A plausible implication is that the method’s behavior depends strongly on the fidelity of the explainer and the speaker classifier used to generate the explanations.

4. Filtering operators: SHAP Noise and SHAP Cropping

InterpTF-SptME applies the SHAP explanations directly to the content embedding and does not retrain the pretrained speech model. The paper presents two filtering operators: SHAP Noise and SHAP Cropping (Zhu et al., 19 Jul 2025).

For SHAP Noise, the SHAP values on the content side are standardized as

$\mathbf{c}_{ij} = f_i(w_j) \in \mathbb{R}^{d_c},$5

The standardized noise is then scaled and shifted,

$\mathbf{c}_{ij} = f_i(w_j) \in \mathbb{R}^{d_c},$6

and added to the original content embedding:

$\mathbf{c}_{ij} = f_i(w_j) \in \mathbb{R}^{d_c},$7

The paper’s interpretation is that the SHAP signal identifies timbre-bearing directions in representation space, and that a suitably loaded perturbation can disrupt those speaker cues while preserving semantic structure.

For SHAP Cropping, dimensions with high SHAP magnitude are thresholded and masked. The threshold is set by percentile:

$\mathbf{c}_{ij} = f_i(w_j) \in \mathbb{R}^{d_c},$8

The cropped attribution is then

$\mathbf{c}_{ij} = f_i(w_j) \in \mathbb{R}^{d_c},$9

which is normalized into a cropping weight

$\mathbf{s}_j \in \mathbb{R}^{d_s},$0

and applied as

$\mathbf{s}_j \in \mathbb{R}^{d_s},$1

The distinction between the two operators is explicit in the paper. SHAP Cropping is a masking-based suppression method, whereas SHAP Noise perturbs the embedding using SHAP-informed noise. The method is described as model-agnostic and layer-agnostic, and the filtering stage is described as label-independent in the sense that it does not require ground-truth labels for the speech data being filtered.

5. Experimental setting and quantitative results

The experiments use the VCTK corpus. For efficiency, the paper uses the first 20 speakers, p225–p246 excluding p235 and p242, for a total of 7,758 audio samples. Preprocessing uses a 16 kHz sampling rate, 16-bit mono WAV, and energy normalization to $\mathbf{s}_j \in \mathbb{R}^{d_s},$2 dB using SOX. Speaker embeddings are extracted with SpeechBrain’s spkrec-ecapa-voxceleb system, based on an ECAPA-TDNN model trained on VoxCeleb (Zhu et al., 19 Jul 2025).

Seven speech pretrained model configurations are benchmarked: HuBERT BASE, HuBERT LARGE, HuBERT-CH, DPHuBERT, ContentVec, WavLM Base+, and Whisper-ppg. Content embeddings are taken from the layer identified as most useful for content-related tasks: layer 9 for HuBERT BASE and HuBERT-CH, layer 21 for HuBERT LARGE, layer 12 for DPHuBERT, ContentVec, and WavLM Base+, and encoder output for Whisper-ppg.

Reported mean timbre residual scores are 5.20% for ContentVec, 7.46% for Whisper-ppg, 7.73% for DPHuBERT, 9.02% for WavLM Base+, 13.72% for HuBERT BASE, 13.93% for HuBERT-CH, and 18.65% for HuBERT LARGE. The paper interprets these results as showing that ContentVec has the best disentanglement and that HuBERT LARGE has the highest timbre residual among the tested English models.

Filtering experiments focus on HuBERT LARGE layer 21, whose original representation has 18.05% timbre residual and an ASR CTC loss baseline of 1.2612. For SHAP Noise, the paper reports the following tradeoffs:

• $\mathbf{s}_j \in \mathbb{R}^{d_s},$3: timbre residual drops to ~0%, with CTC loss 1.3252
• $\mathbf{s}_j \in \mathbb{R}^{d_s},$4: timbre residual 2.21%, with CTC loss 1.2730
• $\mathbf{s}_j \in \mathbb{R}^{d_s},$5: timbre residual 7.23%, with CTC loss 1.2626

The strongest SHAP Noise setting is said to reduce timbre leakage from 18.05% to nearly 0% at the cost of a 5.08% increase in speech recognition loss.

For SHAP Cropping, the reported settings are:

• $\mathbf{s}_j \in \mathbb{R}^{d_s},$6: timbre residual 4.65%, with CTC loss 1.2969
• $\mathbf{s}_j \in \mathbb{R}^{d_s},$7: timbre residual 10.07%, with CTC loss 1.2779

The paper states that cropping is effective but less powerful than SHAP Noise, with a more controlled ASR degradation and a reported loss increase of up to 2.83%.

A further result concerns layer sensitivity within HuBERT LARGE. The paper reports 10.58% timbre residual for layer 18 and 18.05% for layer 21, and concludes that, from a disentanglement standpoint, layer 18 is better suited than layer 21 for content-related processing.

6. Position relative to prior disentanglement approaches and practical significance

The paper contrasts InterpTF-SptME with several earlier families of speaker disentanglement methods (Zhu et al., 19 Jul 2025). Compared with GAN- or VAE-based voice conversion and anonymization approaches, it is described as post-hoc, not as a redesign of the pretrained speech model or a task-specific end-to-end disentanglement architecture. Compared with ContentVec-style learned disentanglement, it does not alter the training recipe of the base model; instead, it analyzes and transforms an existing representation. Compared with indirect evaluation metrics such as ASR WER, speaker identification accuracy, phone purity, or cluster purity, the associated benchmark is presented as a direct estimate of timbre leakage through attribution mass on content dimensions relative to speaker dimensions.

Two practical implications are emphasized. First, the filtered embeddings are proposed as better content-oriented representations for ASR, TTS, speech translation, voice conversion, and semantic speech processing. Second, the method is proposed as a way to mitigate timbre privacy leakage, since raw pretrained embeddings may encode voiceprint information.

The paper’s strongest empirical claim is that explanation-guided filtering can suppress residual speaker information sharply without retraining the underlying speech model. At the same time, the reported CTC-loss increases show that disentanglement is not cost-free. This suggests that InterpTF-SptME is best understood not as a universal replacement for learned disentanglement, but as an interpretable control mechanism for navigating the tradeoff between content preservation and speaker suppression in pretrained speech encodings.