Targeted Speaker Anonymization

• Targeted speaker anonymization is a method of selectively masking designated voices using advanced neural extraction, masking, and signal recombination techniques.
• It employs dedicated modules such as target speaker extraction and identity masking to preserve conversational context and ensure downstream utility in analytics.
• The approach is evaluated using specialized metrics like tcpWER, DER, and SI-SDR, balancing privacy protection and speech quality in complex audio scenarios.

Targeted speaker anonymization refers to a suite of advanced techniques designed to conceal the identity of one or more designated speakers in voice data, while preserving the intelligibility, utility, and context of the speech signal. Unlike generic anonymization schemes that operate indiscriminately on all speakers, targeted methods operate with explicit focus on (and often only on) selected participants, as may be required in domains such as call centers or compliant data publishing. This area encompasses developments in speaker extraction, flexible anonymization pipelines, and specialized evaluation methods suitable for complex multi-speaker conversational environments.

1. Technical Frameworks for Targeted Speaker Anonymization

The primary architecture for targeted speaker anonymization in multi-speaker recordings is the Target Speaker Anonymization (TSA) pipeline, consisting of three main stages:

1. Target Speaker Extraction (TSE):
   • A neural separation model (e.g., conformer-based or band-split RNN) takes the mixed audio and a reference speaker embedding, producing a soft mask (Mask A) that isolates the target speaker's signal even in overlapping speech.
   • The reference embedding is typically derived from a known utterance of the target speaker.
2. Speaker Anonymization Module:
   • The extracted single-speaker waveform is anonymized via an identity-masking subsystem, for instance by converting it to acoustic vector quantized bottleneck (VQ-BN) features and resynthesizing with a speech generator (e.g., HiFi-GAN).
   • The choice of anonymization backbone (e.g., VQ-based, x-vector, or adversarially generated speaker embeddings) affects the degree of privacy and quality.
3. Signal Recombination:
   • The anonymized target speaker waveform is merged with the original non-target signals, reconstructing the full multi-speaker scene while leaving the other speakers untouched.
   • This preserves conversational integrity and enables downstream applications (e.g., ASR, diarization) to operate effectively on the complete recording.

These modules can be adapted to various speaker extraction and anonymization backbones, enabling flexibility depending on the domain and privacy requirements (Tomashenko et al., 10 Oct 2025).

2. Challenges Unique to Multi-Speaker and Conversational Scenarios

Targeted speaker anonymization in complex audio differs fundamentally from single-speaker anonymization due to overlapping speech, crosstalk, and the necessity to mask only specific voices. Major technical challenges include:

• Extracting clean target speech in the presence of overlap. Imperfect TSE can leave traces of non-target speakers; interference can propagate through the anonymization and recombination steps, degrading both privacy and utility.
• Residual leakage and privacy risk. Any non-ideal separation may allow adversaries to exploit artifacts or interfere with privacy guarantees.
• Preserving utility in downstream tasks. Overlapping speech and imperfect mask estimation can lead to increased ASR word error rates or diarization errors, particularly when evaluating anonymized versus original signals.
• Maintaining conversational context. Unlike blanket anonymization, targeted schemes must avoid distorting the dialogue or prosodic patterns of non-target speakers.

The effectiveness of targeted anonymization thus relies critically on both the quality of the extraction and the degree to which anonymization modules can robustly mask identity cues only in the intended segments.

3. Evaluation Methodologies for Targeted Speaker Anonymization

Standard privacy and utility metrics for speaker anonymization are insufficient in multi-speaker scenarios. The proposed evaluation framework incorporates both privacy and utility, adapting existing metrics to permit correct analyses:

• Time-constrained Minimum-Permutation Word Error Rate (tcpWER):
  • This metric concatenates all utterances per speaker (reference or hypothesis), computes WERs across all possible speaker-permutation alignments (using a toolkit such as MeetEval), and applies temporal constraints in Levenshtein distance computation, matching words only if they are temporally adjacent.
  • TcpWER provides a robust, context-aware measure of utility that correctly accounts for overlapping speech and speaker label permutation (Tomashenko et al., 10 Oct 2025).
• Diarization Error Rate (DER):
  • Evaluates the accuracy of “who spoke when,” which is critical in multi-speaker settings.
• Target Speaker ASR WER:
  • The WER computed solely on the cleanly extracted target speaker speech (pre-anonymization or re-mixing) using ASR.
• Privacy Metric (Equal Error Rate, EER):
  • Computed by an ASV system (as in the VoicePrivacy Challenge pipeline), applied after possible attacker-side speaker extraction.
  • The attacker may first attempt to extract the target’s signal before attempting identity recovery.
• Scale-Invariant Signal-to-Distortion Ratio (SI-SDR):
  • Quantifies the fidelity of source separation:

  $$\text{SI-SDR} = 10 \log_{10} \frac{\| \alpha s \|^2}{\| \alpha s - \hat{s} \|^2 }, \qquad \alpha = \underset{\alpha}{\mathrm{argmin}} \|\hat{s} - \alpha s\|_2$$

where $s$ is the target reference, $\hat{s}$ is the extracted signal.

This framework allows for comprehensive assessment of both privacy protection strength and the utility of anonymized data for downstream tasks in realistic, conversational conditions (Tomashenko et al., 10 Oct 2025).

4. Systemic and Practical Implications

Targeted speaker anonymization is of particular importance in privacy-sensitive environments (e.g., customer support call centers, regulatory compliance contexts). Applications include:

• Selective Privacy Compliance: Protecting only the customer’s voice in conversational logs while retaining full operator (agent) speech improves compliance with data protection regulations (e.g., GDPR) without impairing analytics.
• Operational Integrity: Recombining anonymized target and untouched non-target speech allows naturalistic transcript generation and preserves analytical value (such as sentiment analysis and intent understanding).
• Generalization to Meetings and Group Dialogues: Any context in which only a subset of speakers require privacy protection (e.g., multi-party meetings) can leverage this framework.

The approach also accommodates scenarios requiring post-hoc anonymization (e.g., after recording) and is extensible to more fine-grained attribute-driven anonymization.

5. Open Issues, Limitations, and Future Research Directions

Several unresolved questions and avenues for future research persist:

• Extraction–Anonymization Error Cascading: Residual crosstalk or incomplete masking from imperfect TSE degrades both privacy (leading to possible re-identification) and utility (ASR/diarization errors).
• Enhanced Masking Strategies: Improving neural mask estimation and integrating joint training for TSE and downstream anonymization modules may yield gains in both privacy and accuracy.
• Metric Development: Greater metric sophistication is needed for cases with multiple overlapping target speakers, dynamic speaker numbers, or for capturing nuanced privacy–utility tradeoffs in conversational flows.
• Evaluation under Adaptive and Informed Attacks: Realistic attack models that assume the adversary can run their own TSE prior to ASV attacking must be included in protocol definitions.
• Generalizability and Scaling: Ensuring that frameworks perform robustly across languages, domains, and variable channel conditions is a continuing challenge.
• Signal Recombination Artifacts: Imperfect mixing, especially after time–frequency masking, can induce artifacts that affect both privacy (possible leakage) and perceived quality.

Further research is required for joint optimization of TSE and anonymization, development of better benchmarks, robust composable privacy metrics, and efficient low-latency implementations.

6. Comparative Analysis and Methodological Variants

Experimentation with two different neural TSE backbones (conformer-based and band-split RNN) demonstrates sensitivity of anonymization performance to extraction quality. Performance degrades as the overlap ratio between speakers increases, with SI-SDR and subsequent EERs and tcpWERs worsened accordingly. The anonymization backbone (e.g., VQ-BN versus other pseudo-embedding or GAN-based systems) further impacts privacy leakage and utility balance.

The TSA approach stands distinct from generic anonymization pipelines by enabling selective, per-speaker operation and by explicitly addressing compositionality of masking, extraction, and synthesis within multi-speaker contexts.

---

In summary, targeted speaker anonymization in multi-speaker recordings leverages dedicated extraction and anonymization modules, novel evaluation methodologies, and sophisticated privacy–utility assessments to address real-world privacy demands in conversational audio. This paradigm is emerging as a critical area of speech privacy research, with ongoing work aimed at minimizing leakage, maximizing downstream utility, and ensuring robustness against sophisticated adversaries (Tomashenko et al., 10 Oct 2025).