Streaming Speaker Anonymization

• Streaming speaker anonymization is a privacy technique that transforms live speech to preserve intelligibility while effectively hiding the speaker's identity.
• It encompasses diverse architectures including end-to-end causal autoencoders, neural audio codec language models, and feature-domain methods, each balancing latency, quality, and privacy.
• Performance is evaluated through metrics like WER, MOS, and Equal Error Rates, with systems operating under strict latency constraints from 20 ms to 370 ms for various real-world deployments.

Streaming speaker anonymization (SA) is the causal, low-latency transformation of an incoming speech stream into a new signal that preserves linguistic content and quality while hiding the original speaker’s identity (Quamer et al., 2024). In contrast to voice conversion, which maps a source speaker to a specific known target speaker, anonymization only requires the output to sound like some different speaker sufficiently far from the source (Quamer et al., 2024). Earlier anonymization used x-vector replacement and neural waveform models (Fang et al., 2019), later work explored neural audio codec LLMs (Panariello et al., 2023), and recent systems target explicitly streaming, low-latency operation on CPUs, GPUs, edge devices, and always-on speech dialogue systems (Quamer et al., 2024, Quamer et al., 4 Sep 2025, Kuzmin et al., 20 Jan 2026, Quamer et al., 10 Feb 2026, Kuzmin et al., 9 Mar 2026).

1. Problem definition and operating constraints

Speaker anonymization is defined by two simultaneous requirements: preserving linguistic content and concealing speaker identity (Quamer et al., 2024). The literature distinguishes it from voice conversion, de-identification, and general DSP privacy methods. Voice conversion transfers speech to a specific target speaker, whereas anonymization only requires the result to sound like some different speaker; DSP anonymization includes formant-shifting, frequency warping, pitch or rate modification, and related cascaded heuristics, but these are described as insufficient against modern ML-based ASV systems (Quamer et al., 2024).

In the streaming setting, systems must operate in real time or faster, have low end-to-end latency, be streaming or causal with limited or no look-ahead, and be resource-efficient enough for deployment scenarios such as customer service, virtual assistants, telemedicine, live legal proceedings, online tutoring, gaming, and full-duplex speech dialogue (Quamer et al., 2024, Quamer et al., 27 Mar 2026, Kuzmin et al., 9 Mar 2026). One paper defines latency as

$$\text{latency} = \text{chunk size} + \text{average processing time per chunk},$$

and treats a system as real-time when

$\text{Latency} < 2C_{\text{ms}}.$

This framing makes latency a joint function of chunk duration, buffering, look-ahead, and computation (Quamer et al., 2024).

The practical target is typically a few tens to a few hundred milliseconds. PHONOS states that target $\le 250$ ms end-to-end is a practical threshold for interactive speech (Quamer et al., 27 Mar 2026). Zero-VC sharpens the constraint further by treating zero-lookahead as “one-frame-in, one-frame-out,” with a 20 ms frame shift and a theoretical minimum algorithmic latency of 20 ms (Li et al., 18 Jun 2026). This suggests that streaming SA is defined as much by systems constraints as by privacy objectives.

2. Architectural families

One prominent family is the end-to-end waveform autoencoder. The 2024 streaming model uses a lightweight content encoder that extracts HuBERT-like information, a pretrained speaker encoder that extracts speaker identity, a variance encoder that injects pitch and energy information, and a causal HiFiGAN-style decoder that reconstructs anonymized waveform chunks; at inference, a pseudo-speaker generator produces an anonymized embedding constrained to be sufficiently far from the source (Quamer et al., 2024).

A second family is neural audio codec language-model SA. Stream-Voice-Anon adapts modern causal LM-based NAC architectures specifically for streaming SA by using a VQ content encoder, an acoustic encoder with 8 codebooks, Slow AR and Fast AR transformers, prompt conditioning, and anonymized speaker embedding mixing

$$g_\text{anon} = \alpha \frac{1}{K}\sum_{i=1}^{K} g_i + (1 - \alpha)\,g_s,$$

with $\alpha = 0.9$ in the reported system (Kuzmin et al., 20 Jan 2026). Earlier NAC-based anonymization had already argued that quantized semantic and acoustic tokens act as a strong information bottleneck for speaker-related information (Panariello et al., 2023).

A third family addresses the mismatch between time-varying content and static speaker embeddings. TVTSyn introduces a content-synchronous, time-varying timbre representation using a Global Timbre Memory, frame-level content attention, a gate, and spherical interpolation, combined with a factorized vector-quantized bottleneck and a causal SEANet-style decoder (Quamer et al., 10 Feb 2026). Its formulation turns speaker conditioning from a global vector into a frame-synchronous control signal.

A fourth family operates directly on intermediate representations rather than waveforms. One approach places flexible gradient-reversal speaker adversarial layers at targeted layers within a Conformer ASR model so that on-device encoder blocks produce speaker-anonymized acoustic embeddings, which are then transmitted to the cloud while preserving ASR performance (Jalal et al., 2023). In always-on dialogue models, two streaming anonymization setups have been proposed: a waveform-level front-end, Anon-W2W, and a feature-domain replacement, Anon-W2F, both built on Stream-Voice-Anon (Kuzmin et al., 9 Mar 2026).

Paradigm	Core components	Streaming characteristic
End-to-end causal autoencoder (Quamer et al., 2024)	HuBERT-like content encoder, X-vector + ECAPA, variance adapter, causal decoder	Full and lite CPU variants
NAC-LM anonymizer (Kuzmin et al., 20 Jan 2026)	VQ content encoder, Slow AR / Fast AR, prompt conditioning, Firefly-GAN vocoder	Dynamic and fixed delay
TVTSyn (Quamer et al., 10 Feb 2026)	Factorized VQ, Global Timbre Memory, gate, Slerp, causal decoder	Streamable end-to-end
On-device feature anonymization (Jalal et al., 2023)	Conformer split, GRL, speaker adversarial training	Embeddings streamed to cloud

3. Disentanglement, pseudo-speakers, and control signals

Most streaming SA systems are organized around representation disentanglement. In the end-to-end waveform model, content is supervised to match HuBERT pseudo-labels, speaker identity is provided by fixed pretrained X-vector and ECAPA-TDNN encoders, and prosodic or variance information is modeled through pitch and energy predictors. A central design choice is that gradients from the decoder are blocked from flowing back to the content encoder so that the encoder is not driven to encode speaker-specific details needed by the decoder; instead, speaker information must be injected via explicit speaker embeddings (Quamer et al., 2024).

Quantization is repeatedly used as a privacy bottleneck. NAC-based anonymization emphasizes that semantic tokens are discrete and strongly dominated by content, while acoustic tokens capture speaker identity, prosody, and recording conditions; by discarding the original acoustic tokens and regenerating them from semantic tokens plus a pseudo-speaker prompt, the original speaker’s acoustic identity is removed (Panariello et al., 2023). TVTSyn applies a factorized VQ bottleneck that compresses 512-dimensional content vectors to 8 dimensions, quantizes them with a 4096-entry codebook, and reports that within-speaker clusters loosen substantially after bottlenecking and VQ, indicating reduced leakage (Quamer et al., 10 Feb 2026). DarkStream similarly uses a k-means bottleneck over encoder outputs as the principal privacy lever under streaming constraints (Quamer et al., 4 Sep 2025).

Pseudo-speaker construction is itself a major design axis. Earlier work formed a pseudo x-vector by averaging multiple external speaker x-vectors from a pool, either randomly, by nearest neighbors, or by a controlled similarity range, and used the result as a session-stable pseudo identity (Fang et al., 2019). A later WGAN-QC approach generated artificial 704-dimensional speaker embeddings from noise and filtered them with a cosine-distance constraint $> 0.3$, explicitly targeting non-existent voices that remain in-distribution for TTS synthesis (Meyer et al., 2022). Recent streaming systems inherit this logic: Stream-Voice-Anon mixes prompt speaker embeddings and a Gaussian-sampled embedding, while the 2024 end-to-end streaming model uses a GAN-based pseudo-speaker generator with cosine distance $> 0.3$ from the original (Kuzmin et al., 20 Jan 2026, Quamer et al., 2024).

Control signals beyond timbre are increasingly treated as privacy-relevant. PHONOS is explicit that timbre modification alone leaves non-native accents intact, and it therefore inserts a streaming accent translator in token space between content encoder and decoder (Quamer et al., 27 Mar 2026). StreamVoiceAnon+ argues that content tokens discard emotional information and that NAC LLMs trained for audio continuation default to dominant acoustic patterns; it therefore adds frame-level emotion distillation on acoustic token hidden states during finetuning, while leaving inference unchanged (Kuzmin et al., 6 Mar 2026). This suggests that “speaker identity” in streaming SA is operationally distributed across timbre, segmentals, prosody, emotion, and accent, even when systems only expose one explicit speaker embedding.

4. Streaming mechanisms and latency regimes

Causality is implemented through a combination of chunking, bounded context, causal convolutions, and limited-context transformers. The 2024 end-to-end streaming model processes raw waveform in fixed-length chunks from 20 to 140 ms, uses causal 1D convolutions throughout, and reports on-CPU latency of 229.75 ms for the base model at 120 ms chunks and 66.27 ms for the lite model at 40 ms chunks; on GPU, the lite model is real-time even at 20 ms chunks with 38.36 ms latency (Quamer et al., 2024).

PHONOS distributes its future context budget across modules: 40 ms encoder lookahead, 40 ms accent translator lookahead, and 40 ms decoder lookahead, for combined $\le 120$ ms future context. With 80 ms chunks on a single NVIDIA RTX 3090 GPU, it reports RTF 0.6 and end-to-end latency under 241 ms; on CPU, 160 ms chunks plus 120 ms lookahead yield about 370 ms total latency at real time (Quamer et al., 27 Mar 2026).

Stream-Voice-Anon and StreamVoiceAnon+ are both reported at 180 ms latency. Stream-Voice-Anon compares dynamic and fixed delay configurations and reports that privacy is almost flat across the tested latency range, while WER improves up to about 180 ms then saturates (Kuzmin et al., 20 Jan 2026). StreamVoiceAnon+ keeps the same streaming NAC-LM architecture and latency while adding only training-time modifications; its claim is zero inference latency overhead (Kuzmin et al., 6 Mar 2026).

Other systems explore alternative latency-quality trade-offs. TVTSyn uses a causal CNN encoder, a slight future peek of 4 tokens in the encoder, a fully causal decoder, and reports end-to-end synthesis latency under 80 ms on GPU and about 100–200 ms on CPU (Quamer et al., 10 Feb 2026). DarkStream combines a causal waveform encoder, a 140 ms lookahead buffer, causal transformer contextual layers with a 2 s look-back, and direct waveform decoding, reporting 203.0 ms end-to-end latency on RTX3090 with 60 ms chunking at its default operating point (Quamer et al., 4 Sep 2025). Zero-VC, although framed as zero-shot voice conversion, repurposes SA as a perturbation mechanism and reports 20 ms algorithmic latency with RTF 0.063 on CPU in a strictly causal, zero-lookahead network (Li et al., 18 Jun 2026).

System	Reported streaming regime	Reported latency
End-to-end streaming model (Quamer et al., 2024)	Causal chunks on CPU	~230 ms base, ~66 ms lite
PHONOS (Quamer et al., 27 Mar 2026)	80 ms chunks, $\le 120$ ms future context	under 241 ms on single GPU
Stream-Voice-Anon / StreamVoiceAnon+ (Kuzmin et al., 20 Jan 2026, Kuzmin et al., 6 Mar 2026)	Causal NAC-LM streaming SA	180 ms
TVTSyn (Quamer et al., 10 Feb 2026)	Streamable end-to-end	<80 ms GPU
DarkStream (Quamer et al., 4 Sep 2025)	60 ms chunks, 140 ms lookahead	203.0 ms
Zero-VC (Li et al., 18 Jun 2026)	Strictly causal, zero-lookahead	20 ms algorithmic

5. Evaluation protocols and empirical performance

Evaluation is centered on privacy, intelligibility, quality, and increasingly broader paralinguistic dimensions. Privacy is usually measured with ASV-based Equal Error Rate; higher EER indicates harder re-identification. Intelligibility is typically measured with WER. Quality appears as DNSMOS, NISQA-MOS, or subjective MOS. Additional measures include Linkability in always-on dialogue models, Unweighted Average Recall for emotion preservation or suppression, speaker verifiability in AB listening tests, and cosine speaker similarity scores such as SSS or SpkSim (Quamer et al., 2024, Kuzmin et al., 6 Mar 2026, Kuzmin et al., 9 Mar 2026).

The 2024 end-to-end streaming model reports, for its base and lite variants, DNSMOS values close to strong baselines and source speech, WER of 5.12% for the base model and 6.47% for the lite model, MOS of $3.57 \pm 0.59$ for the base model and $\text{Latency} < 2C_{\text{ms}}.$0 for the lite model, and EER values in the low-to-mid 40% range under VoicePrivacy-style O-A and A-A conditions. In subjective verification, “same speaker, anonymized” yields 14.5% verifiability with confidence 2.37, comparable to the “different speaker” condition (Quamer et al., 2024).

PHONOS evaluates accent as a privacy signal. For Indian L2 speakers, it reports $\text{Latency} < 2C_{\text{ms}}.$1, $\text{Latency} < 2C_{\text{ms}}.$2, and $\text{Latency} < 2C_{\text{ms}}.$3, summarized as an 81% reduction in non-native accent confidence. In speaker-embedding space, SpkSim moves from approximately $\text{Latency} < 2C_{\text{ms}}.$4 for TVTSyn-INE reconstruction to $\text{Latency} < 2C_{\text{ms}}.$5 for PHONOS output, indicating reduced linkability even when the same speaker embedding is used (Quamer et al., 27 Mar 2026).

StreamVoiceAnon+ addresses the emotion-preservation deficit of streaming NAC-LM anonymization. On VoicePrivacy 2024 it reports UAR 49.22%, WER 5.77%, EER-L 48.98%, and EER-S 18.30%, improving over a neutral-prompt baseline at 39.72% UAR and over an emotion-prompt variant at 44.59% UAR while keeping the same 180 ms latency (Kuzmin et al., 6 Mar 2026). Stream-Voice-Anon, in turn, reports WER 4.71% and UAR 39.94% for cross-ds-4rnd, and UAR 44.59% for cremad-emo-4rnd, while maintaining lazy-informed EER comparable to DarkStream but showing degradation under semi-informed attackers (Kuzmin et al., 20 Jan 2026).

TVTSyn reports WER 5.35%, EER 47.55% for the lazy-informed attacker, EER 14.57% for the semi-informed attacker, and UAR 37.32%, alongside MOS $\text{Latency} < 2C_{\text{ms}}.$6. DarkStream reports near-chance lazy-informed EER with k-means quantization, specifically 46.75% for Wave+CL at 140 ms lookahead, with WER 9.52% and MOS $\text{Latency} < 2C_{\text{ms}}.$7 in the same quantized condition (Quamer et al., 10 Feb 2026, Quamer et al., 4 Sep 2025). In always-on dialogue, Anon-W2F raises EER from 11.2% to 41.0% relative to the discrete encoder baseline while keeping FRL under 0.8 s, showing that streaming SA has become relevant beyond standalone anonymization systems (Kuzmin et al., 9 Mar 2026).

6. Deployment, limitations, and research directions

A common misconception is that timbre conversion alone is sufficient for anonymization. PHONOS argues that non-native accents and regional accents narrow the anonymity set, affect linkability and profiling, and can therefore remain a strong privacy signal even when timbre has been changed (Quamer et al., 27 Mar 2026). A related misconception is that preserving emotion is always desirable: StreamVoiceAnon+ treats emotion preservation as a utility goal, whereas TVTSyn interprets low emotion UAR as intentional reduction of a paralinguistic identity cue (Kuzmin et al., 6 Mar 2026, Quamer et al., 10 Feb 2026). This suggests that “utility” in streaming SA is task-dependent rather than uniform.

Another recurring limitation is the absence of formal privacy guarantees. Most systems report strong empirical privacy against ASV and human listeners, but explicitly note that they do not provide strict formal guarantees such as differential privacy (Quamer et al., 2024). Differentially Private Speaker Anonymization is the exception: it introduces differentially private BN and pitch extractors based on a TDNN acoustic model and a convolutional autoencoder with Laplace noise, and states that the resulting utterances have a provable upper bound on the speaker information they contain (Shamsabadi et al., 2022). The practical trade-off is that user-level or session-level privacy accounting under composition remains difficult for long streams.

Deployment work increasingly targets edge and always-on settings. The lite version of the 2024 end-to-end model is about 10 MB and 0.1× the number of parameters of the base model, making smartphone deployment plausible (Quamer et al., 2024). Flexible-location GRL anonymization proposes running the initial layers of a Conformer ASR model on-device and transmitting only anonymized acoustic embeddings to the cloud, reporting a 33% reduction in speaker recognition relative accuracy and a 6.2% relative WER reduction (Jalal et al., 2023). In full-duplex dialogue, Anon-W2W and Anon-W2F preserve sub-second response latency while mitigating hidden-state speaker leakage (Kuzmin et al., 9 Mar 2026).

Open directions are consistent across the literature: accent conversion, multilingual extension, stronger attacker models, prosodic modification, more formal privacy guarantees, further latency reduction, robustness in noisy real-world conditions, and integrated on-device adaptation or federated learning (Quamer et al., 2024, Quamer et al., 27 Mar 2026, Kuzmin et al., 20 Jan 2026). Zero-VC additionally suggests that SA can be repurposed as a representation-shaping front end for strictly causal synthesis, not only as a privacy layer (Li et al., 18 Jun 2026). More broadly, the current evidence indicates that streaming SA is no longer a narrow voice conversion variant; it is becoming a systems-level discipline spanning waveform anonymization, discrete token generation, feature-domain privacy, accent and emotion control, and privacy auditing for speech-native large models.