Voice Authentication Systems

• Voice Authentication Systems (VAS) are biometric pipelines that verify or identify speakers by comparing speech to stored voiceprints using advanced acoustic feature extraction.
• Contemporary VAS employ deep neural networks such as CNNs, Transformers, and end-to-end models to generate robust speaker embeddings and counter threats like replay, adversarial, and poisoning attacks.
• Practical applications span mobile devices, call centers, and smart-home systems, emphasizing privacy, continuous liveness detection, and multi-factor authentication.

Voice Authentication Systems (VAS) verify or identify speakers by comparing incoming speech against stored “voiceprints.” Over more than half a century, they have evolved from simple pattern-matching of handcrafted acoustic cues to end-to-end deep learning pipelines that produce rich speaker embeddings. Early systems relied on handcrafted acoustic features such as MFCC and GFCC with Gaussian Mixture Models and Universal Background Models; later systems adopted i-vectors; and contemporary systems use DNNs, CNNs, and Transformer-based models to learn embeddings such as x-vectors, d-vectors, and ECAPA-TDNN. As VAS became more accurate and ubiquitous in mobile phones, call centers, financial services, law enforcement, and smart-home devices, the attack surface expanded from replay and impersonation to data poisoning, adversarial perturbation, deepfake synthesis, and attacks that jointly target speaker verification and anti-spoofing countermeasures (Kamel et al., 22 Aug 2025).

1. Core concepts and reference pipelines

A modern VAS commonly follows a staged pipeline: user audio is pre-processed, transformed into acoustic features or raw-waveform representations, encoded into a fixed-dimensional speaker embedding, scored against enrollment data, and thresholded for accept/reject. The survey literature describes this transition from MFCC/GFCC with GMM-UBM back ends to deep speaker embeddings compared by cosine or learned distance metrics (Kamel et al., 22 Aug 2025). A representative black-box formulation for commercial assistants uses MFCC extraction, a neural embedding front end, and cosine or PLDA scoring, with the decision controlled by a threshold $\theta$ (Bilika et al., 2023).

In embedding-based systems, enrollment often stores multiple utterances per user. One formulation averages embeddings to form a template,

$\mu = \frac{1}{N}\sum_k f_\theta(x_k),$

and accepts when the cosine similarity between a test embedding and $\mu$ exceeds a threshold $\tau$ (Li et al., 2022). The same paper describes a 512-dimensional embedding space produced by a ResCNN-style front end. Other systems use explicit log-likelihood ratio scoring. In Active Voice Authentication, a 1 s sliding window is scored every 10 ms using

$\Gamma(X)=\log p(X|\lambda_0)-\log p(X|\lambda_1),$

with VAD suppressing silent anchor windows (Meng et al., 2020).

The front end varies by application. Some systems operate on 39-dimensional MFCCs from 8 kHz audio with 25 ms frames and 10 ms shifts (Meng et al., 2020). Others compute GFCCs over detected plosive “pop” bursts and classify a 3-dimensional feature vector with an RBF-kernel SVM (Jamdar et al., 13 Feb 2025). Text-dependent mobile protocols can train a separate GMM per passphrase and use encrypted challenge-model pairs, while text-independent systems build per-phoneme or utterance-level templates (Johnson et al., 2012, Zhang et al., 2021). This diversity suggests that “VAS” is better understood as a family of biometric pipelines than as a single architecture.

2. Operational paradigms and deployment models

A central design axis is the text-dependent versus text-independent distinction. Fixed pass-phrases improve stability but limit flexibility, whereas text-independent systems remove prompt constraints at the cost of harder modeling under lexical variation (Johnson et al., 2012). VoiceLive explicitly supports both modes on smartphones: in text-dependent mode, a fixed passphrase is aligned into phonemes; in text-independent mode, enrollment stores per-phoneme TDoA templates for all 44 English phonemes and builds the test-utterance template on the fly (Zhang et al., 2021).

Challenge-response protocols extend text-dependence into template protection. Vaulted Voice Verification adapts vaulted verification to voice by prompting the user with a question set, training a GMM for each phrase, encrypting real and chaff models, and verifying a fresh probe against an $n$-bit random challenge. The server checks whether the returned bitstring matches the challenge within a Hamming-distance threshold, while the biometric models remain encrypted under a user-derived key and the server’s public key (Johnson et al., 2012). This architecture addresses privacy and revocability more directly than conventional static voiceprint storage.

A separate operational mode is continuous or active authentication. Active Voice Authentication performs window-based speaker verification on unconventionally short test signals, reporting a new decision every 10 ms. On its 25-speaker dataset it achieved average windowed-based equal error rates of 3–4% with 1 second of voice data per decision, and on NIST SRE 2001 it provided a 3.88% absolute gain over i-vector when the duration of test segment was 1 second (Meng et al., 2020). VAuth similarly targets continuous authentication for voice assistants, but binds each spoken command to body-surface vibration captured by a wearable, rather than relying on audio alone (Feng et al., 2017).

Recent deployment models also shift verification and privacy controls onto the device. One on-device design extracts MFCCs, deep-speaker embeddings, and spoof/liveness features locally, fuses identity, spoofing, and liveness scores into a single “VoiceID” confidence, derives a FIDO2-style public-key credential from embeddings, and applies a user-configurable privacy filter before any cloud transmission. That system reports 98.63% authentication accuracy under 5-fold cross-validation and end-to-end latency of approximately 30–80 ms, with runtime memory under 10 MB (Aloufi et al., 2022). This suggests that VAS deployment is increasingly shaped by trust boundaries, not only by speaker-discriminative performance.

3. Threat taxonomy and empirical attack surface

The current threat landscape is conventionally divided into four principal classes: data poisoning, adversarial attacks, deepfake/spoofing attacks, and adversarial spoofing attacks (Kamel et al., 22 Aug 2025). Data poisoning manipulates training or enrollment samples; adversarial attacks add norm-bounded perturbations at inference time; deepfake/spoofing attacks use TTS, VC, replay, or partial-fake generation to impersonate a target; and adversarial spoofing attacks jointly optimize against the speaker model and the anti-spoofing countermeasure.

Replay and impersonation remain foundational threats, but recent work emphasizes deepfake audio. In an experimental study against commercial assistants, more than 30% of deepfake attacks were successful, and there was at least one successful attack for more than half of the participants. The same study reported statistically significant variation among vendors and, in one case, gender bias (Bilika et al., 2023). These findings indicate that proprietary voice-match subsystems can remain vulnerable even when they are already deployed for “sensitive” commands.

Attackers also exploit auxiliary defenses rather than only the speaker model. SyntheticPop targets a VA + VoicePop system whose enrollment pipeline detects low-frequency plosive bursts, computes STFT and GFCC features, and classifies with an RBF-kernel SVM. By superimposing a synthetic low-frequency sinusoid with $f\approx 90$ Hz and $A\approx 0.5$ on 20% of spoof samples, SyntheticPop achieved an attack success rate of over 95%; VA + VoicePop accuracy fell from 69% under normal conditions to 14% under the proposed attack (Jamdar et al., 13 Feb 2025).

Black-box adversarial attacks can also bypass cascaded VAS + CM systems without modifying the main voiced content. The Spectral Masking and Interpolation Attack perturbs low-energy, perceptually silent spectral regions via masking and interpolation. Under the reported evaluation, SMIA achieved at least 82% attack success rate against combined VAS/CM systems, at least 97.5% against standalone speaker verification systems, and 100% against countermeasures, while requiring an average 28 s per attack compared with 185 s for the reported baseline SiFDetectCracker (Kamel et al., 9 Sep 2025). The survey’s distinction between “static” and “dynamic, context-aware” countermeasures is directly relevant here, because static detectors are precisely what adaptive perturbation schemes exploit (Kamel et al., 22 Aug 2025).

4. Liveness detection and multi-factor hardening

A major response to spoofing is to add cues that do not exist in conventional replayed or synthesized audio. These cues include articulatory motion, stereo geometry, body-surface vibration, cross-domain speech similarity, and inertial measurements of lower-face motion.

System	Primary cue	Reported result
VoiceGesture (Zhang et al., 2021)	Doppler shifts from articulatory gestures using built-in speaker and microphone	over 99% and around 98% detection accuracy for text-dependent and text-independent liveness detection
VoiceLive (Zhang et al., 2021)	stereo TDoA dynamics with gyroscope and ultrasound distance/angle correction	over 99% detection accuracy at around 1% EER on the text-dependent system and around 99% accuracy and 2% EER on the text-independent one
VAuth (Feng et al., 2017)	matching body-surface vibrations from a wearable with microphone speech	almost perfect matching accuracy with less than 0.1% false positive rate
WearID (Wang et al., 2020)	cross-domain speech similarity between wearable accelerometer and VA microphone	TPR = 99.8%, FPR = 0% in normal usage
Mouth-motion inertial sensing (Ineza et al., 16 Oct 2025)	three accelerometer modules around the mouth capturing lower-face motion	median EER of 0.01 or lower across all scenarios

These systems differ sharply in sensing assumptions. VoiceGesture re-uses commodity speakers and microphones as a Doppler radar by transmitting a roughly 20 kHz probe tone and extracting gesture-induced Doppler contours from phoneme-aligned windows (Zhang et al., 2021). VoiceLive leverages stereo recording, gyroscope readings, and an 18–23 kHz chirp to estimate distance and angle, then tracks per-phoneme TDoA dynamics that “doesn’t exist under replay attacks” (Zhang et al., 2021). VAuth and WearID instead bind speech to the body through accelerometer measurements, rejecting replay, impersonation, hidden command, and ultrasound attacks when the audio-domain signal lacks the corresponding vibration-domain structure (Feng et al., 2017, Wang et al., 2020).

Anti-spoofing can also be implemented as a raw-audio front-end guard rather than as an auxiliary sensor stack. PSA-Net is a lightweight, end-to-end, raw-waveform anti-spoofing network for IoT-enabled smart devices that uses a split-transform-aggregate design with SE-PSA blocks. It reported 3.04% EER / 0.087 t-DCF on ASVspoof19 LA and 1.26% EER / 0.038 t-DCF on ASVspoof19 PA, with a preferred 4×64 model of about 25 MB and roughly 3.25 s inference on Jetson Nano for 4 s audio (Khan et al., 2024). This places liveness and spoof filtering directly into the VAS ingress path.

5. Poisoning, backdoors, and robust training

Training-time compromise is now a central VAS problem. Targeted Data Poisoning Attacks (TDPA) replace a subset of genuine enrollment utterances with attacker-supplied audio so as to shift the decision boundary of the speaker-verification model, while Backdoor-Triggered Attacks (BTA) implant covert triggers that later redirect classification (Mohammadi et al., 6 May 2025). In one threat model, the attacker’s goal is to cause the system to store attacker embeddings in place of a victim’s, with only black-box access and the ability to upload utterances under a victim’s account (Mohammadi et al., 2024).

Guardian is a registration-time defense against targeted poisoning in DNN-based VAS. It treats paired 512-dimensional embeddings from one enrollment as an interleaved 32×32 “image,” applies a CNN discriminator with bias reduction, input augmentation, and ensemble learning, and then aggregates file-level probabilities with an 11-NN decision rule. Under MiTM + poisoning, Guardian distinguished about 95% of attacked accounts from normal accounts, whereas the reported existing approaches achieved only 60% accuracy; more specifically, Guardian reached 0.944 accuracy, 0.952 recall, and 0.057 FPR at $\lambda=5\%$, and 0.939 accuracy, 0.910 recall, and 0.058 FPR at $\lambda=10\%$ (Li et al., 2022).

A second line of work hardens the recognition stack itself. One framework feeds 10 raw utterances per speaker from LibriSpeech through Deep Speaker to obtain 512-dimensional embeddings, processes interleaved victim/attacker pairs with a 12-layer CNN, and classifies the resulting 10-dimensional probability vector with an 11-NN layer. The CNN minimizes a weighted softmax cross-entropy with $\mu = \frac{1}{N}\sum_k f_\theta(x_k),$0 regularization, and the KNN stage uses SMOTE plus stratified 5-fold cross-validation. In the 5% poisoning scenario, the default CNN+KNN baseline reported Accuracy = 0.89, Recall = 0.55, and F1 = 0.68, whereas the proposed Scenario 1 reached Accuracy = 0.98, Recall = 0.92, and F1 = 0.95 (Mohammadi et al., 2024).

Unified defenses now target BTA and TDPA simultaneously. A dual-layer framework places a frequency-focused PBSM detector in front of the normal VAS pipeline to flag pitch boosting and sound masking triggers, then passes suspicious or deferred cases to a CNN whose softmax outputs Legitimate, Attacked, and Triggered. On LibriSpeech, VoxCeleb, and a merged dataset, this framework reduced post-defense ASR to 4.17%, 11.11%, and 15.22%, respectively, while maintaining TDPA Recall of 95%, 94%, and 93%, with false positives of about 1.2%, 1.8%, and 2.1% (Mohammadi et al., 6 May 2025).

The same literature also shows that specialized anti-spoofing cues can themselves be poisoned. SyntheticPop is particularly instructive because it does not solve a gradient-based optimization problem; instead, it directly engineers “pop-detector-friendly” signals, teaching the downstream SVM that the injected low-frequency bursts correlate with genuine speech (Jamdar et al., 13 Feb 2025). This suggests that poisoning in VAS is not limited to speaker embeddings; it also targets the side channels introduced by anti-spoofing design.

6. Evaluation, privacy, and open challenges

VAS and countermeasures are evaluated with a standard metric set: False Acceptance Rate (FAR), False Rejection Rate (FRR), Equal Error Rate (EER), Detection Error Tradeoff (DET) curves, Attack Success Rate (ASR), tandem Detection Cost Function (t-DCF), and perturbation SNR (Kamel et al., 22 Aug 2025). Public benchmarks span speaker-verification corpora such as YOHO, VoxCeleb1 and 2, LibriSpeech, TIMIT/NTIMIT, Common Voice, and AISHELL, and spoofing corpora such as ASVspoof, Fake or Real, the Voice Conversion Challenge, and BackdoorVoice (Kamel et al., 22 Aug 2025). The breadth of these benchmarks reflects the fact that performance claims are highly sensitive to channel, codec, language, replay setting, and attack family.

Privacy is a parallel design objective rather than an afterthought. Vaulted Voice Verification keeps biometric models encrypted under both a user-derived key and a server public key, uses nonce-based fresh challenges, and makes replayed data fail decryption or yield wrong bit indices (Johnson et al., 2012). On-device VoiceID goes further by verifying identity and liveness locally, deriving a public-key credential from speaker embeddings, and transmitting only privacy-filtered audio or a discrete surrogate. In that system, paralinguistic leakage on CPC-kmeans data fell to near-random accuracy for identity, gender, and accent prediction, while cloud ASR word error rate degraded by only about 6 percentage points relative to raw audio (Aloufi et al., 2022).

Current open challenges are explicitly catalogued in the recent survey literature: realism in evaluation, transferability across architectures and languages, defense efficiency on edge devices, standardized threat models, joint spoofing plus adversarial attacks, federated learning and privacy, and explainability and trust (Kamel et al., 22 Aug 2025). A recent edge-oriented proposal responds by combining a 6-dimensional physics-guided feature vector with frozen WavLM-Large embeddings, orthogonal feature fusion, a hybrid ViT/GNN/LightGBM detector, and Monte-Carlo-dropout uncertainty estimates, achieving EERs from 6.80% to 15.05% across ASVspoof 2019/2021 LA/PA and using uncertainty as a trust signal for poisoning-resilient federated aggregation (Mohammadi et al., 4 Dec 2025). A plausible implication is that future VAS will be judged less by isolated speaker-verification accuracy than by their ability to maintain identity assurance, liveness assurance, template protection, and deployment efficiency under evolving multi-stage threat models.