PhiNet: Speaker Verification with Phonetic Interpretability

Abstract: Despite remarkable progress, automatic speaker verification (ASV) systems typically lack the transparency required for high-accountability applications. Motivated by how human experts perform forensic speaker comparison (FSC), we propose a speaker verification network with phonetic interpretability, PhiNet, designed to enhance both local and global interpretability by leveraging phonetic evidence in decision-making. For users, PhiNet provides detailed phonetic-level comparisons that enable manual inspection of speaker-specific features and facilitate a more critical evaluation of verification outcomes. For developers, it offers explicit reasoning behind verification decisions, simplifying error tracing and informing hyperparameter selection. In our experiments, we demonstrate PhiNet's interpretability with practical examples, including its application in analyzing the impact of different hyperparameters. We conduct both qualitative and quantitative evaluations of the proposed interpretability methods and assess speaker verification performance across multiple benchmark datasets, including VoxCeleb, SITW, and LibriSpeech. Results show that PhiNet achieves performance comparable to traditional black-box ASV models while offering meaningful, interpretable explanations for its decisions, bridging the gap between ASV and forensic analysis.

• The paper introduces PhiNet, a self-interpretable ASV framework that decomposes verification outcomes into phoneme-level evidence.
• It employs a hierarchical architecture using ECAPA-TDNN-inspired features with learnable transformations to generate match scores.
• Experiments show that PhiNet achieves verification accuracy comparable to baselines while offering detailed local and global interpretability.

PhiNet: Speaker Verification with Phonetic Interpretability

Overview

The paper introduces PhiNet, a novel automatic speaker verification (ASV) framework that integrates phonetic interpretability into the decision-making process. PhiNet is inspired by forensic speaker comparison (FSC) methodologies, aiming to bridge the gap between the opaque, black-box nature of traditional ASV systems and the evidence-driven, transparent reasoning central to FSC. PhiNet offers both local (trial-level) and global (model-level) interpretability by decomposing verification outcomes into phoneme-level evidence. The proposed system achieves speaker verification performance on par with standard neural ASV models, while providing detailed insight into which phonetic features drive verification decisions.

Motivation and Background

Traditional ASV approaches, including GMM-UBM, i-vector, x-vector, r-vector, and ECAPA-TDNN, provide high accuracy but remain fundamentally opaque, yielding only a single similarity score per trial. In forensic contexts, such lack of transparency undermines trust and limits accountability. FSC, in contrast, leverages granular phonetic evidence (e.g., analysis of vowels, consonants, and intonation) to support decisions, prioritizing interpretability and evidential reasoning. Recent research in explainable AI, including prototype-based and concept bottleneck models, has largely overlooked open-set speaker verification tasks, focusing instead on closed-set recognition or visualization-based attribution. The paper positions PhiNet as a "self-interpretable" ASV system capable of providing explicit phoneme-level rationales for its verdicts.

PhiNet Architecture

PhiNet consists of three principal components: the phonetic trait extractor, trait-level comparison layer, and decision layer.

• Phonetic Trait Extractor: Utilizes ECAPA-TDNN-inspired frame-level convolutional layers to generate 1536-D frame features, which are aggregated into phoneme-level traits via phonetic boundaries detected by a pre-trained recognizer. Each trait encodes speaker-specific information for a given phoneme, with absent phonemes defaulting to zero vectors.

  Figure 1: Block diagram of the phonetic trait extractor, displaying the hierarchical transformation from frame-level features to phonetic trait embeddings.

• Trait-Level Comparison: For each phoneme $i$, PhiNet computes the cosine similarity between enrollment and test traits, mapping the similarity into $[-\infty, +\infty]$ via learnable transformations. This generates phonetic scores reflecting speaker match for each phoneme.

  Figure 2: Illustration of adjusting phonetic scores according to distinctiveness, demonstrating the score-shifting effects of phoneme weights in decision boundaries.

• Decision Layer: Learns a global phonetic weight vector $w^i$ via min-max normalization or alternative strategies. The final match score is a weighted average of phonetic scores, with weights reflecting each phoneme's distinctiveness.

  Figure 3: Block diagram of the proposed framework, showing the flow from input utterances through phonetic trait extraction, trait comparison, and interpretability-driven decision making.

The entire network is trained end-to-end—with dedicated losses enforcing both trial-level verification consistency and intra/inter-speaker phonetic trait discriminability.

Interpretability Methods

PhiNet provides dual interpretability:

• Local Interpretability: For each trial, PhiNet exposes a breakdown of the match score by individual phoneme, allowing inspection of speaker-specific evidence at the phonetic level.

  Figure 4: Visualization of phoneme-segmented evidence in decision making for non-target and target trials, with segment-level weights and scores illustrated.

• Global Interpretability: The learned phonetic weights $w^i$ reflect the systemic decision logic, ranking phonemes by their contribution to speaker discrimination. This aligns with FSC heuristics (e.g., nasals and vowels are most discriminative for identity).

  Figure 5: Distribution of learned phonetic weights across the phonemic inventory, revealing global model biases and highlighting dominant phonetic cues.

PhiNet's interpretability is evaluated by removing phonemic segments or trait vectors and measuring their effects on Equal Error Rate (EER). Faithfulness is quantified by the fidelity score, the mean absolute difference in EER change between spectrogram and trait removals.

Experimental Analysis

Speaker Verification Performance

PhiNet is evaluated across VoxCeleb 1/2, SITW, and LibriSpeech. Results demonstrate that PhiNet attains EERs comparable to ECAPA-TDNN baselines (e.g., 5.67% EER on Vox1-O vs. 3.40% for ECAPA-TDNN). The gap is moderate, and interpretability is achieved with marginal overhead (parameter count: 4.81M vs. 5.40M/6.19M).

Local Evidence Visualization

Trial-level evidence reveals that PhiNet's decision is dominated by high-weight, highly distinctive phonemes (e.g., "N" in nasals), confirmed by spectrogram and trait comparisons.

Global Distinctiveness Trends

Consistent with FSC research, weights for nasals (N, M) and select vowels/fricatives (AY, S, Z) are high, while stops and weak fricatives (B, P, HH) contribute minimally. Fricative weights exhibit subject-level variability, matching known debates in forensic linguistics.

Figure 6: Heatmaps of similarity between phonetic traits and their centroids, visualizing diagonal dominance for discriminative phonemes and trait entanglement for less informative segments.

Effect of Training Sample Duration

Sample duration strongly influences both accuracy and interpretability. Models trained on longer utterances exhibit high accuracy but more concentrated weights (less interpretability), while models trained on shorter utterances distribute weights more evenly but at lower accuracy.

Figure 7: Phonetic weight distributions learned across various training durations, illustrating the tradeoff between phonetic concentration and trait dispersion.

Training with mixed durations yields uniformly distributed weights (~0.5), indicating difficulty in learning meaningful phoneme distinctions.

Fidelity and Faithfulness

Leave-one-phoneme-out experiments consistently confirm that removal of high-weight phonemes dramatically increases EER. Fidelity scores range from 0.053–0.113, indicating moderate faithfulness between trait and segment-based evidence. Increasing training duration or dataset size reduces faithfulness, likely due to expanded receptive fields and imperfect phoneme segmentation.

Figure 8: Performance degradation from leave-$i$th-phoneme-out experiments, validating the contribution ranking provided by global interpretability.

Ablation Studies

Key findings:

• Data augmentation improves both accuracy and faithfulness.
• Min-max normalization for weights yields superior performance compared to Sigmoid or ReLU.
• Parameter tuning for mapping dimension ($c$) and loss weights ($\alpha$, $\beta$) yields optimal trade-offs.
• PhiNet outperforms black-box models in transparency without significant compromise in verification efficacy.

Practical and Theoretical Implications

Practically, PhiNet enables forensic-style critical review of ASV results, facilitating manual inspection and error tracing. Developers can identify system-level biases and optimize training hyperparameters. In legal contexts, PhiNet's evidence chain increases accountability and suitability for high-stakes applications.

Theoretically, the findings suggest that phoneme-level evidence can meaningfully explain recognition outcomes, supporting the hypothesis that ASV may benefit from integrating linguistic and acoustic interpretability. There exists a clear tradeoff between accuracy and interpretability, governed by sample duration, augmentation, and trait extraction architecture. Future research should investigate dynamic weighting strategies, improved phoneme segmentation, and hybrid training for optimizing both accuracy and interpretability.

Conclusion

PhiNet advances the state of speaker verification by embedding phonetic interpretability directly into the ASV workflow. The framework offers detailed, evidence-driven explanations at both the local and global level, narrowing the gap between black-box ASV and evidence-centric FSC. PhiNet maintains competitive accuracy relative to neural baselines while delivering substantial improvements in transparency, faithfulness, and user trust. The study lays foundational groundwork for developing accountable, explainable ASV systems and highlights promising avenues for further integration of linguistic evidence and interpretable training strategies.