ECAPA-TDNNLite Model

• ECAPA-TDNNLite is a compact speaker embedding extractor designed for automatic speaker verification with an efficient SE-Res2Net backbone and depthwise separable convolutions.
• The model employs an asymmetric enroll-verify structure, using a full model for enrollment and the lightweight model for verification to achieve high accuracy with minimal runtime cost.
• It achieves competitive performance with an EER of 3.07% (or 2.31% in the asymmetric configuration) at only 11.6M FLOPS, making it well-suited for resource-constrained environments.

The ECAPA-TDNNLite model is a compact speaker embedding extractor developed for automatic speaker verification (ASV) systems, designed to address computational efficiency constraints while maintaining robust performance. It combines architectural innovations—most notably an optimized SE-Res2Net backbone with depthwise separable convolutions, channel reduction, and architectural simplifications—with an asymmetric enroll-verify structure for speaker verification that enables high accuracy with minimal inference cost. ECAPA-TDNNLite functions as either a standalone lightweight verifier or as the verification branch in an asymmetric pipeline, paired with a more complex ECAPA-TDNN during enrollment, to further enhance accuracy without increasing the verification-time footprint (Lin et al., 2021).

1. Network Architecture of ECAPA-TDNNLite

ECAPA-TDNNLite accepts mean-normalized 80-dimensional MFCC input features (cropped to 2-second segments, typically $T\approx200$ frames, no VAD), with SpecAugment masking applied (up to 5 frames in time and frequency).

The architecture comprises:

1. Initial 1D Convolution:
   • $80\rightarrow144$ channels, kernel size 5, stride 2, dilation 1.
   • Output shape: $(144, \lceil T/2 \rceil)$.
2. Three Stacked SE-Res2Blocks:
   • Each block has 144 channels, scale $s=8$, and a unique dilation rate $d_i$.
   • Each SE-Res2Block proceeds as:
     • 1×1 Conv (bottleneck): $W_1 \in \mathbb{R}^{144\times144}$
     • Res2Net split: Partition features into 8 groups ($18$ channels each).
     • For each group $u_k$ ($k=1,\dots,8$): $y_1 = u_1$ $$y_k = \mathrm{Conv1D}_{\mathrm{dilated}}(u_k) + y_{k-1}$$, $k>1$, using depthwise convolutions ($k=3$, stride 1, dilation $d_i$).
     • Concatenate $\{y_k\}_{k=1}^{8}$ to get output vector.
     • 1×1 Conv (restore): $W_2 \in \mathbb{R}^{144\times144}$
     • Squeeze-Excitation: Squeeze: $s_j = \frac{1}{T}\sum_t v_j(t)$ for $j=1,\dots,144$ Excitation: $$z = \sigma(W_2' \cdot \delta(W_1' s + b_1') + b_2') \in \mathbb{R}^{144}$$, bottleneck $r=128$
     • Scale: $v \odot z$
     • Add input (residual): $x + (\cdots)$
   • Output of the three SE-Res2Blocks is summed.
3. Expansion 1×1 Conv:
   • $144\rightarrow1536$ channels, kernel size 1.
4. Attentive Statistics Pooling (ASP):
   • For feature map $H \in \mathbb{R}^{1536\times L}$, L = sequence length.
   • Attention weights: $a_t = \mathrm{softmax}(w^T\tanh(W H + b))$.
   • Compute statistics: mean $\mu = \sum_t a_t H_t$, std $\sigma = \sqrt{\sum_t a_t (H_t - \mu)^2}$.
   • Output: $\phi = [\mu; \sigma] \in \mathbb{R}^{3072}$.
5. Final Dense Layer:
   • $3072\rightarrow192$ embedding dimension.

The model totals 318K parameters and achieves a measured computational cost of 11.6M FLOPS under optimized implementation. Notable modifications relative to full ECAPA-TDNN include strided first convolution, systematic use of depthwise Conv1D, and channel count reduction (Lin et al., 2021).

2. Computational Complexity and Comparative Analysis

Multiply-add operations are used to evaluate computational complexity. For Conv1D layers with input $C_i$, output $C_{i+1}$, kernel $k$, and sequence length $L$, the computation is $2\cdot L\cdot C_i\cdot C_{i+1}\cdot k$ (Conv1D), $2\cdot L\cdot C\cdot k$ (depthwise), and $2\cdot L\cdot C_i\cdot C_{i+1}$ (pointwise/1x1). For $T=200$, post-stride $T=100$:

Layer/Component	# Operations (M)
Conv1D₁ (input conv)	23
Three SE-Res2Blocks	25
1×1 Conv (144→1536 channels)	44
ASP + Final Dense	1.2
Estimated sum (multiply-adds ×2 = FLOPS)	186
Measured implementation (real FLOPS)	11.6

The optimization is achieved through highly parallel depthwise separable convolutions and efficient pooling/final layers. For comparison, the full ECAPA-TDNN model (C=512 channels) computes approximately 1.8G FLOPS, ~150 times heavier than ECAPA-TDNNLite (Lin et al., 2021).

3. Performance Evaluation

ECAPA-TDNNLite, as a standalone symmetric system, achieves an Equal Error Rate (EER) of 3.07% on the VoxCeleb1 original test set, with 0.296 minDCF. The asymmetric system (enrollment with ECAPA-TDNN, verification with ECAPA-TDNNLite) achieves a reduced EER of 2.31% and minDCF of 0.251 under identical verification-time costs.

Model	O-EER (%)	O-minDCF	E-EER (%)	E-minDCF	H-EER (%)	H-minDCF
ECAPA-TDNNLite (standalone)	3.07	0.296	3.00	0.318	5.20	0.436
Asymmetric (Enroll=ECAPA, Verify=Lite)	2.31	0.251	2.24	0.245	3.77	0.358

These results demonstrate that the asymmetric paradigm provides a substantial reduction in EER compared to the standalone lightweight system, supporting the claim that asymmetric modeling confers accuracy benefits without additional verification cost (Lin et al., 2021).

4. Asymmetric Enroll–Verify Structure

The ECAPA-TDNNLite model is central to the proposed asymmetric enroll-verify structure, which decouples the computation-heavy enrollment phase from the cost-critical verification phase. Speaker representations are extracted at enrollment via the full ECAPA-TDNN ($f_L$), processed server-side and stored as enrollment embeddings ($e$). Verification is performed with the ECAPA-TDNNLite ($f_S$) on-device, generating test embeddings ($v$). Identity matching employs cosine similarity:

$$s(x_\mathrm{enroll}, x_\mathrm{test}) = \cos(e, v) = \frac{e\cdot v}{\|e\|\,\|v\|}$$

Only $f_S$ is evaluated at verification, guaranteeing the 11.6M FLOPS runtime footprint per trial. This structure enhances deployability in resource-constrained environments, as verification efficiency is decoupled from enrollment complexity (Lin et al., 2021).

5. Training Regime and Implementation

Training utilizes the VoxCeleb2 dev dataset (1.09M utterances, 5,994 speakers), with augmentation from MUSAN (music, babble, ambient, TV), RIR (small/medium room impulse responses), and tempo augmentation ($\times 0.9, \times 1.1$). Input features are 80-dim mean-normalized MFCC with 25 ms window, 10 ms shift, and SpecAugment. Segments are 2 s, batch size 256 (all distinct speakers for AP loss).

The loss function is a sum of three terms:

1. AAM-Softmax Loss (on both branches; margin $m=0.2$, scale $s=32$):

$$L_\mathrm{AAM} = -\frac{1}{N}\sum_{i}\log \left(\frac{e^{s(\cos(\theta_{y_i}) - m)}}{e^{s(\cos(\theta_{y_i}) - m)} + \sum_{j\neq y_i} e^{s\cos(\theta_j)}}\right)$$

2. Angular Prototypical (AP) Space-Alignment Loss ($w=32$):

$$L_\mathrm{AP} = -\frac{1}{B}\sum_{i=1}^B \log\left( \frac{e^{w\,\cos\theta_{i,i}}}{\sum_{j=1}^{B} e^{w\,\cos\theta_{i,j}}} \right)$$

where $$\cos\theta_{i,j} = \frac{e_i \cdot v_j}{\|e_i\| \|v_j\|}$$.

3. Total Loss:

$$L = L_\mathrm{S-AAM} + L_\mathrm{L-AAM} + \lambda L_\mathrm{AP}, \quad \lambda=10$$

Optimization uses SGD (momentum not specified), linear learning rate warmup (0 to 0.1 over 5 epochs), followed by halving learning rate on validation-loss plateau (for 3 epochs), with training terminating at 100 epochs (Lin et al., 2021).

6. Context and Impact in Resource-Constrained Speaker Verification

ECAPA-TDNNLite addresses the persistent challenge in ASV of balancing recognition accuracy and real-time deployability under limited computational resources. By leveraging architectural pruning, depthwise separable convolutions, and an asymmetric enrollment-verification protocol, ECAPA-TDNNLite makes it feasible to deploy accurate speaker verification on devices where compute, memory, and energy are limited. Performance results indicate that using the asymmetric protocol achieves state-of-the-art (for this regime) with substantial reductions in both parameter count (318K vs. full ECAPA-TDNN’s multi-million scale) and FLOPS (11.6M vs. 1.8G). These design choices establish ECAPA-TDNNLite as a significant contribution to practical ASV in ambient, embedded, and IoT contexts (Lin et al., 2021).