Duration-Aware Matryoshka Embedding (DAME)

• The paper introduces DAME, which aligns embedding dimensions with speech duration using nested sub-embeddings for robust speaker verification.
• DAME employs duration–dimension alignment via hard and soft weighting, optimizing supervision for both short and long utterances.
• Empirical results show DAME reduces short-utterance EER by 15–25% without adding inference-time overhead across various architectures.

Duration-Aware Matryoshka Embedding (DAME) is a model-agnostic framework designed to align embedding capacity with utterance duration in speaker verification (SV) systems. DAME introduces a nested hierarchy of sub-embeddings, where compact lower-dimensional prefixes encode robust speaker traits for short speech, and higher-dimensional tails capture richer details available in longer utterances. The approach generalizes across various speaker encoder architectures, consistently enhancing short-utterance robustness while maintaining full-length accuracy and incurring no inference-time computational overhead (Jung et al., 20 Jan 2026).

1. Motivation and Conceptual Foundations

Conventional SV systems employ a single, fixed-dimensional embedding $\mathbf{e}\in\mathbb{R}^D$ for all utterance lengths. Short utterances (e.g., $\leq 2$ s) yield limited phonetic and acoustic variability, rendering high-dimensional embeddings inefficient and prone to overfitting. Conversely, long utterances utilize the additional capacity to encapsulate fine-grained speaker traits. Prior empirical findings demonstrate that decreasing embedding dimensionality improves short-utterance SV and larger embeddings favor long utterances.

DAME directly addresses this capacity mismatch by constructing an embedding hierarchy: each longer prefix subsumes all shorter ones, effecting a “matryoshka” (nested doll) structure. The leading dimensions specialize in compact, discriminative cues for short durations, while trailing dimensions refine representation as utterance length increases.

2. Architecture and Mechanism

Let $f(\cdot)$ denote a speaker encoder mapping an utterance chunk $u$ to a $D$-dimensional embedding $z = f(u) \in \mathbb{R}^D$. Define a set of target durations $\mathcal{T} = \{\ell_1, \ldots, \ell_J\}$, and a set of prefix dimensions $\mathcal{D} = \{d_1, \ldots, d_K\}$, with $d_K = D$.

During training, for each speaker-labeled example $i$, $\leq 2$0 truncated chunks $\leq 2$1 are sampled at durations $\leq 2$2, and full embeddings $\leq 2$3 are computed. The $\leq 2$4-th sub-embedding for chunk $\leq 2$5 is the leading $\leq 2$6 components, $\leq 2$7.

Duration–dimension alignment is governed by weights $\leq 2$8, controlling which prefixes supervise each chunk.

$\leq 2$9

with $f(\cdot)$0.

• Hard Weighting (HW): $f(\cdot)$1, $f(\cdot)$2; each duration supervises only its matching prefix.
• Soft Weighting (SW): $f(\cdot)$3, $f(\cdot)$4 softly down-weights off-diagonal supervision.

3. Mathematical Formulation

Each prefix-level sub-embedding $f(\cdot)$5 is classified via a large-margin Softmax (SphereFace2). With speaker label $f(\cdot)$6, the per-prefix loss is: $f(\cdot)$7 where $f(\cdot)$8 is the logit scale (typically $f(\cdot)$9), and $u$0 the angular margin for dimension $u$1.

Chunk-wise aggregation uses alignment weights: $u$2

The DAME batch loss is

$u$3

where $u$4 balances longest-chunk and short-chunk losses.

Relation to Large-Margin Fine-Tuning (LMFT): LMFT applies a single margin $u$5 to the full embedding, potentially degrading short-utterance accuracy when long segments dominate batches. DAME replaces this with dimension- and duration-aligned margin objectives.

4. Training and Inference Regimes

Training consists of sampling multiple (typically $u$6) truncated chunks per speaker, computing full embeddings, extracting all prefix subvectors, and accumulating prefix losses under the DAME loss. Two operating modes are provided:

• DAME-GT (general training): Individual classifier heads $u$7 are learned for each prefix dimension.
• DAME-FT (fine-tuning): Uses a shared, pre-trained classifier $u$8; for prefix $u$9, the top $D$0 rows are used (“weight-tying”).

Inference always utilizes the full $D$1-dimensional vector and its single classifier head; there are no changes or additional costs at test-time relative to conventional SV.

5. Empirical Analysis

Performance is evaluated by equal error rate (EER, %) on VoxCeleb1-O/E/H and VOiCES using five trial types: full–full (f–f), and four short-test conditions (macro-averaged as s–avg). Key findings include:

Setup	f–f EER (%)	s–avg EER (%)	5s–1s EER (%)
Baseline (ECAPA)	1.08	2.54	4.84
DAME-GT-SW (ECAPA)	1.08	2.06 (−18.9%)	3.66 (−24.4%)
LMFT (ResNet34)	0.95	2.35	4.59
DAME-FT-HW (ResNet)	0.92	2.00 (↓)	4.19 (↓)
VOiCES (ECAPA)	4.84	11.18	18.53
DAME-GT-SW (ECAPA)	5.24 (+8.3%)	10.79 (−3.5%)	16.34 (−11.8%)

Across encoder architectures (ResNet34, ECAPA-TDNN, ERes2NetV2) and both training modes (GT, FT), DAME consistently reduced EER on short-duration trials (s–avg, 5s–1s) by 15–25% relative, while preserving or slightly improving full-length performance.

6. Comparative Frameworks and Positioning

Unlike feature aggregation approaches, which still learn a single embedding, DAME reconfigures the loss at the representation level to incorporate duration-awareness. This restructuring is orthogonal and complementary to architectural enhancements in encoder networks. DAME's training procedure does not impose additional inference-time costs or model complexity.

In the context of fine-tuning techniques, DAME acts as a direct replacement for conventional LMFT, mitigating the tendency to overfit longer utterances and enhancing generalization to short speech segments.

7. Insights and Implications

The nested sub-embedding structure enforces the allocation of speaker-discriminative information in the earliest coordinates for short utterances; longer inputs benefit from richer representation without penalizing brevity. Capacity–duration alignment is realized by explicit supervision of embedding prefixes tailored to chunk length.

Because inference always uses the full-length vector, DAME incurs zero runtime or architectural overhead compared to standard SV systems. A plausible implication is that DAME can be widely adopted as a model-agnostic, plug-in alternative to LMFT for applications in forensics, diarization, or any regime where short-utterance robustness is crucial.

By design, DAME improves short-utterance verification rates while maintaining full-utterance accuracy, generalizes across architectures and training paradigms, and highlights the importance of representation-level, duration-aware supervision in deep embedding systems (Jung et al., 20 Jan 2026).