Matryoshka Audio-Text Embeddings

• The paper introduces MATE, a neural embedding framework that uses nested 'matryoshka' sub-embeddings to ensure semantic alignment across audio and text modalities.
• It leverages multi-level contrastive and PCA-guided alignment strategies to maintain informativeness in truncated prefixes for diverse retrieval tasks.
• Empirical results highlight significant gains in ASR rescoring and audio-text retrieval, achieving up to 23% WER reduction and an 8× storage reduction.

Matryoshka Audio-Text Embeddings (MATE) are a family of neural embedding frameworks designed to encode audio and text modalities into a shared, multi-resolution vector space using nested sub-embedding (“matryoshka”) structures. Originating in multi-modal sequence modeling and audio-language retrieval, MATE architectures impose the principle that lower-dimensional sub-vectors (“prefixes”) of the embedding must remain maximally informative and semantically aligned across modalities and tasks. This design supports lossless truncation, parameter-efficient indexing, and robust performance in zero-shot, few-shot, and open-vocabulary retrieval scenarios. MATE is instantiated in several domains, including multi-modal rescoring for ASR, open-vocabulary keyword spotting, and large-scale audio-text retrieval, leveraging supervision strategies ranging from InfoNCE-based contrastive learning to PCA-guided prefix alignment (Cai et al., 2023, Kumar et al., 21 Jan 2026, Jung et al., 20 Jan 2026).

1. Core Principles and Embedding Architecture

The MATE paradigm is rooted in matryoshka representation learning, where a single embedding vector is trained such that all its leading (prefix) subspaces—of length $d_1 < d_2 < ... < d_K = D$—encode increasingly detailed information. Both audio and text encoders map their respective inputs to vectors in $\mathbb{R}^D$; at inference, vectors can be truncated on the fly to a smaller dimension for efficiency, with minimal loss in representational fidelity (Kumar et al., 21 Jan 2026, Jung et al., 20 Jan 2026).

A generic MATE dual-encoder comprises:

• Audio tower: Typically a large pre-trained backbone (e.g., Whisper, WavLM, ECAPA-TDNN) projecting audio features into a fixed-dim space via pooling or a learnable CLS token (Cai et al., 2023, Kumar et al., 21 Jan 2026, Jung et al., 20 Jan 2026).
• Text tower: BERT, CLIP, or sequence RNNs map tokenized text into an embedding of identical or compatible dimension.
• Adapter/projector modules: Linear layers or bottleneck adapters ensure the output dimensionality and alignment.
• Prefix structure: For a given embedding $u\in\mathbb{R}^D$, the $k$th prefix is $u^{(k)} = u[1:d_k]$, with all $u^{(k)}$ nested in $u$.

This nesting directly supports:

• Dynamic retrieval-storage tradeoffs (indexing at different dimensions)
• Multi-granular supervision via loss functions applied at selected prefix levels.

2. Supervision and Alignment Strategies

MATE frameworks employ various supervision signals to enforce semantic alignment at multiple prefix granularities, maximizing the informativeness of truncated sub-embeddings:

• Multi-level Contrastive (InfoNCE) Loss: Enforces cross-modal similarity at all prefix resolutions: for each $k$, a contrastive loss $L^{(k)}$ is computed on the similarities between the first $d_k$ coordinates of normalized audio and text embeddings; the MATE objective averages or sums these losses across $k$ (Kumar et al., 21 Jan 2026).
• PCA-Guided Prefix Alignment: For open-vocabulary KWS, PCA components from the full text embedding are used as teacher signals for each prefix. Alignment losses (MSE + KL divergence) force both audio and text prefixes to match these compressed representations, concentrating high-salience cues in the lowest dimensions (Jung et al., 20 Jan 2026).
• Utterance-Level Pooling and Cross-Modality Fusion: In MLM/ASR tasks, BERT-based models fuse concatenated tokenized text and projected acoustic features, enabling direct cross-attention and joint optimization under combined MLM and contrastive losses (Cai et al., 2023).

These approaches ensure that retrieval and matching tasks at multiple resolutions benefit from robust, semantically aligned features.

3. Model Instantiations and Architectures

3.1 Masked Audio Text Encoder for Rescoring

MATE (Cai et al., 2023) leverages BERT (110M) and WavLM (95M), fusing tokenized text with projected audio (via CNN and adapter) at the transformer input. The complete input sequence $x$ consists of both text tokens and matched-length adapted audio features, enabling cross-attention throughout BERT layers.

3.2 Dual-Encoder Architectures for KWS & Retrieval

• In open-vocabulary KWS (Jung et al., 20 Jan 2026), the architecture comprises ECAPA-TDNN for audio and G2P + Bi-LSTM for text, producing 256-dim vectors with up to 5 nested prefixes.
• WavLink (Kumar et al., 21 Jan 2026) uses Whisper plus a “global” learnable token for pooling, paired with a CLIP text encoder; both are projected and L2-normalized.

Table: Example MATE Encoder Configurations

Application	Audio Encoder	Text Encoder	Prefix Schedule (dims)
ASR Rescoring	WavLM + CNN/Adapter	BERT (WordPiece)	Single-level (full/dims)
Audio-Text Retrieval	Whisper (CLS token)	CLIP (ViT), ModernBERT	$\{d, d/2, d/4, d/8\}$
Open-Vocab KWS	ECAPA-TDNN	G2P + Bi-LSTM	$\{16,32,64,128,256\}$

4. Empirical Performance and Applications

MATE-based models consistently deliver state-of-the-art or competitive accuracy in matched and transfer conditions:

• ASR Rescoring: MATE reduces word error rates over text-only BERT rescorers by 4–16% (in-domain) and 3–7% (out-of-domain). Few-shot adaptation (0.8 h domain data) yields up to 23% WER reduction vs. baselines (Cai et al., 2023).
• Audio-Text Retrieval: In WavLink, slicing the output embedding from 768 to 96 dims produces a <1% drop in Recall@1 on AudioCaps and Clotho, enabling 8× storage reduction without material retrieval loss (Kumar et al., 21 Jan 2026).
• Open-Vocab KWS: Prefix-aligned MATE improves average precision (AP) on WSJ from 78.66% (single embedding) to 80.94% (with 3–5 nested prefixes), surpassing all traditional proxy and triplet-based KWS losses (Jung et al., 20 Jan 2026).

Empirical ablations reveal that:

• Omitting alignment or using non-contrastive alignment (e.g., MSE alone) substantially lowers accuracy compared to full multi-level contrastive alignment (Cai et al., 2023).
• Using multiple prefixes up to an optimal $K$ ($K=3$ in KWS) maximizes AP; further increasing $K$ may plateau or slightly decrease gains (Jung et al., 20 Jan 2026).

5. Training Procedures and Data Regimes

MATE frameworks are trained on large paired audio-text datasets, with multi-stage recipes scaling from millions of web-scale clip-caption pairs down to labeled benchmarks:

• Pre-training employs massive datasets (AudioSet, VGGSound, LibriSpeech, Voxpopuli, and in-house corpora); fine-tuning leverages human-annotated evaluation sets (e.g., AudioCaps, Clotho).
• Batch sizes and optimizer schedules are scaled for data volume and model capacity (e.g., batch=768, 64×H100 GPUs for web-scale data in WavLink) (Kumar et al., 21 Jan 2026).
• Prefix-alignment losses are often scheduled with delayed onset to allow text encoder stabilization before squeezing embeddings (Jung et al., 20 Jan 2026).
• All prefix and multi-level losses are active only during training; inference is cost-equivalent to a standard encoder retrieval system.

6. Extensions, Limitations, and Future Directions

MATE’s matryoshka structure is extensible:

• Hierarchical nesting may be extended to additional modalities (e.g., visual, lip-motion), finer-granularity units (phoneme/syllable/word), or hierarchical tasks (ASR→SLU→MT) (Cai et al., 2023).
• Potential to increase the number of prefix levels, experiment with non-linear slicing, or apply task-specific subspace tuning (Kumar et al., 21 Jan 2026).
• Substituting discriminative projections (e.g., LDA) for PCA in the alignment machinery remains an open area (Jung et al., 20 Jan 2026).

Limitations include:

• Extreme compression (1/16 or smaller prefixes) may not suffice for fine-grained grounding tasks.
• Marginally increased training computation due to supervising multiple levels in parallel.
• All alignment loss machinery is offline; no runtime overhead, but prefix structure must be chosen/trained in advance.

A plausible implication is that further enhancements in multi-modal retrieval or sequence labeling may derive from deeper integration of temporal alignment or margin-based losses in the contrastive regime, as well as from expanding the matryoshka paradigm to structured outputs and richer annotation schemas.

7. Summary Table: MATE Applications and Key Metrics

Paper & Task	Dimensionality Schedule	Training Loss	Key Metric (Improvement)	Inference Overhead
(Cai et al., 2023) ASR Rescoring	Full, single-level	MLM + InfoNCE	WER –15.6% (in-domain)	None
(Kumar et al., 21 Jan 2026) Audio-Text Retrieval	d, d/2, d/4, d/8	Multi-InfoNCE (MATE)	R@1 drop <1% at 1/8 size	None
(Jung et al., 20 Jan 2026) Open-Vocab KWS	$\{16,32,64,128,256\}$	PCA prefix align + RPL	AP +2.28 pp vs. baseline	None

MATE establishes a principle of multi-resolution, matryoshka-style representation alignment for audio-text modeling, providing a practical and theoretically informed route for scalable, efficient, and accurate cross-modal matching.