WavLink: Compact Audio–Text Embeddings

• WavLink is a compact audio–text embedding model that defines separate audio and text towers and uses a learnable global token with multi-scale Matryoshka contrastive supervision.
• Its two-stage training leverages 6M synthetic plus 0.1M quality audio-caption pairs with AdamW and cosine decay, achieving state-of-the-art retrieval and classification results.
• The multi-scale supervision enables up to 8× reduction in embedding size with minimal performance loss, improving storage efficiency and compute requirements in cross-modal tasks.

WavLink is a compact audio–text embedding model that augments the Whisper audio encoder with a learnable global token, jointly trained with a text encoder, and supervised using a multi-scale Matryoshka contrastive loss. WavLink achieves state-of-the-art performance in cross-modal retrieval and strong results in zero-shot classification and multiple-choice question answering benchmarks while emitting up to 8× smaller vectors than prior frame-level audio–LLMs. Its two-stage training recipe, deployment flexibility across model scales, and integration of Whisper's encoder mark a notable advance in deployable, general-purpose audio–text representations (Kumar et al., 21 Jan 2026).

1. Model Architecture

WavLink's architecture consists of distinct "audio" and "text" towers, a shared multi-dimensional embedding space, and dedicated projection and normalization heads:

• Audio Tower: Input log-Mel spectrograms $X\in\mathbb{R}^{B\times F\times T}$ are processed via Whisper's convolutional frontend, yielding intermediate representations $\tilde{H}_0\in\mathbb{R}^{B\times T'\times D}$. A single learnable "global" token $a_{\text{cls}}\in\mathbb{R}^{1\times D}$ is appended to each sequence $[\tilde{H}_0; a_{\text{cls}}]$, which is then passed through Whisper's Transformer encoder. The pooled output at the position of $a_{\text{cls}}$ forms the audio embedding $z_a$.
• Text Tower: Two variants are supported: a CLIP-ViT text encoder or ModernBERT. Input text is tokenized, and the CLS output is extracted as $z_t$.
• Projection & Normalization: Linear projection heads $f_a(\cdot)$ and $f_t(\cdot)$ map $z_a$ and $z_t$ to a shared embedding space, followed by $\ell_2$ normalization:

$$\hat u_a = \frac{f_a(z_a)}{\|f_a(z_a)\|_2},\quad \hat u_t = \frac{f_t(z_t)}{\|f_t(z_t)\|_2}$$

• Similarity: Cosine similarity is computed as $\cos(\hat u_a, \hat u_t)$.

This configuration replaces the costly $1500$-frame Whisper representation per 30-second clip with a single token, enabling substantially more compact audio embeddings.

2. Training Methodology

WavLink employs a two-stage training regimen and comprehensive design sweeps to optimize supervision and architecture:

• Stage 1 (Pre-alignment): The model is trained on $\approx6$ million audio–caption pairs (AudioSetCaps: AudioSet, VGGSound, and YouTube-8M with synthetic captions) for 3 epochs using AdamW (learning rate $10^{-4}$, cosine decay, 5% warmup, BF16, batch size $768$).
• Stage 2 (Finetuning): The model is further fine-tuned on higher-quality, verified datasets ($\approx0.1$ million pairs from AudioCaps v2 and Clotho) for 3 epochs with the same optimizer and continued Matryoshka loss.
• Design Sweep: Systematic evaluation using $\approx2$ million Auto-ACD pairs, batch size $80$, 10 epochs, LoRA rank $8$, $8\times$H100 GPUs, in BF16, is used to choose optimal text encoder, loss function, and adaptation regime.

This two-stage process—large-scale synthetic pre-alignment followed by domain-specific finetuning—proves crucial for both broad generalization and fine-grained alignment.

3. Matryoshka-Style Supervision and Embedding Compactness

A core innovation in WavLink is "Matryoshka" supervision, allowing a single model instance to emit nested embeddings at successively smaller dimensions. During each forward pass, the embedding vector $\hat u\in\mathbb{R}^{d_1}$ is sliced to produce $\hat u^{(k)}=\text{slice}(\hat u,d_k)$ for target embedding dimensions $d_1>d_2>\ldots>d_K$. Each $\hat u^{(k)}$ is supervised with the same contrastive loss, and the average is taken:

$$\mathcal{L} = \frac{1}{K}\sum_{k=1}^K \mathcal{L}_{\text{contrast}}(\hat u^{(k)}_a,\hat u^{(k)}_t)$$

At inference, embeddings may be truncated at any $d_k$, providing 1–8× storage and compute reduction (<1 percentage point performance drop at minimum dimension). Global token usage reduces embedding sizes from typical frame-level representations by 8× (e.g., 768→96 or 512→64).

4. Loss Functions and Training Objectives

WavLink evaluates two primary cross-modal objective functions:

• CLIP-style InfoNCE Loss: For normalized batches $U_a,U_t\in\mathbb{R}^{B\times d}$, similarity matrix $S = U_a U_t^T$, with learnable temperature $\tau$. The loss for audio→text is:

$$L_{a\rightarrow t} = \frac{1}{B} \sum_{i=1}^B -\log \text{softmax}(S_{i, \cdot}/\tau)[i]$$

Text→audio is analogous. Total loss: $L = L_{a\rightarrow t}+L_{t\rightarrow a}$.

• SigLIP (sigmoid-BCE) Variant: Sigmoid activation applied to $S_{ij}/\tau$, with only diagonal (same-pair) entries as positives (1) and off-diagonal as negatives (0), optimized via binary cross entropy.

Ablations revealed CLIP-style InfoNCE with CLIP text encoder and full finetuning delivers optimal results.

5. Model Sizes and Technical Variants

WavLink is implemented in several parameter regimes, each supporting multi-scale embedding emission:

Size	Audio+Text Params	Supported Embedding Dims
Base	84M (20+63)	512, 256, 128, 64
Small	152M (88+63)	512, 256, 128, 64
Large	761M (637+123)	768, 384, 192, 96

The design admits single-token ($d_k$-dimensional) embeddings, down to 96 or 64D, replacing up to 1,500 frame-level features. Base and Small variants remain competitive, with Base (<100M params) outperforming specialized CLAPs on multiple tasks at substantially lower memory and compute.

6. Empirical Performance and Comparative Analysis

• Audio–Text Retrieval: WavLink-Large surpasses prior CLAP variants on AudioCaps and Clotho by 2–6 percentage points across Recall@1/5/10 metrics. WavLink-Small (152M params) is within 1–2 percentage points of Large at approximately 20% of its size. Embedding truncation to 1/8th size produces <1 percentage point drop on R@1/5/10.
• Zero-Shot Classification: On VGGSound, WavLink-Small and WavLink-Large achieve ≈31.7–31.8% accuracy (state-of-the-art, LAION-CLAP: 29.1%). ESC-50 and US8K results are within 5 percentage points of top specialized CLAPs.
• AIR-Bench Multiple-Choice QA: WavLink-Base (84M params, single token) attains 42.0% total average, an improvement of +6 percentage points versus LAION-CLAP (35.8%) and parity with Falcon3-Audio-3B (42.0%) despite being 43–100× smaller. Performance is strongest on classification, with relative deficits in grounding and temporally fine-grained musical tasks.
• Whisper vs. HTS-AT: Whisper significantly outperforms HTS-AT in long-clip retrieval (Clotho R@1: 22.4 vs. 14.0), confirming the advantage of Whisper-derived audio features in the compact embedding regime.

7. Applications, Limitations, and Prospective Directions

Applications:

• Low-latency, storage-efficient retrieval at web scale using a single 96-dimensional vector.
• Zero-shot classifiers for sound event, acoustic scene, music, and speech recognition.
• Plugin encoder for LLMs via single-token injection for audio modality fusion.

Limitations:

• Reduced performance relative to frame-level representations on fine-grained grounding and temporally precise question answering.
• Slightly diminished results on structured speech benchmarks (ESC-50, US8K).
• Current model only supports English.

Future Directions:

• Extension to multilingual global tokens.
• Integration as a single-token plugin into larger audio–LLMs for efficient inference.
• Hybrid models combining global and local tokens for enhanced fine-grained performance.
• Incorporation of task-specific adapters atop compact embeddings.

WavLink demonstrates that appending a learnable global token to a Whisper encoder, combined with Matryoshka-style multi-scale supervision and optimal cross-modal training, results in compact, high-fidelity audio–text embeddings that set new retrieval records and scale flexibly for deployed or research-facing audio-language applications (Kumar et al., 21 Jan 2026).