Speech2Vec: Neural Speech Semantic Embeddings

Updated 4 November 2025

Speech2Vec is a neural model that learns semantic embeddings from raw speech by adapting distributional objectives via an RNN encoder-decoder architecture.
It employs skipgram and CBOW-style training to capture contextual acoustic cues like prosody and intonation, offering richer semantic insights than text-only models.
Empirical evaluations demonstrate that Speech2Vec often outperforms text-based embeddings such as Word2Vec on word similarity tasks, while also revealing challenges in reproducibility and handling rare words.

Speech2Vec is a neural framework for learning fixed-length semantic representations of spoken words directly from continuous speech. Distinct from conventional text-derived embeddings, Speech2Vec aims to capture distributional semantics inherent in speech through an RNN encoder-decoder architecture trained via adaptations of skipgram and continuous bag-of-words (CBOW) mechanisms. The model leverages local sequence context among acoustic word segments without dependence on textual transcriptions, grounding embeddings in speech signal properties—such as prosody and intonation—unavailable in text. The learned embeddings are evaluated and compared to both text-based Word2Vec and GloVe, demonstrating notable performance on standard word similarity tasks.

1. Model Architecture and Design

Speech2Vec employs an RNN-based sequence-to-sequence architecture to process variable-length acoustic word segments. The model comprises two principal components:

Encoder RNN: Processes an input sequence of acoustic features (e.g., MFCC vectors) corresponding to a spoken word segment, mapping it to a fixed-length latent vector representation.
Decoder RNN: Receives the encoder’s output (latent vector) and reconstructs a sequence of acoustic features either for the target word or neighboring word segments in the utterance, depending on the training paradigm.

Both skipgram and CBOW training modes are supported, mirroring the designs of Mikolov et al.’s Word2Vec, but operating on continuous audio sequences rather than discrete tokens.

Component	Function	Details
Encoder RNN	Encodes variable-length acoustic sequence to fixed-length vector	Bidirectional LSTM
Decoder RNN	Generates output sequence of acoustic features conditioned on encoder output	Unidirectional LSTM

This architecture enables the processing of raw speech signals, accommodating variability in word length and acoustic realization.

2. Training Paradigms: Skipgram and CBOW for Speech

Speech2Vec adapts Word2Vec’s distributional objectives to the acoustic domain:

Skipgram-style Training: For each center word segment $x^{(n)}$ , the encoder produces embedding $z^{(n)}$ . The decoder then reconstructs the acoustic sequences of its $k$ neighboring segments ( $x^{(n-k)}, ..., x^{(n-1)}, x^{(n+1)}, ..., x^{(n+k)}$ ). The loss is the sum of mean squared errors (MSE) over all context segments:

$\mathcal{L}_{\mathrm{skipgram}} = \sum_{i} \left\| \mathbf{x}^{(i)} - \mathbf{y}^{(i)} \right\|^2$

CBOW-style Training: Surrounding word segments are encoded, their representations summed, and the decoder reconstructs the center word segment. The loss again uses MSE but targets the reconstruction of the central segment only.

$\mathbf{z}^{(n)} = \sum_{i} \mathbf{h}^{(i)}$

Training is performed on large speech corpora using stochastic gradient descent; each spoken instance yields an embedding, and word-level embeddings are computed by averaging across instances.

Speech2Vec structurally parallels Word2Vec but introduces critical differences:

Input/Output Domain: Speech2Vec’s inputs and outputs are continuous acoustic feature sequences, unlike Word2Vec’s discrete one-hot word vectors.
Semantic Scope: Speech2Vec has access to prosodic, phonetic, and paralinguistic cues, potentially encoding richer semantics.
Embedding Variance: Each spoken instance yields a distinct embedding; variance among instances decreases as instance count increases.

Performance on standard word similarity benchmarks validated the claim that skipgram Speech2Vec often outperforms Word2Vec, especially when leveraging speech-derived information.

Model	Input	Architecture	Training Objective	Best Use
Word2Vec	Text tokens	FCNN	Skipgram/CBOW	Text NLP
Speech2Vec	Audio segments	Seq2Seq RNN	Skipgram/CBOW	Speech semantics

4. Embedding Properties and Evaluation

Speech2Vec embeddings are evaluated intrinsically by calculating cosine similarity between averaged embeddings of word pairs and correlating these against human similarity ratings (Spearman’s $\rho$ ) on 13 standard datasets (e.g., WS-353, SimLex-999, MEN, RG-65). In eight out of thirteen benchmarks, skipgram Speech2Vec exceeded Word2Vec, with notable results such as $\rho=0.846$ (Speech2Vec, MC-30) vs. $0.713$ (Word2Vec). Visualizations (e.g., t-SNE) demonstrate semantic organization along sentiment polarities.

A variance paper found that averaging over more spoken instances stabilizes word-level embeddings, substantiating the aggregation technique.

5. Model Implementation Details and Variants

The Speech2Vec implementation uses:

Encoder: Single-layer bidirectional LSTM for sequence encoding.
Decoder: Single-layer unidirectional LSTM for sequence generation, with optional attention to the encoder output.
Input Features: MFCC vectors, capturing spectral properties of speech.
Context Window: Typically $k=3$ neighboring words.
Training: SGD optimization, 500 epochs, batch size configured for data scale.

Dimensionality studies revealed that a 50-dimensional embedding suffices for strong performance; increasing size did not consistently improve results.

6. Limitations and Directions

Speech2Vec’s reliance on precise word-aligned acoustic segments introduces a dependence on forced alignment, limiting applicability in low-resource or unsegmented scenarios. The model excels at encoding semantic relationships in speech, but its performance for rare words is constrained by data sparsity. Extensions toward unsupervised segmentation and integration with end-to-end acoustic models are suggested as future directions. Practical application to extrinsic speech processing tasks (e.g., spoken language understanding) remains a focus for further research.

7. Legacy and Current Research Context

The foundational results of Speech2Vec (Chung et al., 2018) catalyzed interest in direct-from-speech semantic modeling, inspiring comparisons and critical reproductions (Chen, 2022, Sayeed et al., 2023). Subsequent research has drawn attention to difficulties in reproducibility and semantic isolation from phonetic interference, indicating that deep, end-to-end architectures and robust input representations—such as discretized speech tokens—are necessary to advance the area. The overarching insight is the feasibility of speech-based semantic learning, but with ongoing controversy regarding reported effectiveness and reproducibility of the original model design.

The Speech2Vec framework stands as a significant step toward unsupervised semantic modeling directly from speech, operationalizing distributional principles for continuous acoustic data and furnishing benchmark validation of its comparative advantage over text-only approaches under specific conditions. Its limitations and the ongoing debate surrounding its reproducibility continue to shape research on spoken semantic representations.