Semantic Topic Modeling with Top2Vec

• Semantic topic modeling is a method that uses distributed embeddings and clustering techniques to automatically uncover latent themes in large text corpora.
• Top2Vec employs joint word and document embeddings with frameworks like Word2Vec and Doc2Vec, then applies UMAP and HDBSCAN for coherent topic extraction.
• Top2Vec outperforms traditional models such as LDA by providing improved topic coherence and adaptability through transformer-based and hybrid approaches.

Semantic topic modeling aims to identify latent themes in large text corpora by leveraging distributed representations that encode semantic relationships among words and documents. Top2Vec is a prominent model that discovers topics by embedding words and documents into a shared vector space and locating dense regions corresponding to semantic topics. Unlike probabilistic approaches such as LDA, Top2Vec operates directly in continuous embedding space, enabling automatic determination of the number of topics, improved coherence, and integration with modern clustering techniques. Variants and related models extend this paradigm to exploit transformer embeddings or combine semantic and graph-based cues.

1. Embedding Frameworks and Joint Representations

Top2Vec is grounded in distributed representation learning, specifically extending the Word2Vec skip-gram and Doc2Vec DBOW frameworks. All words and documents are mapped to real-valued vectors in $\mathbb{R}^d$ such that dot products and cosine similarities encode semantic relatedness. After training, semantically similar documents and words are proximal in the embedding space. The skip-gram negative sampling objective captures word co-occurrence patterns: $$\mathcal L_{\rm SGNS} = -\log\sigma(U_{w_O}^\top V_{w_I}) - \sum_{i=1}^M\log\sigma(-U_{w_i^-}^\top V_{w_I}),$$ where %%%%1%%%% are word matrices and $M$ is the number of negative samples. In the Doc2Vec DBOW mode, document vectors $D$ are trained to predict their constituent words via

$$\mathcal L_{\rm DBOW} = -\sum_{w \in d} \left[\log\sigma(U_w^\top D_d) + \sum_{i=1}^M\log\sigma(-U_{w_i^-}^\top D_d)\right].$$

This yields document and word embeddings cohabiting the same $d$-dimensional space, forming the basis for semantic clustering and topic extraction (Angelov, 2020).

2. Topic Discovery Pipeline: Dimensionality Reduction and Clustering

The Top2Vec pipeline proceeds by training the embedding model, followed by dimensionality reduction and density-based clustering. UMAP is used to reduce high-dimensional document embeddings to a lower-dimensional manifold (often $\mathbb{R}^5 \sim \mathbb{R}^{50}$), preserving both local and global structure for subsequent clustering. This reduced representation is clustered using HDBSCAN, which discovers dense regions corresponding to topics and labels outlier documents as noise.

Each non-noise cluster $C_t$ defines a topic, with its centroid $\mathbf t_t = (1/|C_t|)\sum_{d\in C_t} D_d$ serving as a topic vector. The number of topics $T$ is determined automatically by the clustering (Angelov, 2020). For each topic vector, the top $n$ closest word embeddings (by cosine similarity) supply interpretable topic labels.

3. Semantic Similarity and Topic Interpretability

Top2Vec measures all semantic relationships—document–topic, topic–word, and word–word—by normalized cosine similarity: $$\mathrm{sim}(x,y) = \cos(x, y) = \frac{x^\top y}{\|x\|\|y\|}, \quad \mathrm{dist}(x, y) = 1 - \cos(x, y).$$ Documents are assigned to their nearest topic vector, and topic labels are determined by proximity in the shared embedding space. This paradigm avoids problems endemic to LDA, such as high-frequency but semantically generic words dominating topic descriptors. Top2Vec’s nearest-word approach surfaces domain-specific, core-semantic keywords (Angelov, 2020).

4. Comparison to LDA, PLSA, and Other Baselines

A key distinction is Top2Vec’s non-generative, clustering-based interpretation of topics as centroids of dense regions in semantic space. In contrast, LDA and PLSA treat topics as discrete word distributions and require explicit specification of $T$, as well as extensive text preprocessing (stop-words, stemming, lemmatization). Top2Vec trains on raw text and automatically settles on the number of topics, with common “functional” words clumping centrally but not forming their own clusters (Angelov, 2020, Bastola et al., 31 Aug 2025). Table 1 summarizes metric-based comparisons in legal document clustering (Bastola et al., 31 Aug 2025):

Model	Silhouette	DBI	CHS	NMI	ARI
TF-IDF + KMeans	0.031	5.647	20	0.121	0.041
LDA (10 topics)	0.460	0.785	848	0.089	0.036
NMF (10 topics)	0.279	0.949	451	0.143	0.045
Top2Vec + KMeans	0.685	0.452	15,340	0.141	0.046
Top2Vec+Node2Vec	0.927	0.111	29,186	0.153	0.051

Higher Silhouette, Calinski–Harabasz, and lower Davies–Bouldin (DBI) indicate improved topic coherence and cluster separation.

5. Extensions: Hybrid Models and Transformer Embeddings

Hybrid approaches augment Top2Vec with additional structural or semantic signals. By concatenating Top2Vec document embeddings with Node2Vec representations derived from a bipartite document–topic graph, then clustering the result with KMeans, markedly improved clustering quality is observed in specialized domains such as legal text (Bastola et al., 31 Aug 2025). This suggests that structural refinement atop semantic vectors yields synergistic gains in internal metrics.

Recent advances supplant Word2Vec/Doc2Vec with transformer-based encoders. The semantic-driven framework (Mersha et al., 2024) uses SBERT to generate document embeddings, UMAP to reduce dimensionality, and HDBSCAN for density-based clustering, mirroring the Top2Vec pipeline but leveraging contextual transformer representations: $$\mathbf{d}_i = f_{\mathrm{SBERT}}(D_i),\quad \mathbf{w} = f_{\mathrm{SBERT}}(w\,|\,\text{context}).$$ Evaluation on 20Newsgroups finds transformer-based semantic topic modeling achieves higher $C_V$ and NPMI than LDA, CTM, ETM, and BERTopic, with $C_V=0.735$ and $\mathrm{NPMI}=0.211$ (Mersha et al., 2024).

6. Empirical Results and Evaluation Methodologies

Empirical evaluation focuses on both internal clustering metrics (Silhouette, DBI, CHS), topic information gain (PWI), and standard coherence measures ($C_V$, NPMI, $U_\text{Mass}$, $C_{\text{UCI}}$). On 20Newsgroups, Top2Vec achieves PWI of 996.6 versus LDA’s 360.9 (with 20 topics and 10 keywords per topic). On Yahoo Answers, Top2Vec records PWI of 837.6 compared to LDA’s 153.3 (Angelov, 2020). This performance gap persists across topic granularity ($T$) and number of keywords ($n$).

Sensitivity analyses show that embedding dimensionality (default $D=300$) optimizes coherence and cluster separation, while reducing below $D=200$ degrades topic quality. UMAP and HDBSCAN parameters (e.g., $n_{\text{neighbors}}$, min cluster size) can be tuned to match interpretability and domain requirements; in legal data, optimal topic counts range from 25–30 depending on the coherence and Silhouette score (Bastola et al., 31 Aug 2025).

7. Limitations, Future Directions, and Related Models

Top2Vec's reliance on continuous vector-space clustering scales to large corpora and performs well on user-generated and domain-specific text. Limitations include sensitivity to embedding quality—specialized domains may require domain-adapted models (e.g., Legal-BERT) (Bastola et al., 31 Aug 2025)—and potential blurring of topic boundaries in highly specialized corpora. Human-in-the-loop validation and domain-specific preprocessing can address edge cases.

Recent research replaces bag-of-words and traditional embeddings with transformer-based architectures, resulting in even higher topic coherence (Mersha et al., 2024). Extensions such as hierarchical topic graphs, dynamic temporal analysis, and joint graphical-semantic pipelines are identified as promising avenues. Variants including Topic2Vec and Vec2Topic further illustrate the broad adoption of embedding-based topic modeling paradigms, each with specific advantages in interpretability and domain adaptation (Randhawa et al., 2016, Niu et al., 2015).

In summary, semantic topic modeling exemplified by Top2Vec and its successors represents a shift from probabilistic generative models to clustering and ranking in jointly trained semantic embedding spaces. Empirical evidence supports substantial improvements in topic coherence, relevance, and adaptability across diverse domains, particularly as more expressive embedding techniques become standard.