Neural Topic Modeling with BERTopic

• Neural Topic Modeling with BERTopic is a pipeline that uses transformer-based embeddings to generate coherent topics from unstructured text.
• It employs UMAP for dimensionality reduction and HDBSCAN (or K-Means) for adaptive clustering, enhancing topic coherence and filtering noise.
• The class-based TF-IDF (c-TF-IDF) technique selects discriminative keywords, improving interpretability and supporting domain-specific adaptations.

Neural topic modeling with BERTopic is a class of algorithms that discovers latent themes in large, unstructured text corpora using transformer-based contextual embeddings, non-linear manifold reduction, unsupervised clustering, and dense class-based TF-IDF keyword extraction. BERTopic is widely adopted in domains requiring high topic coherence and interpretable clusters, particularly for short, noisy, or morphologically complex texts. This approach surpasses classical bag-of-words latent variable models by leveraging pretrained LLMs to encapsulate polysemy, morphology, and nuanced context.

1. Theoretical Foundation and Architecture

BERTopic formalizes topic modeling as a four-stage pipeline grounded in neural NLP advances:

1. Contextual Embedding: Each document $d$ is passed through a pretrained sentence transformer, such as SBERT, producing an embedding $\mathbf{e}_d \in \mathbb{R}^D$. Models are typically fine-tuned on sentence similarity objectives, and selection is critical for downstream clustering quality—domain-specialized models like FinTextSim or BioClinicalBERT markedly improve coherence and separability in context-rich domains (e.g., financial or biomedical) (Jehnen et al., 22 Apr 2025, Ionescu et al., 17 Jan 2026).
2. Manifold Learning (UMAP): High-dimensional embeddings are projected to a latent space $\mathbb{R}^p$ (commonly $p=2$–$20$) using Uniform Manifold Approximation and Projection (UMAP). The UMAP objective minimizes the cross-entropy between high- and low-dimensional affinities:

$$C = \sum_{i\neq j}\left[p_{ij}\log\frac{p_{ij}}{q_{ij}} + (1-p_{ij})\log\frac{1-p_{ij}}{1-q_{ij}}\right]$$

where $p_{ij}$ encodes neighborhood relationships in the embedding space and $q_{ij}$ in the UMAP space. UMAP hyperparameters, especially $n\_neighbors$, $min\_dist$, and $\mathbf{e}_d \in \mathbb{R}^D$0, must be tuned for language, document length, and desired topic granularity (Kandala et al., 20 Apr 2025, Medvecki et al., 2024, Arfaoui et al., 24 Nov 2025).

1. Clustering (HDBSCAN or K-Means): Reduced embeddings are clustered using HDBSCAN—a density-based, nonparametric algorithm with auto-adaptive cluster size and a principled noise assignment strategy via mutual reachability distance:

$\mathbf{e}_d \in \mathbb{R}^D$1

Alternatives such as K-Means are used when full document coverage is required; HDBSCAN optimizes for cluster purity at the cost of labeling outliers, while K-Means offers deterministic, centroid-based assignments and zero noise (Groot et al., 2022, Ulloa, 2023).

2. Topic Representation (Class-based TF-IDF: c-TF-IDF): For cluster $\mathbf{e}_d \in \mathbb{R}^D$2, the c-TF-IDF weight for term $\mathbf{e}_d \in \mathbb{R}^D$3 is:

$\mathbf{e}_d \in \mathbb{R}^D$4

with $\mathbf{e}_d \in \mathbb{R}^D$5 the frequency in cluster $\mathbf{e}_d \in \mathbb{R}^D$6, $\mathbf{e}_d \in \mathbb{R}^D$7 the total number of clusters, and $\mathbf{e}_d \in \mathbb{R}^D$8 clusters containing $\mathbf{e}_d \in \mathbb{R}^D$9. This formulation emphasizes discriminative keywords within the context of the discovered cluster structure (Grootendorst, 2022, Bhandarkar et al., 8 Oct 2025).

2. Hyperparameter Tuning and Embedding Selection

Performance and interpretability depend strongly on embedding model, dimension reduction, and clustering settings:

• Embedding Model Selection: Domain- or language-adapted encoders (e.g., FinTextSim for finance, MahaBERT-V2 for Marathi, BioClinicalBERT for clinical narratives) consistently surpass general-purpose models in coherence and topic precision, reducing intertopic similarity and outlier rate (Jehnen et al., 22 Apr 2025, Ionescu et al., 17 Jan 2026, Shinde et al., 4 Feb 2025).
• Intermediate Layer Representations: Aggregating hidden states from multiple transformer layers (sum/concatenate last four or all layers) and mean/max pooling can boost both topic coherence and diversity beyond standard last-layer pooling; CLS-token pooling is consistently suboptimal (Koterwa et al., 10 May 2025).
• UMAP and HDBSCAN Grid Search: Optimal values for $\mathbb{R}^p$0, $\mathbb{R}^p$1, $\mathbb{R}^p$2, and $\mathbb{R}^p$3 vary by language, document length, and corpus structure. Outlier rates are minimized and stability maximized by aligning these hyperparameters with corpus characteristics (Medvecki et al., 2024, Arfaoui et al., 24 Nov 2025, Kandala et al., 20 Apr 2025).
• Handling Outliers: HDBSCAN typically labels a significant fraction of documents as noise, especially in fragmented or short text corpora (outlier rates $\mathbb{R}^p$470% possible). Approximate distribution post-processing or use of K-Means can mitigate this when topic coverage is essential (Groot et al., 2022, Kandala et al., 20 Apr 2025).

3. Evaluation Metrics and Validation

Multiple quantitative and human-centric evaluation frameworks are used to assess topic model quality:

• Coherence Metrics: $\mathbb{R}^p$5, UMass, UCI/PMI, and NPMI are used extensively—higher values reflect more semantically consistent topic-word co-occurrence (Kandala et al., 20 Apr 2025, Sangaraju et al., 2022, Murugaraj et al., 12 Dec 2025).
• Diversity: Computed as the ratio of unique words in all top-$\mathbb{R}^p$6 topic keywords to the total possible ($\mathbb{R}^p$7), measuring redundancy and semantic spread (Grootendorst, 2022, Shinde et al., 4 Feb 2025).
• Stability: Bootstrap resampling (e.g., 30 replicates), Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI) measure cluster assignment robustness to corpus perturbations (Arfaoui et al., 24 Nov 2025).
• Human Evaluation: Domain-expert assessment of semantic coherence and cultural specificity often reveals deficiencies in automated metrics, especially in morphologically rich or low-resource languages (Kandala et al., 20 Apr 2025, Arfaoui et al., 24 Nov 2025).
• Soft Assignment and Approximate Distribution: For nuanced corpora (e.g., open-ended narratives), soft labeling via posterior topic probabilities (approximate_distribution method) captures overlap and uncertainty in topic assignments (Ionescu et al., 17 Jan 2026).

4. Applications and Empirical Results

BERTopic has been deployed across diverse textual domains:

• Software Engineering and API Documentation: Applied to millions of Stack Overflow posts, automatically extracting 1,813 topics, with 75% of posts in the top 80 clusters. Extractive summarization produces concise question–solution pairs per topic, supporting rapid knowledge retrieval (Naghshzan et al., 2023).
• Open-Ended Personal Narratives: In Belgian Dutch daily narratives, BERTopic generated semantically tight, culturally resonant topics, outperforming LDA and KMeans in both qualitative and diverse-topic coverage, albeit with lower automated coherence—human rating is crucial for validation in such settings (Kandala et al., 20 Apr 2025).
• Financial and Biomedical Text: Domain-adaptive sentence transformers (FinTextSim, BioClinicalBERT) enable BERTopic to delineate fine-grained economic and clinical phenomena, with near-perfect topic precision and strong improvements in intra-topic similarity (Jehnen et al., 22 Apr 2025, Ionescu et al., 17 Jan 2026).
• Multilingual and Low-Resource Languages: By combining multilingual transformers and minimal preprocessing, BERTopic delivers coherent topics for morphologically complex texts (e.g., Serbian, Marathi), outperforming traditional LDA/NMF even under partial lemmatization or reduced vocabulary (Medvecki et al., 2024, Shinde et al., 4 Feb 2025).
• Large-Scale Historical and Conversational Analysis: BERTopic scales to hundreds of thousands of documents (e.g., historical newspapers, LLM conversations), supporting dynamic topic modeling and time-resolved trend analysis. On impresso archives, it traces nuclear discourse evolution and identifies event-aligned topic shifts (Murugaraj et al., 12 Dec 2025, Bhandarkar et al., 8 Oct 2025).
• Qualitative Research and Focus Groups: Hierarchical pipelines (NMF→BERTopic) and reproducible validation unlock interpretable, stable topic extraction for small sample sizes, enabling domain-expert validation and enabling best-practices for manual coding replacement (Arfaoui et al., 24 Nov 2025, Cheng et al., 2022).

5. Comparative Performance and Methodological Variants

Direct comparison studies highlight the following:

• Superiority of Embedding-based Models: BERTopic systematically outperforms classical LDA, NMF, and LSA on both coherence and topic diversity, especially in the analysis of short, multi-domain, or morphologically rich corpora (Sangaraju et al., 2022, Groot et al., 2022, Medvecki et al., 2024).
• Clustering Alternatives and Trade-offs: HDBSCAN optimizes cluster purity and density but filters outliers aggressively. K-Means (in combination with UMAP) provides full assignment but may yield less coherent or less linguistically tuned clusters (Groot et al., 2022, Ulloa, 2023).
• Hybrid Pipelines: Combining bag-of-words NMF for broad thematic carving with BERTopic for fine-grained, neural subtopic detection yields higher interpretability and computational efficiency in hierarchical analyses (Cheng et al., 2022).
• Impact of Hyperparameter Choices: Coherence, stability, and interpretability are highly sensitive to UMAP and HDBSCAN configurations, necessitating grid search and, ideally, reproducibility protocols with fixed seeds and embedding reuse (Arfaoui et al., 24 Nov 2025, Schäfer et al., 2024).
• Evaluation Shortcomings: Conventional coherence metrics may not correspond to human judgments in high-context, cultural, or domain-specific data; multi-metric and qualitative validation are essential (Kandala et al., 20 Apr 2025, Schäfer et al., 2024).

6. Best Practices and Limitations

Recommendations derived from empirical studies:

• Embed with Domain- and Language-Adaptive Models: Select transformers pre-adapted to your corpus language/genre (FinTextSim, BioClinicalBERT, MahaBERT) for maximal separability and precision (Jehnen et al., 22 Apr 2025, Shinde et al., 4 Feb 2025, Ionescu et al., 17 Jan 2026).
• Pooling and Layer Selection: Mean or max pooling over last four/all layers enhances model robustness and diversity compared to default CLS pooling (Koterwa et al., 10 May 2025).
• Preprocessing and Outlier Reduction: Minimal lemmatization or partial preprocessing suffices with neural encoders; outlier assignment can be reduced post hoc using cosine similarity reallocation (Medvecki et al., 2024, Kandala et al., 20 Apr 2025).
• Hyperparameter Exploration: Conduct grid search over UMAP $\mathbb{R}^p$8, HDBSCAN $\mathbb{R}^p$9; optimize for combined coherence and topic coverage rather than any single score (Arfaoui et al., 24 Nov 2025, Schäfer et al., 2024).
• Evaluate with Multiple Metrics: Pair automated measures (NPMI, $p=2$0, diversity) with qualitative and expert validation, particularly in low-resource or open-ended settings (Kandala et al., 20 Apr 2025, Arfaoui et al., 24 Nov 2025).
• Document and Share Protocols: Ensure reproducibility by storing embedding seeds and grid configurations, sharing code, and using deterministic clustering (Arfaoui et al., 24 Nov 2025).
• Address Limitations: Hard clustering and redundancy in static BERTopic requires post-processing (e.g., topic merging, hierarchical clustering), and current evaluation metrics can bias against embedding-based models in language/genre-diverse corpora (Murugaraj et al., 12 Dec 2025, Schäfer et al., 2024).

7. Future Directions

Technical limitations and open research questions include:

• Soft Assignment and Probabilistic Extensions: Static BERTopic provides hard clustering; future research aims to capture document overlap and nuanced discourse using approximate posterior distributions (Murugaraj et al., 12 Dec 2025, Ionescu et al., 17 Jan 2026).
• Automated Hyperparameter Optimization: Existing results highlight substantial model instability without protocolized grid search; adaptive optimization frameworks are needed for robust deployment across domains (Jehnen et al., 22 Apr 2025, Arfaoui et al., 24 Nov 2025).
• Enhanced Evaluation: Refinement of coherence metrics to reflect cultural or domain relevance, incorporation of semi-automated human-in-the-loop coherence (e.g., CTC), and exploration of clustering validation indices for density models (Schäfer et al., 2024).
• Scalability and Interactive Visualization: Efficient, scalable implementations have demonstrated applicability to corpora $p=2$1100K documents; future exploration into topic-trend visualization and hierarchical exploration tools is ongoing (Murugaraj et al., 12 Dec 2025).
• Domain-Specific Pretraining and Fine-Tuning: Masked LLM pretraining in domain/language space (finance, biomedicine, morphologically rich languages) remains an active area for further coherence and coverage gains (Jehnen et al., 22 Apr 2025, Shinde et al., 4 Feb 2025, Ionescu et al., 17 Jan 2026).

---

Selected Key References:

• (Grootendorst, 2022) (BERTopic: Neural topic modeling with a class-based TF-IDF procedure)
• (Naghshzan et al., 2023) (Enhancing API Documentation through BERTopic Modeling and Summarization)
• (Kandala et al., 20 Apr 2025) (Evaluating BERTopic on Open-Ended Data: A Case Study with Belgian Dutch Daily Narratives)
• (Shinde et al., 4 Feb 2025) (Topic Modeling in Marathi)
• (Koterwa et al., 10 May 2025) (Enhancing BERTopic with Intermediate Layer Representations)
• (Jehnen et al., 22 Apr 2025) (FinTextSim: Enhancing Financial Text Analysis with BERTopic)
• (Murugaraj et al., 12 Dec 2025) (Automating Historical Insight Extraction from Large-Scale Newspaper Archives)
• (Arfaoui et al., 24 Nov 2025) (A Reproducible Framework for Neural Topic Modeling in Focus Group Analysis)
• (Groot et al., 2022) (Experiments on Generalizability of BERTopic on Multi-Domain Short Text)
• (Medvecki et al., 2024) (Multilingual transformer and BERTopic for short text topic modeling: The case of Serbian)
• (Ionescu et al., 17 Jan 2026) (Analyzing Cancer Patients' Experiences with Embedding-based Topic Modeling and LLMs)
• (Schäfer et al., 2024) (Unveiling the Potential of BERTopic for Multilingual Fake News Analysis -- Use Case: Covid-19)
• (Bhandarkar et al., 8 Oct 2025) (Investigating Thematic Patterns and User Preferences in LLM Interactions using BERTopic)
• (Ulloa, 2023) (A Process for Topic Modelling Via Word Embeddings)
• (Cheng et al., 2022) (Multi-scale Hybridized Topic Modeling: A Pipeline for Analyzing Unstructured Text Datasets via Topic Modeling)
• (Sangaraju et al., 2022) (Topic Modelling on Consumer Financial Protection Bureau Data: An Approach Using BERT Based Embeddings)