Dynamic Embedded Topic Model

• Dynamic Embedded Topic Model (DETM) is a probabilistic model that captures the temporal evolution of topics using embedding-based parameterizations.
• It employs a linear Gaussian state-space model and RNN-based variational inference to ensure smooth transitions in topic representations.
• DETM is applied in policy discourse, historical linguistics, and diachronic semantic analysis to enhance topic coherence and capture subtle temporal drifts.

The Dynamic Embedded Topic Model (DETM) is a probabilistic latent variable model designed to capture the temporal evolution of topics in sequential document corpora. DETM integrates the strengths of dynamic topic models with embedding-based parameterizations, enabling it to model both smooth temporal topic drift and the geometric relationships among words in a continuous embedding space. The architecture underpins state-of-the-art approaches in diachronic semantic analysis, policy discourse modeling, and historical linguistics (Badekale et al., 8 Jul 2025, Dieng et al., 2019, Fittschen et al., 27 Apr 2025, Sirin et al., 2024).

1. Generative Framework and Model Structure

The DETM posits a corpus comprised of $D$ documents, each associated with a timestamp $t_d \in \{1,\ldots,T\}$, $K$ topics, vocabulary size $V$, and embedding dimension $L$. Each topic $k$ is assigned a time-indexed embedding trajectory $\{\alpha_{k,1:T}\}$, where $\alpha_{k,t} \in \mathbb{R}^L$, and each vocabulary word $v$ is associated with an embedding vector $\rho_v \in \mathbb{R}^L$ (fixed or learned).

The temporal dynamics are encoded by a linear Gaussian state-space model:

$t_d \in \{1,\ldots,T\}$0

where $t_d \in \{1,\ldots,T\}$1 is typically chosen as diagonal or isotropic.

At each time $t_d \in \{1,\ldots,T\}$2 and topic $t_d \in \{1,\ldots,T\}$3, the topic induces a categorical word distribution via:

$t_d \in \{1,\ldots,T\}$4

The document-level generative process draws a latent topic proportion vector $t_d \in \{1,\ldots,T\}$5, normalized as $t_d \in \{1,\ldots,T\}$6. For each token in document $t_d \in \{1,\ldots,T\}$7, the topic assignment $t_d \in \{1,\ldots,T\}$8 is sampled from $t_d \in \{1,\ldots,T\}$9, and the observed word $K$0 is drawn from $K$1 (Badekale et al., 8 Jul 2025).

2. Priors, Likelihoods, and Temporal Dynamics

The prior over topic trajectories is a discrete-time Gaussian random walk:

$K$2

Document-level priors are independent standard Gaussians:

$K$3

The likelihood of the observed words marginalizes over latent topic assignments:

$K$4

Temporal regularization enforces smoothness of topic evolution; empirically, this prevents dramatic or noisy topic shifts across time points, thus capturing meaningful trajectories in policy or lexical change (Badekale et al., 8 Jul 2025, Dieng et al., 2019, Fittschen et al., 27 Apr 2025, Sirin et al., 2024).

3. Variational Inference and Optimization

Posterior inference in DETM is intractable due to non-conjugate likelihoods and sequential dependencies. Variational inference employs an amortized, mean-field approximation:

$K$5

• Topic-trajectory encoder: An RNN (LSTM or GRU, often with $K$6) parametrizes $K$7 as a sequence of Gaussians conditioned on past states.
• Document encoder: A feedforward network generates parameters of $K$8 given the bag-of-words representation (Badekale et al., 8 Jul 2025, Dieng et al., 2019, Fittschen et al., 27 Apr 2025).

The Evidence Lower Bound (ELBO) incorporates expected log-likelihood and KL divergences:

$K$9

Optimization is performed via stochastic gradient ascent using the reparameterization trick for all latent Gaussian variables, typically with Adam as the optimizer. Mini-batching and backpropagation are utilized throughout (Badekale et al., 8 Jul 2025, Dieng et al., 2019).

4. Role of Embeddings and Semantic Coherence

Word embeddings $V$0 constitute the key innovation of DETM over classical topic models. Embedding-based parameterization allows topics to smoothly interpolate distributions over words according to geometric similarity in $V$1. This realizes two principal benefits:

• Semantic coherence: Words close in embedding space appear together in topics, resulting in higher topic coherence than count-based models.
• Semantic flexibility: Topics can shift continuously, enabling the model to capture subtle semantic drifts, polysemy, and rare word use if embeddings are pre-initialized.

Empirically, DETM achieves superior topic quality, measured by combined topic coherence and diversity metrics, and is robust to large vocabulary sizes when embeddings are pretrained and fixed (Dieng et al., 2019, Fittschen et al., 27 Apr 2025).

5. Empirical Properties, Implementation, and Scalability

Standard implementation details include:

• Embedding dimension $V$2
• Number of topics $V$3 (for policy tasks), up to $V$4–$V$5 for literary or scientific corpora
• Vocabulary sizes $V$6–$V$7, with approximate softmax for very large $V$8
• Training schedules of 5–1000 epochs; learning rate and regularization (dropout, $V$9 penalty) determined by validation loss
• Amortization of document-level variational parameters, RNN-based topic encoders, and the use of mini-batch stochastic optimization

Computationally, DETM requires less training time than dynamic LDA and its reparameterized variants, scaling efficiently to large document collections with temporal structure (Badekale et al., 8 Jul 2025, Dieng et al., 2019, Fittschen et al., 27 Apr 2025).

6. Applications and Model Extensions

DETM has been applied to:

• Climate policy discourse, revealing temporal shifts from greenhouse gas policy to finance and technical collaboration in UNFCCC documents (Badekale et al., 8 Jul 2025)
• Historical linguistics, tracing diachronic changes in classical and early Christian Latin, capturing standardization trends and lexical semantic shifts (Sirin et al., 2024)
• Comparative diachronic corpora, including UN debates, scientific literature, and literary texts, where it outperforms dynamic LDA on document completion and topic quality benchmarks (Dieng et al., 2019, Fittschen et al., 27 Apr 2025)

Extensions documented in recent research include integration with change-point detection via the “bimodality” statistic to reveal lexical sense shifts, the adaptation to non-uniform or sparse temporal windows, and scalable vocabulary handling via approximate normalization techniques (Sirin et al., 2024, Fittschen et al., 27 Apr 2025).

7. Limitations, Practical Recommendations, and Future Directions

Key practical findings indicate that:

• Most performance gains are due to increasing topic count and embedding dimension; other hyperparameters (loss reweighting, temporal binning, drift ratio) are less critical (Fittschen et al., 27 Apr 2025)
• Vocabulary size can be maximized up to GPU memory limits without degradation; recommended defaults are $L$0, pretrain and freeze embeddings
• DETM is robust to bin coarsening; 2–4 time bins suffice for most applications, though extensions to continuous-time or interpolated priors are under investigation

Current limitations include the necessity for curated, lemmatized corpora, sensitivity to OCR noise, and basic change-point detection that assumes a single shift per term. Proposed future work includes integrating sparsity-inducing priors, modeling multiple change points, and generalizing temporal modeling to continuous-time variants (Fittschen et al., 27 Apr 2025, Sirin et al., 2024).

---

References

• (Badekale et al., 8 Jul 2025)
• (Dieng et al., 2019)
• (Fittschen et al., 27 Apr 2025)
• (Sirin et al., 2024)