---
title: Dynamic Bernoulli Embeddings
url: https://www.emergentmind.com/topics/dynamic-bernoulli-embeddings
type: topic
---

# Dynamic Bernoulli Embeddings

Dynamic Bernoulli Embeddings (DBE) are a probabilistic framework for modeling temporally-evolving word or node embeddings based on observed co-occurrences. Originally developed for diachronic word semantics, DBE combines a Bernoulli likelihood for observed context with a time-indexed Gaussian random-walk prior, enforcing temporal smoothness while capturing lexical or structural drift. The DBE model has been applied to language evolution [1703.08052], sparse temporal text [1909.01863, 1907.09169], and dynamic networks [1906.09860].

## 1. Probabilistic Formulation

Let \( T \) denote the number of time slices, \( L \) the vocabulary size (or number of nodes), and \( d \) the embedding dimension. For each word (or node) \( w \) at time \( t \), DBE introduces a temporal embedding \( \rho_{t, w} \in \mathbb{R}^d \) and a context embedding \( \alpha_i \in \mathbb{R}^d \), with \( i \) ranging over word or node indices. For each observed pair \((w, i)\) at time \( t \):

\[
p(x_{t,w,i}=1 \mid \rho_{t,w}, \alpha_i) = \sigma(\rho_{t,w}^\top \alpha_i)
\]
where \( \sigma(z) = \frac{1}{1 + e^{-z}} \).

The complete-likelihood for all observations is given by:

\[
\prod_{t=1}^T \prod_{w=1}^L \prod_{i=1}^L \sigma(\rho_{t,w}^\top \alpha_i)^{x_{t,w,i}} [1-\sigma(\rho_{t,w}^\top \alpha_i)]^{1-x_{t,w,i}}
\]

Temporal smoothness is encoded via a zero-mean Gaussian prior on \( \rho_{1,w} \) and a Markovian random-walk prior for \( t>1 \):

\[
p(\rho_{1,w}) = \mathcal{N}(0, \lambda_0^{-1} I), \quad p(\rho_{t,w} \mid \rho_{t-1,w}) = \mathcal{N}(\rho_{t-1,w}, \lambda^{-1} I)
\]
and
\[
p(\alpha_i) = \mathcal{N}(0, \lambda_0^{-1} I)
\]

This coupling enforces that most words/nodes evolve gradually. Equivalent objectives arise in both word embedding and dynamic network contexts [1703.08052, 1906.09860, 1909.01863].

## 2. MAP Learning and Negative Sampling

Maximum a posteriori (MAP) estimation is used by minimizing:

\[
\mathcal{L}_{\text{DBE}}(\rho, \alpha) = -\sum_{t=1}^T \sum_{w=1}^L \sum_{i=1}^L \Big[ x_{t,w,i} \log \sigma(\rho_{t,w}^\top \alpha_i) + (1-x_{t,w,i}) \log(1-\sigma(\rho_{t,w}^\top \alpha_i)) \Big] + \mathcal{R}(\rho, \alpha)
\]

where the quadratic regularizer is

\[
\mathcal{R}(\rho, \alpha) = \frac{\lambda_0}{2} \sum_w \|\rho_{1,w}\|^2 + \frac{\lambda}{2} \sum_{t=2}^T \sum_w \|\rho_{t,w} - \rho_{t-1,w}\|^2 + \frac{\lambda_0}{2} \sum_i \|\alpha_i\|^2
\]

Exact likelihood sums are replaced by negative sampling: for each positive \((w, i)\), a fixed number of negatives are sampled (from a unigram or power-law distribution). The stochastic objective is optimized using minibatch SGD or Adam. Notably, DBE requires no Kalman filtering, variational Bayes, or explicit post-hoc alignment across time [1703.08052, 1909.01863, 1906.09860].

## 3. Extensions: Data Scarcity, Initialization, and Drift Regularization

When corpora are sparse, DBE model robustness depends crucially on initialization and regularization [1909.01863, 1907.09169]:

- **Initialization**:
  - *Random*: Embeddings are initialized independently from a Gaussian.
  - *Internal*: A static Bernoulli embedding is fit to the full corpus, initializing \( \rho_{1,w} \), or all \( \rho_{t,w} \).
  - *Backward external*: Pretrained embeddings (e.g. Wikipedia) are loaded for \( \rho_{T,w} \), then the model is run backwards in time.

- **Drift Threshold Regularization**: To better discriminate between stable and high-drift words/nodes under high sparsity, a drift-penalty using the HardShrink operator is added:
  \[
  \mathrm{Reg}_\beta = \alpha_{\rm reg} \sum_{t>1} \sum_w \mathrm{HardShrink}(\| \rho_{t,w} - \rho_{t-1,w} \|, \beta)
  \]
  where \(\beta\) is the mean empirical drift, and HardShrink suppresses minor fluctuations while capping outliers.

- **Drift Prior Variants**: Comparative evaluation of chronological (DBE), non-chronological (\(\ell_2\) anchoring to \( t=0 \)), semi-chronological, and incremental (no penalty) priors reveals only the standard DBE model achieves directed, smooth, and robust drift trajectories in empirical word drift detection [1907.09169].

## 4. Empirical Performance and Qualitative Analysis

### Predictive power:

DBE achieves superior or comparable held-out log-likelihood on positive examples under data scarcity:

| Model            | Random init | Internal | Backward external |
|------------------|:----------:|:--------:|:-----------------:|
| ISG (SGNS incr.) | –3.17      | –2.59    | –2.69             |
| DSG (dyn. SGNS)  | –0.749     | –0.686   | –0.695            |
| DBE              | –2.935     | –2.236   | –2.459            |

Internal initialization is most beneficial for short, low-variance periods, while backward external becomes advantageous with longer periods or larger slices [1909.01863].

### Directed drift and stability:

- **Directed drift:** On moderate sparsity (10% data), both DBE and dynamic SGNS (DSG) recover monotonic drift, whereas at extreme sparsity (1%), only DBE/DSG maintain directionality—incremental SGNS fails due to noise dominance.
- **Stability vs. extreme drifts:** The Gaussian-walk prior clusters the majority of words around small drift values. Under extreme scarcity, drift distribution tails contract and model discriminability for highly drifting entities drops, but introducing HardShrink-based penalization restores separation between extreme and stable elements [1909.01863, 1907.09169].

### Semantic and node trajectory case studies:

In congressional debates or scientific corpora, DBE captures smooth trajectories: e.g., "computer" transitions from occupational to technological senses; "bush" evolves from plant-names to political context [1703.08052]. In dynamic networks, evolving nodes (e.g., students during recess, researchers switching departments) trace interpretable paths in the learned space, corroborated by metadata [1906.09860].

## 5. Applications and Evaluation Protocols

### Textual data:

- Analyzing semantic change across decades of legislative, scientific, or news text.
- Cross-lingual comparison using alignment-initialized DBEs, revealing both convergent and divergent semantic evolution (e.g., "Barbie" anchored to a war criminal in French vs. rapid drift toward the fashion toy in English) [1907.09169].

### Network data:

- **Link prediction:** DBE outperforms static and other dynamic baselines on network evolution benchmarks, as measured by AUC in predicting emerging links across multiple real-world graphs [1906.09860].
- **Evolving-node detection:** Nodes with top drift metrics are interpretable as "active" or "evolving"; DBE achieves higher mean average precision and recall relative to alternatives, especially at low sample sizes.
- **Visualization:** Embedding trajectories reveal dynamic group structure, mobility, and organizational change.

## 6. Hyperparameter Choices and Training Details

Practical DBE instantiations use embedding dimensions of \( d=100 \) (language) or \( d=128 \) (network), context windows \( c=4 \), negative samples per positive ranging from 1 to 20, and drift regularization coefficients (e.g., \( \lambda = 1 \), \( \lambda_0 = \lambda / 1000 \)). Optimization is performed with SGD or Adam, using mini-batches interleaved over time slices and early stopping on validation log-likelihood [1703.08052, 1909.01863, 1906.09860, 1907.09169].

## 7. Significance and Theoretical Positioning

Dynamic Bernoulli Embeddings provide a principled method for learning temporally-indexed embeddings that achieve temporal continuity, directed semantic drift, and robust separation between stable and rapidly-evolving entities. The Gaussian random-walk prior is essential for this behavior. Unlike approaches requiring explicit time-step alignment or expensive variational frameworks, DBE leverages a simple and scalable MAP objective with negative sampling. This balance of statistical structure, computational tractability, and empirical effectiveness positions DBE as a canonical method for temporally-aware representation learning on both text and network data [1703.08052, 1909.01863, 1906.09860, 1907.09169].

Source: https://www.emergentmind.com/topics/dynamic-bernoulli-embeddings