Dynamic KeyNMF Framework

• Dynamic KeyNMF is a topic modeling framework that unifies transformer-based keyword importance scoring with NMF to capture evolving topics in large text corpora.
• It factorizes a non-negative keyword-importance matrix using coordinate descent, yielding semantically rich topics compared to traditional token-count models.
• The dynamic phase slices the corpus over time and optionally enforces temporal smoothness, enabling fine-grained analysis of topic emergence and resonance.

Dynamic KeyNMF is a topic modeling framework that unifies transformer-based keyword importance scoring with non-negative matrix factorization (NMF) to enable tracking the evolution of topics across contiguous time periods in large text corpora. The central innovation lies in constructing a non-negative “keyword-importance” matrix using contextualized embeddings from pretrained transformers, enabling more semantically meaningful topic induction compared to token-count-based approaches. Dynamic KeyNMF incorporates both static and dynamic phases, with optional temporal regularization, and was designed to support high-resolution analysis of topical information dynamics in settings such as Chinese diaspora media in the leadup to electoral events (Kristensen-McLachlan et al., 2024).

1. Construction of the Keyword-Importance Matrix

Given a corpus of $D$ documents and a vocabulary $V$ of candidate keywords, an embedding model $f: \text{text} \to \mathbb{R}^d$ (e.g., multilingual Sentence-BERT) generates dense representations for each document and each token. For each document $d$, its embedding $x_d = f(d)$ is computed, as well as $v_w = f(w)$ for candidate tokens $w$. The set $K_d$ of top-$N$ tokens for $d$ is selected to maximize the sum of cosine similarities $\cos(x_d, v_w)$. The $D \times V$ keyword-importance matrix $M$ is then defined by:

$$M_{d, w} = \begin{cases} \max\{0, \cos(x_d, v_w)\}, & w \in K_d \ 0, & \text{otherwise} \end{cases}$$

This matrix is non-negative by construction. Unlike traditional document-term matrices filled by TF–IDF or raw counts, $M$ encodes semantic strength directly via contextual similarities.

2. Static KeyNMF Decomposition

The static phase factorizes $M$ into $W \in \mathbb{R}_{\ge 0}^{D \times K}$ and $H \in \mathbb{R}_{\ge 0}^{K \times V}$, where $K$ is the number of topics. The factorization minimizes the Frobenius norm:

$\min_{W \ge 0, H \ge 0} \| M - W H \|_F^2$

Optimization is performed via non-negative least squares, using coordinate-descent updates. Convergence is declared when the relative change in loss drops below $\epsilon$ (e.g., $10^{-4}$).

The coordinate-descent update for $H$ (for fixed $W$) is:

$H = \max\{0, H - \eta [W^\top W H - W^\top M]\}$

where $\eta$ is a small step-size. Updates for $W$ are symmetric.

3. Dynamic Slicing and Topic Evolution

To track topic evolution over $T$ time slices, Dynamic KeyNMF operates as follows:

• Compute the static factorization over the entire corpus to obtain $W$ and $H$.
• For each time slice $t$, extract $M_t$ (documents from slice $t$) and the corresponding $W_t$ (rows of $W$).
• Fix $W_t$ and re-solve for slice-specific $H_t$:

$H_t = \arg\min_{H' \ge 0} \| M_t - W_t H' \|_F^2$

• Define the raw “temporal importance” of topic $j$ at time $t$:

$I_{t j} = \sum_{d=1}^{D_t} (W_t)_{d j}$

• Normalize $I_{t \bullet}$ to obtain a pseudo-distribution:

$$\hat{P}_{t j} = \frac{I_{t j}}{\sum_{i=1}^K I_{t i}}$$

This pseudo-distribution enables downstream information dynamics analyses such as novelty, transience, and resonance.

4. Optional Temporal Smoothness Regularization

Dynamic KeyNMF supports, but does not require, temporal smoothness regularization over topic assignments across time slices. The joint dynamic-NMF objective adds slice-to-slice continuity penalties:

$$\min_{\{W_t, H_t\} \ge 0} \sum_{t=1}^T \| M_t - W_t H_t \|_F^2 + \lambda_W \sum_{t=2}^T \| W_t - W_{t-1} \|_F^2 + \lambda_H \sum_{t=2}^T \| H_t - H_{t-1} \|_F^2$$

where $\lambda_W$ and $\lambda_H$ are non-negative smoothness weights, typically chosen small ($\approx 0.01$–$0.1$) via cross-validation or grid search. The objective is optimized by alternating projected-gradient or coordinate-descent updates.

5. Preprocessing, Hyperparameters, and Practical Pipeline

Preprocessing steps include tokenization (e.g., jieba for Chinese), stopword removal, and candidate vocabulary construction. The main pipeline follows:

1. Embedding & Keyword Extraction: For each $d$, compute $x_d = f(d)$ and $v_w = f(w)$ for all $w$. Select top $N$ tokens per doc by $\cos(x_d, v_w)$.
2. Matrix Construction: Assemble $M$ as above.
3. Static NMF: Initialize $W$ and $H$ randomly, update via coordinate descent to convergence.
4. Dynamic Slicing: For each time $t$, extract $M_t$, $W_t$, solve for $H_t$; compute $\hat{P}_{t \bullet}$.
5. (Optional) Information Dynamics: Compute windowed Jensen–Shannon divergence between topic distributions for novelty, transience, and resonance analysis.

Key hyperparameters include $K$ (topics, typically 10–50), $N$ (keywords per doc, typically 15), $\epsilon$ (convergence threshold), $\eta$ (step size), $\lambda_W$, $\lambda_H$ (smoothness), $n$ (window size for novelty calculations), and the smoothing span for dynamics curves.

6. Computational Complexity and Scaling

• Embedding: $O(D \cdot \text{cost}_f)$ for $D$ documents.
• Keyword Selection: $O(D \cdot |d| \cdot d)$.
• Static NMF: Each iteration $O(KDV)$, for $I$ iterations.
• Dynamic H$_t$ Re-Estimation: Per slice, $O(K D_t V)$.
• Total: $O(I K D V + T K \bar{D} V)$, with $D$ documents, $T$ time slices, average slice size $\bar{D}$, and vocabulary size $V$.

For corpora with $D \approx 10^4$, $T \approx 200$, and $V \approx 5 \times 10^3$, static NMF computations dominate resource usage.

7. Evaluation Metrics and Benchmarks

Dynamic KeyNMF is evaluated using several metrics:

Metric	Description	Computed As
Topic-diversity ($d$)	Fraction of unique words in union of top $L$ topic words	Standard toolkit
Internal coherence ($C_{in}$)	Avg. pairwise cosine among top $L$ topic words	Standard toolkit
External coherence ($C_{ex}$)	Same as $C_{in}$ but using external embeddings	Standard toolkit
(Optional) NPMI	Normalized pointwise mutual information (coherence)	Standard toolkit

These allow comparison against LDA, CTM, BERTopic, Top2Vec, etc., using e.g., the topic-benchmark Python package.

Dynamic KeyNMF has demonstrated competitive performance on several Chinese datasets, substantiating its suitability for high-resolution analysis of topical information dynamics in multilingual and non-Western corpora (Kristensen-McLachlan et al., 2024).