Atom-Based Topic Modeling

• Atom-based topic representation is a framework that unifies sparse dictionary learning, word embeddings, and probabilistic generative modeling to define latent discourse atoms.
• The method employs K-SVD for sparse coding and smooth inverse frequency embeddings to derive and assign continuous topic vectors within text.
• Empirical evaluations show that DATM produces coherent, diverse topics that capture domain-specific axes such as gender, offering actionable insights for text mining.

Atom-based topic representation, as operationalized in Discourse Atom Topic Modeling (DATM), unifies sparse dictionary learning, word embeddings, and probabilistic generative modeling to characterize latent topics within text corpora. This approach represents topics as “discourse atoms”—vectors in the same continuous word embedding space as lexical items—supporting both sequence-based and distributional document representations. DATM offers computationally tractable tools for topic extraction and assignment, yields interpretable latent structures, and aligns quantitative topic features with domain-relevant axes such as gender.

1. Extraction of Discourse Atoms via Sparse Dictionary Learning

DATM identifies topics as “discourse atoms,” which are basis vectors derived through sparse dictionary learning applied to word embeddings. Given a word embedding matrix $W \in \mathbb{R}^{d \times V}$, with $V$ the vocabulary size and $d$ embedding dimensionality, DATM seeks an overcomplete dictionary $A \in \mathbb{R}^{d\times K}$ (with $K$ atoms) and a sparse coefficient matrix $S \in \mathbb{R}^{K \times V}$ such that

$$\min_{A,S} \| W - AS \|_F^2 \quad \text{subject to} \quad \forall v: \|s_v\|_0 \leq T,\ \forall k: \|a_k\|_2 = 1,$$

where $s_v$ is the $v$-th column of $S$, $\|\cdot\|_0$ counts nonzeros (enforcing sparsity $T$), and each atom $a_k$ is $\ell_2$-normalized. This problem is solved using K-SVD (Aharon et al.), alternating between sparse coding (Orthogonal Matching Pursuit) and dictionary updates via rank-one SVD. Hyperparameters $K$ and $T$ are selected to balance coherence, diversity, and coverage (Arseniev-Koehler et al., 2021).

2. Generative Modeling: Mapping Atoms to Word Distributions

Each discourse atom $a_k$ is interpreted as a latent “topic” vector $c$. The generative model for text emission follows Arora et al., postulating that the probability of word $w$ given context $c$ is:

$\Pr[w|c] \propto \exp(\langle c, w \rangle),$

with normalization $Z(c) = \sum_{v=1}^V \exp(\langle c, w_v \rangle)$. An enhanced emission model incorporates unigram frequencies:

$$\Pr[w|c_t] = \alpha p(w) + (1-\alpha) \frac{\exp(\langle \hat{c}_t, w \rangle)}{Z(\hat{c}_t)},$$

where $\hat{c}_t = \beta c_0 + (1-\beta) c_t$ and $c_0 \perp c_t$. By setting $c_t = a_k$, one derives a probabilistic word distribution for each atom.

3. Inference: Assigning Atoms to Text and Document Representation

DATM represents text fragments and documents via their closest discourse atoms:

• Window Embedding: For a context window $C = \{w_i\}$, the SIF (Smooth Inverse Frequency) embedding

$$\hat{c} = \sum_{w \in C} \left[ \frac{a}{p(w)+a} \right] w$$

is calculated, where $a = (1-\alpha)/(\alpha Z)$, followed by projection of the top principal component $u$ (global component) to obtain $c = \hat{c} - (u^T \hat{c})u$.

• Atom Assignment: Each window is associated to atom $k^* = \arg\max_k \cos(a_k, c)$.
• Document Modeling: Documents are encoded either as ordered atom sequences or as normalized histograms $\theta \in \Delta^{K-1}$ of atom frequency. Windows are typically of fixed length (e.g., 10 tokens).

4. Computational Workflow and Complexity

The DATM pipeline comprises iterative stages:

Stage	Core Steps	Complexity
Training	Preprocessing, word2vec training, K-SVD	$$O(\textrm{iter} \cdot V \cdot T \cdot d + K \cdot d^2)$$ per K-SVD iter.
Inference	Sliding window SIF, atom assignment	SIF: $O(L \cdot d)$; Atom search: $O(K \cdot d)$ per window

Preprocessing includes phrase merging and tokenization. Principal component analysis is employed to identify global components for SIF. Efficient sparse coding and indexed nearest-neighbor search can accelerate atom assignment.

5. Evaluation Metrics and Empirical Findings

DATM’s performance is assessed via:

Metric	Definition
Coherence	Average pairwise Pointwise Mutual Information among atom top-N words
Diversity	Fraction of unique words across all atom top lists
Coverage	$1 - \|W-AS\|_F^2/\|W\|_F^2$ (dictionary fit to embedding space)

In the NVDRS study ($\sim$300,000 narratives), $K=225$ atoms was identified as optimal. The resulting topics reflect interpretable themes (e.g., weapons, substance use, family). A latent “gender” axis (difference of feminine vs. masculine pronoun/noun average vectors) reveals strong correlation (Spearman $\rho = 0.69$, $p < 10^{-4}$) with atom prevalence in male vs. female victim narratives. Case studies confirm that atoms capture expected gender-linked themes (e.g., “rifles & shotguns” as masculine, “pain medication” as feminine) (Arseniev-Koehler et al., 2021).

6. Advantages and Limitations

DATM provides several notable strengths:

• Coexistence of topics and words in a shared continuous space enables fine-grained measurement of topic–topic and topic–dimension relations.
• SIF weighting reduces reliance on manually curated stopword lists.
• Framework leverages theoretically grounded latent variable (Arora et al.) and efficient sparse dictionary learning (K-SVD).
• Flexible document representations: sequence or distributional. Key limitations include:
• Topic quality is contingent on the embedding quality; domain mismatch or poor embedding hampers results.
• Hyperparameter selection (K, T) introduces tuning complexity.
• Scalability of K-SVD is limited for very large vocabulary or atom sets.
• Unlike fully Bayesian topic models, DATM does not yield posterior uncertainty estimates.

7. Context and Scope

DATM constitutes a unified, generative, and inferential framework for atom-based topic representation. Topics emerge from the structure of the embedding space rather than discrete assignment models, enabling quantitative examination of semantic axes (e.g., gender, class) and latent patterns not explicated by structured variables. Empirical findings indicate that atom-based topic representation produces coherent, diverse, and semantically robust topical structures that can reveal latent social patterns. A plausible implication is that such techniques may inform large-scale text mining applications where interpretable, embedding-driven topic vectors are necessary for downstream quantitative analyses (Arseniev-Koehler et al., 2021).