Partitioned-LDA: Scalable Parallel LDA

• Partitioned-LDA is a parallelization strategy that divides the document–word matrix into non-overlapping blocks, enabling concurrent topic sampling and reducing synchronization delays.
• It introduces deterministic (A1, A2) and randomized (A3) algorithms that optimize workload distribution, resulting in improved load-balancing ratios and near-linear speedup.
• The method extends to LDA variants like Bag of Timestamps, maintaining model quality and statistical fidelity while scaling efficiently to large datasets.

Partitioned-LDA is a parallelization strategy and set of partitioning algorithms for improving the computational efficiency and load balancing of Latent Dirichlet Allocation (LDA) and LDA-like topic models. Partitioned-LDA operates by dividing the document–word (or related) matrix into non-overlapping blocks so that computations—including Gibbs sampling for topic assignments—can proceed in parallel with minimized waiting time and overhead. Central to Partitioned-LDA are three partitioning algorithms that optimize the distribution of the workload, quantified by the load-balancing ratio, across concurrent processes. This enables scalable and efficient inference, particularly in large-scale data applications.

1. Parallelization of Topic Modeling: Motivation and Problem Statement

Parallelizing LDA presents fundamental challenges related to data dependencies and process synchronization. In standard approaches, the document–word matrix is split into $P \times P$ blocks for $P$ parallel processes. Yan et al.'s diagonal partitioning allows groups of partitions to be sampled synchronously, provided their respective document and word subsets are disjoint. However, workload imbalances—where one process must handle disproportionately many tokens—lead to bottlenecks, as all processes must wait for the slowest partition. Formally, for a workload matrix $R = (r_{jw})$ (number of times word $w$ appears in document $j$), the partition cost $C_{m n}$ for the block indexed by document group $J_m$ and word group $V_n$ is: $C_{m n} = \sum_{j \in J_m, w \in V_n} r_{j w}$ Each diagonal epoch's cost is taken as the maximum among its blocks, and total cost: $$C = \sum_{l=0}^{P-1} \max_{(m, n): m \oplus l = n} C_{m n}$$ The ideal balanced cost is: $C_{\text{opt}} = \frac{\sum_{j, w} r_{j w}}{P}$ and the load-balancing ratio is $\eta = C_{\text{opt}}/C$. $\eta$ close to 1 ensures minimal excess waiting and almost linear speedup.

2. Partitioning Algorithms for Load Balancing

Three algorithms are introduced for partitioning the workload matrix:

A. Deterministic Partitioning (A1, A2)

• A1 (Heuristic 1): Sort rows and columns by token count, interleave longest and shortest elements (e.g., longest, shortest, 2nd longest, etc.), and partition into $P$ groups each with approximately equal token sum. This method quickly achieves balanced partitions with a single pass.
• A2 (Heuristic 2): Interleave from both ends more thoroughly (e.g., longest, shortest, second longest, second shortest), then partition as in A1. This variant addresses cases with more extreme token imbalances.

Algorithm	Approach	Partitioning Strategy
A1	Heuristic, Interleave	Pair longest-shortest, one pass
A2	Heuristic, Bidirectional	Deeper interleave from both ends

B. Randomized Partitioning (A3)

• A3 (Heuristic 3): Sort rows/columns, split into groups, randomly shuffle within each, and concatenate. This is repeated multiple times and the partition with the highest $\eta$ is selected. Though randomized, it maintains comparable runtime to prior methods but yields consistently higher load-balancing ratios.

Algorithm	Approach	Main Advantage
A3	Randomized	Attains highest $\eta$

Partitioning steps are applied independently to both rows (documents) and columns (words), preparing the matrix for parallel block-diagonal sampling.

3. Extension to LDA Variants: Bag of Timestamps (BoT)

Partitioned-LDA extends naturally to LDA-like models which incorporate additional modalities. Bag of Timestamps (BoT) represents each document not only by its words but also by associated timestamps (both sharing the document-topic distribution $\theta$; timestamps possess their own topic-specific distribution $\pi$ with prior $\gamma$). Partitioning proceeds independently for both the standard document–word matrix and the document–timestamp matrix. Blocks are sampled in parallel by applying the same strategies, and load balancing is achieved for both modalities.

4. Performance Analysis: Load-Balancing Ratio, Speed, and Quality

Experimental results across classical datasets (NIPS, NYTimes) and a large publication corpus (MAS, with >1 million documents for BoT) demonstrate consistent improvements:

• For NIPS (P=60): Baseline $\eta \approx 0.57$; A1 $\eta \approx 0.7126$; A2 $\eta \approx 0.7097$; A3 $\eta \approx 0.7553$.
• Near-linear speedup: effective speedup is $\eta \times P$.
• Partitioning time: deterministic A1/A2 are two orders of magnitude faster than previous randomized approaches; A3 provides higher $\eta$ at similar total effort.
• Model quality: no degradation in topic quality or perplexity. For BoT (MAS dataset) perplexity is 595 (serial) and 593.9–595.1 (parallel), indicating statistical fidelity is maintained, if not slightly improved.

Dataset	Baseline $\eta$	A1	A2	A3
NIPS	0.57	0.7126	0.7097	0.7553

Partitioned-LDA minimizes process waiting and maximizes utilization, enabling practical parallelization for large datasets and complex topic models.

5. Operational Significance and Extensibility

Partitioned-LDA's partitioning paradigm is applicable beyond standard LDA, benefiting extensions including models that incorporate temporal, spatial, or other structured information. The permutation-and-partition principle is generic and can be used for any model where the sampling or update structure admits non-conflicting groupings. The approach is not tied to a particular sampler: the improved load balancing can be plugged into any parallel LDA implementation, including those leveraging Pólya Urn techniques or clustered allocations. The extensibility is confirmed by direct experiments on models such as BoT.

6. Mathematical Formulation and Interpretation

Key formulas:

• Cost per epoch: $$C = \sum_{l=0}^{P-1} \max_{(m, n): m \oplus l = n} C_{mn}$$
• Ideal cost: $C_{\text{opt}} = (\sum_{j, w} r_{j w}) / P$
• Load-balancing ratio: $\eta = C_{\text{opt}} / C$
• Perplexity: ${\text{Perp}(x) = \exp( - \frac{1}{N} \log p(x) )}$, with $$\log p(x) = \sum_{j, i} \log \sum_k (\theta_{k|j} \phi_{x_{ji}|k})$$

These metrics quantify both computational efficiency (through $\eta$ and speedup) and statistical model fidelity (through perplexity).

7. Summary and Impact

Partitioned-LDA introduces a systematic solution to parallelization bottlenecks in topic modeling. By optimizing the distribution of tokens across processes using deterministic and randomized algorithms, it achieves superior load balancing and runtime performance without sacrificing model quality. Its extensibility to advanced topic models underscores its utility as a scalable backbone for large-scale text analysis, providing near-linear speedup and robust, statistically sound outcomes (Tran et al., 2015).