FastAMI: Scalable Monte Carlo Estimation

• FastAMI is a Monte Carlo-based algorithm that efficiently estimates adjustment-for-chance metrics like AMI without building full contingency tables.
• It uses a low-dimensional sampling scheme with hypergeometric draws and Welford’s online algorithm to achieve controlled absolute or relative precision.
• Empirical benchmarks show FastAMI can be up to 50× faster than exact methods while maintaining high ranking accuracy in clustering comparisons.

FastAMI is a Monte Carlo-based algorithm designed for scalable, high-precision estimation of adjustment-for-chance metrics in clustering comparison, most notably Adjusted Mutual Information (AMI) and its variance-standardized extension, Standardized Mutual Information (SMI). The method directly addresses the computational bottleneck inherent to classical AMI on large or fine-grained clusterings by providing an efficient, statistically principled estimator that operates without building the full contingency table, thereby enabling reliable comparison at scales previously intractable with exact or pairwise methods (Klede et al., 2023).

1. Foundations: Mutual Information and Adjustment for Chance

Given a dataset $S=\{s_1,\dots,s_N\}$ and two hard clusterings $U\colon S\to\{1,\dots,R\}$ and $V\colon S\to\{1,\dots,C\}$, the empirical Mutual Information (MI) quantifies the shared information between the respective cluster assignments. For the contingency matrix $n_{ij} = |\{s: U(s) = i, V(s)=j\}|$, row and column marginals $a_i$, $b_j$, the MI is

$$I(U,V) = \sum_{i=1}^R\sum_{j=1}^C \frac{n_{ij}}{N} \log\frac{n_{ij} N}{a_i b_j}\,,$$

with null terms omitted. Adjustment for chance addresses the baseline similarity that arises even under random labeling. The core correction is to subtract the expected MI under random permutation of one clustering ($\pi(V)$), while fixing cluster-size marginals $A = \{a_i\}$, $B = \{b_j\}$:

$U\colon S\to\{1,\dots,R\}$0

AMI is then normalized by the mean entropy:

$U\colon S\to\{1,\dots,R\}$1

where $U\colon S\to\{1,\dots,R\}$2 and $U\colon S\to\{1,\dots,R\}$3 are the empirical entropies of clusterings $U\colon S\to\{1,\dots,R\}$4 and $U\colon S\to\{1,\dots,R\}$5.

2. FastAMI Monte Carlo Estimation

The core innovation in FastAMI is a reduction of the EMI estimation problem to a low-dimensional Monte Carlo (MC) sampling scheme, bypassing the $U\colon S\to\{1,\dots,R\}$6 cost of explicit permutations. Each MC sample is parameterized by three random variables: the size $U\colon S\to\{1,\dots,R\}$7 of a cluster in $U\colon S\to\{1,\dots,R\}$8, the size $U\colon S\to\{1,\dots,R\}$9 in $V\colon S\to\{1,\dots,C\}$0, and the overlap $V\colon S\to\{1,\dots,C\}$1 between them after permutation, with overlap distributed hypergeometrically. The expected value of the MI under random permutation can be represented as

$V\colon S\to\{1,\dots,C\}$2

where $V\colon S\to\{1,\dots,C\}$3 and $V\colon S\to\{1,\dots,C\}$4 are empirical distributions of cluster sizes and $V\colon S\to\{1,\dots,C\}$5 is the corresponding hypergeometric probability. Each draw produces

$V\colon S\to\{1,\dots,C\}$6

with online mean and variance tracked via Welford’s algorithm. Samples are drawn as follows:

• $V\colon S\to\{1,\dots,C\}$7 and $V\colon S\to\{1,\dots,C\}$8 by Walker’s alias method ($V\colon S\to\{1,\dots,C\}$9 per sample, $n_{ij} = |\{s: U(s) = i, V(s)=j\}|$0 setup)
• $n_{ij} = |\{s: U(s) = i, V(s)=j\}|$1, set $n_{ij} = |\{s: U(s) = i, V(s)=j\}|$2
• Online updates to mean and squared deviations

Sampling ceases when the estimated standard error $n_{ij} = |\{s: U(s) = i, V(s)=j\}|$3 satisfies $n_{ij} = |\{s: U(s) = i, V(s)=j\}|$4, for user-specified precision $n_{ij} = |\{s: U(s) = i, V(s)=j\}|$5. This ensures either absolute or relative precision.

3. Algorithmic Complexity and Comparative Analysis

A summary of algorithmic complexities is shown below:

Method	Time Complexity	Memory Complexity
Exact AMI	$n_{ij} = |\{s: U(s) = i, V(s)=j\}|$6	$n_{ij} = |\{s: U(s) = i, V(s)=j\}|$7
Pairwise AMI	$n_{ij} = |\{s: U(s) = i, V(s)=j\}|$8	$n_{ij} = |\{s: U(s) = i, V(s)=j\}|$9
FastAMI	$a_i$0	$a_i$1

For fixed relative or absolute error ($a_i$2), FastAMI scales as $a_i$3 in the regime of many clusters, with each MC sample $a_i$4 amortized. The theoretical upper bound on required samples is $a_i$5, though empirical convergence is often faster when cluster sizes are small.

4. Practical Implementation and Empirical Benchmarks

Workflow for practitioners implementing FastAMI includes:

• Precomputing cluster-size histograms and constructing alias tables for $a_i$6 and $a_i$7.
• Running the core MC loop with desired precision $a_i$8 to obtain $a_i$9 as an estimate for EMI.
• Computing $b_j$0 and applying the AMI formula:

$b_j$1

Empirical evaluation demonstrates:

• On synthetic data with $b_j$2 and large cluster counts ($b_j$3), FastAMI is up to 50× faster than exact methods, always meeting error targets (e.g., $b_j$4) and vastly outpacing pairwise AMI, which suffers from model-dependent bias.
• On medium UCI datasets (up to $b_j$5), FastAMI exactly matches the ranking (Spearman 1.0) of exact AMI, at about 4× runtime, still sub-millisecond per clustering pair.
• On large-scale SNAP graphs ($b_j$6), exact AMI is infeasible, pairwise AMI is memory intensive, while FastAMI completes in seconds with modest memory use, Spearman $b_j$7, and MC mean absolute error $b_j$8.

Recommended parameter values are $b_j$9 for AMI (guaranteeing $$I(U,V) = \sum_{i=1}^R\sum_{j=1}^C \frac{n_{ij}}{N} \log\frac{n_{ij} N}{a_i b_j}\,,$$0 absolute/relative error) and $$I(U,V) = \sum_{i=1}^R\sum_{j=1}^C \frac{n_{ij}}{N} \log\frac{n_{ij} N}{a_i b_j}\,,$$1 for SMI (absolute error $$I(U,V) = \sum_{i=1}^R\sum_{j=1}^C \frac{n_{ij}}{N} \log\frac{n_{ij} N}{a_i b_j}\,,$$2 sd).

5. Extension to Standardized Mutual Information (SMI)

SMI is defined as

$$I(U,V) = \sum_{i=1}^R\sum_{j=1}^C \frac{n_{ij}}{N} \log\frac{n_{ij} N}{a_i b_j}\,,$$3

and quantifies the adjusted MI in units of its own standard deviation under permutation. FastAMI extends to SMI (FastSMI) via either:

• Separate MC estimation of $$I(U,V) = \sum_{i=1}^R\sum_{j=1}^C \frac{n_{ij}}{N} \log\frac{n_{ij} N}{a_i b_j}\,,$$4 alongside $$I(U,V) = \sum_{i=1}^R\sum_{j=1}^C \frac{n_{ij}}{N} \log\frac{n_{ij} N}{a_i b_j}\,,$$5, followed by variance computation.
• Direct MC draws of full contingency tables (Patefield’s $$I(U,V) = \sum_{i=1}^R\sum_{j=1}^C \frac{n_{ij}}{N} \log\frac{n_{ij} N}{a_i b_j}\,,$$6 sampler), empirically yielding faster and more stable estimates.

6. Accuracy Guarantees, Error Bounds, and Limitations

FastAMI employs Welford’s online algorithm, providing normal-approximation confidence intervals: $$I(U,V) = \sum_{i=1}^R\sum_{j=1}^C \frac{n_{ij}}{N} \log\frac{n_{ij} N}{a_i b_j}\,,$$7, and guarantees that $$I(U,V) = \sum_{i=1}^R\sum_{j=1}^C \frac{n_{ij}}{N} \log\frac{n_{ij} N}{a_i b_j}\,,$$8 through the stopping rule. Theoretical sample size bounds can be pessimistic as fewer samples are empirically sufficient when cluster sizes are small. FastAMI achieves its memory efficiency by requiring only $$I(U,V) = \sum_{i=1}^R\sum_{j=1}^C \frac{n_{ij}}{N} \log\frac{n_{ij} N}{a_i b_j}\,,$$9 storage, avoiding explicit contingency tables. A plausible implication is that FastAMI is particularly well-suited for datasets with many small clusters.

7. Significance and Application Scope

FastAMI enables unbiased, scalable ground-truth evaluation and comparison of clusterings across the full spectrum of data sizes and cluster granularities. By restoring “adjustment for chance” to the large-data regime with a fully tunable accuracy–runtime trade-off, FastAMI outperforms the pairwise AMI shortcut both in bias and interpretability. Its performance characteristics—scalability, controlled statistical error, and moderate memory requirements—extend its applicability to problems on tens of millions of points, removing a key bottleneck in empirical clustering research (Klede et al., 2023).