Anchored Bayesian Gaussian Mixture Models

• ABGMMs introduce fixed anchor points to break label symmetry, ensuring coherent and interpretable inference on component-specific parameters.
• The approach employs methods like A-EM and case-deletion to select representative anchors that guide the allocation of data points reliably.
• Empirical evaluations confirm that ABGMMs generate unimodal posteriors and enhanced allocation certainty, aligning mixture components with true group structures.

Anchored Bayesian Gaussian Mixture Models (ABGMMs) address the issue of label identifiability in Bayesian finite mixture models by introducing a small set of fixed component allocations known as anchors. These predetermined assignments eliminate the posterior equivalence of label permutations—commonly known as label-switching—enabling direct, interpretable inference on component-specific parameters. The anchored approach represents a mathematically coherent alternative to post hoc relabeling, with well-defined probability models and rigorous guarantees on identifiability and inference quality (Kunkel et al., 2018, Kunkel et al., 2019).

1. Standard Exchangeable Bayesian Gaussian Mixture Models

Bayesian Gaussian Mixture Models (GMMs) for data $y_1, \ldots, y_n$ model the observed distribution as a weighted sum of $k$ Gaussian components. This can be expressed both marginally,

$$p(\mathbf y\mid\bm\mu, \bm\sigma^2, \bm\eta) = \prod_{i=1}^n \sum_{j=1}^k \eta_j\;\mathcal N(y_i\mid\mu_j,\sigma_j^2)$$

and with latent allocations $s_i \in \{1,...,k\}$,

$$P(s_i=j \mid \bm\eta) = \eta_j, \qquad y_i \mid s_i = j \sim \mathcal N(\mu_j, \sigma_j^2).$$

With exchangeable priors—such as $\mu_j \overset{\mathrm{iid}}{\sim} N(m_0, v_0)$, $$\sigma_j^{-2} \overset{\mathrm{iid}}{\sim} \Gamma(a, b)$$, and $\bm\eta \sim \mathrm{Dir}(\alpha, \ldots, \alpha)$—the model's joint distribution is invariant under permutation of component labels. Consequently, the posterior distribution is multimodal, each mode corresponding to a different permutation of the labels (a total of $k!$ modes). In MCMC-based inference, this symmetry produces label-switching in the Markov chain, yielding marginal distributions for $(\mu_j, \sigma_j^2)$ that are non-interpretable without further processing (Kunkel et al., 2019, Kunkel et al., 2018).

2. Anchor Sets and Breaking Label-Exchangeability

The central innovation of ABGMMs is the inclusion of anchor sets $A_1, \ldots, A_k$, where $A_j \subset \{1, \ldots, n\}$ contains indices of observations forced to be allocated to component $j$ with probability one. For $i \in A_j$, the allocation variable is deterministically set: $P(s_i = j) = 1$. For $i \notin A = \bigcup_j A_j$, the prior remains $P(s_i = j) = \eta_j$.

This modification yields the following structure for the complete-data distribution:

$$\widetilde\eta_{i,j} = \begin{cases} 1, & i \in A_j \ 0, & i \in A_{j'},\, j' \neq j \ \eta_j, & i \notin A \end{cases}$$

$$p(\bm\mu, \bm\sigma^2, \bm\eta, \mathbf s \mid \mathbf y) \propto \pi(\bm\eta) \prod_{j=1}^k \pi(\mu_j)\pi(\sigma_j^2) \prod_{i=1}^n \widetilde\eta_{i,s_i} \mathcal N(y_i \mid \mu_{s_i}, \sigma_{s_i}^2)$$

This anchoring is equivalent to encoding a strong, data-dependent informative prior on the labelings. Any allocation violating the anchor constraints has prior probability zero. Once each component has at least one (preferably a small number of) anchor assigned, the $k!$-fold symmetry is destroyed, and each component is affiliated to a unique cluster mode in the posterior (Kunkel et al., 2018).

3. Anchor Point Selection Methodologies

Two primary methodologies are established for the selection of effective anchor points:

3.1 Anchored Expectation–Maximization (A-EM)

This iterative approach alternates between responsibility calculation and anchor assignment:

• E-step: Compute responsibilities $$r_{i,j} = P(s_i = j | \bm\mu, \bm\sigma^2, \bm\eta)$$.
• Anchor-step: For each component $j$, select the $m_j$ observations with highest $r_{i,j}$, forming $A_j$ so as to maximize $\sum_{j=1}^k \sum_{i \in A_j} r_{i,j}$ under the constraint $A_j \cap A_{j'} = \varnothing$.
• M-step: Fit updated parameters given anchored responsibilities.

This scheme returns a locally optimal anchor assignment and a local posterior mode (Kunkel et al., 2019, Kunkel et al., 2018).

3.2 Case-Deletion Weight (CDW) Methods

• Fit a base model (e.g., least-squares regression), sample parameter draws via MCMC.
• For each case $i$, compute normalized case-deletion importance weights $\bar{w}_i(\theta_\ell)$.
• Compute the covariance (or correlation) matrix of $\{\log w_i(\theta_\ell)\}$ across draws.
• Cluster cases in the PCA-projected influence-profile space into $k$ groups, then designate $m_j$ anchor points per group using clustering centroids.

Variants include "CDW-cov" (covariance-based) and "CDW-cor" (correlation-based). The methodology ensures the selection of anchor points that typify or extremize component-specific influence profiles, yielding data-adaptive, robust anchor assignments (Kunkel et al., 2019).

4. Posterior Computation and Identifiability

Having fixed anchor sets, posterior inference proceeds with a standard Gibbs sampler:

• For $i \notin A$, sample allocation:

$$P(s_i = j \mid \mathbf y, \theta) \propto \eta_j \mathcal N(y_i \mid \mu_j, \sigma_j^2)$$

For $i \in A_j$, allocation is fixed: $s_i = j$.

• Given allocations, sample $(\mu_j, \sigma_j^2)$ from their conjugate normal–gamma posteriors, and $\bm\eta$ from a Dirichlet with counts augmented by anchor and non-anchor allocations.

The resulting posterior is approximately unimodal with respect to component parameters, and $P(s_i = j \mid \mathbf y)$ are sharply separated. This precludes label-switching during MCMC and produces componentwise parameter summaries that are directly interpretable (Kunkel et al., 2019, Kunkel et al., 2018).

Asymptotic identifiability is characterized by the quasi-consistency coefficient

$$\alpha = \max_{q=1, \ldots, K!} P\{\mathrm{mode} = q\}$$

where high $\alpha$ (close to 1) indicates that one labeling dominates the posterior. Empirically, one or two anchors per component suffice to approach near-perfect identifiability, while excessive anchoring degrades performance when component overlap is substantial (Kunkel et al., 2018).

5. Application: Allometric Data and Empirical Evaluation

A comprehensive case study involves modeling log brain mass versus log body mass for $n=100$ placental mammalian species. Standard linear regression reveals systematic residuals related to taxonomic order. A $k=3$ component anchored Bayesian regression mixture is fit, with $m_j = 3$ anchors per component. Priors are $\mu_j \sim N((3.5, 0.6), \mathrm{diag}(1, 0.5))$, $\sigma^{-2} \sim \Gamma(5, 1)$, and $\bm\eta \sim \mathrm{Dir}(1, 1, 1)$.

Anchor selection via A-EM yields confident, representative points per component; case-deletion methods (CDW-cov and CDW-cor) show tradeoffs regarding distribution of anchors and influence variance, but confirm that random or naive anchor selection fails to deliver well-separated allocations.

Posterior means correspond to:

• Component 1: slope ≈ 0.70 (covers Rodentia)
• Component 2: slope ≈ 0.74 (covers Artiodactyla, Carnivora, etc.)
• Component 3: slope ≈ 0.88–0.92 (isolates Primates, Cetacea, etc.)

A-EM achieves the highest allocation certainty, with similar qualitative results for CDW-cor; random anchors yield much lower certainty. Interpretability and biological taxonomic alignment are improved through anchoring, with component allocations reflecting true group structure (Kunkel et al., 2019).

Empirical evaluation on benchmark datasets (e.g., galaxies, SisFall) corroborates that anchored models match or exceed the interpretability of post hoc relabeling while being more principled and computationally efficient (Kunkel et al., 2018).

6. Comparison to Post Hoc Relabeling and Extensions

Traditional relabeling algorithms process samples from exchangeable models in an attempt to retroactively assign consistent labels to sampled parameters. Such transformations lack derivation from a coherent joint distribution and result in marginal inferences that do not correspond to a single Bayesian model.

In contrast, ABGMMs formalize label identifiability at the modeling stage. Marginals for $(\mu_j, \sigma_j^2)$ directly reflect the posterior distribution under the non-exchangeable anchored mixture. Extensions of the anchored approach include:

• Application to mixtures of non-Gaussian families (Poisson, skew-normal).
• Hierarchical and group-level mixture models, where a subset of labels (subjects) is anchored to known classes.
• Accommodation of improper priors once empty components are prevented via anchoring.

A plausible implication is that ABGMM methodology generalizes to any finite mixture of continuous densities, wherever data-dependent identifiability is required (Kunkel et al., 2018).

7. Summary and Practical Guidance

Anchored Bayesian Gaussian Mixture Models provide a rigorous, practical mechanism for achieving label identifiability in Bayesian mixture modeling. By enforcing small, data-driven anchor assignments, the approach eliminates the need for ad hoc post-processing, yields interpretable, unimodal posterior inferences for component parameters, and aligns mixture components with structured patterns of scientific or practical interest. Anchor selection via EM-responsibility or case-deletion diagnostics produces robust assignments, and one to two anchors per component typically suffice for identifiability. Excessive anchoring should be avoided in settings of substantial component overlap to preserve predictive accuracy (Kunkel et al., 2018, Kunkel et al., 2019).