Autoencoder Diversity Score (ADS)

• ADS is a family of metrics that quantifies how broadly and uniformly data or generated outputs occupy a learned latent space.
• Methods include entropy-based measures, kernel discrepancy, posterior diversity, and bottleneck decorrelation, each targeting different aspects of diversity.
• Applications range from anomaly detection and image generation to text and molecular autoencoding, emphasizing trade-offs between fidelity and feature diversity.

Searching arXiv for papers relevant to “Autoencoder Diversity Score (ADS)” and closely related latent-space diversity metrics. Autoencoder Diversity Score (ADS) is not a standardized term in the arXiv literature. Across the relevant work, the phrase is either absent, used for a different concept, or only approximately applicable. What does recur is a set of technically related constructions that assess diversity through encoded representations, posterior dispersion, bottleneck decorrelation, sparse feature coverage, or distributional coverage in latent space. This suggests that ADS is best treated as a family resemblance concept: a score derived from an autoencoder or autoencoder-like representation that quantifies how broadly, distinctly, or uniformly data or generations occupy a learned space (Koizumi et al., 2019, Ibarrola et al., 2024, Turinici, 2022).

1. Terminological status and major formulations

The supplied literature does not contain a single canonical metric literally named “Autoencoder Diversity Score.” In “Batch Uniformization for Minimizing Maximum Anomaly Score of DNN-based Anomaly Detection in Sounds,” ADS means anomaly detection in sounds, and the core issue is not generative diversity but the unequal anomaly scores assigned to frequent-normal and rare-normal sounds by an autoencoder normal model (Koizumi et al., 2019). In “Measuring Diversity in Co-creative Image Generation,” the authors explicitly state that they do not define a metric literally called Autoencoder Diversity Score (ADS) and do not use an autoencoder; instead they introduce entropy-based diversity measures in pretrained latent spaces (Ibarrola et al., 2024). Several other papers similarly provide the closest equivalent to ADS rather than the exact term: kernel discrepancy in VAE latent space (Turinici, 2022), Mutual Posterior Diversity (MPD) in VAE posteriors (Shen et al., 2021), pairwise bottleneck decorrelation in deterministic autoencoders (Laakom et al., 2022), and sparse feature coverage in sparse autoencoders (Yang et al., 19 Feb 2025).

Formulation	Core quantity	Representative paper
Score uniformization	Weighted anomaly score over normal support	(Koizumi et al., 2019)
Embedding entropy	Truncated entropy of latent covariance	(Ibarrola et al., 2024)
Measure discrepancy	Kernel distance between empirical latent measure and target	(Turinici, 2022)
Posterior diversity	Expected symmetric KL between posteriors	(Shen et al., 2021)
Bottleneck decorrelation	Sum of pairwise covariance terms	(Laakom et al., 2022)
Sparse feature coverage	Novelty and overlap of active SAE features	(Yang et al., 19 Feb 2025)

Taken together, these works suggest that “ADS” is not a single metric class but a design space whose members differ in whether they measure diversity of outputs, diversity of latent codes, diversity of encoder posteriors, or diversity of feature usage.

2. Uniformity and support coverage as early autoencoder-related precursors

The sound-anomaly paper is important because it links an autoencoder score to support coverage rather than to mean behavior. In frame-wise unsupervised anomaly detection in sounds, the anomaly score is the reconstruction error,

$$\mathcal{A}_{\theta} \left( \bm{x}_t \right) = \left\lVert \bm{x}_t - \mathcal{D}_{\theta_D}( \mathcal{E}_{\theta_E} (\bm{x}_t) ) \right\rVert _2 ^2,$$

and sequence-level detection is governed by the worst frame,

$$\mathcal{A}_{\theta}( \bm{X} ) = \max_{ \bm{x}_t } \mathcal{A}_{\theta} \left( \bm{x}_t \right).$$

The paper argues that minimizing the mini-batch mean anomaly score biases training toward frequent-normal samples and leaves rare-normal samples with larger scores, which then dominate the maximum over time (Koizumi et al., 2019).

The proposed batch uniformization (BU) reweights normal samples by the reciprocal of an estimated density so that rare-normal sounds receive larger weights. The concrete objective is

$\mathcal{J}_{\theta}^{\mbox{\tiny BU}} = \frac{1}{\sum_{i=1}^{M_u} w_i } \sum_{i=1}^{M_u} w_i \mathcal{A}_{\theta} \left( \bm{x}_i^{(u)} \right) - \mathcal{L}_a,$

with

$$w_i = \left( \mathcal{K} ( \bm{x}_i^{(u)} ) + \epsilon \right)^{-1}, \qquad \mathcal{K} \left( \bm{x}_i^{(u)} \right) = \frac{1}{M_u} \sum_{j=1}^{M_u} \exp \left\{ -\sigma \left\lVert \bm{x}_{i}^{(u)} - \bm{x}_{j}^{(u)} \right\rVert_2^2 \right\}.$$

The target intuition is a model density that is more uniform over the support of normal data. Although the paper is not about diversity scoring in the generative sense, it supplies a strong precursor: diversity can be operationalized as equitable coverage of low-density yet legitimate regions rather than as concentration on the modal region alone (Koizumi et al., 2019).

This suggests one ADS lineage in which “diversity” means support uniformization: encoded or reconstructed data should not overfit frequent modes at the expense of rare but valid structure.

3. Entropy and discrepancy in latent space

A more direct diversity literature measures the spread of latent codes. “Measuring Diversity in Co-creative Image Generation” defines Truncated Inception Entropy (TIE) and Truncated CLIP Entropy (TCE). Under a Gaussian approximation to encoded image features, the full differential entropy is

$h(\pi_f) = \frac{1}{2}\log\det(2\pi e \Sigma_i),$

and the truncated diversity scores for small sample sets are

$$\text{TIE}_K(A) \doteq \frac{K}{2}\log(2\pi e) + \frac{1}{2}\sum_{k=1}^{K}\log\lambda^{(i)}_k,$$

$$\text{TCE}_K(A) \doteq \frac{K}{2}\log(2\pi e) + \frac{1}{2}\sum_{k=1}^{K}\log\lambda^{(c)}_k.$$

These are not autoencoder-based, but they are an explicit latent-space diversity formalism in which diversity is the entropy of the covariance structure of a set of encodings (Ibarrola et al., 2024).

A second latent-space view appears in “Diversity in deep generative models and generative AI.” There, diversity is defined by how well a generated latent set approximates a target latent distribution as a whole. For generated latent points $X_1,\dots,X_J$ and reference points $z_1,\dots,z_B$, the central discrepancy is

$$d\left( \frac{1}{J}\sum_{j=1}^J \delta_{X_j}, \frac{1}{B}\sum_{b=1}^B \delta_{z_b} \right)^2 = \frac{\sum_{j,b=1}^{J,B} h(X_j-z_b) }{J B} - \frac{\sum_{j,j'=1}^{J} h(X_j-X_{j'})}{2 J^2} - \frac{\sum_{b,b'=1}^{B} h(z_b-z_{b'})}{2 B^2},$$

with

$$\mathcal{A}_{\theta}( \bm{X} ) = \max_{ \bm{x}_t } \mathcal{A}_{\theta} \left( \bm{x}_t \right).$$0

The paper’s closest ADS-like construction is the negative kernel distance between the empirical distribution of generated latent codes and the target latent distribution (Turinici, 2022).

A third, non-autoencoder but structurally transferable formulation appears in SDICE, which compares similarity-score distributions between synthetic and real datasets in a learned representation space. Its normalization uses

$$\mathcal{A}_{\theta}( \bm{X} ) = \max_{ \bm{x}_t } \mathcal{A}_{\theta} \left( \bm{x}_t \right).$$1

with separate intra-class and inter-class scores. This provides a template for an ADS-style metric based on matching latent similarity distributions rather than only counting distinct samples (Alam et al., 2024).

4. Posterior diversity and bottleneck decorrelation

Within variational autoencoders, diversity is often defined at the level of encoder posteriors rather than decoded outputs. “Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness” adopts Mutual Posterior Diversity (MPD),

$$\mathcal{A}_{\theta}( \bm{X} ) = \max_{ \bm{x}_t } \mathcal{A}_{\theta} \left( \bm{x}_t \right).$$2

together with Conditional Entropy (CE),

$$\mathcal{A}_{\theta}( \bm{X} ) = \max_{ \bm{x}_t } \mathcal{A}_{\theta} \left( \bm{x}_t \right).$$3

Its key claim is that a better latent representation should be more diverse across samples and less uncertain within each sample. The paper also gives a lower bound,

$$\mathcal{A}_{\theta}( \bm{X} ) = \max_{ \bm{x}_t } \mathcal{A}_{\theta} \left( \bm{x}_t \right).$$4

which makes the variance of posterior means a simple surrogate for posterior diversity (Shen et al., 2021).

A deterministic counterpart appears in “Reducing Redundancy in the Bottleneck Representation of the Autoencoders,” which adds a bottleneck regularizer based on centered pairwise covariance,

$$\mathcal{A}_{\theta}( \bm{X} ) = \max_{ \bm{x}_t } \mathcal{A}_{\theta} \left( \bm{x}_t \right).$$5

and augments the reconstruction loss as

$$\mathcal{A}_{\theta}( \bm{X} ) = \max_{ \bm{x}_t } \mathcal{A}_{\theta} \left( \bm{x}_t \right).$$6

The paper’s interpretation is explicit: penalizing cross-covariance forces the encoder to learn a more diverse and richer representation and less correlated non-redundant features (Laakom et al., 2022).

These two lines are conceptually distinct. MPD measures how much different inputs induce different posterior distributions; bottleneck decorrelation measures how non-redundant latent coordinates are across the dataset. A plausible implication is that ADS can target either sample separation or feature independence, and the two need not coincide.

5. Sparse autoencoders, feature coverage, and domain-specific diversity

Sparse autoencoders introduce a feature-coverage notion of diversity. “Diversity-driven Data Selection for LLM Tuning through Sparse Autoencoder” trains a TopK-SAE on transformer residual streams and uses activated sparse features to measure data diversity. The encoding is

$$\mathcal{A}_{\theta}( \bm{X} ) = \max_{ \bm{x}_t } \mathcal{A}_{\theta} \left( \bm{x}_t \right).$$7

with reconstruction loss

$$\mathcal{A}_{\theta}( \bm{X} ) = \max_{ \bm{x}_t } \mathcal{A}_{\theta} \left( \bm{x}_t \right).$$8

and inference-time thresholding via

$$\mathcal{A}_{\theta}( \bm{X} ) = \max_{ \bm{x}_t } \mathcal{A}_{\theta} \left( \bm{x}_t \right).$$9

The closest equivalent to ADS is the novelty and overlap of active sparse feature sets $\mathcal{J}_{\theta}^{\mbox{\tiny BU}} = \frac{1}{\sum_{i=1}^{M_u} w_i } \sum_{i=1}^{M_u} w_i \mathcal{A}_{\theta} \left( \bm{x}_i^{(u)} \right) - \mathcal{L}_a,$0 and $\mathcal{J}_{\theta}^{\mbox{\tiny BU}} = \frac{1}{\sum_{i=1}^{M_u} w_i } \sum_{i=1}^{M_u} w_i \mathcal{A}_{\theta} \left( \bm{x}_i^{(u)} \right) - \mathcal{L}_a,$1, with example-level diversity captured by new feature coverage rather than by distances or entropy (Yang et al., 19 Feb 2025).

In molecular autoencoding, diversity is evaluated through repeated stochastic decoding around a latent seed. “Improving Chemical Autoencoder Latent Space and Molecular De novo Generation Diversity with Heteroencoders” does not define a unified ADS, but it separates string-level variation, molecular-level diversity, latent-space organization, and alignment between latent distance and chemically meaningful similarity. Heteroencoders using enumerated SMILES in the decoder increase output variability and improve the similarity between latent-space distance and molecular similarity measured as circular fingerprints, but they also raise the rate of decoding to a different molecule than encoded (Bjerrum et al., 2018).

For text generation, “Recurrence Boosts Diversity!” connects diversity to latent-variable utilization in Transformer VAEs. Its recurrent segment-wise latent variables, residual posterior parameterization, and non-zero lower bound on the KL term are presented as a theoretical guarantee against latent collapse. Empirically, diversity is reported through Self-BLEU, Dist, and JS, while latent usage is tracked through KL, MI, and AU (Hu et al., 2022). This establishes another ADS-related pattern: output diversity should be interpreted jointly with evidence that the latent space is actually being used.

6. Empirical behavior, trade-offs, and adjacent latent-quality scores

Across domains, the literature repeatedly shows that stronger diversity usually comes with a trade-off against some other desideratum. In sound anomaly detection, weighting rare-normal samples improves AUC and reduces false positives on rare-normal operating conditions, but the method depends on within-mini-batch KDE, bounded support assumptions, and a surrogate rather than exact maximum-score optimization (Koizumi et al., 2019). In chemical heteroencoders, enumerated-decoder training increases local generation diversity but also increases wrong-molecule decoding, and more complex architectures are needed to recover fidelity (Bjerrum et al., 2018). In recurrent text VAEs, shorter segment length raises diversity, while parallel approximations can shift the quality-diversity balance, especially on long sequences (Hu et al., 2022). In SDICE, low diversity can coexist with plausible fidelity, particularly as poor intra-class coverage despite acceptable inter-class separation (Alam et al., 2024).

The literature also shows that diversity depends on the representation in which it is measured. TIE and TCE are explicitly not numerically comparable because they live in different latent spaces, and the choice between InceptionV3 and CLIP changes whether diversity is primarily visual or semantic (Ibarrola et al., 2024). Sparse-autoencoder feature coverage is model-relative because the active features are extracted from the residual stream of a particular base model (Yang et al., 19 Feb 2025). Posterior-diversity metrics depend on the approximate posterior family and on how uncertainty is controlled (Shen et al., 2021). Bottleneck-decorrelation scores can mistake inactive or collapsed units for diversity unless accompanied by activity diagnostics (Laakom et al., 2022).

A neighboring but non-diversity framework appears in “Autoencoder Architectures for Athlete Performance Scoring from Wearable Telemetry.” That work combines reconstruction error with latent-score interpretability through Spearman, Kendall, Mutual Information, and Permutation Importance in a composite selection criterion,

$\mathcal{J}_{\theta}^{\mbox{\tiny BU}} = \frac{1}{\sum_{i=1}^{M_u} w_i } \sum_{i=1}^{M_u} w_i \mathcal{A}_{\theta} \left( \bm{x}_i^{(u)} \right) - \mathcal{L}_a,$2

It is not an ADS, but it demonstrates a closely related principle: autoencoder quality should not be selected on reconstruction alone (Kubita et al., 26 Jun 2026).

The cumulative implication is that ADS is best viewed as a composite latent-quality notion whose exact form depends on the application. Some formulations reward entropy or coverage, some reward posterior separation, some reward feature non-redundancy, and some penalize collapse or overconcentration. What unifies them is the rejection of reconstruction accuracy alone as a sufficient description of what a good autoencoder representation should preserve or generate.