Image-Information Score for Visual Semantics

• Image-Information Score is a family of measures that assess the semantic content of images by capturing realism, variety, and fidelity beyond standard metrics.
• It integrates methods such as PCA, t-SNE, fuzzy C-Means clustering, and covariance-weighted norms to quantify semantic accuracy in both text-to-image synthesis and contrastive models.
• The score operationalizes semantic informativeness through clustering accuracy and KL divergence approximations, serving as a diagnostic for model performance in capturing image semantics.

Searching arXiv for the cited papers and closely related context. Image-Information Score (IS) is not the literal name of a single standardized metric in the cited literature. In the available work, the expression is best understood as an umbrella for methods that quantify how much semantic information an image carries or preserves, especially beyond what the traditional Inception Score captures. One formulation, developed for text-to-image synthesis, uses Inception features, PCA, t-SNE, fuzzy C-Means clustering, and clustering accuracy to estimate realism, variety, and semantic accuracy of generated images (Sommer et al., 2019). A later formulation defines semantic informativeness as an instance-wise KL divergence between model-induced prior and posterior distributions in a contrastive vision-LLM, and shows that this quantity is tightly approximated by a covariance-weighted norm of the embedding (Uchiyama et al., 28 Jun 2025).

1. Terminological scope and motivation

The central motivation for an image-information-oriented score is that the standard Inception Score evaluates only part of what matters. In the text-to-image setting, the conventional Inception Score is

$$\text{IS} = \exp\big(\mathbb{E}_{x}[\mathrm{KL}(p(y\mid x) \;\|\; p(y))]\big),$$

where $p(y\mid x)$ is the class distribution predicted by Inception for image $x$, and $p(y) = \mathbb{E}_x[p(y\mid x)]$ is the marginal label distribution over generated images. Its usual interpretation is that high values arise when each generated image is confidently classified and the marginal label distribution is diverse. The cited text explicitly notes that IS correlates well with human judgment as a quality metric for unconditional image generation, but “solely takes into account the realism and variety of the generated images” and “fails to capture important information when the image synthesis task is conditioned on classes or text descriptions” (Sommer et al., 2019).

A distinct motivation appears in contrastive vision-language modeling. There, the question is not whether an image is merely realistic or diverse, but how much conditioning on that image distorts the distribution of plausible captions relative to a prior distribution over captions. This motivates an instance-wise notion of semantic informativeness: an image is informative when the conditional text distribution induced by the image differs strongly from the global text prior (Uchiyama et al., 28 Jun 2025).

These two motivations define two different but related uses of the phrase “Image-Information Score.” The first is class- and dataset-relative, targeting semantic fidelity in conditional generation. The second is prior-posterior relative, targeting semantic informativeness in a shared embedding space. This suggests that “image information” is not a single primitive, but a family of operational measures tied to the evaluation setting.

2. Cluster-based semantic information in text-to-image synthesis

In the text-to-image formulation, real images are denoted $X = \{x_1, x_2, \dots, x_n\}$, text descriptions are $T = \{t_1, t_2, \dots, t_n\}$, and a generator $G$ produces synthetic images

$S = \{s_1, s_2, \dots, s_n\}, \quad s_i = G(t_i).$

The evaluation datasets are disjoint from the training data of $G$ (Sommer et al., 2019).

The method begins by training an Inception network on the real images $X$ using their class labels, achieving high but not perfect classification accuracy, approximately $p(y\mid x)$0. The cited description states that the authors intentionally avoid $p(y\mid x)$1 accuracy to reduce overfitting. This network is then used as a feature extractor: for each real image and each synthetic image, activations from the next-to-last layer, with $p(y\mid x)$2 neurons, are extracted. These define high-dimensional representations

$p(y\mid x)$3

and

$p(y\mid x)$4

To visualize and structure these representations, the method combines $p(y\mid x)$5 and $p(y\mid x)$6, reduces dimensionality by PCA from $p(y\mid x)$7 to $p(y\mid x)$8, and then applies t-SNE from $p(y\mid x)$9 to $x$0 dimensions. The cited description states that this PCA preprocessing follows the recommendation in the original t-SNE paper. The resulting embeddings are

$x$1

In the reported experiments on the CUB-200-2011 test set, the perplexity is set to $x$2, approximately the number of synthetic images per class, because it is “the expected number of neighbors” (Sommer et al., 2019).

Clustering is then applied only to the real-image embeddings $x$3 using fuzzy C-Means. The number of clusters is set equal to the number of classes in the dataset; in the main experiment, this yields $x$4 clusters for $x$5 species in the test subset. Each cluster is assigned a unique class label by taking the most frequent ground-truth class among the real images assigned to that cluster, with an explicit priority rule when multiple clusters would map to the same class. The clusters are interpreted as “visual concepts” approximating real semantic classes.

Generated images are classified by computing fuzzy C-Means membership probabilities $x$6 to each cluster in the 2D t-SNE space and predicting

$x$7

Because each cluster already has a class label, the predicted cluster determines a predicted semantic class. The main scalar metric is clustering accuracy,

$x$8

where $x$9 is the class of the real image whose caption generated the synthetic image. The cited text characterizes this clustering accuracy as the effective gauge for semantic accuracy and as a practical measure of how much class information is present in the generated image representations (Sommer et al., 2019).

3. Realism, variety, and empirical behavior of the clustering-based score

The cluster-based method is designed to evaluate three properties simultaneously: realism, variety, and semantic accuracy. Realism is assessed visually: for fairly realistically looking synthetic images, the distribution of their low-dimensional representations $p(y) = \mathbb{E}_x[p(y\mid x)]$0 should show a significant degree of overlap with the distribution of the low-dimensional representations of real images $p(y) = \mathbb{E}_x[p(y\mid x)]$1. Variety is also assessed visually: the synthetic image representations should be spread out to a reasonable degree, comparable to the distribution of $p(y) = \mathbb{E}_x[p(y\mid x)]$2. Semantic accuracy is the clustering accuracy defined above (Sommer et al., 2019).

The reported experiments use CUB-200-2011, with approximately $p(y) = \mathbb{E}_x[p(y\mid x)]$3k test images from $p(y) = \mathbb{E}_x[p(y\mid x)]$4 species and approximately $p(y) = \mathbb{E}_x[p(y\mid x)]$5k captions, with $p(y) = \mathbb{E}_x[p(y\mid x)]$6 captions per image. Each caption generates one synthetic image, yielding approximately $p(y) = \mathbb{E}_x[p(y\mid x)]$7k synthetic images per method. Four state-of-the-art text-to-image methods are evaluated: AttnGAN, MirrorGAN, HDGAN, and StackGAN++. A pre-trained Inception model from StackGAN, trained on CUB-200-2011, is used as the feature extractor.

The paper reports that AttnGAN and MirrorGAN have the highest clustering accuracy on synthetic images, approximately $p(y) = \mathbb{E}_x[p(y\mid x)]$8 and $p(y) = \mathbb{E}_x[p(y\mid x)]$9, while HDGAN and StackGAN++ are lower, approximately $X = \{x_1, x_2, \dots, x_n\}$0. Direct Inception classifier accuracy on synthetic images is similar: approximately $X = \{x_1, x_2, \dots, x_n\}$1, $X = \{x_1, x_2, \dots, x_n\}$2, $X = \{x_1, x_2, \dots, x_n\}$3, and $X = \{x_1, x_2, \dots, x_n\}$4, respectively. The cited description states that this is used as a sanity check and as “a strong indicator that the overall structure and clustering … has not been lost after t-SNE” (Sommer et al., 2019).

Method	Clustering acc. (synthetic)	Inception acc. (synthetic)
AttnGAN	0.22	0.23
MirrorGAN	0.20	0.22
HDGAN	0.11	0.13
StackGAN++	0.11	0.12

The same experiments also report Inception Scores that are relatively close across methods: AttnGAN $X = \{x_1, x_2, \dots, x_n\}$5, HDGAN $X = \{x_1, x_2, \dots, x_n\}$6, MirrorGAN $X = \{x_1, x_2, \dots, x_n\}$7, and StackGANv2 $X = \{x_1, x_2, \dots, x_n\}$8. The stated interpretation is that clustering accuracy distinguishes models in semantic accuracy even when IS values are similar. This is the principal sense in which the method behaves like an image-information score: it captures semantic correctness that a realism-and-diversity metric can miss (Sommer et al., 2019).

Interpretability is a prominent feature of the method. In the reported visualizations, real images are shown in black, correctly classified synthetic images in blue, and incorrectly classified synthetic images in red. AttnGAN and MirrorGAN exhibit many blue points around most real clusters, whereas HDGAN and StackGAN++ show more red points. Regions of t-SNE space far from any real cluster are associated with synthetic images containing “strange content,” while regions close to real clusters contain visually plausible images. This gives the measure a diagnostic role rather than only a ranking role.

4. KL-based semantic informativeness in contrastive vision-LLMs

A different formulation of image information is introduced in the context of CLIP and SigLIP. Here, the semantic informativeness of an image is defined as how much conditioning on that image distorts the distribution of plausible captions relative to the prior caption distribution. Let $X = \{x_1, x_2, \dots, x_n\}$9 be a text sample and $T = \{t_1, t_2, \dots, t_n\}$0 an image, with prior distributions $T = \{t_1, t_2, \dots, t_n\}$1 and $T = \{t_1, t_2, \dots, t_n\}$2 and conditionals $T = \{t_1, t_2, \dots, t_n\}$3 and $T = \{t_1, t_2, \dots, t_n\}$4. The semantics of an image $T = \{t_1, t_2, \dots, t_n\}$5 are captured by how the conditional distribution $T = \{t_1, t_2, \dots, t_n\}$6 differs from the prior text distribution $T = \{t_1, t_2, \dots, t_n\}$7 (Uchiyama et al., 28 Jun 2025).

Following prior work in NLP, the paper defines the Information Gain of an image as the KL divergence

$T = \{t_1, t_2, \dots, t_n\}$8

and, symmetrically, the Information Gain of a text as

$T = \{t_1, t_2, \dots, t_n\}$9

The cited description explicitly states that these are not mutual information in the Shannon sense; they are instance-wise KL divergences between model-induced posterior and prior distributions.

The model-induced distributions are obtained from contrastive embeddings. CLIP and SigLIP map images and texts into a shared embedding space, with an image encoder producing $G$0 and a text encoder producing $G$1. For CLIP, given an image $G$2 and candidate texts $G$3,

$G$4

For SigLIP,

$G$5

The text prior is estimated by averaging the posterior over a sample set of images:

$G$6

Then

$G$7

The theoretical development in the paper adapts SGNS-style analysis to SigLIP and states that, under ideal optimization and ignoring higher-order terms, the Information Gain is approximately a covariance-weighted quadratic form:

$G$8

where

$G$9

and $S = \{s_1, s_2, \dots, s_n\}, \quad s_i = G(t_i).$0. The cited description interprets this as a Mahalanobis-like squared norm of the centered image embedding with respect to the covariance of text embeddings (Uchiyama et al., 28 Jun 2025).

5. Covariance-weighted norm as a practical Image-Information Score

The covariance-weighted formulation turns semantic informativeness into a directly computable scalar score. After choosing a pre-trained contrastive model such as OpenCLIP ViT-B/32 or SigLIP ViT-B/16 and selecting sample sets $S = \{s_1, s_2, \dots, s_n\}, \quad s_i = G(t_i).$1 and $S = \{s_1, s_2, \dots, s_n\}, \quad s_i = G(t_i).$2, one computes normalized embeddings for the images and texts, their means, and the covariance matrices

$S = \{s_1, s_2, \dots, s_n\}, \quad s_i = G(t_i).$3

and

$S = \{s_1, s_2, \dots, s_n\}, \quad s_i = G(t_i).$4

For a new image $S = \{s_1, s_2, \dots, s_n\}, \quad s_i = G(t_i).$5, the score is computed from its normalized embedding $S = \{s_1, s_2, \dots, s_n\}, \quad s_i = G(t_i).$6 by centering

$S = \{s_1, s_2, \dots, s_n\}, \quad s_i = G(t_i).$7

and evaluating

$S = \{s_1, s_2, \dots, s_n\}, \quad s_i = G(t_i).$8

The cited description gives the canonical scaling

$S = \{s_1, s_2, \dots, s_n\}, \quad s_i = G(t_i).$9

and states that the factor $G$0 can be dropped if only rankings are needed (Uchiyama et al., 28 Jun 2025).

The same framework is symmetric for text:

$G$1

with $G$2. This symmetry is central to the paper’s presentation: image informativeness is defined through the distortion of text distributions, and text informativeness through the distortion of image distributions.

The empirical validation is unusually tight. For normalized embeddings, the paper reports that the coefficient of determination between $G$3 and $G$4 is approximately $G$5 for images and approximately $G$6 for texts in the normalized SigLIP experiments, while for pre-trained open-weight CLIP and SigLIP the corresponding $G$7 values range from $G$8 to $G$9. The reported interpretation is that the covariance-weighted norm is a very accurate proxy for KL-based Information Gain (Uchiyama et al., 28 Jun 2025).

Qualitatively, the lowest-Information images in the reported CC12M experiments are often “image not found” icons, blank or placeholder thumbnails, and extremely low-content graphics. The lowest-Information texts are often placeholders such as <PERSON> or generic template titles. The cited description also reports little to no correlation between Information Gain and caption length, word count, negative log probability under a small LLaMA LLM, or number of Visual Genome attributes. This indicates that the score is not reducible to superficial complexity or rarity measures.

6. Computational profile, limitations, and conceptual boundaries

The computational advantage of the covariance-weighted score is that, after means and covariances have been estimated once, the per-instance inference cost no longer depends on the size of the candidate set in the opposite modality. The paper states the complexity comparison as follows (Uchiyama et al., 28 Jun 2025):

Quantity	Pre-inference	Inference
Information Gain	$X$0	$X$1
Norm-based metric	$X$2	$X$3

This is significant because exact KL-based Information Gain for a new image requires computing a conditional distribution over all candidate texts, whereas the norm-based score only requires the embedding, the precomputed mean, and the covariance matrix.

Both lines of work impose important constraints on interpretation. In the text-to-image setting, the scalar metric focuses on class semantics rather than fine-grained caption semantics such as pose, background, or attributes; clustering is performed in t-SNE space rather than in the original $X$4-dimensional Inception feature space; and the method depends on a strongly performing Inception model, so biases or failures in that model propagate into the evaluation (Sommer et al., 2019). The cited text also notes that no explicit human correlation study is performed for the proposed clustering metric, even though the motivation is partly to stand in for semantic aspects that humans care about.

In the contrastive setting, the score depends on the pre-trained model and on the data used to estimate means and covariances. The cited description explicitly warns that means and covariances estimated on CC12M or LAION might not transfer perfectly to medical images, satellite imagery, and other shifted domains; that the error between true distributions and model-induced approximations is not fully analyzed; and that “Information Gain” is not Shannon information but KL between model-based prior and posterior distributions (Uchiyama et al., 28 Jun 2025).

A common misconception is to treat Image-Information Score as simply another name for Inception Score. The cited material does not support that equivalence. In one formulation, the score is clustering accuracy derived from real-image semantic structure in Inception-t-SNE space; in the other, it is a covariance-weighted norm approximating KL distortion in a contrastive embedding space. Another misconception is that a high score must imply complete semantic correctness. The first method measures alignment to dataset classes, and the second measures model-internal semantic informativeness relative to a learned prior. Both are operational and useful, but neither is an unrestricted measure of meaning.

Taken together, these formulations establish two rigorous senses in which an image may be said to contain “information.” In conditional image synthesis, information is the preservation of intended class semantics relative to real-image clusters. In contrastive multimodal modeling, information is the extent to which an image changes the posterior over plausible texts relative to the prior. The phrase “Image-Information Score” is therefore best understood as a conceptual label for these operationalizations rather than as the canonical title of a single metric.