Representation Quality Index (RQI)

• Representation Quality Index (RQI) is defined as a quantitative framework that evaluates neural feature representations through dual metrics measuring clustering tightness and prediction alignment.
• It employs a Raw Zero-Shot test approach alongside Davies–Bouldin (DBM) and Amalgam (AM) metrics to assess model robustness against adversarial attacks.
• Empirical findings reveal a trade-off between high classification accuracy and robust feature representations, with stronger correlations to defense effectiveness and attack distortions.

The Representation Quality Index (RQI) provides a quantitative framework for evaluating the efficacy of neural network feature representations, specifically linking representational structure to model robustness against adversarial attacks and the capacity for generalization to unseen classes. Originating from the need to explain neural network vulnerability to adversarial examples, the RQI methodology leverages a bespoke Raw Zero-Shot learning test with metrics that explicitly measure cluster tightness and proximity to an “oracle” prediction, thus connecting internal feature geometry to downstream robustness and transfer capabilities (Kotyan et al., 2019).

1. Raw Zero-Shot Test Formulation

The Raw Zero-Shot test is central to RQI computation. For an $N$-class classification task, $N$ separate classifiers are trained, each omitting one class $i$ (yielding a model $C_{-i}$). For each excluded class $i$, all its test samples are presented to $C_{-i}$, generating soft label vectors in $\mathbb{R}^{N-1}$. The key requirement is that, if learned features are genuinely reusable, the outputs for withheld class $i$ should form a coherent cluster in the output space, and this cluster should align closely with the aggregate prediction a fully trained classifier would provide for class $i$ (Kotyan et al., 2019). This dual requirement—tight clustering and proximity to an “amalgam” ground-truth—is tested empirically using two complementary metrics.

2. Representation Quality Metrics: DBM and AM

The RQI is defined via:

Davies–Bouldin Metric (DBM):

Measures the root-mean-square Euclidean dispersion of Raw Zero-Shot outputs for omitted class $i$,

$$\mathrm{DBM}_i = \sqrt{\frac{1}{n} \sum_{j=1}^n \left\| z_j - \mu \right\|_2^2}$$

where $z_j$ are the soft-label outputs for each sample, and $\mu$ their centroid in $\mathbb{R}^{N-1}$. Low $\mathrm{DBM}_i$ indicates tight, coherent clustering of omitted-class samples, interpreted as evidence of shared and generalizable feature recognition.

Amalgam Metric (AM):

Measures the $L_1$-distance between the sum of Raw Zero-Shot outputs and the sum of oracle-softmax outputs (withheld class removed and probabilities renormalized),

$$\mathrm{AM}_i = \frac{1}{N-1} \left\| H' - H \right\|_1$$

with $H = \sum_{j=1}^n z_j$, $H' = \sum_{j=1}^n z'_j$. Small $\mathrm{AM}_i$ reflects close agreement between Raw Zero-Shot outputs and the “full information” classifier’s prediction, indicating that the omitted class is interpretable as a convex combination of known-class feature responses.

Model-level scores are obtained via averaging: $$\mathrm{DBM} = \frac{1}{N} \sum_{i=1}^N \mathrm{DBM}_i$$, $$\mathrm{AM} = \frac{1}{N} \sum_{i=1}^N \mathrm{AM}_i$$. The original methodology does not prescribe canonical fusion of these metrics into a single scalar index; normalization and (optional) averaging across models is possible but not standardized (Kotyan et al., 2019).

3. Experimental Protocols and Benchmarks

Empirical evaluations employed datasets including Fashion-MNIST, CIFAR-10, and a 10-superclass Sub-Imagenet. Classifiers tested encompass LeNet, MLP, ConvNets (AllConv, ResNet, WideResNet, DenseNet, VGG), and dynamic routing architectures (CapsNet).

RQI was assessed in the presence and absence of standard adversarial defenses: Gaussian Augmentation, Feature Squeezing, Spatial Smoothing, Label Smoothing, and Thermometer Encoding. Multiple white-box attacks (FGM, BIM, PGD, DeepFool, NewtonFool) were used, with mean $L_2$ perturbation and changes in classifier confidence/accuracy as attack strength indicators.

Procedure for each model:

• For each class $i$, train $C_{-i}$ on $N-1$ classes, evaluate Raw Zero-Shot outputs $z_j$ for omitted class $i$.
• Collect and aggregate $\mathrm{DBM}_i$ and $\mathrm{AM}_i$ metrics.
• Average across all $N$ classes to produce model-level metrics (Kotyan et al., 2019).

4. Empirical Findings and Observed Correlations

• Architectural Differentiation: CapsNet achieved the lowest (i.e., best) DBM and AM scores on CIFAR-10, indicative of high representation quality, closely followed by the shallow LeNet. Contemporary deep architectures (ResNet, DenseNet, VGG) exhibited substantially worse RQI metrics, trading representation quality for marginal gains in top-1 classification accuracy.
• Defensive Interventions: Adversarial defenses—except Gaussian noise augmentation—consistently reduced (improved) both DBM and AM values. Label Smoothing especially resulted in tight DBM clusters and better AM scores, while Thermometer Encoding sparsified DBM clusters but still improved AM.
• Correlation with Robustness: Across five attacks and ten CIFAR-10 classes, Pearson correlation coefficients between precomputed DBM/AM and attack mean-$L_2$ distortion ranged up to $|\rho|\approx 0.8$–$0.9$ ($p\ll 0.05$). Specifically, DBM was negatively correlated with required attack distortion ($\rho\approx -0.5$ to $-0.8$), indicating that tighter feature clusters are more robust, while AM had strong positive correlation ($\rho\approx 0.7$ to $0.98$), meaning that poorer amalgam alignment signals greater vulnerability.
• Trade-off Phenomenon: Very deep networks tuned for highest classification accuracy suffered from elevated DBM/AM, suggesting a representation–robustness trade-off detrimental to adversarial resilience (Kotyan et al., 2019).

5. Implementation Methodology

The canonical implementation steps for RQI calculation are:

1. For the model under consideration, train $N$ Raw Zero-Shot variants, omitting each class $i$ in turn.
2. For each $C_{-i}$, evaluate its predictions on the withheld class $i$; collect softmax outputs $z_j \in \mathbb{R}^{N-1}$.
3. Compute the cluster centroid ($\mu$), aggregate sums ($H$, $H'$), and then derive DBM$_i$ and AM$_i$.
4. Average DBM$_i$, AM$_i$ over all $i$ to obtain the global DBM and AM.
5. For cross-model comparison or benchmarking, optionally normalize both metrics to $[0,1]$ and average for a single scalar RQI, though this practice is not canonically endorsed (Kotyan et al., 2019).

This methodology enables systematic comparison of architectures, hyperparameters, and the impact of defense mechanisms on feature representation structure. The use of DBM or AM as a differentiable regularizer in the training objective is proposed as a means to directly optimize representation quality and thus model robustness.

6. Broader Implications and Prospective Directions

The empirical evidence linking RQI metrics to adversarial robustness suggests that improved “zero-shot” generalization—quantified by low DBM and AM—implies greater resistance to adversarial manipulation. Dynamic routing and non-linear feature grouping (exemplified by CapsNet) demonstrated superior representation quality without compromising accuracy, indicating architectural pathways for future research.

Possible extensions to the RQI framework include consideration of alternative cluster indices (Silhouette score, Dunn index), unsupervised manifold-based distance measures, and confidence drop metrics for granular class-wise analysis. Incorporating RQI metrics as explicit training constraints may foster networks with inherently robust internal feature geometries.

The practical utility of RQI lies in its ability to diagnose, compare, and guide the development of both adversarial defense methodologies and novel neural architectures, anchoring representation quality as a core determinant of both generalization and robustness (Kotyan et al., 2019).