Converge to Surprise: Evolutionary Self-supervised Image Clustering

Abstract: Most self-supervised image clustering models, actually almost all deep learning approaches, are based on gradient descent: In order to calculate the loss, every optimization step requires a clearly defined target, whether a contrastive split, a masked patch or entity, an EMA-teacher output, a pseudo-label, or a differentiable information-theoretic functional. We propose a self-supervised framework that drops this requirement for image clustering. Without any prior knowledge, we have to assume that each pixel is i.i.d. according to the Principle of Maximum Entropy. Taking this as our null hypothesis H0, we define a "surprise score" that measures how unlikely the model's output representation would be under H0. Maximizing the surprise score forces the deep learning model to reject H0 - equivalently, to discover non-random feature from data. Also, here is our fundamental assumption: a surprise score cannot, in general, be reduced to a per-step loss. Hence, we propose the "converge-to-surprise" scheme to optimize our model: an evolution-strategy (ES) outer loop, which directly maximizes the surprise score without needing its gradient, paired with a periodic gradient-descent inner loop, which uses the surprising clusters already discovered by ES as surrogate targets. On standard image benchmarks, our framework achieves new state-of-the-art results in non-parametric self-supervised image clustering - the strictest deep-clustering setting, in which the number of ground-truth classes is not given to the model.

• The paper introduces a novel evolutionary framework that uses a global surprise score to challenge the null hypothesis of pixel independence.
• It strategically combines an outer evolution strategy with inner-loop gradient descent and complementary chessboard masking to enforce semantic feature learning.
• Experimental results on MNIST, Fashion-MNIST, and USPS show superior cluster purity and accuracy, validating the method’s nonparametric and unsupervised design.

Evolutionary Self-supervised Image Clustering Without Parametric Loss

Motivation and Context

The paper "^^^^0^^^^" (2607.06887) systematically challenges the standard paradigm of self-supervised image clustering based solely on gradient descent optimization. While prior approaches invariably require differentiable per-step loss functions tied to clearly-defined targets, this work's core assertion is that not all forms of semantic structure discovery in unlabeled data can be reduced to such losses. The authors posit that, under the maximum entropy principle, the null hypothesis for image data is total pixel independence; meaningful representation learning thus requires actively rejecting this hypothesis through the discovery of structured, non-random clusters.

Framework and Methodology

The framework employs a hybrid optimization strategy, termed "converge-to-surprise," combining an outer loop of evolution strategy (ES) and an inner loop of gradient descent. The ES directly maximizes a global, non-differentiable surprise score derived from cluster co-occurrence statistics, while the periodic inner-loop gradient descent uses clusters previously discovered by ES as surrogate targets for further consolidation.

A key architectural component is the complementary masking strategy. Each image is split into two disjoint views via a chessboard pattern, with independent augmentation applied to each, thereby ensuring views share no pixels and, under $\mathcal{H}_0$, possess zero mutual information. This forces the network to rely solely on global, semantic features for prediction, rather than trivial pixel-level correlations.

Figure 1: Chessboard masking partitions each image into two disjoint pixel sets, defining independent input views for clustering.

The clustering pipeline processes both views through a shared deep network (ResNet-9), with cluster assignment determined by the $\arg\max$ of the output logit. The surprise score for the model parameterization $\theta$ aggregates across clusters $k$ the binary KL divergence between observed view-matching frequency $\hat{q}_k$ and random null $q_k = p_k^2$, summing only over clusters that exhibit statistically significant over-matching relative to the null hypothesis.

Figure 2: The pipeline builds masked and augmented views, applies the network, collects cluster assignments, and computes a surprise score over the batch.

The ES optimizes $\mathcal{S}(\theta)$ as a black-box fitness, perturbing parameter vectors and updating via rank-shaped rewards over the batch. After sufficient evolution epochs, the inner-loop gradient descent is triggered, using the discovered "surprising" clusters as pseudo-labels to reinforce representation consolidation. Notably, $K$ is set much higher than the expected number of clusters; the framework is non-parametric and naturally deactivates unused output dimensions.

Empirical Results and Qualitative Observations

Experiments are conducted on MNIST, Fashion-MNIST, and USPS, each with ten ground-truth classes but without providing this information to the model. The framework achieves superior cluster purity, accuracy, and agreement across metrics, consistently matching or surpassing previous state-of-the-art non-parametric methods such as DeepDPM and UNSEEN. On USPS, the evolution-plus-gradient descent paradigm yields notably improved performance, increasing ACC from DeepDPM's $89\%$ to $94.9\%$ and NMI by four points.

The method robustly infers the number of clusters, converging to ten on MNIST and USPS, and slightly above ten on Fashion-MNIST, paralleling the dataset's inherent complexity. Ablation studies confirm the criticality of independent augmentation, ES warm-up, and, for USPS, anisotropic augmentation to address class imbalance and geometric variation in digits.

Cluster visualization on MNIST demonstrates nearly perfect one-to-one mapping between clusters and digits at $\arg\max$0--$\arg\max$1 purity.

Figure 3: MNIST cluster visualization; each cluster corresponds cleanly to a digit identity.

On USPS, cluster splitting in the dominant "0" class arises due to aspect ratio variation—mitigated but not eliminated by anisotropic augmentation.

Figure 4: USPS clusters: digit 0 split across two clusters characterized by shape differences.

Fashion-MNIST clusters reflect both coarse and fine attributes: bag clusters are split by handle/strap presence, garment clusters by texture (patterned vs. plain), indicating the framework captures subtle image semantics beyond label information.

Figure 5: Fashion-MNIST clusters encode finer-grained structure in bags and garments.

Analysis of cluster count evolution during training reveals that ES initially over-produces clusters; periodic gradient descent consolidates the cluster set, stabilizing at the expected number. This dynamic is visualized for MNIST, USPS, and Fashion-MNIST.

Figure 6: MNIST cluster count; pure ES creates redundant clusters, consolidated by gradient descent.

Figure 7: USPS cluster count; weak inner-loop training phase shows sawtooth consolidation and exploration.

Figure 8: Fashion-MNIST cluster count; stabilization above 10 clusters reflects dataset complexity.

Theoretical and Practical Implications

The central theoretical claim is that a "surprise score"—as a global property of model output—cannot generally be reduced to an element-wise loss suitable for gradient descent. This claim aligns with classical information theory and the principle of maximum entropy, formalizing learning as the iterative rejection of an ever-strengthening null hypothesis. The method can be viewed as building order from chaos, consistently increasing mutual information across views even as output dimensions are diversified.

Practically, the framework is applicable to settings where parametric assumptions (such as cluster number) are unavailable and where differentiable targets cannot be reliably constructed. The hybrid ES plus gradient descent pipeline demonstrates scalable, high-quality semantic clustering directly from raw data, without recourse to pretraining or external supervision.

Future Directions

Further research may explore multi-round iterative null hypothesis updating, more complex masking schemes, and application to higher-resolution or more diverse datasets. Extending the surprise-driven optimization to modalities beyond vision (e.g., sequential or graph data) may yield new unsupervised learning paradigms for robust semantic discovery. The fundamental insight—that meaningful structure may not be expressible as a differentiable per-step target—has broad implications for model design and learning theory, particularly in self-supervised and open-ended environments.

Conclusion

This work establishes an alternative to conventional loss-based self-supervised learning for image clustering, leveraging evolutionary optimization and surprise scores to drive nonparametric semantic discovery. The framework's strong empirical results and theoretical stance against reducibility to per-step losses highlight its utility for strictly unsupervised settings, and motivate broader exploration of information-theoretic principles in deep learning optimization.