Self-Supervised Selective Diffusion

• Self-Supervised Selective-Guided Diffusion is a framework that replaces explicit supervision with self-driven guidance signals in diffusion models for controllable generation.
• It integrates advanced feature extraction, unsupervised clustering, and selective conditioning to achieve semantic consistency and improved visual fidelity.
• Empirical results demonstrate that SSDiff enhances metrics like FID and IS, proving its potential for scalable, annotation-free generative modeling.

Self-Supervised Selective-Guided Diffusion (SSDiff) is a class of diffusion model methodologies that utilize self-supervised, data-driven mechanisms to derive guidance signals for the generative process, thereby removing or minimizing reliance on explicit external annotation. SSDiff encompasses a family of frameworks with the common objective of learning semantically meaningful conditioning purely from unlabelled data, enabling selective, fine-grained, and controllable generation across a range of modalities and tasks. These approaches structurally modify the guidance pipeline in diffusion models to incorporate self-supervision, clustering, and, in advanced variants, selective regional or semantic mechanisms for improved generative fidelity, diversity, and downstream utility.

1. Framework Foundations: Replacing Supervised Annotation with Self-Supervision

Traditional guided diffusion methods typically require conditioning signals drawn from human-annotated datasets (e.g., class labels, segmentation masks, or box coordinates), formalized as an annotation function $\xi(x; \mathcal{H}, \mathcal{C})$, where $x$ is an input image, $\mathcal{H}$ the feature space, and $\mathcal{C}$ the label space. SSDiff eliminates the dependence on annotation by introducing a self-supervised feature extraction function $g_{\phi}$ followed by a self-annotation function $f_{\psi}$:

• $g_{\phi}(x)$ extracts semantic features from input $x$ using a self-supervised backbone (e.g., Vision Transformer models trained with DINO, SimCLR, MSN).
• $f_{\psi}$ maps these features into concise, semantically interpretable “guidance signals” $k$, typically obtained via $k$-means clustering or unsupervised localization/segmentation.

The guidance signal $k$ can be generated at various levels of spatial granularity:

• Image-level: One-hot pseudo-labels encoding semantic cluster membership.
• Object-level/boxed: Binary masks for detected regions via unsupervised object detectors (e.g., LOST).
• Segmentation masks: Multi-channel pixel-level masks derived from unsupervised segmentation (e.g., STEGO).

For each forward pass $x \rightarrow k = f_{\psi}(g_{\phi}(x))$, $k$ is injected into the denoising network, typically by concatenation with the time-step embedding, thus modulating the noise prediction throughout the generative trajectory.

2. Generation of Self-Supervision Signals and Self-Annotation

The self-supervision signals in SSDiff result from modern unsupervised learning frameworks that yield feature representations rich in semantic content. For image-level guidance, the extracted features $h_x = g_{\phi}(x)$ are clustered non-parametrically (e.g., via $k$-means) to produce pseudo-labels $k \in \mathbb{R}^K$ (a one-hot vector for $K$ clusters). For spatially detailed guidance, $f_{\psi}$ leverages methods such as LOST to localize objects or STEGO for segmentation, allowing the guidance signal to encode higher granularity details—e.g., bounding-box masks $k_s$ or full segmentation grids $k_s \in \mathbb{R}^{W \times H \times K'}$.

This process ensures that the guidance tokens used in the diffusion process are constructed from the data distribution itself, and are structurally aligned with the patterns present in the training set, not from exogenous, potentially biased annotation.

3. Integration with the Diffusion Process

SSDiff retains the formalism of denoising diffusion probabilistic models (DDPMs). The forward process is defined as

$$q(x_t \mid x_{t-1}) = \mathcal{N}\bigl(x_t; \sqrt{1-\beta_t} \, x_{t-1}, \beta_t I\bigr),$$

with the cumulative product $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$, $\alpha_t = 1 - \beta_t$. The reverse process is learned by training a noise-prediction network $\varepsilon_\theta(x_t, t; k)$:

Conditioning Strategies

• Image-level guidance: $$\varepsilon_{\theta}(x_t, \operatorname{concat}[t, k])$$
• Object-level guidance: $$\varepsilon_{\theta}(\operatorname{concat}[x_t, k_s], \operatorname{concat}[t, k])$$
• Segmentation guidance: $$\varepsilon_{\theta}(\operatorname{concat}[x_t, k_s], \operatorname{concat}[t, \hat{k}])$$, where $\hat{k}$ is an aggregated (e.g., average-pooled) version of $k_s$.

Guided sampling employs classifier-free guidance as

$$\tilde{\varepsilon}_\theta(x_t, t; k, w) = (1 - w)\varepsilon_\theta(x_t, t) + w\varepsilon_\theta(x_t, t; k),$$

with $w$ the guidance strength. In this context, $k$ is derived self-supervised rather than from an external label.

4. Performance Characteristics and Empirical Results

Extensive experiments demonstrate the efficacy of SSDiff across a spectrum of datasets:

• ImageNet32/64, CIFAR100 (image-level guidance):
  • Self-supervised pseudo-label guidance outperforms unconditional models on Fréchet Inception Distance (FID) and Inception Score (IS).
  • With optimized cluster counts (e.g., $K\in[1000, 5000]$), SSDiff may match or even surpass ground-truth label guidance, especially in class-imbalanced scenarios.
• Pascal VOC / COCO_20K (self-boxed and self-segmented guidance):
  • Object and segmentation-based self-supervised guidance delivers lower FID and better content control relative to unconditional generation, without requiring any class, box, or segment labels.
  • Fine-grained mask-based guidance enables selective, spatially controlled sampling and reduction of artifacts, even when the masks are noisy.
• Scalability: SSDiff is implemented at the scale of $256 \times 256$ images (e.g., LSUN-Churches) and is compatible with latent diffusion models, offering improved semantic fidelity and visual diversity.

5. Visual Diversity, Semantic Consistency, and Control

The self-guided approach introduces two critical advantages for generative modeling:

1. Semantic Consistency: Clustering (or fine-grained localization) ensures that the generated samples adhere to semantic similarity within a cluster or spatial region, promoting content coherence.
2. Diversity: The composition of clusters and their within-group variance enables broader sample diversity. When multiple spatial guidance signals are mixed, SSDiff can generate hybrid or controlled samples not explicitly present in training data.

Box or mask conditioning further allows users to specify target regions for object placement, synthesis, or editing, imparting local control without explicit object or pixel labels.

6. Implementation Nuances, Modularity, and Scaling

SSDiff is architecturally modular—feature extractors ($g_\phi$), annotation functions ($f_\psi$), and denoising models are independent components. This modularity enables:

• Easy substitution or improvement as self-supervised learning progresses (e.g., more advanced ViT backbones or unsupervised segmentation).
• Flexible adaptation to alternative spatial granularities, domain-specific requirements, or varying compute budgets.
• Applicability to large, unlabelled datasets, as it eschews the annotation bottleneck that restricts standard conditioned models.
• Co-deployment with latent diffusion models for high-resolution image synthesis scenarios with negligible annotation overhead.

7. Mathematical Summary

The SSDiff pipeline is mathematically characterized by:

Training objective:

$$\mathcal{L}(\theta) = \mathbb{E}_{x, t, \varepsilon \sim \mathcal{N}(0, I)} \left[ \left\| \varepsilon_\theta(x_t, t; f_\psi(g_\phi(x))) - \varepsilon \right\|_2^2 \right]$$

Self-guided classifier-free guidance:

$$\tilde{\varepsilon}_{\theta}(x_t, t; f_\psi(g_\phi(x)), w) = (1-w)\varepsilon_\theta(x_t, t) + w\varepsilon_\theta(x_t, t; f_\psi(g_\phi(x)))$$

Condition concatenation at inference:

• Image-level: $$\varepsilon_\theta(x_t, \operatorname{concat}[t, k])$$
• Box: $$\varepsilon_\theta(\operatorname{concat}[x_t, k_s], \operatorname{concat}[t, k])$$
• Segmentation: $$\varepsilon_\theta(\operatorname{concat}[x_t, k_s], \operatorname{concat}[t, \hat{k}])$$

This framework is conceptually grounded in the classifier-free guidance paradigm, with externally provided annotation signals replaced by self-supervised, data-driven cluster or spatial proxies.

---

In summary, Self-Supervised Selective-Guided Diffusion (SSDiff) constitutes a set of methods that utilize self-supervised representations to derive conditioning signals for diffusion models, enabling scalable, annotation-free, and spatially controllable generative modeling. Empirical results demonstrate state-of-the-art performance on standard benchmarks, sometimes outperforming fully supervised guidance. The approach is designed to scale with advances in self-supervised learning, and is adaptable across domains or granularity levels, positioning SSDiff as a robust strategy for controllable, high-fidelity generation in the absence of human-labeled data (Hu et al., 2022).