Stop-Gradient Targets in Self-Supervised Learning

• Stop-gradient Targets are defined as techniques that block gradient flow in one branch of a Siamese network to prevent trivial representation collapse, as seen in BYOL and SimSiam.
• They employ symmetric loss formulations where gradients are selectively frozen to stabilize latent space geometry, with analyses showing distinct eigenvalue sectors and converging class separability.
• These methods enhance transferability and robustness in representation learning, notably improving performance in settings with small batches and facilitating guided stop-gradient strategies.

Stop-gradient targets are a class of techniques in self-supervised representation learning that prevent trivial representational collapse by selectively blocking gradient flow through one branch of symmetric (Siamese) training objectives. This mechanism, first formulated to address the collapse in positive-only self-supervised methods such as BYOL and SimSiam, is now recognized as a foundational tool for stabilizing latent space geometry in the absence of explicit negative samples. Recent theoretical frameworks and algorithmic innovations—including spectral analyses of gradient-flow dynamics and guided stop-gradient variants—have established the central role of stop-gradient targets in both geometric stabilization and improved transferability of learned representations (Yao et al., 11 Apr 2026, Lee et al., 12 Mar 2025).

1. Mathematical Framework and Training Objectives

In self-supervised frameworks that use Siamese or twin-branched architectures, the stop-gradient operation acts as follows: given a loss term involving two feature vectors—typically a prediction branch and a target branch—the gradient is backpropagated only through the prediction branch, while the target is “frozen” using the stop-gradient operation $\mathrm{sg}(\cdot)$. For example, in the minimal embedding-only model with projection head $W \in \mathbb{R}^{p \times d}$, the loss is

$$\ell = \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{(1-r)N} \Big[(W u_{i\alpha} - v_i)^2 + (W v_i - u_{i\alpha})^2\Big] + \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{rN} \Big[(W s_\alpha - v_i)^2 + (W v_i - s_\alpha)^2\Big]$$

with stop-gradient applied symmetrically: for each term, gradient flow to the “target” argument (either $v_i$ or $u_{i\alpha}$) is blocked. This symmetry is characteristic of the BYOL/SimSiam family (Yao et al., 11 Apr 2026).

In practical algorithmic settings, such as SimSiam or BYOL, the loss adopts the structure: $$\mathcal{L}_{\mathrm{base}} = \frac12\,\mathcal{D}(p_1, \mathrm{sg}(z_2)) + \frac12\,\mathcal{D}(p_2, \mathrm{sg}(z_1))$$ where $p_i$ is a predictor output, $z_i$ is a projection output, and $\mathcal{D}$ denotes a similarity metric, typically negative cosine similarity (Lee et al., 12 Mar 2025). The key operational principle is selective blocking of gradients to one side of each paired term.

2. Gradient Dynamics: Spectral Pathways and Collapse

Stop-gradient targets modify the gradient-flow ODEs by removing reciprocal feedback between the branches. In the embedding-only setting, the system of ODEs for weights, sample embeddings, and class embeddings under full gradient flow contains reciprocal terms: $$\dot{u}_{i\alpha} = -\gamma[W^T(W u_{i\alpha} - v_i) + (u_{i\alpha} - W v_i)]$$ With stop-gradient, all $W \in \mathbb{R}^{p \times d}$0-type feedback terms vanish: $W \in \mathbb{R}^{p \times d}$1 This structure guarantees that each embedding variable is updated only via the projection matrix and never by gradients passing back through its paired target. As shown in (Yao et al., 11 Apr 2026), this eliminates geometric contraction mechanisms responsible for representation collapse: all branches communicate through $W \in \mathbb{R}^{p \times d}$2, but never directly influence their “target anchors.”

3. Fixed Point Analysis and Eigenspectral Structure

The fixed-point conditions under stop-gradient admit a precise spectral decomposition: $W \in \mathbb{R}^{p \times d}$3 where $W \in \mathbb{R}^{p \times d}$4 is the mean of class embeddings. $W \in \mathbb{R}^{p \times d}$5 displays at most two eigenvalues on the class-embedding subspace:

• $W \in \mathbb{R}^{p \times d}$6 (collapsed sector)
• $W \in \mathbb{R}^{p \times d}$7 (non-collapsed sector)

Projectors $W \in \mathbb{R}^{p \times d}$8 and $W \in \mathbb{R}^{p \times d}$9 define these sectors. In the non-collapsed sector, as long as the global mean is centered ($$\ell = \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{(1-r)N} \Big[(W u_{i\alpha} - v_i)^2 + (W v_i - u_{i\alpha})^2\Big] + \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{rN} \Big[(W s_\alpha - v_i)^2 + (W v_i - s_\alpha)^2\Big]$$0), label embeddings retain finite non-zero separation even at infinite training time. Without stop-gradient, only the collapsed fixed point ($$\ell = \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{(1-r)N} \Big[(W u_{i\alpha} - v_i)^2 + (W v_i - u_{i\alpha})^2\Big] + \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{rN} \Big[(W s_\alpha - v_i)^2 + (W v_i - s_\alpha)^2\Big]$$1) remains stable (Yao et al., 11 Apr 2026).

4. Time Scales, Collapse Prevention, and Mean-Field Dynamics

Time-evolution under stop-gradient can be formulated via dynamical mean-field theory (DMFT). The projection sector obeys a linear ODE in $$\ell = \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{(1-r)N} \Big[(W u_{i\alpha} - v_i)^2 + (W v_i - u_{i\alpha})^2\Big] + \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{rN} \Big[(W s_\alpha - v_i)^2 + (W v_i - s_\alpha)^2\Big]$$2 with time-dependent correlators; embedding sectors evolve in a time-dependent "medium" $$\ell = \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{(1-r)N} \Big[(W u_{i\alpha} - v_i)^2 + (W v_i - u_{i\alpha})^2\Big] + \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{rN} \Big[(W s_\alpha - v_i)^2 + (W v_i - s_\alpha)^2\Big]$$3. In unprojected models or models lacking stop-gradient, two separated time scales appear:

• fast alignment mode ($$\ell = \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{(1-r)N} \Big[(W u_{i\alpha} - v_i)^2 + (W v_i - u_{i\alpha})^2\Big] + \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{rN} \Big[(W s_\alpha - v_i)^2 + (W v_i - s_\alpha)^2\Big]$$4)
• slow "frustration" mode ($$\ell = \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{(1-r)N} \Big[(W u_{i\alpha} - v_i)^2 + (W v_i - u_{i\alpha})^2\Big] + \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{rN} \Big[(W s_\alpha - v_i)^2 + (W v_i - s_\alpha)^2\Big]$$5)

The slow frustration mode contracts class geometry toward collapse at rate $$\ell = \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{(1-r)N} \Big[(W u_{i\alpha} - v_i)^2 + (W v_i - u_{i\alpha})^2\Big] + \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{rN} \Big[(W s_\alpha - v_i)^2 + (W v_i - s_\alpha)^2\Big]$$6. The stop-gradient mechanism stabilizes a persistent, nonzero component of $$\ell = \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{(1-r)N} \Big[(W u_{i\alpha} - v_i)^2 + (W v_i - u_{i\alpha})^2\Big] + \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{rN} \Big[(W s_\alpha - v_i)^2 + (W v_i - s_\alpha)^2\Big]$$7 in the $$\ell = \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{(1-r)N} \Big[(W u_{i\alpha} - v_i)^2 + (W v_i - u_{i\alpha})^2\Big] + \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{rN} \Big[(W s_\alpha - v_i)^2 + (W v_i - s_\alpha)^2\Big]$$8 sector, blocking further contraction and ensuring the preservation of finite class separability (Yao et al., 11 Apr 2026).

5. Guided Stop-Gradient and Implicit Negative Repulsion

The “guided stop-gradient” (GSG) strategy extends basic stop-gradient by conditionally deciding (per-pair, per-batch) which branch is fixed and which is trainable, based on relative proximity in embedding space. For a reference pair, GSG selects the two projections from distinct images that are closest and applies the stop-gradient there. The resulting loss function (per pair) is: $$\ell = \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{(1-r)N} \Big[(W u_{i\alpha} - v_i)^2 + (W v_i - u_{i\alpha})^2\Big] + \frac12 \sum_{i=1}^n \sum_{\alpha=1}^{rN} \Big[(W s_\alpha - v_i)^2 + (W v_i - s_\alpha)^2\Big]$$9 where indices $v_i$0 correspond to the minimum cross-pair distance. By blocking gradient flow in exactly those directions, GSG induces implicit negative-sample repulsion, enforcing uniformity in the learned representations without explicit negatives or memory banks (Lee et al., 12 Mar 2025).

6. Empirical Phenomena, Robustness, and Applications

Empirical studies in both minimal models and full teacher–student networks confirm:

• Nontrivial class-separation persists (e.g., $v_i$1 plateaus at nonzero value, even under frustration).
• Predictive accuracy tracks the best achievable under the frustration-induced classification bound (converging to $v_i$2).
• Guided stop-gradient eliminates collapse even when the predictor head is removed; conventional stop-gradient requires the predictor for stability (Lee et al., 12 Mar 2025).
• Representation learning with GSG is robust to batch-size reduction, outperforming explicit contrastive approaches under small batches and improving transfer performance in linear probing, k-NN, and detection/segmentation benchmarks.

7. Limitations and Future Directions

Current validation of stop-gradient targets and GSG is confined to positive-only architectures. Efficacy under multi-crop, clustering objectives, or vision transformers remains to be demonstrated. Future research directions include integration with transformer backbones, formal characterization of the implicit uniformity alignment induced by GSG, and extensions to dense prediction tasks (Lee et al., 12 Mar 2025).

---

References

• "A Minimal Model of Representation Collapse: Frustration, Stop-Gradient, and Dynamics" (Yao et al., 11 Apr 2026)
• "Implicit Contrastive Representation Learning with Guided Stop-gradient" (Lee et al., 12 Mar 2025)