Power Self-Distillation: Enhanced Self-Teaching

• Power self-distillation is a method where a model iteratively refines itself by distilling knowledge from its previous iterations with intensified teacher signals.
• Variants include repeated, amplified, competence-adaptive, and sequence-level self-distillation that adjust training rounds, weightings, or target distributions.
• Empirical results demonstrate improved accuracy and robustness, especially under high-noise or complex data conditions, by leveraging intensified self-generated supervision.

Searching arXiv for recent and foundational papers on self-distillation and repeated/self/power distillation. arxiv_search(query="power self-distillation self-distillation repeated self-distillation", max_results=10) Power self-distillation is a family of self-distillation procedures in which a model is distilled into another model of the same architecture under an intensified transfer rule. The literature does not present a single universal definition. Instead, recent work uses the idea in several closely related senses: repeated or multi-round self-distillation, where each student becomes the next teacher; amplified self-distillation, where the teacher term can receive weight beyond the standard convex range; competence-adaptive self-distillation, where training budget is concentrated on the student’s frontier of competence; and sequence-level power self-distillation, where the distilled target is a power-transformed distribution rather than the base model distribution itself (Pham et al., 2022, Das et al., 2023, Tomihari et al., 6 May 2026). Across these variants, the teacher and student are separated not by architecture, but by training round, weighting, information access, temporal slice, or internal branch.

1. Definition and scope

In the standard formulation, self-distillation is the special case of knowledge distillation in which teacher and student share the same architecture. A canonical iterative construction begins with a model trained from ground-truth labels only, denoted $f^{(0)}$, and then repeatedly trains $f^{(1)}, f^{(2)}, \dots$ so that the previous model acts as teacher for the next one (Pham et al., 2022). This round-by-round recursion is the most direct precursor of what is often meant by a “power” form of self-distillation: the transfer process is not a single teacher-to-student event, but an iterated refinement.

A second sense of “power” arises when the teacher term itself is amplified. In noisy-label learning, the student objective

$$\xi\,\ell(\text{teacher predictions}, \text{student predictions}) + (1-\xi)\,\ell(\text{given labels}, \text{student predictions})$$

need not be restricted to $\xi\in[0,1]$. When $\xi>1$, the teacher receives more than convex-combination weight and the label term acquires negative weight; the paper on noisy-label self-distillation explicitly interprets this regime as beneficial in high-noise settings because it can “anti-learn” corrupted labels (Das et al., 2023).

A third sense is distributional rather than iterative. In "Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation" (Tomihari et al., 6 May 2026), power self-distillation means distilling from the sequence-level power distribution

$$\pi_\alpha(y\mid x)=\frac{\pi(y\mid x)^\alpha}{\sum_{y'\in\mathcal{Y}}\pi(y'\mid x)^\alpha}, \qquad \alpha>1.$$

Here the “power” is literal: the base sequence probability is raised to a power before normalization.

A plausible synthesis is that power self-distillation denotes self-distillation regimes that strengthen the teacher signal beyond ordinary one-step, uniformly weighted, architecture-matched imitation. The strengthening may come from extra rounds, stronger teacher weighting, selective example weighting, or target sharpening.

2. Objectives, target distributions, and operational forms

The conventional self-distillation objective inherits the standard knowledge-distillation decomposition

$$\mathcal{L}_{KD} = \alpha \mathcal{L}_{CE}(\boldsymbol{z}_s,\boldsymbol{y}) + (1-\alpha) \mathcal{L}_{KL}(\boldsymbol{z}_s, \boldsymbol{z}_t),$$

where the student is trained jointly on ground-truth cross-entropy and KL divergence to the teacher’s softened logits (Pham et al., 2022). In repeated self-distillation, this same objective is applied across rounds $f^{(0)}\to f^{(1)}\to f^{(2)}\to \cdots$, with the previous student becoming the next teacher.

Under label noise, the same template becomes a tunable tradeoff between teacher imitation and label fitting. The theoretical analysis for regularized linear regression and logistic regression shows that the optimal imitation parameter can exceed one in high-noise regimes, with

$\lim_{\gamma \to \infty}\xi^{*} > 1,$

and with $\partial \xi^*/\partial \gamma^2 > 0$, so stronger label noise implies stronger optimal teacher amplification (Das et al., 2023). This establishes a formally non-convex power regime for self-distillation.

A different operational form appears in competence-adaptive distillation. PACED weights each prompt by a Beta kernel of the student’s pass rate,

$f^{(1)}, f^{(2)}, \dots$0

with default $f^{(1)}, f^{(2)}, \dots$1, so training concentrates on the zone of proximal development rather than on already-mastered or hopeless examples (Xu et al., 11 Mar 2026). In self-distillation on instruction-tuned models, PACED uses reverse KL,

$f^{(1)}, f^{(2)}, \dots$2

with pass rates estimated from student rollouts only:

$f^{(1)}, f^{(2)}, \dots$3

Sequence-level power self-distillation replaces the teacher target altogether. The teacher samples are drawn from $f^{(1)}, f^{(2)}, \dots$4, and the student minimizes the forward KL from $f^{(1)}, f^{(2)}, \dots$5 to the student, which reduces to maximum likelihood on teacher samples:

$f^{(1)}, f^{(2)}, \dots$6

This turns expensive inference-time power sampling into offline supervised training (Tomihari et al., 6 May 2026).

Policy self-distillation introduces yet another target design. DemoPSD does not match the privileged teacher distribution directly; instead it distills to a reverse-KL barycenter target

$f^{(1)}, f^{(2)}, \dots$7

with the tokenwise blending coefficient determined by teacher-student disagreement measured by Jensen–Shannon divergence (Li et al., 2 Jul 2026). This is a selective rather than unconditional power-up of teacher guidance.

3. Explanatory frameworks

The literature offers several distinct explanations for why intensified self-distillation can help, and the explanations are not mutually equivalent.

A major empirical-theoretical account links self-distillation to loss-landscape geometry. "Revisiting Self-Distillation" (Pham et al., 2022) argues that self-distilled students converge to flatter minima than their teachers, supported by smaller Hessian trace, smaller largest eigenvalue $f^{(1)}, f^{(2)}, \dots$8, and eigenspectra concentrated nearer zero. The same work explicitly critiques the multi-view hypothesis as insufficient: if repeated self-distillation simply accumulated more “views,” then performance should improve progressively across rounds, yet empirical performance fluctuates, and Born-Again Neural Networks underperform simple ensembles.

In the fixed-feature linear-probing setting, "Rethinking Self-Distillation: Label Averaging and Enhanced Soft Label Refinement with Partial Labels" (Jeong et al., 2024) gives a different mechanism. Under a linear approximation of softmax, multi-round self-distillation yields

$f^{(1)}, f^{(2)}, \dots$9

so the student acts as a graph-smoothing or label-averaging operator over feature-correlated instances. In the balanced class-structured case,

$$\xi\,\ell(\text{teacher predictions}, \text{student predictions}) + (1-\xi)\,\ell(\text{given labels}, \text{student predictions})$$0

which makes the denoising effect explicit: a sample’s label is progressively mixed with labels of correlated same-class points, followed eventually by uniform wash-out if too many rounds are used.

A third explanation is spectral. In overparameterized networks, AIR—Anisotropic Information Retrieval—states that informative components are learned before non-informative components. With NTK eigendecomposition

$$\xi\,\ell(\text{teacher predictions}, \text{student predictions}) + (1-\xi)\,\ell(\text{given labels}, \text{student predictions})$$1

the residual component along eigenvector $$\xi\,\ell(\text{teacher predictions}, \text{student predictions}) + (1-\xi)\,\ell(\text{given labels}, \text{student predictions})$$2 decays as

$$\xi\,\ell(\text{teacher predictions}, \text{student predictions}) + (1-\xi)\,\ell(\text{given labels}, \text{student predictions})$$3

so larger-eigenvalue directions are fitted earlier (Dong et al., 2019). Distillation is therefore interpreted as approximately early stopping, and sequential self-distillation preserves earlier, less-memorized predictions while avoiding later memorization of noise.

A fourth explanation appears in linear regression. "Understanding the Gains from Repeated Self-Distillation" (Pareek et al., 2024) shows that $$\xi\,\ell(\text{teacher predictions}, \text{student predictions}) + (1-\xi)\,\ell(\text{given labels}, \text{student predictions})$$4-step repeated self-distillation produces a richer polynomial preconditioner of the ridge operator:

$$\xi\,\ell(\text{teacher predictions}, \text{student predictions}) + (1-\xi)\,\ell(\text{given labels}, \text{student predictions})$$5

The extra polynomial degrees of freedom allow finer spectral shaping than ridge regression or one-step self-distillation.

Finally, power-distribution self-distillation explains gains in terms of target sharpening. In that framework, self-reward KL-regularized RL has the power distribution as its exact optimizer, and downstream improvement depends on the covariance

$$\xi\,\ell(\text{teacher predictions}, \text{student predictions}) + (1-\xi)\,\ell(\text{given labels}, \text{student predictions})$$6

This suggests that sharpened self-distillation helps when self-reward and true reward are aligned, and need not help when they are misaligned (Tomihari et al., 6 May 2026).

4. Architectural and domain-specific realizations

Self-distillation need not be restricted to teacher–student retraining across separate checkpoints. Several papers internalize the same idea within one network or one temporal process.

In domain-agnostic clustering, a DeepCluster-v2 framework is augmented with three bottleneck branches, where the deepest classifier $$\xi\,\ell(\text{teacher predictions}, \text{student predictions}) + (1-\xi)\,\ell(\text{given labels}, \text{student predictions})$$7 supervises shallower branches through KL divergence and feature-level hint matching:

$$\xi\,\ell(\text{teacher predictions}, \text{student predictions}) + (1-\xi)\,\ell(\text{given labels}, \text{student predictions})$$8

with total loss

$$\xi\,\ell(\text{teacher predictions}, \text{student predictions}) + (1-\xi)\,\ell(\text{given labels}, \text{student predictions})$$9

using $\xi\in[0,1]$0 and $\xi\in[0,1]$1 in the appendix (Adnan et al., 2021). The important structural point is that no separate student network is created.

Feature self-distillation can also be phrased in explicitly information-theoretic terms. MUSE replaces feature matching by dependency maximization through

$\xi\in[0,1]$2

and

$\xi\in[0,1]$3

arguing that intermediate layers should share information without collapsing to identical distributions (Gong et al., 2021).

Spiking neural networks admit an internal temporal decomposition. "Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers" (Ding et al., 9 Oct 2025) treats each timestep submodel

$\xi\in[0,1]$4

as a candidate teacher or student, ranking them by confidence $\xi\in[0,1]$5. This yields Strong2Weak and Weak2Strong distillation losses,

$\xi\in[0,1]$6

without external teachers or extra heads.

In reasoning-oriented LLM training, self-distillation is increasingly conditioned on policy state. PACED reallocates training budget to the competence frontier (Xu et al., 11 Mar 2026), whereas DemoPSD changes the target itself to attenuate privileged-information leakage (Li et al., 2 Jul 2026). These works suggest that modern policy self-distillation is less about copying a stronger static teacher than about controlling when, where, and to what extent self-generated supervision should be trusted.

5. Empirical performance patterns

The first robust empirical pattern is that one round of self-distillation can improve even a strong teacher. On CIFAR-10 and CIFAR-100 with strengthened teachers using cosine learning rate schedule, early stopping, Cutout, and AutoAugment, round-1 students outperform teachers in all reported cases. Representative examples include ResNet18 on CIFAR-10, where accuracy improves from $\xi\in[0,1]$7 to $\xi\in[0,1]$8; ResNet18 on CIFAR-10 with augmentation, from $\xi\in[0,1]$9 to $\xi>1$0; and ResNet18 on CIFAR-100 with augmentation, from $\xi>1$1 to $\xi>1$2 (Pham et al., 2022). The same paper also reports that gains are not monotonic across later rounds.

The second pattern is that repeated self-distillation can be substantially stronger than one-step self-distillation when the data geometry is favorable. In fixed-design linear regression, excess risk can improve over one-step self-distillation by a factor as large as $\xi>1$3, the input dimension, and UCI experiments report test-MSE reductions up to $\xi>1$4, with Air Quality dropping from $\xi>1$5 for optimal ridge to $\xi>1$6 for optimal 2-step self-distillation (Pareek et al., 2024).

The third pattern is that intensified self-distillation is particularly effective under noisy supervision. In the noisy-label analysis, the best-performing $\xi>1$7 in the reported linear-probing experiments is always greater than $\xi>1$8. Examples include Caltech-256 random corruption with ResNet-34, best at $\xi>1$9 with improvement $$\pi_\alpha(y\mid x)=\frac{\pi(y\mid x)^\alpha}{\sum_{y'\in\mathcal{Y}}\pi(y'\mid x)^\alpha}, \qquad \alpha>1.$$0; Caltech-256 random corruption with VGG-16, best at $$\pi_\alpha(y\mid x)=\frac{\pi(y\mid x)^\alpha}{\sum_{y'\in\mathcal{Y}}\pi(y'\mid x)^\alpha}, \qquad \alpha>1.$$1 with improvement $$\pi_\alpha(y\mid x)=\frac{\pi(y\mid x)^\alpha}{\sum_{y'\in\mathcal{Y}}\pi(y'\mid x)^\alpha}, \qquad \alpha>1.$$2; and Flowers-102 adversarial corruption at $$\pi_\alpha(y\mid x)=\frac{\pi(y\mid x)^\alpha}{\sum_{y'\in\mathcal{Y}}\pi(y'\mid x)^\alpha}, \qquad \alpha>1.$$3, best at $$\pi_\alpha(y\mid x)=\frac{\pi(y\mid x)^\alpha}{\sum_{y'\in\mathcal{Y}}\pi(y'\mid x)^\alpha}, \qquad \alpha>1.$$4 with improvement $$\pi_\alpha(y\mid x)=\frac{\pi(y\mid x)^\alpha}{\sum_{y'\in\mathcal{Y}}\pi(y'\mid x)^\alpha}, \qquad \alpha>1.$$5 (Das et al., 2023). In the linear-probing theory of label averaging, multi-round self-distillation improves for a few rounds and then degrades as predictions drift toward uniformity; the proposed PLL student approximates the benefits of multi-round distillation in one round and is especially strong at high corruption rates (Jeong et al., 2024).

The fourth pattern is that selective weighting helps in LLM reasoning. On Qwen2.5-Math-7B-Instruct self-distillation with reverse KL, PACED raises MATH-500 from $$\pi_\alpha(y\mid x)=\frac{\pi(y\mid x)^\alpha}{\sum_{y'\in\mathcal{Y}}\pi(y'\mid x)^\alpha}, \qquad \alpha>1.$$6 under unweighted reverse KL to $$\pi_\alpha(y\mid x)=\frac{\pi(y\mid x)^\alpha}{\sum_{y'\in\mathcal{Y}}\pi(y'\mid x)^\alpha}, \qquad \alpha>1.$$7, AIME 2024 from $$\pi_\alpha(y\mid x)=\frac{\pi(y\mid x)^\alpha}{\sum_{y'\in\mathcal{Y}}\pi(y'\mid x)^\alpha}, \qquad \alpha>1.$$8 to $$\pi_\alpha(y\mid x)=\frac{\pi(y\mid x)^\alpha}{\sum_{y'\in\mathcal{Y}}\pi(y'\mid x)^\alpha}, \qquad \alpha>1.$$9, and AIME 2025 from $$\mathcal{L}_{KD} = \alpha \mathcal{L}_{CE}(\boldsymbol{z}_s,\boldsymbol{y}) + (1-\alpha) \mathcal{L}_{KL}(\boldsymbol{z}_s, \boldsymbol{z}_t),$$0 to $$\mathcal{L}_{KD} = \alpha \mathcal{L}_{CE}(\boldsymbol{z}_s,\boldsymbol{y}) + (1-\alpha) \mathcal{L}_{KL}(\boldsymbol{z}_s, \boldsymbol{z}_t),$$1; MMLU rises from $$\mathcal{L}_{KD} = \alpha \mathcal{L}_{CE}(\boldsymbol{z}_s,\boldsymbol{y}) + (1-\alpha) \mathcal{L}_{KL}(\boldsymbol{z}_s, \boldsymbol{z}_t),$$2 to $$\mathcal{L}_{KD} = \alpha \mathcal{L}_{CE}(\boldsymbol{z}_s,\boldsymbol{y}) + (1-\alpha) \mathcal{L}_{KL}(\boldsymbol{z}_s, \boldsymbol{z}_t),$$3, corresponding to only $$\mathcal{L}_{KD} = \alpha \mathcal{L}_{CE}(\boldsymbol{z}_s,\boldsymbol{y}) + (1-\alpha) \mathcal{L}_{KL}(\boldsymbol{z}_s, \boldsymbol{z}_t),$$4 forgetting from the base model’s $$\mathcal{L}_{KD} = \alpha \mathcal{L}_{CE}(\boldsymbol{z}_s,\boldsymbol{y}) + (1-\alpha) \mathcal{L}_{KL}(\boldsymbol{z}_s, \boldsymbol{z}_t),$$5 (Xu et al., 11 Mar 2026). In the same study, a two-stage Paced forward-KL-then-reverse-KL schedule reaches MATH-500 $$\mathcal{L}_{KD} = \alpha \mathcal{L}_{CE}(\boldsymbol{z}_s,\boldsymbol{y}) + (1-\alpha) \mathcal{L}_{KL}(\boldsymbol{z}_s, \boldsymbol{z}_t),$$6, AIME 2024 $$\mathcal{L}_{KD} = \alpha \mathcal{L}_{CE}(\boldsymbol{z}_s,\boldsymbol{y}) + (1-\alpha) \mathcal{L}_{KL}(\boldsymbol{z}_s, \boldsymbol{z}_t),$$7, AIME 2025 $$\mathcal{L}_{KD} = \alpha \mathcal{L}_{CE}(\boldsymbol{z}_s,\boldsymbol{y}) + (1-\alpha) \mathcal{L}_{KL}(\boldsymbol{z}_s, \boldsymbol{z}_t),$$8, with MMLU forgetting $$\mathcal{L}_{KD} = \alpha \mathcal{L}_{CE}(\boldsymbol{z}_s,\boldsymbol{y}) + (1-\alpha) \mathcal{L}_{KL}(\boldsymbol{z}_s, \boldsymbol{z}_t),$$9.

The fifth pattern is that target sharpening can amortize inference-time sampling improvements into a cheaper student. For Qwen2.5-Math-7B on MATH500, the reported accuracies are $f^{(0)}\to f^{(1)}\to f^{(2)}\to \cdots$0 for Base + standard decoding, $f^{(0)}\to f^{(1)}\to f^{(2)}\to \cdots$1 for Base + temperature, $f^{(0)}\to f^{(1)}\to f^{(2)}\to \cdots$2 for Base + power sampling, and $f^{(0)}\to f^{(1)}\to f^{(2)}\to \cdots$3 for Power-distilled + temperature (Tomihari et al., 6 May 2026). The same work reports that teacher generation with Metropolis–Hastings power sampling takes more than a day per dataset/model, whereas student SFT finishes in under an hour per model on one GPU.

The sixth pattern is that internal self-distillation is not confined to supervised classification. In augmentation-free clustering on CIFAR-10, DeepCluster-v2 improves from $f^{(0)}\to f^{(1)}\to f^{(2)}\to \cdots$4 to $f^{(0)}\to f^{(1)}\to f^{(2)}\to \cdots$5 with self-distillation (Adnan et al., 2021). In SNNs, CIFAR10-DVS accuracy rises from $f^{(0)}\to f^{(1)}\to f^{(2)}\to \cdots$6 for a vanilla VGG-9 SNN to $f^{(0)}\to f^{(1)}\to f^{(2)}\to \cdots$7 with Strong2Weak and $f^{(0)}\to f^{(1)}\to f^{(2)}\to \cdots$8 with Weak2Strong; at $f^{(0)}\to f^{(1)}\to f^{(2)}\to \cdots$9 low-latency inference, accuracy rises from $\lim_{\gamma \to \infty}\xi^{*} > 1,$0 to $\lim_{\gamma \to \infty}\xi^{*} > 1,$1 and $\lim_{\gamma \to \infty}\xi^{*} > 1,$2, respectively (Ding et al., 9 Oct 2025). In feature self-distillation, MUSE improves the last module of ResNet34 on CIFAR-100 from $\lim_{\gamma \to \infty}\xi^{*} > 1,$3 baseline to $\lim_{\gamma \to \infty}\xi^{*} > 1,$4 with MI+SI, and improves YOLOv5-L on COCO from $\lim_{\gamma \to \infty}\xi^{*} > 1,$5 mAP to $\lim_{\gamma \to \infty}\xi^{*} > 1,$6 with MI$\lim_{\gamma \to \infty}\xi^{*} > 1,$7SI (Gong et al., 2021).

6. Limitations, controversies, and unresolved issues

Several misconceptions are explicitly challenged in the literature. One is that repeated self-distillation should improve monotonically with more rounds. Both loss-landscape analysis and label-averaging analysis contradict this: later rounds often fluctuate or degrade, and excessive repetition can wash out class structure (Pham et al., 2022, Jeong et al., 2024). Another is that the multi-view hypothesis is a complete explanation. It fails to explain why multi-round gains are not stepwise, why ensembles outperform distilled students, why BAN underperforms simple ensembling, and why some synthetic non-multi-view settings do not exhibit the same behavior (Pham et al., 2022).

Theoretical guarantees are also highly regime-dependent. The strongest repeated-self-distillation separation in linear regression requires distinct nonzero singular values and a “peaky” signal direction aligned with the top eigenvector; if singular values are equal, or if the signal is spread across directions, the separation disappears (Pareek et al., 2024). The label-averaging theory relies on fixed features and class-structured correlations $\lim_{\gamma \to \infty}\xi^{*} > 1,$8, so it does not directly transfer to end-to-end feature learning (Jeong et al., 2024). AIR-based noisy-label guarantees require overparameterization, random initialization, clusterability, and bounded corruption $\lim_{\gamma \to \infty}\xi^{*} > 1,$9 (Dong et al., 2019).

For sequence-level power self-distillation, local approximations are structurally limited. The odds-ratio identity

$\partial \xi^*/\partial \gamma^2 > 0$0

shows that sequence-level power depends on suffix Rényi entropies and cannot, in general, be reproduced by tokenwise temperature transforms without suffix information (Tomihari et al., 6 May 2026). A related limitation in policy self-distillation is privileged-information leakage: dense token-level imitation of a teacher conditioned on $\partial \xi^*/\partial \gamma^2 > 0$1 can encode answer-dependent shortcuts unavailable at test time. DemoPSD addresses this by reducing teacher influence where teacher-student disagreement is high, but it still requires tuning of $\partial \xi^*/\partial \gamma^2 > 0$2 and $\partial \xi^*/\partial \gamma^2 > 0$3 (Li et al., 2 Jul 2026).

Practical costs and heuristics remain significant. PACED adds rollout-based pass-rate estimation overhead, with default $\partial \xi^*/\partial \gamma^2 > 0$4, even though no architectural changes are required (Xu et al., 11 Mar 2026). SNN self-distillation uses confidence as a label-free proxy for strength, but the paper explicitly notes that confidence is only a heuristic and that simultaneous distillation can reduce diversity too much (Ding et al., 9 Oct 2025). Augmentation-free clustering results remain preliminary and are confined to CIFAR-10 in the reported study (Adnan et al., 2021).

Taken together, the literature suggests that power self-distillation is best understood not as a single mechanism, but as a design space for strengthening self-generated supervision. Depending on the regime, the strengthening operates through flatter minima, graph-like label averaging, NTK spectral filtering, richer polynomial shrinkage, competence-frontier weighting, or target sharpening. The shared empirical lesson is that intensified self-distillation can improve generalization and efficiency, but only when the intensified teacher signal remains aligned with the information the student can legitimately and productively absorb.