Prediction-Only Distillation in Linear and Logistic Regression

Abstract: Self-distillation (SD) is typically studied when the student is retrained on the teacher's original training inputs. In many practical deployments, however, the labeled training data are no longer available, and one has access only to the trained predictor and fresh unlabeled covariates. We study SD in this prediction-only regime through a fresh-X prediction-mixed scheme: a pure-distilled student is trained on fresh covariates pseudo-labeled by the teacher, and the final predictor is an affine combination of the teacher and student predictions. For ridge regression under proportional asymptotics, we derive deterministic equivalents for the optimally mixed prediction risk under general anisotropic covariance and deterministic signal. We show that this risk is strictly smaller than the teacher risk for almost every pair of teacher and student regularization levels, including when the fresh covariates are out-of-distribution and even when their covariance is isotropic. We further show that the optimal mixing weight cannot be identified from unlabeled data alone, but can be consistently estimated in a single post-training step using a small independent labeled calibration set, without additional model fitting. Finally, for binary logistic regression, we show that prediction mixing can outperform both the teacher and the pure-distilled classifier.

• The paper introduces fresh-X prediction-mixed self-distillation, combining a teacher with a ridge-trained pseudo-label student to generically reduce prediction risk even with OOD covariates and untuned penalties.
• The analysis derives high-dimensional risk formulas and proves strict improvement for almost every teacher–student penalty pair, while showing that more unlabeled data can initially hurt before improving performance.
• The paper establishes that unlabeled data cannot identify the optimal mixing weight, but a small labeled calibration set enables consistent tuning and logistic experiments show extrapolating mixtures can recover near-perfect accuracy after pseudo-label collapse.

Setting and motivation

The paper studies self-distillation (SD) in a prediction-only regime: the original labeled training data of a deployed teacher are unavailable, and the practitioner can only query the trained predictor on fresh unlabeled covariates, which may be out-of-distribution (OOD) relative to the teacher's training distribution. The proposed procedure is fresh-$X$ prediction-mixed self-distillation (PMSD): a pure-distilled (PD) student is trained by ridge regression on the teacher's pseudo-labels over fresh covariates $\tilde X$, and the final predictor is an affine combination $(1-\xi)f^{\te}(x)+\xi f_{\pd}(x)$ with real-valued mixing weight $\xi$ that need not lie in $[0,1]$. This reduces post-deployment adaptation to a one-dimensional aggregation problem requiring no retraining and no access to the labeled sample.

The central question is whether such mixing can provably improve a fixed teacher when the same-design geometric identities underlying prior same-$X$ SD guarantees (Das et al., 2023, Pareek et al., 2024) no longer hold because the student's design is independent of the teacher's.

Oracle decomposition and high-dimensional characterization

The structural core is an exact oracle identity: writing $C(\lambda_t,\lambda_s)$ for the conditional correlation between teacher and PD residuals and $D = R_{\te}+R_{\pd}-2C$ for their mean squared prediction difference, the optimal weight and risk satisfy

$\xi^\star = \frac{R_{\te}-C}{D}, \qquad R^\star_{pmsd} = R_{\te} - \frac{(R_{\te}-C)^2}{D},$

so $R^\star_{pmsd}\le \min\{R_{\te},R_{\pd}\}$ whenever $\tilde X$0, which holds generically. Under proportional asymptotics ($\tilde X$1 at fixed aspect ratios), with general anisotropic covariances $\tilde X$2 assumed simultaneously diagonalizable and deterministic signal, the paper derives deterministic equivalents for $\tilde X$3, $\tilde X$4, and $\tilde X$5 in terms of two coupled resolvents $\tilde X$6, $\tilde X$7, fixed-point parameters $\tilde X$8, and finitely many trace and quadratic-form functionals $\tilde X$9. These expressions are more involved than their same-$(1-\xi)f^{\te}(x)+\xi f_{\pd}(x)$0 counterparts but remain explicit, and empirical risks match the predictions closely at moderate dimensions.

Strict improvement guarantees

Because strict improvement fails exactly when $(1-\xi)f^{\te}(x)+\xi f_{\pd}(x)$1, the analysis reduces this gap to a rational function in $(1-\xi)f^{\te}(x)+\xi f_{\pd}(x)$2 whose numerator is a polynomial of degree at most $(1-\xi)f^{\te}(x)+\xi f_{\pd}(x)$3, where $(1-\xi)f^{\te}(x)+\xi f_{\pd}(x)$4 is the number of distinct eigenvalues of $(1-\xi)f^{\te}(x)+\xi f_{\pd}(x)$5. Two consequences follow:

• Teacher-degeneracy: for any fixed $(1-\xi)f^{\te}(x)+\xi f_{\pd}(x)$6, the set of "bad" student penalties has cardinality at most $(1-\xi)f^{\te}(x)+\xi f_{\pd}(x)$7; moreover, the set of teacher penalties for which no $(1-\xi)f^{\te}(x)+\xi f_{\pd}(x)$8 helps is bounded above by $(1-\xi)f^{\te}(x)+\xi f_{\pd}(x)$9 where $\xi$0 counts distinct eigenvalues of $\xi$1. Hence, under mild nondegeneracy, PMSD strictly improves the teacher for almost every pair $\xi$2 — including when $\xi$3 is unknown and $\xi$4 is not tuned.
• Concrete cases: for isotropic design, degeneracy occurs only at the ridge-optimal value $\xi$5; for spiked covariances with $\xi$6 spikes, failure occurs for at most $\xi$7 values of $\xi$8; and notably, even isotropic Gaussian fresh covariates (which commute with any $\xi$9) yield strict improvement for all $[0,1]$0 unless a non-generic alignment condition holds. In the balanced regime $[0,1]$1, $[0,1]$2, the fresh-$[0,1]$3 construction recovers the tangent identity $[0,1]$4, connecting back to same-$[0,1]$5 results, with the sign rule $[0,1]$6.

Against the PD student itself, in the fully isotropic setting PMSD strictly improves for all but at most two values of $[0,1]$7.

Non-monotonicity in unlabeled data

A notable negative result: in the same-$[0,1]$8 isotropic setting, the optimal PMSD risk is not monotone in the unlabeled aspect ratio $[0,1]$9. It is unimodal with a unique maximum at $X$0, so adding more unlabeled data can initially hurt performance before helping. This phenomenon is absent from same-$X$1 distillation and contrasts with risk-monotonization results from subsampling and aggregation.

Data-dependent tuning

The oracle weight cannot be identified from unlabeled data alone: the paper constructs two response models sharing identical observables (teacher predictor and unlabeled design) but with oracle weights $X$2 versus $X$3, establishing an information-theoretic impossibility rather than a computational one. However, a small independent labeled calibration set suffices: plug-in estimates of $X$4, $X$5, and $X$6 from calibration residuals yield consistent estimators of $X$7 and $X$8 as $X$9, uniformly over finite candidate collections of students. Crucially, since calibration evaluates fixed predictors, no high-dimensional condition relating $C(\lambda_t,\lambda_s)$0 to $C(\lambda_t,\lambda_s)$1 is required, and tuning costs only three empirical averages after one training run.

Logistic regression

For binary classification with hard-label teachers of class-balanced error rate $C(\lambda_t,\lambda_s)$2, under the block-correlation feature assumption validated empirically on six image datasets, the paper proves three results. First, there exists $C(\lambda_t,\lambda_s)$3 such that for $C(\lambda_t,\lambda_s)$4 the PD student matches the teacher's $C(\lambda_t,\lambda_s)$5 accuracy, yet some $C(\lambda_t,\lambda_s)$6 makes the PMSD classifier achieve asymptotic $C(\lambda_t,\lambda_s)$7 accuracy for every $C(\lambda_t,\lambda_s)$8. Second, mixing soft predictions across two successive distillation rounds achieves $C(\lambda_t,\lambda_s)$9 accuracy for all $D = R_{\te}+R_{\pd}-2C$0, extending the noise-tolerance thresholds of prior work. Third, and most striking, when $D = R_{\te}+R_{\pd}-2C$1 both successive PD students collapse to $D = R_{\te}+R_{\pd}-2C$2 population accuracy, while a PMSD classifier with extrapolating weight $D = R_{\te}+R_{\pd}-2C$3 still attains asymptotic $D = R_{\te}+R_{\pd}-2C$4 accuracy. Linear probing experiments on Caltech-101, Caltech-256, and CIFAR-100 with corrupted labels confirm gains across regularization levels, with estimated weights frequently far outside $D = R_{\te}+R_{\pd}-2C$5.

Scope, limitations, and open questions

The paper is explicit about boundaries. Compared with same-$D = R_{\te}+R_{\pd}-2C$6 loss-mixed SD, fresh-$D = R_{\te}+R_{\pd}-2C$7 PMSD operates with strictly less information: it cannot recover information lost in a mistuned teacher regardless of unlabeled sample size, and in isotropic designs it generally cannot match the optimally tuned ridge risk, though in spiked models with additional unlabeled data it can outperform the optimal same-$D = R_{\te}+R_{\pd}-2C$8 spectral shrinkage estimator. The theoretical analysis requires simultaneous diagonalizability of $D = R_{\te}+R_{\pd}-2C$9 and $\xi^\star = \frac{R_{\te}-C}{D}, \qquad R^\star_{pmsd} = R_{\te} - \frac{(R_{\te}-C)^2}{D},$0, and the logistic results rely on the idealized block-correlation Gram structure. Within its scope, the paper formulates the design problem of choosing the fresh covariance $\xi^\star = \frac{R_{\te}-C}{D}, \qquad R^\star_{pmsd} = R_{\te} - \frac{(R_{\te}-C)^2}{D},$1: in the underparameterized ridgeless limit, maximizing the risk reduction reduces to a linearly constrained quadratic program, and matching the teacher's covariance need not be optimal. Joint optimization of $\xi^\star = \frac{R_{\te}-C}{D}, \qquad R^\star_{pmsd} = R_{\te} - \frac{(R_{\te}-C)^2}{D},$2 and $\xi^\star = \frac{R_{\te}-C}{D}, \qquad R^\star_{pmsd} = R_{\te} - \frac{(R_{\te}-C)^2}{D},$3 beyond this setting remains open, as does reducing the label cost of the calibration step.

Conclusion

This paper establishes that prediction-only distillation with affine prediction mixing yields generic strict improvement over both teacher and pure-distilled students in high-dimensional ridge regression — even with OOD or isotropic fresh covariates and untuned penalties — characterizes the exact dependence on unlabeled sample size (including non-monotonicity), identifies a fundamental identifiability barrier overcome by lightweight labeled calibration, and demonstrates, in logistic regression, that mixing can rescue classifiers from regimes where iterated pseudo-label training collapses entirely.