RCE-KD: Rejuvenated CE for Recommender KD

• The paper introduces RCE-KD, a novel method that re-engineers cross-entropy loss to better approximate NDCG in ranking distillation for recommendation systems.
• It employs a split–sample–fuse design that divides teacher top items based on student alignment and adaptively weights losses to satisfy the closure condition required for effective ranking.
• Empirical results demonstrate 3–12% relative improvements over existing KD methods across multiple backbones and datasets while maintaining efficient training and robust performance.

Searching arXiv for the specified papers to ground the article in the cited sources. Rejuvenated Cross-Entropy for Knowledge Distillation (RCE-KD) is a knowledge-distillation method for recommender systems that re-examines cross-entropy (CE) as a ranking distillation objective under partial-item training. It is introduced in “Rejuvenating Cross-Entropy Loss in Knowledge Distillation for Recommender Systems” (Zhu et al., 25 Sep 2025). The method starts from the observation that recommender-system KD is ranking-centric, especially for the teacher’s top items, and that full-item distillation is infeasible because the item universe is huge, so CE is computed on only a small sampled subset. The central claim is that CE is theoretically aligned with ranking quality through a lower-bound relationship to NDCG, but only when the sampled item subset satisfies a closure condition that is typically violated by naïve distillation on the teacher’s top-$K$ items. RCE-KD is designed to restore that theoretical connection by splitting teacher top items into overlap and non-overlap subsets, sampling additional student-side items for the non-overlap case, and adaptively combining the resulting losses (Zhu et al., 25 Sep 2025).

1. Problem setting and conceptual motivation

RCE-KD is formulated for recommender-system knowledge distillation rather than the dense-output settings more typical of CV and NLP. In this setting, the task is ranking-centric, especially for the teacher’s top items, and CE can only be computed on a small subset of items. The paper explicitly notes that when CE is naively applied to the teacher’s top-$K$ items, it often performs worse than response-based KD baselines such as RRD, DCD, and HetComp (Zhu et al., 25 Sep 2025).

The method is therefore motivated by a mismatch between theoretical desiderata and practical usage. The paper asks whether CE is intrinsically unsuitable for KD in recommendation or whether it is being used incorrectly. Its answer is the latter: CE is presented as theoretically sensible for ranking distillation, but only under a subset condition that typical top-$K$ teacher sampling does not satisfy. This reframes the problem from replacing CE to repairing the conditions under which CE is applied.

A plausible implication is that the method belongs to a broader line of work that treats KD objectives not merely as empirical heuristics but as surrogates whose validity depends on the geometry of the distilled subset. In the recommender setting, the subset itself becomes part of the objective design rather than a neutral implementation detail.

2. The CE–NDCG connection

The theoretical core of RCE-KD is the claim that minimizing CE maximizes a lower bound on NDCG when the teacher score distribution is used as the soft relevance target (Zhu et al., 25 Sep 2025). For a user $u$, with teacher score vector $r_u^T$ and student score vector $r_u^S$, the CE loss over an item set $J^u$ is defined as

$$L_{CE}=-\frac{1}{| U|}\sum_{u\in U}\sum_{i\in J^u}\sigma(r_{ui}^T, J^u)\log\sigma(r_{ui}^S, J^u),$$

where

$$\sigma(r_{ui}, J^u)=\frac{\exp(r_{ui})}{\sum_{j\in J^u}\exp(r_{uj})}.$$

The corresponding ranking metric is written in the usual form: $$\mathrm{NDCG}(\pi,y)=\frac{\mathrm{DCG}(\pi,y)}{\mathrm{DCG}(\tilde\pi,y)},$$ with

$K$0

For the full-item case $K$1, the paper states that if the relevance scores are set to

$K$2

then minimizing CE on all items maximizes a lower bound of NDCG. The proof sketch given in the source summary states that after substituting this relevance definition into DCG and using standard inequalities of the $K$3 type together with Jensen’s inequality, the resulting lower bound becomes exactly the negative of CE (Zhu et al., 25 Sep 2025).

This result matters because it gives CE a direct ranking interpretation in KD for recommendation. Rather than viewing CE as merely a probabilistic matching objective, the paper interprets it as a lower-bound surrogate for NDCG when the teacher distribution provides the soft labels. This suggests that poor empirical performance of naïve CE in recommendation arises from subset mismatch rather than from an inherent incompatibility between CE and ranking.

3. Closure and the failure of naïve partial-item distillation

In practice, CE cannot be computed over all items, so KD is performed on a subset $K$4. The paper therefore introduces partial NDCG: $K$5 where the truncated relevance vector is

$K$6

The crucial condition is the closure of $K$7: $K$8 The interpretation given in the source is that if an item $K$9 is in the subset, then every item that the student ranks above $K$0 must also be in the subset (Zhu et al., 25 Sep 2025). Under this condition, minimizing CE on $K$1 maximizes a lower bound of $K$2.

The paper’s proof idea makes closure operational rather than purely formal. Starting from partial NDCG, it lower-bounds DCG, replaces $K$3 by the teacher-induced soft label $K$4, and uses a rank bound involving $K$5. Under closure, the items ranked above $K$6 by the student are contained inside $K$7, so the rank denominator can be upper-bounded by the sum over $K$8, producing a softmax-style probability and hence CE plus a constant. The source summary writes the resulting inequality as

$K$9

with $u$0 fixed for a given $u$1 (Zhu et al., 25 Sep 2025).

The practical difficulty is that the teacher’s top items and the student’s top items differ substantially, especially early in training. The paper reports that teacher top-$u$2 items are frequently ranked far lower by the student, often below rank 100 or even 200, and that this mismatch is strongest at the beginning of training. Consequently, using only the teacher’s top items as the CE subset usually violates closure, so the CE–NDCG guarantee no longer applies (Zhu et al., 25 Sep 2025).

4. Construction of RCE-KD

RCE-KD addresses the closure problem through a split–sample–fuse design. Let

$u$3

The teacher’s top items are split into

$u$4

The first subset contains teacher top items that the student also ranks highly; the second contains teacher top items that the student does not currently rank highly (Zhu et al., 25 Sep 2025).

For the overlapping subset, the loss is computed on the student’s top items: $u$5 Because $u$6 is the student’s top set, the paper states that it satisfies the closure assumption. The method therefore uses a subset with exact theoretical support for the portion of teacher top items already covered by the student.

For the non-overlapping subset, direct application of the theorem is impossible because $u$7 violates closure. RCE-KD therefore samples additional items from the student side to form a new set $u$8. The source summary describes the procedure as follows: for each item $u$9, consider items ranked above $r_u^T$0 by the student; for each such higher-ranked item $r_u^T$1, increment a score $r_u^T$2 by 1; then sample items $r_u^T$3 with probability

$r_u^T$4

where $r_u^T$5 is a temperature hyperparameter, fixed to 10 in experiments. After sampling $r_u^T$6 items, they are unioned with $r_u^T$7 to form $r_u^T$8, and the second loss is

$r_u^T$9

The final KD loss is

$r_u^S$0

with adaptive weight

$r_u^S$1

The total training loss is

$r_u^S$2

where $r_u^S$3 is the recommender’s normal loss, for example BPR (Zhu et al., 25 Sep 2025).

The adaptive fusion has a specific interpretation in the source material. If the overlap $r_u^S$4 is small, then $r_u^S$5 is larger and $r_u^S$6 receives more weight, because the model still needs help aligning low-overlap teacher items. If the overlap is large, $r_u^S$7 receives more weight, because the student can already use the exact closure-compatible subset to refine fine-grained ranking.

5. Empirical findings, scope, and efficiency

The experiments are reported on three public datasets: CiteULike, Gowalla, and Yelp2018. All are filtered and split chronologically into train, validation, and test. The method is evaluated in both homogeneous and heterogeneous teacher–student settings: MF $r_u^S$8 MF, LightGCN $r_u^S$9 LightGCN, HSTU $J^u$0 HSTU, LightGCN $J^u$1 MF, and HSTU $J^u$2 MF. The baselines are CD, RRD, DCD, and HetComp. Metrics are Recall@10, NDCG@10, Recall@20, and NDCG@20 (Zhu et al., 25 Sep 2025).

The paper reports that RCE-KD is best in essentially all reported settings. The stated improvements over the best baseline are about 3–12% relative improvement depending on dataset, backbone, and metric. Examples given in the source summary include MF$J^u$3MF on CiteULike with +9.02% R@10 and +11.76% N@10, LightGCN$J^u$4LightGCN on Yelp with +6.42% R@10 and +8.20% N@10, and HSTU$J^u$5HSTU on CiteULike with +11.49% R@10 and +8.70% N@10 (Zhu et al., 25 Sep 2025).

The paper also reports efficiency claims: training time per epoch is comparable to or better than other KD methods, GPU memory usage is similarly competitive, and overhead is modest because the method only adds a sampling step on top of CE. The ablation study is reported to show that splitting teacher top items into two subsets, using student top items in the distillation process, and using the adaptive weighting scheme all matter. The source summary further states that variants denoted w/o S, w/o T, w/ const, and w/o sep all underperform the full method, and that w/o S usually performs worse than w/o T, which is interpreted there as evidence that student top items are essential for satisfying closure (Zhu et al., 25 Sep 2025).

An additional empirical point concerns how well the constructed set approximates the ideal closure set. The overlap rate

$J^u$6

is reported as around 0.52–0.73 at the beginning depending on dataset and setting, around 0.88–0.96 by mid-training, and around 0.95–0.98 by the end. This is presented as support for the claim that teacher-student collaboration can effectively approximate closure as training proceeds. The same source also states that RCE-KD generalizes to sequential recommendation, where it continues to outperform baselines on the same datasets (Zhu et al., 25 Sep 2025).

6. Relation to other cross-entropy reformulations of KD

RCE-KD should be distinguished from earlier cross-entropy-based re-analyses of KD outside recommender systems. In particular, “Rethinking Knowledge Distillation via Cross-Entropy” (Yang et al., 2022) reformulates classical KD by decomposing the KD loss into the ordinary CE loss plus an extra CE-like term, argues that the extra term mismatches teacher absolute probabilities with student relative probabilities, and proposes a distributed loss, a soft loss, New Knowledge Distillation (NKD), and teacher-free NKD (tf-NKD). The source material explicitly states that the authors of that paper do not use the names “rejuvenated cross-entropy,” “revised cross-entropy,” or “cross-entropy reformulation” for the main proposal; the relevant terms are distributed loss, soft loss, NKD, and tf-NKD (Yang et al., 2022).

The distinction is substantive as well as terminological. In (Yang et al., 2022), the central issue is the decomposition of classical logit-based KD and the mismatch between teacher absolute probabilities and student relative probabilities once the target class is factored out. The proposed correction normalizes non-target distributions: $J^u$7 and defines the distributed loss

$J^u$8

The final NKD loss is

$J^u$9

with teacher-free tf-NKD replacing the teacher-derived soft target with a smoothed version of the student’s own target output (Yang et al., 2022).

By contrast, RCE-KD in recommender systems does not arise from decomposing classical KD into CE plus an auxiliary CE-like term. Its defining issue is partial-item ranking distillation and the closure condition required for CE to lower-bound NDCG. This suggests that the two papers share a family resemblance at the level of “rethinking KD via cross-entropy,” but they solve different mismatches in different domains: (Yang et al., 2022) addresses probability-space mismatch in classification KD, whereas (Zhu et al., 25 Sep 2025) addresses subset-selection mismatch in ranking KD for recommendation.

7. Interpretation and significance

RCE-KD’s main significance lies in rehabilitating CE as a principled KD objective for recommender systems under partial-item training. The paper’s bottom-line message is that CE loss is not the problem; misuse is (Zhu et al., 25 Sep 2025). Within the paper’s own framework, CE becomes theoretically justified because it maximizes a lower bound of NDCG when the distilled subset obeys closure, and the method is specifically engineered to approximate that condition when naïve teacher-top-item distillation fails.

This perspective also clarifies a potential misconception. Poor performance of vanilla CE on teacher top-$$L_{CE}=-\frac{1}{| U|}\sum_{u\in U}\sum_{i\in J^u}\sigma(r_{ui}^T, J^u)\log\sigma(r_{ui}^S, J^u),$$0 items does not, in the paper’s account, imply that CE is intrinsically misaligned with ranking. Rather, it indicates that the subset does not respect the student’s current ranking structure. RCE-KD therefore makes student-side information central to distillation, not merely as an optimization convenience but as a condition for preserving the CE–NDCG relationship.

A plausible implication is that RCE-KD exemplifies a broader methodological pattern in recommendation distillation: the objective, the sampled subset, and the evolving student ranking should be designed jointly. In this reading, the contribution is not only a new loss construction, but a re-specification of what it means for a sampled KD objective to remain ranking-faithful under realistic computational constraints.