Concrete Score Distillation (CSD)

• The paper introduces CSD, a discrete score-matching objective that trains a student model to align pairwise logit differences with a teacher’s, overcoming the smoothing effects of softmax.
• CSD leverages a factorized weighting scheme to reduce computations from quadratic to linear complexity while enabling various fidelity–diversity trade-offs using teacher, student, or uniform distributions.
• Empirical results demonstrate that CSD outperforms traditional KD methods on metrics like ROUGE-L and COMET, making it a robust approach for autoregressive language modeling.

Concrete Score Distillation (CSD) most explicitly denotes a discrete score-matching objective for knowledge distillation in autoregressive LLMs, where the student is trained to match the teacher’s pairwise logit differences rather than only its softmax probabilities (Kim et al., 30 Sep 2025). The expression sits within a broader literature on score distillation and concrete scores, but the acronym itself is not standardized: in adjacent work, “CSD” also refers to Classifier Score Distillation, Collaborative Score Distillation, Coupled Score Distillation, Cascaded Score Distillation, and Canonical Score Distillation (Yu et al., 2023, Kim et al., 2023, Yang et al., 7 May 2025, Decatur et al., 2023, Wang et al., 2023). In that sense, Concrete Score Distillation is both a specific LLM distillation method and part of a wider attempt to make score-based objectives explicit, ratio-based, and structurally matched to discrete or multi-view targets.

1. Terminological scope and acronym overload

In the LLM distillation setting, “Concrete Score Distillation” is the name of a discrete score-matching objective that overcomes both softmax-induced smoothing and restrictions on the optimal solution set, while aligning relative logit differences across all vocabulary pairs between student and teacher with flexible weighting (Kim et al., 30 Sep 2025). In the 3D and diffusion literature, however, the same acronym has several unrelated expansions. “Text-to-3D with classifier score distillation” uses CSD for Classifier Score Distillation and defines it as the classifier-score term induced by classifier-free guidance (Yu et al., 2023). “Collaborative Score Distillation for Consistent Visual Synthesis” uses CSD for a Stein Variational Gradient Descent formulation that couples multiple images through kernelized score sharing (Kim et al., 2023). “Bridging Geometry-Coherent Text-to-3D Generation with Multi-View Diffusion Priors and Gaussian Splatting” uses CSD for Coupled Score Distillation, a multi-view joint optimization rule for text-to-3D (Yang et al., 7 May 2025). “3D Paintbrush” uses CSD for Cascaded Score Distillation across all stages of a cascaded diffusion model (Decatur et al., 2023), and “AnimatableDreamer” uses CSD for Canonical Score Distillation in non-rigid 4D generation and reconstruction (Wang et al., 2023).

This naming instability is itself part of the subject. In “NeRF Inpainting with Geometric Diffusion Prior and Balanced Score Distillation,” the term CSD is explicitly described as “Classifier Score Distillation” following Yu et al., but is also called Conditional in some places, and the paper notes that this is a naming inconsistency in the literature (Zhang et al., 2024). As a result, discussions of Concrete Score Distillation require distinguishing a discrete-logit distillation objective from several score-distillation methods in vision that share the same acronym but optimize different quantities.

2. Formal objective in autoregressive language modeling

The concrete score used by the LLM formulation is the discrete analogue of a continuous score. For a discrete distribution $q_\theta$ over a finite set $\mathcal{Y}$, the concrete score is defined as

$$s_\theta(y) := \left[ \frac{q_\theta(x)}{q_\theta(y)} \right]_{x\in\mathcal{Y}},$$

so it is a collection of probability ratios rather than a gradient in input space (Kim et al., 30 Sep 2025). In autoregressive distillation, the teacher distribution $p_T$ and the student distribution $q_\theta$ are defined tokenwise from logits $f_T$ and $f_\theta$ through the standard softmax: $$q_{\theta}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_\theta[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_\theta[x])},\quad p_{T}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_{T}[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_{T}[x])}.$$

The motivation is twofold. First, softmax-based KD blurs fine logit structure; the paper reports that in large vocabularies only $0.0023\%$ of token probabilities exceed $0.01$, so most of the teacher’s tail information is heavily compressed by softmax (Kim et al., 30 Sep 2025). Second, Direct Logit Distillation (DLD) avoids softmax smoothing but ignores logit shift invariance, because any student logits of the form $\mathcal{Y}$0 induce exactly the same probabilities as the teacher under softmax, whereas DLD still forces pointwise equality (Kim et al., 30 Sep 2025).

Concrete Score Distillation addresses both issues by matching pairwise log-ratios: $\mathcal{Y}$1 This makes the target of the loss explicit: CSD matches all pairwise logit differences $\mathcal{Y}$2 to the corresponding teacher differences $\mathcal{Y}$3 (Kim et al., 30 Sep 2025).

Because the objective depends only on differences of logits, it is shift-invariant. If $\mathcal{Y}$4 for all $\mathcal{Y}$5, every pairwise difference is matched exactly, so $\mathcal{Y}$6 and the student probabilities equal the teacher probabilities under softmax (Kim et al., 30 Sep 2025). The paper proves that the CSD solution set strictly contains the DLD solution set: $\mathcal{Y}$7 That is a central property of the method: it preserves logit-level information without imposing the unnecessary pointwise constraints of DLD.

3. Weighting, optimization, and computational form

A naive implementation of the double sum in $\mathcal{Y}$8 is quadratic in vocabulary size and numerically unstable if one works directly with raw probability ratios. The LLM CSD paper resolves both issues by replacing ratios with log-ratios and by assuming a rank-1 factorization of the pairwise weights,

$\mathcal{Y}$9

which yields an $$s_\theta(y) := \left[ \frac{q_\theta(x)}{q_\theta(y)} \right]_{x\in\mathcal{Y}},$$0 gradient formula (Kim et al., 30 Sep 2025).

Under this factorization, the gradient can be written in terms of weighted-centered logits. Define

$$s_\theta(y) := \left[ \frac{q_\theta(x)}{q_\theta(y)} \right]_{x\in\mathcal{Y}},$$1

Then

$$s_\theta(y) := \left[ \frac{q_\theta(x)}{q_\theta(y)} \right]_{x\in\mathcal{Y}},$$2

where $$s_\theta(y) := \left[ \frac{q_\theta(x)}{q_\theta(y)} \right]_{x\in\mathcal{Y}},$$3 and the tilde-vectors collect centered logits under $$s_\theta(y) := \left[ \frac{q_\theta(x)}{q_\theta(y)} \right]_{x\in\mathcal{Y}},$$4 and $$s_\theta(y) := \left[ \frac{q_\theta(x)}{q_\theta(y)} \right]_{x\in\mathcal{Y}},$$5 (Kim et al., 30 Sep 2025). Operationally, the computation reduces to a few weighted averages and centered-logit residuals, rather than an explicit pairwise matrix.

The weighting choice determines the fidelity–diversity profile. The paper studies three distributions over the vocabulary: teacher probabilities $$s_\theta(y) := \left[ \frac{q_\theta(x)}{q_\theta(y)} \right]_{x\in\mathcal{Y}},$$6, detached student probabilities $$s_\theta(y) := \left[ \frac{q_\theta(x)}{q_\theta(y)} \right]_{x\in\mathcal{Y}},$$7, and uniform $$s_\theta(y) := \left[ \frac{q_\theta(x)}{q_\theta(y)} \right]_{x\in\mathcal{Y}},$$8. This yields a family of practical variants:

Variant	Weighting choice	Reported behavior
CSD(S,S)	$$s_\theta(y) := \left[ \frac{q_\theta(x)}{q_\theta(y)} \right]_{x\in\mathcal{Y}},$$9	Highest fidelity; mode-seeking
CSD(U,S)	$p_T$0	More diversity; more mode-covering
CSD(T,S)	$p_T$1	Different fidelity–diversity trade-off

The paper explicitly characterizes these as providing both mode-seeking and mode-covering instances within the same framework (Kim et al., 30 Sep 2025). It also notes that CSD’s gradient resembles KL in having a “student-normalized minus teacher-normalized” structure, but the normalization is a centering in logit space rather than a softmax normalization in probability space. This preserves relative spacing among logits rather than compressing it through exponentiation and normalization (Kim et al., 30 Sep 2025).

4. Concrete scores beyond language: discrete diffusion and ratio-based modeling

The broader “concrete score” program is older and more general than the LLM distillation use of the term. In Target Concrete Score Matching, the concrete score of a discrete distribution $p_T$2 over sequence space is defined on a neighborhood graph as

$p_T$3

and the paper states that if the graph induced by $p_T$4 is weakly connected, matching concrete scores uniquely determines the distribution $p_T$5 (Zhang et al., 23 Apr 2025). TCSM uses this idea to train discrete diffusion models in both pre-training and post-training settings, including fine-tuning using reward functions or preference data, and distillation of knowledge from pre-trained autoregressive models (Zhang et al., 23 Apr 2025).

A closely related CTMC-based formulation appears in “Sampling from Energy distributions with Target Concrete Score Identity,” where the concrete score is

$p_T$6

That paper derives the Target Concrete Score Identity,

$p_T$7

which allows Monte Carlo estimation of discrete scores using only the unnormalized target density and the forward uniform diffusion kernel, without requiring samples from the target distribution or computation of the partition function (Kholkin et al., 27 Oct 2025). That identity is then used to train neural approximations of concrete scores and to define reverse-time CTMC samplers (Kholkin et al., 27 Oct 2025).

In graph generation, “Composable Score-based Graph Diffusion Model for Multi-Conditional Molecular Generation” extends score matching to discrete molecular graphs via concrete scores, then introduces Composable Guidance and Probability Calibration. The paper states that “Concrete Score Distillation (CSD)” is not a term used explicitly there, but the model builds directly on concrete scores and uses them as the basis for conditional recombination and calibrated reverse transitions (Qiao et al., 11 Sep 2025). This suggests a broader interpretation of CSD as any procedure that learns, matches, or composes ratio-valued discrete score fields and then distills them into a generator or reverse process.

5. Relation to score distillation in vision and 3D

The phrase “score distillation” emerged primarily in continuous image and 3D settings, where it typically means using a pretrained diffusion model’s score or noise predictor to optimize another parameterized object. In that literature, CSD often names non-concrete objectives. “Text-to-3D with classifier score distillation” decomposes SDS under classifier-free guidance into a generative term and a classifier-like term,

$p_T$8

and then defines CSD by keeping only the classifier score

$p_T$9

The core claim is that “guidance alone is enough” for effective text-to-3D generation (Yu et al., 2023).

In NeRF inpainting, CSD appears again as Classifier Score Distillation or Conditional Score Distillation, but the paper’s main refinement is Balanced Score Distillation (BSD). Its refined CSD form introduces independent weights on the positive, negative, and unconditional predictions,

$q_\theta$0

and the paper reports that positive $q_\theta$1 leads to blurry reconstructions, negative $q_\theta$2 introduces artifacts, and best reconstruction quality occurs as $q_\theta$3 approaches zero (Zhang et al., 2024). BSD then removes the unconditional term entirely.

Other 3D formulations use the acronym differently. Collaborative Score Distillation treats multiple images as SVGD particles and couples their gradients to improve inter-sample consistency (Kim et al., 2023). Coupled Score Distillation combines a single-view score term, a multi-view score term, and a noise term to optimize geometry-consistent 3D Gaussian Splatting from a joint multi-view KL (Yang et al., 7 May 2025). Cascaded Score Distillation distills all stages of a cascaded text-to-image model for local mesh texturing (Decatur et al., 2023). Canonical Score Distillation backpropagates multi-view diffusion scores through rendering and a learned warping field into a shared canonical 3D model and its articulation parameters (Wang et al., 2023).

A separate theoretical line, “Rethinking Score Distillation as a Bridge Between Image Distributions,” interprets SDS-like updates as first-order approximations to an optimal-cost transport path from a source distribution to a target distribution. It identifies two recurring failure modes—linear approximation of the optimal path and poor estimates of the source distribution—and proposes calibrating the text conditioning of the source distribution to improve image, NeRF, and 3D optimization quality (McAllister et al., 2024). Although that work does not use the name Concrete Score Distillation, it reinforces the idea that making the source and target scores explicit is central to stabilizing distillation.

6. Empirical profile of the LLM objective

The LLM CSD paper evaluates the method on task-agnostic instruction-following and task-specific distillation using GPT-2-1.5B, OpenLLaMA-7B, and GEMMA-7B-IT teachers (Kim et al., 30 Sep 2025). On GPT-2-1.5B $q_\theta$4 GPT-2-0.1B pure distillation, CSD(S,S) reaches an average ROUGE-L of $q_\theta$5, compared with $q_\theta$6 for SRKL(0.1), $q_\theta$7 for TV, and $q_\theta$8 for ABKD(0.2,0.7) (Kim et al., 30 Sep 2025). Under on-policy distillation, replacing KL-based objectives with CSD remains complementary: for GPT-2-1.5B $q_\theta$9 GPT-2-0.1B, DistiLLM improves from $f_T$0 to $f_T$1 average ROUGE-L when combined with CSD, and ImitKD improves from $f_T$2 to $f_T$3 (Kim et al., 30 Sep 2025). For GPT-2-1.5B $f_T$4 GPT-2-0.3B, ImitKD + CSD reaches $f_T$5, exceeding the best reported baseline of $f_T$6 from DistiLLM SRKL (Kim et al., 30 Sep 2025). For OpenLLaMA-7B $f_T$7 OpenLLaMA-3B, ImitKD + CSD reaches $f_T$8, above $f_T$9 for DistiLLM SRKL (Kim et al., 30 Sep 2025).

On task-specific distillation from GEMMA-7B-IT to GEMMA-2B-IT, CSD attains $f_\theta$0 ROUGE-L on DialogSum summarization, $f_\theta$1 COMET on Flores-200 translation, and $f_\theta$2 GSM8K accuracy, outperforming KL at $f_\theta$3, $f_\theta$4, and $f_\theta$5, respectively (Kim et al., 30 Sep 2025). The paper also reports that several highly mode-seeking probability-based baselines, including TV, SKL, and SRKL, collapse to $f_\theta$6 GSM8K accuracy, while CSD remains stable (Kim et al., 30 Sep 2025).

A brief summary of representative results is given below.

Setting	Representative baselines	CSD result
GPT-2-1.5B $f_\theta$7 GPT-2-0.1B, pure distillation	SRKL(0.1): $f_\theta$8, TV: $f_\theta$9, ABKD(0.2,0.7): $$q_{\theta}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_\theta[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_\theta[x])},\quad p_{T}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_{T}[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_{T}[x])}.$$0	CSD(S,S): $$q_{\theta}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_\theta[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_\theta[x])},\quad p_{T}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_{T}[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_{T}[x])}.$$1
GPT-2-1.5B $$q_{\theta}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_\theta[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_\theta[x])},\quad p_{T}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_{T}[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_{T}[x])}.$$2 GPT-2-0.1B, on-policy	DistiLLM: $$q_{\theta}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_\theta[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_\theta[x])},\quad p_{T}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_{T}[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_{T}[x])}.$$3, ImitKD: $$q_{\theta}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_\theta[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_\theta[x])},\quad p_{T}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_{T}[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_{T}[x])}.$$4	DistiLLM + CSD: $$q_{\theta}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_\theta[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_\theta[x])},\quad p_{T}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_{T}[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_{T}[x])}.$$5, ImitKD + CSD: $$q_{\theta}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_\theta[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_\theta[x])},\quad p_{T}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_{T}[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_{T}[x])}.$$6
GEMMA-7B-IT $$q_{\theta}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_\theta[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_\theta[x])},\quad p_{T}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_{T}[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_{T}[x])}.$$7 GEMMA-2B-IT	KL: $$q_{\theta}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_\theta[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_\theta[x])},\quad p_{T}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_{T}[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_{T}[x])}.$$8 / $$q_{\theta}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_\theta[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_\theta[x])},\quad p_{T}(y_t\mid\mathbf{c},\mathbf{y}_{<t})=\frac{\exp(f_{T}[y_t])}{\sum_{x\in\mathcal{V}}\exp(f_{T}[x])}.$$9 / $0.0023\%$0	CSD: $0.0023\%$1 / $0.0023\%$2 / $0.0023\%$3

These results support the paper’s claim that CSD consistently surpasses recent KD objectives, achieves favorable fidelity-diversity trade-offs, and yields complementary gains when combined with on-policy techniques (Kim et al., 30 Sep 2025).

7. Limitations and prospective directions

The present LLM formulation of Concrete Score Distillation has several explicit constraints. Its efficient implementation assumes factorized pairwise weights $0.0023\%$4, so the full space of pairwise weighting schemes is not explored (Kim et al., 30 Sep 2025). The method requires extra centered-logit computations beyond a straightforward KL loss, and its validation loss is less directly interpretable than probability-based objectives, so the paper often relies on external metrics such as ROUGE-L or COMET for model selection (Kim et al., 30 Sep 2025). Like other logit-level distillation methods, it also inherits whatever calibration or structural biases are present in the teacher.

The same paper identifies a large remaining design space in $0.0023\%$5 and $0.0023\%$6, and points to extensions to larger models and combinations with reinforcement-learning-based alignment (Kim et al., 30 Sep 2025). In neighboring discrete diffusion work, Target Concrete Score Matching already supports pre-training, fine-tuning using reward functions or preference data, and distillation of knowledge from pre-trained autoregressive models (Zhang et al., 23 Apr 2025). Target Concrete Score Identity Sampler further shows that, in CTMC settings, concrete scores can be estimated from unnormalized energies and then turned into reverse samplers without requiring samples from the target distribution (Kholkin et al., 27 Oct 2025). Taken together, these developments suggest a broader trajectory in which “concrete” score objectives are used not only for autoregressive KD, but also for reward-guided post-training, discrete diffusion, and score-based generation on graphs and other structured discrete spaces.

In that broader sense, Concrete Score Distillation is best understood as a ratio-based, shift-invariant approach to distillation that treats relative structure—between logits, neighboring states, or reverse-transition alternatives—as the primary object of matching.