---
title: Compressed Token Distillation
url: https://www.emergentmind.com/topics/compressed-token-distillation-ctd
type: topic
---

# Compressed Token Distillation

Compressed Token Distillation (CTD) is a non-unified term in recent machine learning literature that denotes several related but distinct procedures for transferring capability under a reduced token budget. In one usage, CTD refers to distilling students on post-hoc compressed chain-of-thought traces so that training consumes fewer tokens and inference produces shorter rationales [2606.05988]. In another, CTD denotes cross-tokenizer distillation, where teacher and student use different vocabularies and the transfer is mediated through a shared byte-level interface rather than a shared token space [2604.07466]. Other papers use the term, or map naturally onto it, for pre-attention sequence compression in embedding models, spatio-temporally compressed supervision for video encoders, and related compressed-token or latent-token schemes [2511.14405][2605.17260]. This suggests that CTD is best understood as an umbrella for token-efficiency-oriented distillation rather than as a single standardized algorithm.

## 1. Terminological scope and conceptual boundaries

Recent papers attach the label CTD to different compression targets, supervision objects, and deployment goals. The common denominator is that a student or downstream model is trained to preserve useful behavior after some reduction, remapping, or compression of token-level supervision. What varies is whether the compressed object is a reasoning trace, a tokenizer interface, a pre-attention sequence, or a spatio-temporally pooled latent representation.

| Usage of CTD | Compressed or aligned object | Primary goal |
|---|---|---|
| "Compress-Distill" [2606.05988] | Teacher-produced chain-of-thought traces | Reduce training tokens and inference verbosity |
| "Cross-Tokenizer LLM Distillation through a Byte-Level Interface" [2604.07466] | Teacher and student token spaces via bytes | Enable distillation across mismatched tokenizers |
| "Jasper-Token-Compression-600M" [2511.14405] | Input sequences before attention | Preserve embedding quality while reducing latency |
| "LiteFrame" [2605.17260] | Teacher vision tokens after spatio-temporal compression | Bypass redundant visual-token computation |

The ambiguity is explicit in the byte-level distillation paper, which states that CTD there means cross-tokenizer distillation, not compression of tokens [2604.07466]. By contrast, the reasoning-trace and video papers use CTD in the more literal sense of compressing teacher supervision or teacher representations before student training [2606.05988][2605.17260]. A plausible implication is that any encyclopedia treatment of CTD must distinguish the acronym’s local paper-specific meaning from its broader role as shorthand for compression-aware distillation.

## 2. Post-hoc compression of reasoning traces

In "Compress-Distill: Reasoning Trace Compression for Efficient Knowledge Distillation" [2606.05988], CTD means training smaller student models on teacher-produced chain-of-thought traces that have been post-hoc compressed to preserve the essential reasoning and the final answer while being much shorter. The pipeline has three stages. First, a teacher $M_T$ generates chain-of-thought within `<think>…</think>` plus a final answer, and only verified-correct traces are retained. The retained sets comprise 283,335 correct traces from Qwen3.5-397B-A17B and 281,911 correct traces from gpt-oss-120B. Second, an instruction-tuned compressor $M_C$ rewrites each correct triple $(q_i, t_i, a_i)$ into a shorter trace using a single, generic prompt at temperature $0.3$; the compressors are Llama-3.3-70B-Instruct and Ministral-3-14B-Instruct-2512. Third, students are trained on raw traces, compressed traces, or answer-only targets with next-token prediction on assistant tokens only, using the chat template `User: q_i → Asst: <think> t̃_i </think> a_i`, where $t̃_i \in \{\text{raw trace } t_i,\text{ compressed trace } \hat t_i,\varnothing\}$.

The students are Qwen3.5-0.8B-Base, Llama-3.1-8B, Qwen3.5-9B-Base, and gpt-oss-20B. Training uses either LoRA with rank $64$, $\alpha=32$, dropout $0.05$, learning rate $1e{-4}$, and one epoch, or full fine-tuning with learning rate $2e{-5}$ and one epoch; FSDP v2 is used for 8B, 9B, and 20B. Tokenization and formatting use 16,384-token sequences, sample packing, BF16, FlashAttention 2, CutCrossEntropy, a consistent chat template, and greedy decoding at inference with an 8,192-token cap. The main grid contains 48 runs, with seven additional Qwen-teacher truncation ablations [2606.05988].

The reported compression is substantial. Character-level mean compression ratios under Qwen3.5-397B are $\rho = 0.142$ for Llama-70B and $\rho = 0.086$ for Ministral-14B; under gpt-oss-120B they are $\rho = 0.210$ and $\rho = 0.147$. Across the full study, compressed traces are reduced to 8.6–21.0% of the original character length, training tokens fall to 12–30% of raw, training speeds up by $2.0$–$7.6\times$, and inference outputs become $3$–$19\times$ shorter. At the same time, raw traces retain the highest downstream accuracy at every scale and for both teachers. Representative rows are Qwen teacher, Qwen-9B Full: raw $0.866$ vs L70 $0.834$ vs M14 $0.817$, and gpt-oss teacher, gpt-oss-20B Full: raw $0.844$ vs L70 $0.776$ vs M14 $0.767$ [2606.05988].

The paper therefore characterizes CTD as an accuracy–efficiency trade-off rather than a free improvement. Students retain up to 96% of raw-trace accuracy while achieving up to 18× higher per-token efficiency, and compressed traces dominate on cost and latency. A length-matched truncation ablation further shows that the benefit is not explained by “just fewer tokens”: model-compressed traces usually beat or match naive truncation at equal length, especially for smaller students, while also producing shorter inference outputs. At the 0.8B scale under LoRA, compressed traces narrow the raw-versus-compressed gap but do not exceed raw [2606.05988].

## 3. Self-distilled Long2Short reasoning compression

"TokenSqueeze: Performance-Preserving Compression for Reasoning LLMs" [2511.13223] operationalizes the central CTD idea in a self-distillation-style Long2Short setting rather than a teacher–student pipeline. The method uses only self-generated data and combines adaptive reasoning depth selection, distribution-aligned intra-step refinement, and a composite preference-learning objective. Its stated objective is to reduce chain-of-thought token usage while preserving reasoning fidelity and task accuracy.

Adaptive depth selection begins from multiple self-sampled responses per problem. If $N$ is the number of sampled responses and $c$ the number of correct ones, the method defines
$$
q = \alpha \cdot (1 - p), \qquad p = \frac{c}{N}, \qquad k = \lceil q \cdot c \rceil.
$$
Correct traces are sorted by token length, and the shortest correct traces up to index $k$ are selected as preferred positives. The intuition given in the paper is that easier problems, with higher $p$, should favor shorter chains, whereas harder problems should preserve more depth. The paper reports that $\alpha = 0.2$ provides the best balance [2511.13223].

The second component rewrites each reasoning step into a shorter form while constraining the model’s continuation distribution. For a step $s_i$, the paper samples $K = 64$ candidate rewrites at temperature $1.0$ and selects the shortest candidate satisfying a KL-divergence constraint with threshold $\varepsilon = 0.005$ over a window $L = 512$. The optimization is written as
$$
\min_{s_i' \in \{s_i^{(k)}\}} \ell(s_i') \quad \text{subject to} \quad D_{\mathrm{KL}\left( P_{\theta}(\cdot | p, s_{\leq i}) \,\|\, P_{\theta}(\cdot | p, s_{<i}, s_i') \right) < \varepsilon.
$$
The final training objective mixes length-aware DPO-L and SFT with $\eta = 0.5$, learning rate $5 \times 10^{-6}$, batch size $128$, full-parameter fine-tuning, and context length $9000$ [2511.13223].

Empirically, the paper reports that DeepSeek-R1-Distill-Qwen-7B fine-tuned with TokenSqueeze achieves a 50% average token reduction while preserving accuracy on MATH500: baseline accuracy $92.8$ with Len-T $3638$ versus TokenSqueeze accuracy $92.4$ with Len-T $1773$. On AIME24 for the same model, TokenSqueeze improves accuracy from $55.5$ to $57.5$ while reducing Len-T from $7543$ to $5157$. The paper also reports up to 15.5% higher accuracy on AIME24 at 3K tokens and 43.1% higher accuracy on MATH500 at 1K tokens versus the base model. Ablations indicate that “No Refinement” mainly reduces the number of steps, whereas full TokenSqueeze additionally shortens per-step expression; DPO-L plus SFT yields the best balance relative to DPO or SFT alone [2511.13223].

Within the broader CTD landscape, TokenSqueeze differs from trace-compression distillation in that it does not rely on an external teacher. Its compression is enforced through self-generated preference data and KL-constrained rewrites rather than through teacher-authored compressed traces. The paper nevertheless frames this as preserving logical content under token reduction, which closely aligns with the broader CTD objective [2511.13223].

## 4. Cross-tokenizer distillation through a byte-level interface

In "Cross-Tokenizer LLM Distillation through a Byte-Level Interface" [2604.07466], CTD means cross-tokenizer distillation: transferring knowledge from a teacher language model to a student language model when the two use different tokenizers. If the teacher uses vocabulary $\mathcal{V}_T$ and tokenizer $T_T$, and the student uses $\mathcal{V}_S$ and tokenizer $T_S$ with $\mathcal{V}_T \neq \mathcal{V}_S$ and $T_T \neq T_S$, then the standard shared-vocabulary KL objective is not well-defined. The proposed baseline, Byte-Level Distillation (BLD), uses the byte level as a common interface. Teacher token probabilities are converted into byte-level probabilities, and a lightweight byte-level decoder head is attached to the student.

The method formalizes a byte alphabet $\mathbb{B} = \{0,\ldots,255\}$ and derives teacher next-byte probabilities by summing over teacher tokenization paths compatible with a byte prefix. Exact computation is expensive, so the paper adopts the approximation of Vieira et al. (2025) via beam search with beam width $K$ and pruning threshold $\varepsilon$. The reported setting $K=10$, $\varepsilon=0.01$ achieves Jensen–Shannon divergence $\approx 0.0045$ to a high-precision reference with $K=100$, $\varepsilon=1e{-6}$, requires $\approx 10.4$ s/sample for 100–150 byte sequences on 4×RTX 3090, and takes $\approx 2$ days to precompute byte probabilities for the Tulu-3 dataset using parallelization [2604.07466].

On the student side, BLD adds a byte-level decoder head $O_b$ in parallel to the token-level head. In experiments, $N_b$ is fixed to $10$, so only the first 10 bytes of a token receive supervision. The aggregate loss combines token-level supervised next-token prediction, byte-level cross-entropy on ground-truth bytes, and byte-level KL to match the teacher’s next-byte distribution. For tokenizer transfer, embeddings and LM head are reinitialized with Fast Vocabulary Transfer, LoRA rank is $64$, and a representative loss setting uses $\lambda_{\mathrm{KL}}=0.1$ and $\lambda_{\mathrm{byte\text{-}CE}}=1.0$ [2604.07466].

The reported results are mixed but competitive. In BPE→BPE transfer from Llama3.2-3B-Instruct to a Qwen2 tokenizer, BLD attains PiQA $75.68$, ARC-C $43.26$, BoolQ $77.34$, MMLU $58.29$, AGI-EN $31.98$, AGI-ZH $35.97$, and IFEval $30.58$; the paper notes that it is competitive but struggles on instruction-following. In BPE→byte transfer, all methods degrade substantially, and no method dominates. In cross-model CTD from OpenMath2-Llama3.1-8B to Gemma2-2B, BLD reaches GSM8K $62.55 \pm 1.33$ and MATH $20.08 \pm 1.82$, outperforming SFT on GSM8K but not on MATH. The paper’s conclusion is explicit: consistent improvements across all tasks and benchmarks remain elusive, and CTD remains an open problem [2604.07466].

## 5. Pre-attention and spatio-temporal compressed representations

A separate line of work applies CTD to internal representations before the dominant compute stage, rather than to reasoning traces or tokenizer interfaces. In "Jasper-Token-Compression-600M Technical Report" [2511.14405], the model is a bilingual embedding system that inserts a compression module between token embeddings and Transformer attention blocks. The module consists of a randomly initialized Qwen3MLP (SwiGLU) FFN followed by a training-free AdaptiveAvgPool1d. If input length is $L_{\mathrm{in}}$, threshold is $L_{\mathrm{th}}$, and compression ratio is $\rho$, the target length rule is
$$
L' = \text{NULL if } L_{\mathrm{in}} \leq L_{\mathrm{th}}, \qquad
L' = L_{\mathrm{th}} + (L_{\mathrm{in}} - L_{\mathrm{th}})\cdot \rho \text{ if } L_{\mathrm{in}} > L_{\mathrm{th}}.
$$
Stage 2 uses fixed compression with $\rho=0.33$ and $L_{\mathrm{th}}=80$, while Stage 3 samples $\rho$ dynamically across several ranges. Distillation uses cosine alignment to teacher embeddings, then adds a batchwise similarity-preservation MSE; Stage 4 adds InfoNCE-style contrastive learning and soft KL distillation over similarity scores [2511.14405].

The reported outcome is a 600M embedding model with English Mean(Task) $74.75$ and Chinese Mean(Task) $73.51$, compared with baseline Qwen3-Embedding-0.6B scores of $70.70$ and $66.33$. Latency at batch size 32 falls from $24.24$ ms to $9.38$ ms at length $1024$ for $\rho=0.33$, and from $49.99$ ms to $17.52$ ms at length $2048$. The paper states that at $\rho \approx 0.33$–$0.50$, performance stays essentially flat versus the best setting while halving latency at long inputs [2511.14405].

In "LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs" [2605.17260], CTD trains a compact student vision encoder to predict spatio-temporally compressed teacher representations directly. The teacher is InternViT-300M from InternVL3-8B, which emits dense tokens; the compression operator is Weighted Average Pooling (WAP), applied to teacher features to produce information-dense compressed targets. The student is a ViT-Base encoder with 12 layers, width $768$, depth-wise 1D temporal convolutions, and progressive spatio-temporal downsampling. The core loss is
$$
\mathcal{L}_{\mathrm{CTD}}(\theta) = \| E_S(x) - \phi(E_T(x)) \|_2^2.
$$
Training uses AdamW, cosine schedule, linear warmup, global batch size $512$ on 8× H100 GPUs, learning rate $4e{-5}$, warmup $100$ epochs, and total CTD pretraining of $1800$ epochs. A later Language Model Adaptation stage fine-tunes both student encoder and LLM with LoRA rank $4$, $\alpha=8$, and dropout $0.05$ [2605.17260].

The paper reports a new latency–accuracy Pareto frontier. At the high-frame regime, the teacher baseline processes 32 frames at 256 tokens/frame with total latency $814.5$ ms and average accuracy $65.3\%$, whereas LiteFrame processes 256 frames at 16 tokens/frame with total latency $532.3$ ms and average accuracy $65.7\%$. At 64 frames, the figure caption reports LLM prefilling $12.7\times$ faster and ViT encoding $5.3\times$ faster than InternVL3-8B. An ablation compares CTD with a reconstructive variant, RTD, and finds that CTD without LMA already exceeds RTD plus LMA. Here CTD is not about language tokenization at all; it is about teaching a student to emit the teacher’s compressed latent tokens directly, thereby bypassing dense teacher computation [2605.17260].

## 6. Comparative context, diagnostics, and unresolved issues

Several adjacent papers do not define their method as CTD in the narrow sense but are important for understanding the broader compressed-token landscape. "Position IDs Matter: An Enhanced Position Layout for Efficient Context Compression in Large Language Models" [2409.14364] studies compressed or gist tokens that summarize long contexts and shows that assigning them uniformly spread position identifiers inside the original input span improves memorization under RoPE. A closed-form rule consistent with the paper’s description is
$$
g(j) = \left\lfloor \frac{j \cdot (m+1)}{n+1} \right\rfloor.
$$
With this position layout and a compression loss, the method reaches 15× compression with 96.9 BLEU4 and 31× compression with 84.4 BLEU4, whereas an ICAE-style reproduction without the position-ID design and without compression loss falls to 52.7 BLEU4 at 15× [2409.14364].

"Large Language Model as Token Compressor and Decompressor" [2603.25340] develops a self-expressive autoencoding framework in which a pretrained LLM translates surface text into a variable-length sequence of discrete Z-tokens and reconstructs the original text from them. The compressor is autoregressive in the latent alphabet, the decompressor is constrained to the base vocabulary, and the total loss is $L_{\mathrm{total}} = L_{\mathrm{tr}} + \lambda L_{\mathrm{KL}} + \beta L_{\mathrm{com}}$. The method reports up to 18 times token reduction and near-exact reconstruction at moderate compression ratios, including BLEU-4 $99.31$ at 4× on Wikipedia. The paper explicitly contrasts this with CTD-style methods, noting that its primary objective is self-reconstruction rather than teacher-driven downstream supervision [2603.25340].

A further diagnostic perspective appears in "Compressed code: the hidden effects of quantization and distillation on programming tokens" [2601.02563], which maps CTD to token-level preservation of code-relevant distributions under compression. The paper defines cold-start measures such as Programming Keywords Probability (PKP) and Special Tokens Probability (STP) from $p(\cdot \mid \mathrm{BOS})$. It reports that DeepSeek-R1-Distill-Qwen-1.5B has PKP $0.0042$ and STP $0.7246$, whereas Qwen2.5 base models across sizes show much more balanced PKP, approximately $0.113$–$0.143$, with STP approximately $0.12$–$0.15$. It also reports that moderate quantization can improve the PKP–STP balance relative to more aggressive quantization, for example Q4_K_S versus Q2_K on Qwen2.5-Coder-7B. This line of work does not propose a new CTD algorithm, but it shows that compressed or distilled students can substantially redistribute probability mass over token categories [2601.02563].

Across these papers, several limitations recur. Reasoning-trace compression reduces training tokens and latency but does not overtake raw-trace accuracy, and medicine shows the largest raw advantage in the reported study [2606.05988]. Byte-level cross-tokenizer distillation remains inconsistent across tasks, especially for instruction-following and BPE→byte transfer [2604.07466]. Jasper still trails its 8B teacher on retrieval, LiteFrame depends on teacher quality and the suitability of WAP targets, TokenSqueeze remains offline-only and sensitive to the KL threshold $\varepsilon$, and cold-start token diagnostics do not replace contextual evaluation [2511.14405][2605.17260][2511.13223][2601.02563]. The aggregate picture is therefore stable: CTD methods can yield large savings in token count, latency, or context length, but the dominant empirical pattern is a controlled trade-off between efficiency and fidelity rather than a universal accuracy gain.

Source: https://www.emergentmind.com/topics/compressed-token-distillation-ctd