Textual Frequency Distillation (TFD)

• Textual Frequency Distillation (TFD) is a method that distills an LLM’s internal sentence frequency through model-generated story completions to overcome discrepancies with open-corpus data.
• It fuses traditional frequency estimates with LLM-derived metrics using tunable hyperparameters, thereby creating model-aware and adaptive metrics.
• TFD supports applications like prompt selection and curriculum fine-tuning, driving significant performance gains in translation, reasoning, and other NLP tasks.

Textual Frequency Distillation (TFD) is a central mechanism within the Textual Frequency Law (TFL) framework, developed to estimate and utilize sentence-level frequency information in LLMs whose pretraining corpora are proprietary. TFD refines raw, off-the-shelf frequency counts into model-aware metrics by using story completions generated directly by the LLM of interest. This process enables the calibration of frequency measures to reflect the “knowledge” and token distribution internal to closed-source LLMs and thereby improves both prompting and fine-tuning strategies across diverse NLP tasks (Lu et al., 2 Apr 2026).

1. Definition and Role within the Textual Frequency Law

TFD is the process by which an LLM’s sense of sentence frequency is distilled through interaction. Standard sentence frequency estimates, $\mathcal F_{1}(x)$, computed from an open-domain corpus may be insufficient due to mismatch with the (often undisclosed) data used to train modern LLMs. TFD addresses this limitation by prompting the LLM to generate extended stories for each candidate sentence, synthesizing a distilled corpus, $\mathcal D'$. A new frequency estimate, $\mathcal F_{2}(x)$, is computed based on this LLM-generated data.

The resulting frequency metric,

$$\mathcal F(x) = \alpha\,\mathcal F_{1}(x) + \bigl(1+\zeta\,\mathbf1[\mathcal F_{1}(x)=0]\bigr)\,\beta\,\mathcal F_{2}(x),$$

integrates both the open-corpus-based and the LLM-distilled estimates, with $\alpha$, $\beta$, and $\zeta$ as tunable hyperparameters. This fused measure is used within TFL to select among paraphrases and to drive the curriculum in fine-tuning setups.

2. Theoretical Motivation and Objectives

TFD is motivated by the premise that an LLM’s ability to naturally extend an input sentence is correlated with its exposure to similar examples during pretraining. If a sentence was frequent in an LLM’s actual training data, the model should be more capable of reliably generating coherent continuations that preserve or reuse key vocabulary and phrasing. TFD thus serves the following objectives:

• Domain Adaptivity: Mitigates bias in $\mathcal F_{1}$ due to discrepancies between open corpora and the LLM’s training data by producing a frequency estimate endogenous to the model.
• Model Alignment: Amplifies scores for sentences recognized as “common” by the model, while down-weighting rare or unfamiliar constructions.
• Zero-Frequency Recovery: Substitutes reliable frequency estimates for sentences unseen in the open corpus by leveraging $\mathcal F_{2}$, with the $\zeta$ hyperparameter boosting confidence in such cases (Lu et al., 2 Apr 2026).

3. Algorithmic Structure and Implementation

The TFD algorithm comprises four key steps:

1. Off-the-Shelf Frequency Calculation: Compute initial sentence frequencies, $\mathcal D'$0, from an open-domain corpus, using the geometric mean of inverse word frequencies:

$\mathcal D'$1

where $\mathcal D'$2.

1. Distilled Corpus Generation: For each sentence $\mathcal D'$3 in the dataset $\mathcal D'$4, issue a story-completion prompt to the LLM. The returned stories are parsed into sentences, producing the synthetic corpus $\mathcal D'$5.
2. Distilled Frequency Computation: Compute $\mathcal D'$6 identically to $\mathcal D'$7 but based on unigram counts from $\mathcal D'$8.
3. Frequency Fusion and Storage: Integrate the two frequency measures to obtain $\mathcal D'$9, which is stored for downstream use. Efficiency considerations include batching queries, truncating excessively long outputs, reusing stored frequencies, and consistent use of the wordfreq library.

Pseudocode Sketch:

$\alpha$2

4. Mathematical Formalization

TFD’s frequency metrics can be summarized as follows:

• Sentence-level frequency in corpus $\mathcal F_{2}(x)$0:

$\mathcal F_{2}(x)$1

• Distilled frequency from $\mathcal F_{2}(x)$2:

$\mathcal F_{2}(x)$3

• Final combined frequency:

$\mathcal F_{2}(x)$4

These formulations are specifically used to re-rank candidate paraphrases (prompt selection) and to design fine-tuning curricula (see next section).

5. Practical Integration and Application

After TFD-derived $\mathcal F_{2}(x)$5 values are computed, two principal applications follow within the TFL pipeline:

• Prompt Selection: For a paraphrase set $\mathcal F_{2}(x)$6, select the sentence $\mathcal F_{2}(x)$7 that maximizes $\mathcal F_{2}(x)$8 as the LLM prompt.
• Curriculum Textual Frequency Training (CTFT): Fine-tune the LLM with the dataset sorted by increasing $\mathcal F_{2}(x)$9 at each epoch, ensuring the model sees lower-frequency sentences before higher-frequency ones.

Table: Summary of TFD Workflow

Step	Input	Output
Off-the-shelf freq. ($$\mathcal F(x) = \alpha\,\mathcal F_{1}(x) + \bigl(1+\zeta\,\mathbf1[\mathcal F_{1}(x)=0]\bigr)\,\beta\,\mathcal F_{2}(x),$$0)	Open corpus $$\mathcal F(x) = \alpha\,\mathcal F_{1}(x) + \bigl(1+\zeta\,\mathbf1[\mathcal F_{1}(x)=0]\bigr)\,\beta\,\mathcal F_{2}(x),$$1, dataset $$\mathcal F(x) = \alpha\,\mathcal F_{1}(x) + \bigl(1+\zeta\,\mathbf1[\mathcal F_{1}(x)=0]\bigr)\,\beta\,\mathcal F_{2}(x),$$2	Initial frequencies
Story completion	Dataset $$\mathcal F(x) = \alpha\,\mathcal F_{1}(x) + \bigl(1+\zeta\,\mathbf1[\mathcal F_{1}(x)=0]\bigr)\,\beta\,\mathcal F_{2}(x),$$3, LLM	Distilled corpus $$\mathcal F(x) = \alpha\,\mathcal F_{1}(x) + \bigl(1+\zeta\,\mathbf1[\mathcal F_{1}(x)=0]\bigr)\,\beta\,\mathcal F_{2}(x),$$4
Distilled freq. ($$\mathcal F(x) = \alpha\,\mathcal F_{1}(x) + \bigl(1+\zeta\,\mathbf1[\mathcal F_{1}(x)=0]\bigr)\,\beta\,\mathcal F_{2}(x),$$5)	$$\mathcal F(x) = \alpha\,\mathcal F_{1}(x) + \bigl(1+\zeta\,\mathbf1[\mathcal F_{1}(x)=0]\bigr)\,\beta\,\mathcal F_{2}(x),$$6, dataset $$\mathcal F(x) = \alpha\,\mathcal F_{1}(x) + \bigl(1+\zeta\,\mathbf1[\mathcal F_{1}(x)=0]\bigr)\,\beta\,\mathcal F_{2}(x),$$7	Model-aware frequencies
Fusion and application	$$\mathcal F(x) = \alpha\,\mathcal F_{1}(x) + \bigl(1+\zeta\,\mathbf1[\mathcal F_{1}(x)=0]\bigr)\,\beta\,\mathcal F_{2}(x),$$8, $$\mathcal F(x) = \alpha\,\mathcal F_{1}(x) + \bigl(1+\zeta\,\mathbf1[\mathcal F_{1}(x)=0]\bigr)\,\beta\,\mathcal F_{2}(x),$$9	Final $\alpha$0 for selection/training

Implementation practices such as batching LLM queries, managing API rate limits, and disk-based storage of synthesized corpora and frequency statistics are recommended for scalability and reproducibility.

6. Empirical Impact and Evaluation

Empirical results confirm the centrality of TFD to the overall effectiveness of TFL strategies (Lu et al., 2 Apr 2026):

• Mathematical Reasoning (GSM8K): Accuracy improvements with TFL+TFD include GPT-4o-mini (60.7% → 68.7%), DeepSeek-V3 (63.6% → 71.5%), LLaMA-3.3-70B (80.5% → 88.8%).
• Machine Translation (100 Languages): BLEU, chrF, and COMET metrics all improve in 91–100% of language directions using high-frequency paraphrases from TFD.
• Commonsense Reasoning (CommonsenseQA): Accuracy gains of 2–3 points across multiple LLMs.
• Agentic Tool Calling: Tool selection accuracy and correctness improve by 3–6 points.
• Fine-Tuning (CTFT using TFD): Up to +30% relative BLEU improvement on low-resource translation directions compared to standard methods.

Ablation studies reveal that omitting TFD (i.e., using only $\alpha$1) consistently degrades performance, especially in machine translation evaluated with COMET, establishing the indispensability of the TFD component.

7. Significance within Model Understanding and Data Selection

TFD provides a robust methodology for inferring model-sensitive frequency statistics when direct access to training data is unavailable. By constructing a synthetic, model-generated corpus that reflects the LLM’s internal distribution, TFD enables nuanced selection and ordering of data for downstream tasks. This mechanism, by bridging the gap between external corpus statistics and the model’s latent knowledge, is critical for enhancing both prompt engineering and systematic fine-tuning, underpinning the performance improvements described in empirical evaluations. The architecture and workflow of TFD thus constitute a significant methodological advance in aligning data selection strategies with the idiosyncrasies of large, closed-source LLMs (Lu et al., 2 Apr 2026).