---
title: Curriculum Textual Frequency Training (CTFT)
url: https://www.emergentmind.com/topics/curriculum-textual-frequency-training-ctft
type: topic
---

# Curriculum Textual Frequency Training (CTFT)

Curriculum Textual Frequency Training (CTFT) is a supervised training paradigm in which training examples are presented to a model according to a time-varying schedule induced by their textual frequency. The core principle is to sequence fine-tuning or pretraining data in an “easy-to-hard” order defined by statistical frequency measures—such as unigram or geometric sentence-level frequencies—so that the model encounters high-frequency (easier) examples first, and lower-frequency (harder, tail) examples later. CTFT spans applications from sequence learning in cognitively-plausible neural network models to large-scale language model pretraining and adaptation, and is empirically validated to stabilize optimization and improve generalization, especially in capacity-constrained regimes [2006.16470][2601.21698][2604.02176].

## 1. Formal Definition and Frequency Metrics

CTFT requires a per-sample scalar frequency score; all curricula constructions and pacing policies rest on this signal.

- **Word Frequency (LLM Pretraining):**
  For a sample $z$ of length $N$ (e.g., a 2048-token slice), the word-level frequency score is
  $$
  \mathrm{Score}_{\mathrm{freq}}(z) = \frac{1}{N} \sum_{i=1}^N \mathrm{Zipf}(w_i)
  $$
  where $\mathrm{Zipf}(w)$ is the log$_{10}$-frequency per billion words (SUBTLEX-US or equivalent) [2601.21698].
- **Sentence Frequency (Supervised Fine-tuning):**
  For a sentence $x = (x_1, ..., x_K)$, the geometric mean frequency is
  $$
  \mathrm{sfreq}(x ; D) = \left( \prod_{k=1}^K w\mathrm{freq}(x_k, D) \right)^{1/K}
  $$
  where $w\mathrm{freq}(x_k, D)$ is the (unigram or bigram) token frequency drawn from a reference corpus $D$ [2604.02176].
- **Time-Varying Distribution (Combinatorial Curriculum):**
  For sequences of training steps $t = 0, ..., T-1$, the sampling distribution is a convex combination of endpoint distributions $P, Q$:
  $$
  p_t(w_i) = R_t(i) = \left( \frac{T-t}{T} \right) P_i + \left( \frac{t}{T} \right) Q_i
  $$
  allowing curriculum pacing to evolve linearly over time [2006.16470].

## 2. Algorithmic Construction and Pacing Strategies

CTFT is implemented by sorting the training set according to the computed frequency scores and imposing deterministic or probabilistic pacing on batch selection:

- **Preprocessing:** Compute scalar frequency for each sample using the chosen metric (Zipf-average for pretraining, geometric mean for sentence-level finetuning).
- **Sorting:** Arrange all $N$ samples from easiest (highest frequency) to hardest (lowest frequency).
- **Pacing Policy:**
  - **Linear exposure:** At normalized training step $t \in [0,1]$, allow sampling of subsets up to quantile $p(t) = t$ of the ordered list.
  - **Full sorting:** For fine-tuning, iterate over the entire dataset sorted by frequency each epoch [2604.02176].
- **Batch Selection:** Draw batches uniformly from the current allowed prefix or follow a fully sorted one-pass sequence.
- **Hyperparameters:** LLM fine-tuning typically uses AdamW ($\text{lr}=1e{-4}$, cosine decay, 10\% warmup), batch size $=8$, 10 epochs; neural network teaching tasks use sequence lengths $T=10,000$ [2006.16470][2604.02176].

## 3. Comparative Baselines and Related Curricula

CTFT is situated among several curriculum learning approaches:

| Curriculum            | Ordering Signal                  | Reference Papers      |
|-----------------------|----------------------------------|----------------------|
| Random/Uniform        | None                             | [2006.16470][2601.21698][2604.02176]           |
| Frequency-based       | Static freq. (unigram/Zipf)      | [2006.16470][2604.02176][2601.21698]           |
| Age-of-Acquisition    | Avg. human AoA per word          | [2006.16470][2601.21698]           |
| Verb Variation        | Verb class/type count per sample | [2601.21698]         |
| Dependency-Tree Depth | Parse depth as example difficulty| [2604.02176]         |
| Reverse CTFT          | Hard-to-easy (high-to-low freq.) | [2604.02176]         |

CTFT is empirically superior to static reweighting and other easy-to-hard baselines; reverse (hard-to-easy) orderings consistently underperform.

## 4. Theoretical Foundations and Optimization Dynamics

CTFT's effectiveness is theoretically supported by gradient-variance control in SGD optimization for overparameterized neural models:

- **Gradient Noise Reduction:** Initial training on high-frequency (easy) data leads to lower gradient noise scale $\mathcal{B}(t) = \frac{\text{tr}(\Sigma_t)}{||G_t||_2^2}$, as rare words produce more stochastic gradients [2601.21698].
- **Spectral Stability:** CTFT delays singular entropy collapse $\displaystyle H_{\text{sing}}(W)$ in model output heads, mitigating softmax bottleneck saturation in late training.
- **Phase Exposure:** CTFT alters exposure within underlying learning phases—extending critical phase access to easy data—without introducing new optimization phases (verified by joint HMM latent-phase analysis).
- **Variance Bounds:** For piecewise population splits $P = (1-\rho) P_{\text{easy}} + \rho P_{\text{hard}}$, curriculum pacing controls the effective variance, ensuring
  $$
  \mathbb{E} \|\theta_t - \theta^*\|^2 \leq (1-\mu \eta)^t \|\theta_0 - \theta^*\|^2 + \frac{\eta}{\mu} \sigma^2
  $$
  so long as the proportion of "hard" samples is tightly managed.

## 5. Empirical Results and Performance Impact

The benefits of CTFT in a range of domains are supported by controlled experiments:

- **LLM Fine-Tuning (Qwen2.5-7B-Instruct, TFPD Corpus):** On machine translation, CTFT yields up to +20–30% higher BLEU and chrF scores over high-frequency selection without curriculum ordering [2604.02176]. Math reasoning and commonsense tasks show +5–8 point accuracy gains.
- **Pretraining (Pythia Models, 14M–410M):** Averaged downstream accuracy improvements:
  ```
  Model    Random     CTFT
  14M      35.7%      37.7%
  70M      39.0%      39.2%
  160M     42.2%      44.6%
  ```
  Gains diminish at scale; at 410M parameters, random ordering matches CTFT [2601.21698].
- **Supervised Machine Teaching Tasks (Monosyllabic Word Reading):** Time-varying frequency curricula surpass static frequency, AoA, and random baselines. Reported held-out accuracies reach 96% for adult-corpus, $K=1000$ training pool [2006.16470].
- **Ablations:** Removing frequency-based curriculum or reversing its order consistently reduces downstream task accuracy, spectral stability, and efficiency [2604.02176][2601.21698].

## 6. Implementation Guidelines and Practical Insights

Robust implementation of CTFT relies on precise metric computation, sorted data streaming, and phase-aware pacing:

- **Metric Selection:** Use up-to-date Zipf or wordfreq statistics for the corpus at hand; for sentence-level fine-tuning, the geometric mean is preferred.
- **Data Handling:** For large-scale distributed training, group sorted samples into $K$ quantile bins (e.g., $K=4$–$8$) and sequence bins deterministically [2601.21698].
- **Phase Adaptation:** Optionally, monitoring singular entropy or gradient noise can trigger adaptive pacing.
- **Cost:** Frequency computation overhead is negligible compared to training; the primary challenge is sorting/preprocessing.
- **Generalization:** CTFT is complementary to syntactic-complexity-based or AoA curricula, as frequency correlates only weakly with those measures (Pearson $r \approx 0.73$–$0.75$ with sample loss) [2601.21698][2604.02176].
- **Model Scale Sensitivity:** Gains are highest for memory- or compute-constrained models (sub-160M parameters); larger models exhibit diminishing returns due to the softmax bottleneck effect [2601.21698].

## 7. Extensions and Limitations

CTFT is adaptable and extensible:

- **Curriculum Mixing:** Time-varying mixing of multiple frequency endpoints ($P, Q$) via linear schedules or more breakpoints generalizes the scheduling for complex domains [2006.16470].
- **Domain Adaptation:** Augmenting the frequency signal with domain- or language-specific norms enables cross-lingual or multimodal curriculum learning [2601.21698].
- **Limitations:** For very large models (>410M parameters), CTFT’s optimization advantages—gradient variance reduction and spectral stability—are marginal, and random ordering suffices [2601.21698]. A plausible implication is that as model capacity increases, the benefits of curriculum-based optimization diminish due to intrinsic regularization ("softmax bottleneck eases").
- **Complementarity:** Because textual frequency is only partially aligned with lexical or syntactic complexity, CTFT can be combined with curricula based on other difficulty signals to balance stability and diversity [2604.02176].

CTFT provides a principled framework for data pacing in both cognitive modeling and large-scale language learning, translating psycholinguistic insights into practical machine learning improvements. Its methodological simplicity—frequency computation, sorting, and sequential exposure—facilitates adoption and experimental analysis across modalities and scales. 

[2006.16470]: https://arxiv.org/abs/2006.16470  
[2601.21698]: https://arxiv.org/abs/2601.21698  
[2604.02176]: https://arxiv.org/abs/2604.02176

Source: https://www.emergentmind.com/topics/curriculum-textual-frequency-training-ctft