---
title: Basic Reading Distillation (BRD)
url: https://www.emergentmind.com/topics/basic-reading-distillation-brd
type: topic
---

# Basic Reading Distillation (BRD)

Basic Reading Distillation (BRD) is a distillation paradigm for transferring reading competence from large language models to smaller models. In the generic-text formulation, BRD “educates” a small autoregressive student on unlabeled sentences by making it imitate a teacher’s explicit sentence-level reading behaviors—named entity recognition (NER), question raising (QR), and question answering (QA)—before downstream adaptation [2507.19741]. In a task-specific reading-comprehension formulation, a BRD pipeline is described as a teacher–student system that combines multi-level distillation losses, architectural sequence-length reduction, and layer-by-layer training to obtain a compact model with high inference efficiency, illustrated with WaLDORf [1912.06638]. Across these uses, BRD denotes a family of methods that treat reading behavior itself as the object of distillation rather than relying only on latent-feature imitation or downstream pseudo-labeling.

## 1. Conceptual definition and relation to other distillation paradigms

Conventional distillation is separated into two standard categories. Knowledge distillation (KD) trains a small student $S$ to mimic latent features of a large teacher LLM $T$, such as soft-logits, attention maps, or hidden states. Task distillation (TD) trains $S$ on teacher-generated input–output pairs for one or more downstream tasks. BRD differs from both by focusing on explicit, sentence-level reading behaviors on generic texts that are unrelated to downstream tasks [2507.19741].

In the generic-text formulation, the goal is first to “educate” $S$ in fundamental reading behaviors by mimicking $T$’s interactions with each sentence—NER, QR, and QA. The stated rationale is that humans acquire general reading skills before tackling specific tests; correspondingly, an “educated” student is expected to learn deeper text comprehension, improving zero- and few-shot generalization and complementing KD or TD. The paper further states that BRD is conceptually and empirically orthogonal to both KD and TD.

A common misconception is to treat BRD as merely another variant of soft-target matching. The generic-text formulation is broader: the supervision is textual and behaviorally explicit, and the student is trained on passages that interleave original text and teacher annotations. In the task-specific reading-comprehension formulation, BRD also exceeds ordinary logit distillation by including embedding, hidden-state, attention-score, and output-logit objectives together with ground-truth supervision. This suggests that BRD is best understood as a reading-centered distillation family rather than a single loss function.

## 2. Formal setup and objective functions

In the generic-text formulation, the teacher LLM is denoted $T$ and is a large pretrained autoregressive model, with Vicuna-13B or Llama-3-8B given as examples. The student model is denoted $S$ and is a much smaller autoregressive decoder, instantiated in experiments as XGLM-564M and initialized from the same family as larger XGLM-7.5B. The unlabeled corpus $D$ is a generic text collection, specifically a subset of CC-100 containing 5–10 million sentences and unrelated to any downstream task [2507.19741].

For each sentence $s \in D$, the teacher produces three basic reading behaviors: $\mathrm{NER}(s)$, a structured description of entities in $s$; $\mathrm{QR}(s)$, a question about $s$; and $\mathrm{QA}(s)$, the answer to $\mathrm{QR}(s)$ drawn from $s$. Each behavior is treated as conditional generation. If $y^*$ is the teacher output for input $x$, the student is trained by minimizing
$$
L(x,y^*) = -\sum_{t=1}^{|y^*|} \log p_S(y_t^* \mid x, y_{<t}^*).
$$

The paper defines four aggregate datasets that are mixed into the final training set $D_{\mathrm{TRAIN}}$:

| Dataset | Contents | Role |
|---|---|---|
| $D_{\mathrm{ORI}}$ | original passages | retain original language modeling signal |
| $D_{\mathrm{NER}}$ | $s \langle\mathrm{sep}\rangle \mathrm{NER}(s)$ | entity-focused reading behavior |
| $D_{\mathrm{QR}}$ | $s \langle\mathrm{sep}\rangle \mathrm{QR}(s)$ | question-raising behavior |
| $D_{\mathrm{QA}}$ | $s \langle\mathrm{sep}\rangle \mathrm{QR}(s) \langle\mathrm{sep}\rangle \mathrm{QA}(s)$ | answer generation conditioned on question |

The subtask losses are given explicitly. For NER, with target text sequence $\mathrm{NER}(s)$,
$$
L_{\mathrm{NER}}(s) = -\sum_{t=1}^{T_{\mathrm{NER}}} \log p_S(y_t^{\mathrm{NER}} \mid s, y_{<t}^{\mathrm{NER}}).
$$
An equivalent tokenized form is also provided:
$$
\mathcal{L}_{\mathrm{NER}} = -\sum_{i=1}^n y_i \log p_S(y_i \mid x_i).
$$

For QR,
$$
L_{\mathrm{QR}}(s) = -\sum_{t=1}^{T_{\mathrm{QR}}} \log p_S(y_t^{\mathrm{QR}} \mid s, y_{<t}^{\mathrm{QR}}).
$$

For QA, the formulation allows a mild teacher–student distribution-matching term:
$$
\mathcal{L}_{\mathrm{QA}}(s) = \alpha \cdot \mathrm{CE}(A,\hat{y}_S) + (1-\alpha)\cdot \mathrm{KL}\!\left(p_T(\cdot \mid s,\mathrm{QR}(s)) \,\|\, p_S(\cdot \mid s,\mathrm{QR}(s))\right).
$$
In the main experiments, $\alpha = 1$, so QA is trained as standard cross-entropy on teacher-generated text.

The overall BRD objective is
$$
\mathcal{L}_{\mathrm{BRD}} = \lambda_1 \mathcal{L}_{\mathrm{NER}} + \lambda_2 \mathcal{L}_{\mathrm{QR}} + \lambda_3 \mathcal{L}_{\mathrm{QA}},
$$
with default setting $\lambda_1 = \lambda_2 = \lambda_3 = 1$. All passages—original, NER, QR, and QA—are shuffled and used equally.

## 3. Student architecture and training procedure

The student in the generic-text BRD experiments is XGLM-564M, described as a 6-layer, 8-head, 512-dim decoder-only transformer with a context window of up to 2048 tokens [2507.19741]. Training proceeds in two stages. Stage 1 is data synthesis, in which the teacher is prompted on 5 million CC-100 sentences to produce $\mathrm{NER}(s)$, $\mathrm{QR}(s)$, and $\mathrm{QA}(s)$. Stage 2 is student training, where $D_{\mathrm{ORI}} : D_{\mathrm{NER}} : D_{\mathrm{QR}} : D_{\mathrm{QA}}$ are mixed in a $1:1:1:1$ ratio.

The optimization setup is specified as AdamW with learning rate $3 \times 10^{-4}$, batch size $8$, max steps $40\,000$, and checkpoints saved every $1\,000$ steps. Regularization uses weight decay $0.01$ and no dropout. Original sentences are constantly mixed to prevent forgetting. Sampling is uniform random over passages. No synthetic paraphrasing or adversarial augmentation is used.

The paper also reports a targeted ablation in which removing sentiment-style QR/QA reduced SST-2 gains but did not eliminate BRD’s overall benefit. This is relevant because it locates part of BRD’s effect in the type of reading drills supplied by the teacher rather than in volume alone. A plausible implication is that BRD data design is an independent axis of control alongside model size and optimizer settings.

## 4. Empirical performance and analytical findings

Evaluation covers Natural Language Inference benchmarks XNLI, RTE, and CB; PAWS-X for paraphrase; BOOLQ for Boolean QA; SST-2 for sentiment; and 73 multiple-choice Google BIG-bench tasks. The principal blind-test result, with no task supervision, is reported as average accuracy across 79 tasks [2507.19741].

| Setting | Model | Average (%) |
|---|---|---:|
| Blind test, no task supervision | XGLM-564M (baseline) | 48.0 |
| Blind test, no task supervision | SKD (KD) | 46.6 |
| Blind test, no task supervision | MiniLLM (SOTA KD) | 51.8 |
| Blind test, no task supervision | XGLM-BRD | 54.2 |
| Blind test, no task supervision | XGLM-7.5B | 52.3 |
| Blind test, no task supervision | Vicuna-13B (teacher) | 67.6 |

These results show BRD yielding +6.2 points over the same-size baseline and outperforming XGLM-7.5B. In a relaxed test, BRD is first applied on generic data to obtain XGLM-BRD, then applied again on downstream inputs without gold labels to obtain XGLM-BRD$^2$; the reported average is 67.1% versus Vicuna-13B’s 67.6%. With gold task supervision, supervised fine-tuning averages 80.3%, while BRD-augmented SFT, denoted XGLM-BRD$^2$-SFT, reaches 81.8%.

The paper’s central analytical claim is orthogonality to KD and TD. Table 4 is summarized as showing that injecting BRD into GPT-2† or MiniLLM students yields +6–8 points on average, and combining BRD with pseudo-label task distillation boosts performance by ~2–3 points. Cross-entropy analysis reinforces this interpretation: $\mathrm{CE}(T\|S)$ is computed on 1,000 samples per task, and XGLM-BRD reduces cross-entropy by 15–20% over the XGLM-564M baseline, indicating better alignment to the teacher’s predictive distribution.

Ablations identify the contribution of the constituent reading behaviors and data structure. Removing QRA costs ~2.5 points on average, and removing NER costs ~2.5 points; QRA is stated to have slightly higher impact because it focuses on sentence understanding. Passage-level training exceeds sentence-level training by ~2 points. Data scale shows steady gains up to ~1 million passages, then plateau. The paper concludes from these analyses that BRD provides its largest gains under zero- and few-shot conditions, where general reading ability is most exposed.

## 5. Task-specific reading-comprehension BRD and the WaLDORf formulation

A second formulation of BRD is presented as a step-by-step pipeline for reading-comprehension distillation, illustrated with WaLDORf, a “hybrid convolutional + transformer” student for SQuAD v2.0 [1912.06638]. The teacher is a full-size transformer such as BERT-Large (Whole-Word-Masking) or RoBERTa-Large, with hidden size $d \approx 1024$, feed-forward size $\approx 4096$, and $L=24$ layers for BERT-Large. The student uses embedding size $e' = 96$, hidden size $d' = 480$, feed-forward size $1440$, and $J=8$ transformer blocks. Two 1D conv + pool layers shrink sequence length by factor 4, and two 1D conv + upsample layers restore it.

The student architecture is specified in detail. Input tokenization is standard BERT word-piece to 96-dimensional vectors. The conv encoder shrink path maps sequence length $l \to l/4$ with Conv$_1$ (1D, 96 filters) + max-pool($\times 2$) and Conv$_2$ (1D, 192 filters) + max-pool($\times 2$). The transformer backbone contains 8 standard encoder blocks with hidden size $480$, FF inner dimension $1440$, and 16 attention heads. The conv decoder expand path maps $l/4 \to l$ with Conv$_{3-4}$ (1D, 480 filters each) + up-sampling ($\times 2$ twice), using skip-connections and layer-norm after upsampling. Output heads are two linear layers over length $l$ for start and end logits. Total parameters are approximately 24.6 M, about one quarter of BERT-Base, while self-attention costs are approximately one sixteenth per block because of the length reduction.

The distillation objective combines five losses:
$$
L_e = \frac{1}{B}\sum_{i=1}^B \|E_i^t - E_i^s W^e\|^2,
$$
$$
L_{h,j} = \frac{1}{B}\sum_{i=1}^B \|\mathrm{AVG}_4(H_{3j+2,i}^t) - H_{j,i}^s W^h\|^2,
$$
$$
L_{a,j} = \frac{1}{B}\sum_{i=1}^B \|\mathrm{MAX}_{4\times 4}(\xi_{3j+2,i}^t) - \xi_{j,i}^s\|^2,
$$
together with output-logits distillation $L_d$ using temperature $T$ and ground-truth cross-entropy $L_g$. The total loss is
$$
L(\tau) = \alpha L_e + \sum_{j=0}^{J-1}\chi_j(\tau)\left[\beta L_{h,j} + \gamma L_{a,j}\right] + \chi_J(\tau)\left[\delta L_d + \epsilon L_g\right],
$$
where $\chi_j$ is a step function that turns on losses for layer $j$ after $\tau > j \cdot \tau^*$.

Training uses SQuAD v2.0 with fine-tuning on approximately 130 K QA pairs and evaluation on the dev set. Data augmentation adds 500 K extra contexts whose questions are generated by T5-Large; because these examples have no ground-truth labels, only distillation losses apply. The implementation details specify batch size 24 on TPU v3 with TF 1.14, Adam with $\epsilon = 1\mathrm{e}{-5}$, initial learning rate $2\mathrm{e}{-4}$ held constant during build-up and then linearly decayed to zero once all layers are active, temperature starting at 5 and linearly decaying to 1, dropout 0, and total training of 35 epochs, approximately 1 M steps, with the first approximately 0.5 M spent turning on layers one by one.

On SQuAD v2.0 dev, WaLDORf reports EM = 66.0 and F1 = 70.3. The teacher, BERT-Large-WWM, reports EM = 82.6 and F1 = 85.6. TinyBERT-4 reports EM = 65.3 and F1 = 68.6, while BERT-4 variants such as DistilBERT and PKD report approximately EM = 60.6 and F1 = 64.4. Inference, tested on a V100 GPU with sequence length 384 and batch size 32, is reported relative to BERT-Base = 1×: BERT-Base requires 106.2 s, TinyBERT-4 with 14.4 M parameters requires 14.4 s for 7.4× speed, and WaLDORf with 24.6 M parameters requires 11.6 s for 9.1× speed.

The ablation sequence isolates the contribution of each BRD component. A baseline using only $L_g$ gives EM 34.0 / F1 37.6. Adding softmax distillation $L_d$ yields EM 42.9 / 45.8. Adding all-layer distillation $L_e + L_h + L_a$ yields EM 45.9 / 49.7. Adding the “slow build” schedule yields EM 57.7 / 62.2. Adding data augmentation yields the final EM 66.0 / 70.3. The paper’s blueprint summarizes the key building blocks as teacher–student pairing, multi-level distillation losses, architectural shrinking of self-attention, layer-by-layer training to mitigate covariate shift, task-specific data augmentation, and careful hyperparameter tuning.

## 6. Interpretation, limitations, and scope

Taken together, the two formulations place BRD at the intersection of distillation, curriculum design, and reading-oriented supervision. The generic-text BRD paper argues that explicit reading drills on generic corpora influence the student’s token distribution directly and measurably, with cross-entropy reductions of 15–20% over baseline and strong blind-test gains [2507.19741]. The WaLDORf formulation, by contrast, demonstrates that a task-specific BRD pipeline can combine intermediate-feature matching, output distillation, architectural compression, and staged optimization to achieve 5–10× faster inference while preserving substantial reading-comprehension accuracy [1912.06638].

The two uses of BRD should not be conflated. In the generic-text setting, the central primitive is explicit sentence-level behavior on unlabeled corpora: NER, QR, and QA. In the WaLDORf setting, the central primitive is a reading-comprehension teacher–student pipeline with embedding-level, hidden-state, attention-score, and logit-level objectives plus ground-truth supervision. This suggests that BRD names a broader strategy of distilling reading behavior, instantiated either as pre-task basic education or as task-specific comprehension compression.

The stated limitations in the generic-text BRD work are narrow teacher coverage and generation cost: only two teacher models, Vicuna-13B and Llama-3-8B, are explored, and creating BRD data can be expensive for larger teachers. Future directions listed there include expanding basic behaviors to discourse parsing and summarization, adaptive weighting $\lambda_i$ per subtask, and interactive or multi-turn reading drills. In the WaLDORf blueprint, the emphasis falls instead on tuning the loss weights $\alpha \ldots \epsilon$, temperature decay, learning-rate schedule, and batch size. A plausible implication is that future BRD systems may combine both strands: generic reading education to improve broad generalization and task-specific architectural distillation to meet deployment constraints.

Source: https://www.emergentmind.com/topics/basic-reading-distillation-brd